A method and system for processing low-altitude images of unmanned aerial vehicles

Abstract

The present invention discloses a method and system for processing low-altitude images of unmanned aerial vehicles, which relates to the field of image processing technology, and solves the technical problem that it is difficult to extract image features from different areas of an image separately, and it is also difficult to accurately splice different image frames while retaining the features of a moving object; by using an inter-frame difference method, the sub-image blocks belonging to the moving area in the image frame are determined, and the area that has changed in the image frame can be effectively distinguished, so as to accurately locate and extract the moving target, and by secondary subdivision, the moving area can be further refined to improve the detection accuracy. At the same time, splicing is performed based on the principle of the highest overlap of the divided highly similar image areas, which can ensure that the visual effect of the spliced ​​image is natural and smooth, and the sense of discontinuity at the splicing point is reduced. When the background area fusion processing is performed on the spliced ​​image, the integrity and continuity of the background information can be effectively maintained.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a method and system for processing low-altitude images of unmanned aerial vehicles. Background Art

[0002] With the rapid development and popularization of drone technology, drones are increasingly used in various fields. Among them, using drones for low-altitude image acquisition and processing has become a hot field. By carrying high-definition cameras or other sensors to obtain high-quality image data during low-altitude flight, combined with advanced image processing algorithms, it is possible to quickly acquire and analyze information such as ground targets and scenes.

[0003] The UAV low-altitude image processing system can effectively make up for the deficiencies of traditional remote sensing technology in terms of resolution and real-time performance, and has higher flexibility and adaptability. By processing the collected image data, it can realize functions such as target detection, recognition, and tracking, and provide users with more comprehensive and accurate information services. Therefore, the UAV low-altitude image processing system has broad application prospects in agricultural production, urban management, environmental protection and other fields.

[0004] Most of the existing UAV low-altitude image processing technologies perform feature extraction and image processing on image frames. It is difficult to extract image features from different areas of the image separately, and it is also difficult to accurately stitch different image frames while retaining the characteristics of moving objects. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a low-altitude image processing method and system for unmanned aerial vehicles, which are used to solve the technical problem that it is difficult to extract image features from different areas of an image separately, and it is also difficult to accurately splice different image frames while retaining the characteristics of moving objects.

[0006] To solve the above problems, the first aspect of the present invention provides a method and system for processing low-altitude images of unmanned aerial vehicles, comprising:

[0007] Obtain an image sequence collected by a drone at low altitude, and pre-process the image frames in the image sequence;

[0008] Dividing the image frames of the preprocessed image sequence into blocks, and dividing each image frame into a plurality of sub-image blocks;

[0009] Detecting the grayscale value of the sub-image block of each image frame, determining the sub-image block belonging to the motion area in the image frame by using the inter-frame difference method, performing secondary subdivision on the sub-image block belonging to the motion area, and obtaining the subdivided image sub-block of the motion area and the image block of the background area;

[0010] A detection window is set in the motion region, and a subdivided image subblock of the motion region where an initial position of the detection window is located is determined, and a feature vector of each subdivided image subblock of the motion region is extracted;

[0011] A search window is set in the background area, and an image block in the background area where the initial position of the search window is located is determined, and image features of the image block in the background area are extracted;

[0012] By using the image features of the image blocks in the background area and the similarity analysis between the image blocks, a highly similar image area in the background area is determined;

[0013] Based on the principle of the highest overlap of the divided highly similar image regions, adjacent image frames in the image sequence are spliced, and the spliced ​​images are subjected to background region image fusion processing and motion region feature retention processing.

[0014] As a further solution of the present invention, obtaining an image sequence collected by a drone at low altitude and preprocessing the image frames in the image sequence comprises the following steps:

[0015] Obtain the image sequence collected by the drone at low altitude, as well as the corresponding attitude data when the drone collects the image sequence;

[0016] According to the corresponding posture data when the drone collects the image sequence, the pixel coordinates and the actual world coordinates of the drone are marked in the image frames of the image sequence collected by the drone at low altitude;

[0017] Through the pixel coordinates of the drone and the actual world coordinates, coordinate matching is performed to establish a mapping relationship between the image coordinates and the actual world coordinates;

[0018] According to the correspondence between the pixel coordinates of the drone and the actual world coordinates, the affine transformation matrix is ​​solved by the least squares method;

[0019] The orthographic image is obtained by multiplying the affine transformation matrix by the inter-frame image coordinates taken by the drone at low altitude.

[0020] As a further solution of the present invention: detecting the gray value of the sub-image block of each image frame, determining the sub-image block belonging to the motion area in the image frame by using the inter-frame difference method, performing secondary subdivision on the sub-image block belonging to the motion area, and obtaining the subdivided image sub-block of the motion area and the background area image block, including the following steps:

[0021] Detecting the grayscale value of the sub-image block of each image frame, determining the sub-image block belonging to the motion area in the image frame by using the inter-frame difference method, dividing the sub-image block belonging to the motion area into the motion area sub-image block, and dividing the remaining image blocks in the image frame into the background area sub-image blocks;

[0022] The motion area sub-image block of the video frame is divided into N subdivided image sub-blocks of fixed size, and the subdivided image sub-blocks belonging to the motion area in the subdivided image sub-blocks are determined by the inter-frame difference method. The subdivided image sub-blocks belonging to the motion area are defined as motion area subdivided image sub-blocks, and the remaining sub-image blocks and subdivided image sub-blocks in the image frame are defined as background area image blocks.

[0023] As a further solution of the present invention: detecting the gray value of the sub-image block of each image frame, and determining the sub-image block belonging to the motion area in the image frame by using the inter-frame difference method, including the following steps:

[0024] Detecting the grayscale values ​​of all pixels within the sub-image block of each image frame;

[0025] Performing a difference operation on two adjacent image frames to obtain the difference between the grayscale values ​​of each pixel in the corresponding sub-image blocks between the adjacent image frames;

[0026] Calculate the average value of the difference between the grayscale values ​​of each pixel in the corresponding sub-image block, and count the number of pixels whose difference between the grayscale value of each pixel in the sub-image block and the average value of the difference between the grayscale values ​​of the pixels is greater than a preset threshold;

[0027] If the number of pixels counted is greater than one percent of the total number of pixels in the sub-image block, the grayscale change degree inside the sub-image block is determined to be 1;

[0028] Otherwise, after eliminating the pixels whose grayscale value and the average value of the difference between the grayscale values ​​of the pixels are greater than the preset threshold, the difference between the grayscale values ​​of each pixel in the corresponding sub-image blocks between adjacent image frames is analyzed, and the grayscale change degree inside the sub-image block is calculated by the following formula:

[0029]

[0030] Where D is the grayscale variation within the sub-image block, d i is the difference between the grayscale values ​​of the i-th corresponding pixel between the current image frame and the adjacent image frame, S is the variance of the difference between the grayscale values ​​of the pixels between the current image frame and the adjacent image frame, and f i is the grayscale value of the i-th corresponding pixel in the current image frame, and n is the number of pixels to be calculated in the corresponding sub-image block between the current image frame and the adjacent image frame;

[0031] If the grayscale variation within the sub-image block exceeds a set threshold, it is determined that the corresponding sub-image block belongs to a sub-image block in a motion area.

[0032] As a further solution of the present invention: determining the subdivided image subblocks belonging to the motion area in the subdivided image subblocks by using the inter-frame difference method, comprising the following steps:

[0033] For N subdivided image sub-blocks, the average grayscale value of all pixels in each subdivided image sub-block is calculated as the feature value of the corresponding subdivided image sub-block;

[0034] Performing a difference operation on two adjacent image frames to obtain the difference between the feature values ​​of all pixel points in the corresponding subdivided image sub-blocks between the adjacent image frames;

[0035] Calculate the average value of the difference between the characteristic values ​​of all pixels in the corresponding subdivided image sub-block;

[0036] If the average value of the difference between the characteristic values ​​of all pixels inside the subdivided image subblock is greater than a preset threshold, the subdivided image subblock is determined to belong to the motion area, otherwise, the subdivided image subblock is determined to belong to the background area.

[0037] As a further solution of the present invention: by using the image features of the image blocks in the background area and by similarity analysis between the image blocks, a highly similar image area in the background area is determined, comprising the following steps:

[0038] Analyze the similarity between the background area image blocks of adjacent image frames through the image features of the background area image blocks, and divide the similarity between the background area image blocks of adjacent image frames according to the analysis results of the similarity between the background area image blocks, and count and mark the image blocks with high similarity in the background area image blocks of adjacent image frames;

[0039] Adjacent annotated highly similar image blocks in the image frame are grouped, and when the number of image blocks in the highly similar image block group is greater than or equal to N, the corresponding highly similar image block group is divided into highly similar image regions.

[0040] As a further solution of the present invention: setting a search window in the background area, determining the background area image block where the initial position of the search window is located, and extracting image features of the background area image block, including the following steps:

[0041] A search window is set in the background area, and an image block in the background area where the initial position of the search window is located is determined, and the search window slides in the image blocks in the background area in sequence;

[0042] The search window extracts the color histogram and texture features of the background area image blocks of adjacent image frames through the OpenCV library;

[0043] The color histogram of the background area image block of the adjacent image frame is divided into 8 bins for each color channel in the RGB color space, and the color histogram of each channel is represented as an array of length 8, and the color histogram of the entire image block is represented as an array of length 24;

[0044] Extract the LBP values ​​of the pixels of the background area image blocks in the texture features of the background area image blocks of the adjacent image frames through the LBP local binary pattern, and divide the LBP values ​​of the pixels of the background area image blocks into 24 groups, calculate the average LBP value of each group, and form an array with a length of 24;

[0045] The two obtained arrays are normalized and then concatenated to serve as image features of the background area image block.

[0046] As a further solution of the present invention, the similarity between the background area image blocks of adjacent image frames is analyzed by using the image features of the background area image blocks, including the following steps:

[0047] Get the vector arrays of image features of background area image blocks of adjacent image frames, which are two n-dimensional vectors A(a1, a2,…, an) and B(b1, b2,…, bn). Calculate the Euclidean distance and cosine similarity between the corresponding image blocks in the background area of ​​adjacent frames using the following formula:

[0048]

[0049] Where d(A,B) is the Euclidean distance between corresponding image blocks in the background area between adjacent frames, i∈(1, 2,…,n);

[0050]

[0051] Among them, cos(A, B) is the cosine similarity between the corresponding image blocks in the background area between adjacent frames;

[0052] The obtained Euclidean distances between the corresponding image blocks in the background area between adjacent frames are normalized so that the Euclidean distance range is between [0,1]. The normalized Euclidean distance and cosine similarity are added to obtain the similarity analysis value between the image blocks in the background area of ​​adjacent image frames.

[0053] In this solution, according to the analysis result of the similarity between the background area image blocks, the similarity degree between the background area image blocks of adjacent image frames is divided, including the following steps:

[0054] According to the analysis results of the similarity between the image blocks in the background area, the image blocks whose analysis values ​​of the similarity between the image blocks in the background area are greater than 1.6 are divided into highly similar image areas;

[0055] The image blocks whose similarity analysis values ​​between the image blocks in the background area are less than or equal to 1.6 and greater than 1.2 are divided into moderately similar image areas;

[0056] Image blocks whose similarity analysis values ​​between image blocks in the background area are less than or equal to 1.2 are divided into low-similarity image areas.

[0057] As a further solution of the present invention: performing background region image fusion processing and motion region feature retention processing on the spliced ​​images, comprising the following steps:

[0058] Performing image fusion on background area image blocks in the stitched adjacent image frames;

[0059] The motion region subdivision image sub-blocks are superimposed, and the feature vectors of the corresponding motion region subdivision image sub-blocks are extracted and added as labels to the corresponding motion region subdivision image sub-blocks.

[0060] As another aspect of the present invention: a UAV low-altitude image processing system, the system adopts the above-mentioned UAV low-altitude image processing method to implement UAV low-altitude image processing.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] The present invention divides the image frame into blocks, and after the motion area is determined by the frame difference method, the remaining part is the background area, which is convenient for accurately dividing the motion area and the background area, and can effectively suppress background interference, so that the final image result is clearer and more accurate. By using the frame difference method, the sub-image blocks belonging to the motion area in the image frame are determined, and the areas that have changed in the image frame can be effectively distinguished, so as to accurately locate and extract the moving target. At the same time, through secondary subdivision, the motion area can be further refined to improve the detection accuracy.

[0063] The present invention analyzes the similarity between background area image blocks, selects highly similar image areas and performs splicing, thereby ensuring the consistency of the background area in the entire image sequence and improving the accuracy of image splicing. At the same time, the splicing is performed based on the principle of the highest overlap of the divided highly similar image areas, which can ensure that the visual effect of the spliced ​​image is natural and smooth, and reduce the sense of discontinuity at the splicing. When the background area fusion processing is performed on the spliced ​​image, the integrity and continuity of the background information can be effectively maintained. By retaining the motion area features, the feature data of the moving target in the spliced ​​image is fully retained, which helps to improve the accuracy of moving target detection and recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0066] Figure 2 The figure is a flow chart of a method for determining a highly similar image region in a background region according to the present invention. DETAILED DESCRIPTION

[0067] The technical solution of the present invention will be clearly and completely described in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] See also Figure 1-Figure 2 The first aspect of the present invention provides a method and system for processing low-altitude images of unmanned aerial vehicles, including:

[0069] Obtain an image sequence collected by a drone at low altitude, and pre-process the image frames in the image sequence;

[0070] Dividing the image frames of the preprocessed image sequence into blocks, and dividing each image frame into a plurality of sub-image blocks;

[0071] Detecting the grayscale value of the sub-image block of each image frame, determining the sub-image block belonging to the motion area in the image frame by using the inter-frame difference method, performing secondary subdivision on the sub-image block belonging to the motion area, and obtaining the subdivided image sub-block of the motion area and the image block of the background area;

[0072] A detection window is set in the motion region, and a subdivided image subblock of the motion region where an initial position of the detection window is located is determined, and a feature vector of each subdivided image subblock of the motion region is extracted;

[0073] A search window is set in the background area, and an image block in the background area where the initial position of the search window is located is determined, and image features of the image block in the background area are extracted;

[0074] By using the image features of the image blocks in the background area and the similarity analysis between the image blocks, a highly similar image area in the background area is determined;

[0075] Based on the principle of the highest overlap of the divided highly similar image regions, adjacent image frames in the image sequence are spliced, and the spliced ​​images are subjected to background region image fusion processing and motion region feature retention processing.

[0076] Specifically, in this embodiment, the image frames of the preprocessed image sequence are divided into blocks, and each image frame is divided into several sub-image blocks; the grayscale value of the sub-image block of each image frame is detected, and the sub-image block belonging to the motion area in the image frame is determined by the inter-frame difference method, and the sub-image block belonging to the motion area is subdivided twice to obtain the subdivided image sub-blocks of the motion area and the background area image block; after the motion area is determined by the inter-frame difference method through the image frame block, the remaining part is the background area, which is convenient for accurately dividing the motion area and the background area, and can effectively suppress background interference, so that the final image result is clearer and more accurate. By using the inter-frame difference method, the sub-image blocks belonging to the motion area in the image frame are determined, and the area that has changed in the image frame can be effectively distinguished, so as to accurately locate and extract the moving target. At the same time, through the secondary subdivision, the motion area can be further refined to improve the detection accuracy.

[0077] A detection window is set in the motion area, and the subdivided image subblock of the motion area where the initial position of the detection window is located is determined, and the feature vector of each subdivided image subblock of the motion area is extracted; a search window is set in the background area, and the image block of the background area where the initial position of the search window is located is determined, and the image features of the image block of the background area are extracted; by setting the detection window and the search window, it is convenient to accurately extract the feature vectors of the target area and the background area, ensuring that the extracted features are more closely related to the actual target or background, thereby improving the accuracy of feature extraction. By setting the detection window and the search window, the range of the area to be processed is limited, avoiding the unified feature extraction of the entire image, and facilitating more targeted extraction of feature vectors of different areas. By accurately setting the detection window position and combining it with a specific feature extraction method, the target can be effectively distinguished from the background and the accuracy of target detection can be maximized.

[0078] Through the image features of the background area image blocks and the similarity analysis between the image blocks, the highly similar image areas in the background area are determined; based on the principle of the highest overlap of the divided highly similar image areas, the adjacent image frames in the image sequence are spliced, and the spliced ​​image is subjected to background area image fusion processing and motion area feature retention processing; by analyzing the similarity between the background area image blocks, highly similar image areas are screened and spliced, thereby ensuring the consistency of the background area in the entire image sequence and improving the accuracy of image splicing. At the same time, based on the principle of the highest overlap of the divided highly similar image areas, the splicing can ensure that the visual effect of the spliced ​​image is natural and smooth, and the sense of discontinuity at the splicing is reduced. When the background area fusion processing is performed on the spliced ​​image, the integrity and continuity of the background information can be effectively maintained. By retaining the motion area features, the feature data of the moving target in the spliced ​​image is fully retained, which helps to improve the accuracy of moving target detection and recognition.

[0079] In one embodiment of the present invention, obtaining an image sequence collected by a drone at low altitude and preprocessing the image frames in the image sequence include the following steps:

[0080] Obtain the image sequence collected by the drone at low altitude, as well as the corresponding attitude data when the drone collects the image sequence;

[0081] According to the corresponding posture data when the drone collects the image sequence, the pixel coordinates and the actual world coordinates of the drone are marked in the image frames of the image sequence collected by the drone at low altitude;

[0082] Through the pixel coordinates of the drone and the actual world coordinates, coordinate matching is performed to establish a mapping relationship between the image coordinates and the actual world coordinates;

[0083] According to the correspondence between the pixel coordinates of the drone and the actual world coordinates, the affine transformation matrix is ​​solved by the least squares method;

[0084] The orthographic image is obtained by multiplying the affine transformation matrix by the inter-frame image coordinates taken by the drone at low altitude.

[0085] In one embodiment of the present invention, the grayscale value of the sub-image block of each image frame is detected, and the sub-image block belonging to the motion area in the image frame is determined by the inter-frame difference method, and the sub-image block belonging to the motion area is subdivided twice to obtain the subdivided image sub-block of the motion area and the background area image block, including the following steps:

[0086] Detecting the grayscale value of the sub-image block of each image frame, determining the sub-image block belonging to the motion area in the image frame by using the inter-frame difference method, dividing the sub-image block belonging to the motion area into the motion area sub-image block, and dividing the remaining image blocks in the image frame into the background area sub-image blocks;

[0087] The motion area sub-image block of the video frame is divided into N subdivided image sub-blocks of fixed size, and the subdivided image sub-blocks belonging to the motion area in the subdivided image sub-blocks are determined by the inter-frame difference method. The subdivided image sub-blocks belonging to the motion area are defined as motion area subdivided image sub-blocks, and the remaining sub-image blocks and subdivided image sub-blocks in the image frame are defined as background area image blocks.

[0088] In one embodiment of the present invention, the grayscale value of the sub-image block of each image frame is detected, and the sub-image block belonging to the motion area in the image frame is determined by the inter-frame difference method, including the following steps:

[0089] Detecting the grayscale values ​​of all pixels within the sub-image block of each image frame;

[0090] Performing a difference operation on two adjacent image frames to obtain the difference between the grayscale values ​​of each pixel in the corresponding sub-image blocks between the adjacent image frames;

[0091] Calculate the average value of the difference between the grayscale values ​​of each pixel in the corresponding sub-image block, and count the number of pixels whose difference between the grayscale value of each pixel in the sub-image block and the average value of the difference between the grayscale values ​​of the pixels is greater than a preset threshold;

[0092] If the number of pixels counted is greater than one percent of the total number of pixels in the sub-image block, the grayscale change degree inside the sub-image block is determined to be 1;

[0093] Otherwise, after eliminating the pixels whose grayscale value and the average value of the difference between the grayscale values ​​of the pixels are greater than the preset threshold, the difference between the grayscale values ​​of each pixel in the corresponding sub-image blocks between adjacent image frames is analyzed, and the grayscale change degree inside the sub-image block is calculated by the following formula:

[0094]

[0095] Where D is the grayscale variation within the sub-image block, d i is the difference between the grayscale values ​​of the i-th corresponding pixel between the current image frame and the adjacent image frame, S is the variance of the difference between the grayscale values ​​of the pixels between the current image frame and the adjacent image frame, and f i is the grayscale value of the i-th corresponding pixel in the current image frame, and n is the number of pixels to be calculated in the corresponding sub-image block between the current image frame and the adjacent image frame;

[0096] If the grayscale variation within the sub-image block exceeds a set threshold, it is determined that the corresponding sub-image block belongs to a sub-image block in a motion area.

[0097] In one embodiment of the present invention, determining the subdivided image subblocks belonging to the motion area in the subdivided image subblocks by using an inter-frame difference method includes the following steps:

[0098] For N subdivided image sub-blocks, the average grayscale value of all pixels in each subdivided image sub-block is calculated as the feature value of the corresponding subdivided image sub-block;

[0099] Performing a difference operation on two adjacent image frames to obtain the difference between the feature values ​​of all pixel points in the corresponding subdivided image sub-blocks between the adjacent image frames;

[0100] Calculate the average value of the difference between the characteristic values ​​of all pixels in the corresponding subdivided image sub-block;

[0101] If the average value of the difference between the characteristic values ​​of all pixels inside the subdivided image subblock is greater than a preset threshold, the subdivided image subblock is determined to belong to the motion area, otherwise, the subdivided image subblock is determined to belong to the background area.

[0102] In one embodiment of the present invention, determining a highly similar image area in the background area by image features of the image blocks in the background area and by similarity analysis between the image blocks includes the following steps:

[0103] Analyze the similarity between the background area image blocks of adjacent image frames through the image features of the background area image blocks, and divide the similarity between the background area image blocks of adjacent image frames according to the analysis results of the similarity between the background area image blocks, and count and mark the image blocks with high similarity in the background area image blocks of adjacent image frames;

[0104] Adjacent annotated highly similar image blocks in the image frame are grouped, and when the number of image blocks in the highly similar image block group is greater than or equal to N, the corresponding highly similar image block group is divided into highly similar image regions.

[0105] In one embodiment of the present invention, a search window is set in the background area, and an image block in the background area where the initial position of the search window is located is determined, and image features of the image block in the background area are extracted, including the following steps:

[0106] A search window is set in the background area, and an image block in the background area where the initial position of the search window is located is determined, and the search window slides in the image blocks in the background area in sequence;

[0107] The search window extracts the color histogram and texture features of the background area image blocks of adjacent image frames through the OpenCV library;

[0108] The color histogram of the background area image block of the adjacent image frame is divided into 8 bins for each color channel in the RGB color space, and the color histogram of each channel is represented as an array of length 8, and the color histogram of the entire image block is represented as an array of length 24;

[0109] Extract the LBP values ​​of the pixels of the background area image blocks in the texture features of the background area image blocks of the adjacent image frames through the LBP local binary pattern, and divide the LBP values ​​of the pixels of the background area image blocks into 24 groups, calculate the average LBP value of each group, and form an array with a length of 24;

[0110] The two obtained arrays are normalized and then concatenated to serve as image features of the background area image block.

[0111] In one embodiment of the present invention, analyzing the similarity between background area image blocks of adjacent image frames by using image features of background area image blocks includes the following steps:

[0112] Get the vector arrays of image features of background area image blocks of adjacent image frames, which are two n-dimensional vectors A(a1, a2,…, an) and B(b1, b2,…, bn). Calculate the Euclidean distance and cosine similarity between the corresponding image blocks in the background area of ​​adjacent frames using the following formula:

[0113]

[0114] Where d(A,B) is the Euclidean distance between corresponding image blocks in the background area between adjacent frames, i∈(1, 2,…,n);

[0115]

[0116] Among them, cos(A, B) is the cosine similarity between the corresponding image blocks in the background area between adjacent frames;

[0117] The obtained Euclidean distances between the corresponding image blocks in the background area between adjacent frames are normalized so that the Euclidean distance range is between [0,1]. The normalized Euclidean distance and cosine similarity are added to obtain the similarity analysis value between the image blocks in the background area of ​​adjacent image frames.

[0118] According to the analysis result of the similarity between the background area image blocks, the similarity degree between the background area image blocks of adjacent image frames is divided, including the following steps:

[0119] According to the analysis results of the similarity between the image blocks in the background area, the image blocks whose analysis values ​​of the similarity between the image blocks in the background area are greater than 1.6 are divided into highly similar image areas;

[0120] The image blocks whose similarity analysis values ​​between the image blocks in the background area are less than or equal to 1.6 and greater than 1.2 are divided into moderately similar image areas;

[0121] Image blocks whose similarity analysis values ​​between image blocks in the background area are less than or equal to 1.2 are divided into low-similarity image areas.

[0122] In one embodiment of the present invention, background region image fusion processing and motion region feature retention processing are performed on the spliced ​​image, including the following steps:

[0123] Performing image fusion on background area image blocks in the stitched adjacent image frames;

[0124] The motion region subdivision image sub-blocks are superimposed, and the feature vectors of the corresponding motion region subdivision image sub-blocks are extracted and added as labels to the corresponding motion region subdivision image sub-blocks.

[0125] In another embodiment of the present invention, a UAV low-altitude image processing system is provided, which adopts the above-mentioned UAV low-altitude image processing method to implement UAV low-altitude image processing.

[0126] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A method for processing low-altitude images of unmanned aerial vehicles, characterized in that: include: Obtain an image sequence collected by a drone at low altitude, and pre-process the image frames in the image sequence; Dividing the image frames of the preprocessed image sequence into blocks, and dividing each image frame into a plurality of sub-image blocks; Detecting the grayscale value of the sub-image block of each image frame, determining the sub-image block belonging to the motion area in the image frame by using the inter-frame difference method, performing secondary subdivision on the sub-image block belonging to the motion area, and obtaining the subdivided image sub-block of the motion area and the image block of the background area; A detection window is set in the motion region, and a subdivided image subblock of the motion region where an initial position of the detection window is located is determined, and a feature vector of each subdivided image subblock of the motion region is extracted; A search window is set in the background area, and an image block in the background area where the initial position of the search window is located is determined, and image features of the image block in the background area are extracted; By using the image features of the image blocks in the background area and the similarity analysis between the image blocks, a highly similar image area in the background area is determined; Based on the principle of the highest overlap of the divided highly similar image regions, adjacent image frames in the image sequence are spliced, and the spliced ​​images are subjected to background region image fusion processing and motion region feature retention processing; The grayscale value of the sub-image block of each image frame is detected, and the sub-image block belonging to the motion area in the image frame is determined by the inter-frame difference method, and the sub-image block belonging to the motion area is subdivided twice to obtain the subdivided image sub-block of the motion area and the background area image block, including the following steps: Detecting the grayscale value of the sub-image block of each image frame, determining the sub-image block belonging to the motion area in the image frame by using the inter-frame difference method, dividing the sub-image block belonging to the motion area into the motion area sub-image block, and dividing the remaining image blocks in the image frame into the background area sub-image blocks; The motion region sub-image block of the video frame is divided into N subdivided image sub-blocks of fixed size, and the subdivided image sub-blocks belonging to the motion region among the subdivided image sub-blocks are determined by using an inter-frame difference method, and the subdivided image sub-blocks belonging to the motion region are defined as motion region subdivided image sub-blocks, and the remaining sub-image blocks and subdivided image sub-blocks in the image frame are defined as background region image blocks; The stitched images are subjected to background region image fusion processing and motion region feature retention processing, including the following steps: Performing image fusion on background area image blocks in the stitched adjacent image frames; The motion region subdivision image sub-blocks are superimposed, and the feature vectors of the corresponding motion region subdivision image sub-blocks are extracted and added as labels to the corresponding motion region subdivision image sub-blocks.

2. The method for processing low-altitude images of unmanned aerial vehicles according to claim 1, characterized in that: Obtain an image sequence collected by a drone at low altitude and preprocess the image frames in the image sequence, including the following steps: Obtain the image sequence collected by the drone at low altitude, as well as the corresponding attitude data when the drone collects the image sequence; According to the corresponding posture data when the drone collects the image sequence, the pixel coordinates and the actual world coordinates of the drone are marked in the image frames of the image sequence collected by the drone at low altitude; Through the pixel coordinates of the drone and the actual world coordinates, coordinate matching is performed to establish a mapping relationship between the image coordinates and the actual world coordinates; According to the correspondence between the pixel coordinates of the drone and the actual world coordinates, the affine transformation matrix is ​​solved by the least squares method; The orthographic image is obtained by multiplying the affine transformation matrix by the inter-frame image coordinates taken by the drone at low altitude.

3. The method for processing low-altitude images of unmanned aerial vehicles according to claim 1, characterized in that: Detecting the grayscale value of the sub-image block of each image frame and determining the sub-image block belonging to the motion area in the image frame by using the inter-frame difference method includes the following steps: Detecting the grayscale values ​​of all pixels within the sub-image block of each image frame; Performing a difference operation on two adjacent image frames to obtain the difference between the grayscale values ​​of each pixel in the corresponding sub-image blocks between the adjacent image frames; Calculate the average value of the difference between the grayscale values ​​of each pixel in the corresponding sub-image block, and count the number of pixels whose difference between the grayscale value of each pixel in the sub-image block and the average value of the difference between the grayscale values ​​of the pixels is greater than a preset threshold; If the number of pixels counted is greater than one percent of the total number of pixels in the sub-image block, the grayscale change degree inside the sub-image block is determined to be 1; Otherwise, after eliminating the pixels whose grayscale value and the average value of the difference between the grayscale values ​​of the pixels are greater than the preset threshold, the difference between the grayscale values ​​of each pixel in the corresponding sub-image blocks between adjacent image frames is analyzed, and the grayscale change degree inside the sub-image block is calculated by the following formula: Where D is the grayscale variation within the sub-image block, d i is the difference between the grayscale values ​​of the i-th corresponding pixel between the current image frame and the adjacent image frame, S is the variance of the difference between the grayscale values ​​of the pixels between the current image frame and the adjacent image frame, and f i is the grayscale value of the i-th corresponding pixel in the current image frame, and n is the number of pixels to be calculated in the corresponding sub-image block between the current image frame and the adjacent image frame; If the grayscale variation within the sub-image block exceeds a set threshold, it is determined that the corresponding sub-image block belongs to a sub-image block in a motion area.

4. The method for processing low-altitude images of unmanned aerial vehicles according to claim 1, characterized in that: Determining the subdivided image subblocks belonging to the motion area in the subdivided image subblocks by using an inter-frame difference method includes the following steps: For N subdivided image sub-blocks, the average grayscale value of all pixels in each subdivided image sub-block is calculated as the feature value of the corresponding subdivided image sub-block; Performing a difference operation on two adjacent image frames to obtain the difference between the feature values ​​of all pixel points in the corresponding subdivided image sub-blocks between the adjacent image frames; Calculate the average value of the difference between the characteristic values ​​of all pixels in the corresponding subdivided image sub-block; If the average value of the difference between the characteristic values ​​of all pixels inside the subdivided image subblock is greater than a preset threshold, the subdivided image subblock is determined to belong to the motion area, otherwise, the subdivided image subblock is determined to belong to the background area.

5. The method for processing low-altitude images of unmanned aerial vehicles according to claim 1, characterized in that: By using the image features of the image blocks in the background area and the similarity analysis between the image blocks, a highly similar image area in the background area is determined, including the following steps: Analyze the similarity between the background area image blocks of adjacent image frames through the image features of the background area image blocks, and divide the similarity between the background area image blocks of adjacent image frames according to the analysis results of the similarity between the background area image blocks, and count and mark the image blocks with high similarity in the background area image blocks of adjacent image frames; Adjacent annotated highly similar image blocks in the image frame are grouped, and when the number of image blocks in the highly similar image block group is greater than or equal to N, the corresponding highly similar image block group is divided into highly similar image regions.

6. The method for processing low-altitude images of unmanned aerial vehicles according to claim 1, characterized in that: Setting a search window in the background area, determining an image block in the background area where the initial position of the search window is located, and extracting image features of the image block in the background area, includes the following steps: A search window is set in the background area, and an image block in the background area where the initial position of the search window is located is determined, and the search window slides in the image blocks in the background area in sequence; The search window extracts the color histogram and texture features of the background area image blocks of adjacent image frames through the OpenCV library; The color histogram of the background area image block of the adjacent image frame is divided into 8 bins for each color channel in the RGB color space, and the color histogram of each channel is represented as an array of length 8, and the color histogram of the entire image block is represented as an array of length 24; Extract the LBP values ​​of the pixels of the background area image blocks in the texture features of the background area image blocks of the adjacent image frames through the LBP local binary pattern, and divide the LBP values ​​of the pixels of the background area image blocks into 24 groups, calculate the average LBP value of each group, and form an array with a length of 24; The two obtained arrays are normalized and then concatenated to serve as image features of the background area image block.

7. The method for processing low-altitude images of unmanned aerial vehicles according to claim 6, characterized in that: Analyzing the similarity between the background area image blocks of adjacent image frames by using the image features of the background area image blocks includes the following steps: Get the vector arrays of image features of background area image blocks of adjacent image frames, which are two n-dimensional vectors A(a1, a2, …, an) and B(b1, b2, …, bn). Calculate the Euclidean distance and cosine similarity between the corresponding image blocks in the background area of ​​adjacent frames using the following formula: Where d(A,B) is the Euclidean distance between corresponding image blocks in the background area between adjacent frames, i∈(1, 2,…, n); Among them, cos(A, B) is the cosine similarity between the corresponding image blocks in the background area between adjacent frames; The obtained Euclidean distances between the corresponding image blocks in the background area between adjacent frames are normalized so that the Euclidean distance range is between [0,1]. The normalized Euclidean distance and cosine similarity are added to obtain the similarity analysis value between the image blocks in the background area of ​​adjacent image frames.

8. A UAV low-altitude image processing system, characterized in that: The system implements UAV low-altitude image processing by using a UAV low-altitude image processing method as described in any one of claims 1 to 7.

Images