Image processing method and device, unmanned equipment and storage medium

Abstract

The invention discloses an image processing method and device, unmanned equipment and a storage medium, and relates to the technical field of unmanned equipment, and the method comprises the steps: determining a first image and a second image according to two frames of adjacent aerial images, and carrying out the three-dimensional correction of the first image and the second image, and obtaining a first corrected image and a second corrected image; determining a non-blank area in the first corrected image according to a parallax range between the first corrected image and the second corrected image; and generating a disparity map of the two frames of adjacent aerial images according to the non-blank area and a semi-global matching algorithm. Through the technical means, the problem that the unmanned device performs parallax calculation on the blank area in the prior art is solved, redundant calculation is avoided, parallax noise is reduced, and the quality and generation efficiency of the parallax image are improved.

Description

Technical Field

[0001] The present application relates to the technical field of unmanned equipment, and in particular to an image processing method, device, unmanned equipment and storage medium. Background Art

[0002] SGBM (Semi-Global Block Matching) is a semi-global matching algorithm for calculating disparity in binocular vision. It is a compromise algorithm proposed based on the advantages and disadvantages of global matching algorithms and local matching algorithms to ensure the quality of the disparity map of binocular images and reduce the computational complexity. In the field of unmanned equipment technology, the disparity map of binocular images can be used to reconstruct dense point clouds, so the quality and generation efficiency of the disparity map can indirectly affect the quality and reconstruction efficiency of dense point clouds.

[0003] In the prior art, unmanned equipment can perform stereoscopic correction on aerial images according to the shooting posture of two adjacent aerial images, and determine the disparity map of two adjacent aerial images through the corrected aerial images and the SGBM algorithm. However, since the aerial images are taken by unmanned equipment along the route, the two adjacent aerial images are not conventional left and right binocular images. Therefore, after the stereoscopic correction of the two adjacent aerial images, the width and height of the aerial images are swapped, resulting in blank areas. As a result, the SGBM algorithm will also calculate the blank areas without disparity when processing the corrected aerial images, resulting in redundant calculations and the introduction of disparity noise, which reduces the quality and generation efficiency of the disparity map. Summary of the invention

[0004] The present application provides an image processing method, apparatus, unmanned equipment and storage medium, which identify non-blank areas of corrected aerial images to perform stereo matching on the non-blank areas, thereby solving the problem of unmanned equipment calculating blank areas in the prior art, avoiding redundant calculations and reducing disparity noise, and improving the quality and generation efficiency of disparity maps.

[0005] In a first aspect, the present application provides an image processing method, comprising:

[0006] Determine a first image and a second image according to two adjacent aerial image frames, and perform stereo correction on the first image and the second image to obtain a first corrected image and a second corrected image;

[0007] determining a non-blank area in the first corrected image according to a parallax range between the first corrected image and the second corrected image;

[0008] A disparity map of the two adjacent aerial images is generated according to the non-blank area and a semi-global matching algorithm.

[0009] In a second aspect, the present application provides an image processing device, comprising:

[0010] A stereo correction module is configured to determine a first image and a second image according to two adjacent aerial images, and perform stereo correction on the first image and the second image to obtain a first corrected image and a second corrected image;

[0011] an area determination module, configured to determine a non-blank area in the first corrected image according to a parallax range between the first corrected image and the second corrected image;

[0012] The disparity map construction module is configured to generate a disparity map of the two adjacent aerial images according to the non-blank area and a semi-global matching algorithm.

[0013] In a third aspect, the present application provides an unmanned device, including:

[0014] One or more processors; a memory storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in the first aspect.

[0015] In a fourth aspect, the present application provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to perform the image processing method as described in the first aspect.

[0016] In the present application, a first image and a second image are determined by two adjacent aerial images, and the first image and the second image are stereo-corrected to obtain a first corrected image and a second corrected image; a non-blank area in the first corrected image is determined according to the disparity range between the first corrected image and the second corrected image; a disparity map of two adjacent aerial images is generated according to the non-blank area and a semi-global matching algorithm. Through the above-mentioned technical means, after stereo-correcting the left and right binocular images, the non-blank area in the corrected first image is identified based on the disparity range between the left and right binocular images. Stereo matching is performed between the non-blank area in the corrected first image and the corrected second image through the SGBM algorithm, so as to avoid introducing blank areas without disparity into the calculation process of stereo matching, thereby solving the problem of disparity calculation of blank areas by unmanned equipment in the prior art, avoiding redundant calculation and reducing disparity noise, and improving the quality and generation efficiency of the disparity map. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic diagram of two adjacent aerial images provided in an embodiment of the present application;

[0018] Figure 2 is a flow chart of an image processing method provided by an embodiment of the present application;

[0019] Figure 3 is a flow chart of performing stereoscopic correction on two adjacent aerial images provided by an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of a first corrected image provided in an embodiment of the present application;

[0021] Figure 5 is a schematic diagram of a second corrected image provided in an embodiment of the present application;

[0022] Figure 6 is a flowchart of determining a non-blank area in a first corrected image provided by an embodiment of the present application;

[0023] Figure 7 is a schematic diagram of a mapping area of ​​a first corrected image provided in an embodiment of the present application;

[0024] Figure 8 is a flow chart of determining a non-blank area in a mapping area provided by an embodiment of the present application;

[0025] Fig. 9 is a schematic diagram of a non-blank area provided in an embodiment of the present application;

[0026] Fig.10 is a flow chart of generating a disparity map of two adjacent aerial images provided by an embodiment of the present application;

[0027] Fig.11 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0028] Fig.12 It is a structural schematic diagram of an unmanned device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0030] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0031] The image processing method provided in this embodiment can be executed by an unmanned device, which can be implemented by software and / or hardware, and can be composed of two or more physical entities or one physical entity. The unmanned device refers to a flying device that operates according to remote control instructions or preset instructions, such as a drone.

[0032] The unmanned device is installed with at least one type of operating system, and the unmanned device can install at least one application based on the operating system, and the application can be an application that comes with the operating system, or an application downloaded from a third-party device or server. In this embodiment, the unmanned device has at least an application that can execute the image processing method.

[0033] For ease of understanding, this embodiment is described by taking an unmanned device as an example of a subject that executes the image processing method.

[0034] In one embodiment, Figure 1 is a schematic diagram of two adjacent aerial images provided in an embodiment of the present application. Figure 1 As shown, when the unmanned device flies along the flight route 13, the camera captures image frame 11 when it flies to the shooting point B, and the camera captures image frame 12 when it continues to fly to the shooting point A. Image frames 11 and 12 are two adjacent aerial images collected by the unmanned device. The width of image frame 11 and image frame 12 is W, and the length is H. After the unmanned device performs stereo correction on image frame 11 and image frame 12, the SGBM algorithm is used to perform stereo matching on the corrected image frame 11 and image frame 12 to obtain a disparity map between image frame 11 and image frame 12. When the flight direction of the unmanned device is perpendicular to the width direction of the aerial image, the width and height of the aerial image are swapped after the stereo correction of the aerial image, resulting in a blank area with no pixel value in the corrected aerial image. Among them, the ratio of the blank area can be simply calculated as 1-H 2 / W 2Since the SGBM algorithm performs stereo matching on the corrected aerial images, that is, it calculates the blank areas where there is no disparity, thereby introducing disparity noise and causing redundant calculations, resulting in edge hypertrophy in the disparity map, which greatly reduces the quality and generation efficiency of the disparity map.

[0035] In order to solve the problems existing in the above-mentioned prior art, the present embodiment provides an image processing method, which avoids redundant calculations and reduces disparity noise by identifying the non-blank areas of the corrected aerial image and performing calculations on the non-blank areas, thereby improving the quality and generation efficiency of the disparity map.

[0036] Figure 2 A flowchart of an image processing method provided by an embodiment of the present application is given. Figure 2 , the image processing method specifically includes:

[0037] S110 , determining a first image and a second image according to two adjacent aerial image frames, and performing stereo correction on the first image and the second image to obtain a first corrected image and a second corrected image.

[0038] Among them, two frames of adjacent aerial images are two images continuously captured by the unmanned equipment through the camera, one of which can be determined as the first image and the other as the second image. The two frames of adjacent aerial images can be regarded as left and right binocular images, and accordingly, the first image can be regarded as the left image in the left and right binocular images, and the second image can be regarded as the right image in the left and right binocular images. Generally, the image captured first is regarded as the first image, and the image captured later is regarded as the second image.

[0039] This embodiment is based on Figure 1 The stereo matching of the image frame 11 and the image frame 12 shown is described as an example. Image frame 11 is determined as the first image, and image frame 12 is determined as the second image. The image frame 11 and the image frame 12 are subjected to distortion correction by using the distortion parameters of the camera carried by the unmanned device, and the image frame 11 and the image frame 12 after the distortion correction are subjected to stereo correction to obtain a first corrected image and a second corrected image. Among them, the first corrected image is the first image after stereo correction, and the second corrected image is the second image after stereo correction. Since both the image frame 11 and the image frame 12 are taken by the camera of the unmanned device, the distortion parameters for the distortion correction of the image frame 11 and the image frame 12 are the same.

[0040] The unmanned device can perform stereo correction on the dedistorted image frames 11 and 12 according to the shooting posture of the camera when shooting image frame 11 and the shooting posture of the camera when shooting image frame 12, so as to adjust the epipolar lines of image frame 11 and image frame 12 to the same level and map the antipodes to infinity, so that image frame 11 and image frame 12 only have parallax in the horizontal direction, that is, the stereo matching problem of image frame 11 and image frame 12 is reduced from two dimensions to one dimension, thereby improving the subsequent stereo matching speed.

[0041] In this embodiment, Figure 3 1 is a flow chart of performing stereoscopic correction on two adjacent aerial images provided by an embodiment of the present application. Figure 3 As shown, the step of performing stereoscopic correction on two adjacent aerial images specifically includes S1101-S1102:

[0042] S1101 . Determine a first stereo correction matrix for the first image and a second stereo correction matrix for the second image according to shooting positions of the first image and the second image.

[0043] Exemplarily, according to the shooting postures of the first image and the second image, the rotation matrix R and the translation matrix T between the shooting postures of the first image and the second image are determined. In order to minimize the image reprojection distortion, the rotation matrix R that rotates the camera plane of the second image to the camera plane of the first image is divided into two parts, called the two synthetic rotation matrices r1 and r2 of the first image and the second image. Each camera plane is rotated by half. Such a rotation can make the two camera planes coplanar but not aligned. In order to transform the pole corresponding to the first image to infinity and align the pole line horizontally, a matrix R starting from the pole direction can be created. rect . Calculate e1 = T / ||T||, where e1 is the pole in the same direction of the translation matrix T. Select the cross product of the principal optical axis direction (0, 0, 1) and e1 and perform normalization to obtain e2. e2 is orthogonal to e1. ​​The calculation formula of e2 is as follows:

[0044]

[0045] T=[T x T y T z ] T , T x 、T y and T z are the translation vectors in the x, y and z directions respectively, and e2 is the vector in the direction of the camera plane.

[0046] Calculate e3 = e1 * e2, and then transform the pole corresponding to the first image to the matrix R at infinity rect =[e1 T e2T e3 T ] T , the matrix R rect The first stereo correction matrix for stereo correction of the first image can be obtained by multiplying the rotation matrix r1 on the left. rect The second stereo correction matrix for performing stereo correction on the second image can be obtained by left-multiplying the rotation matrix r2.

[0047] S1102 . Map the first image using a first stereo correction matrix to obtain a first corrected image, and map the second image using a second stereo correction matrix to obtain a second corrected image.

[0048] Exemplarily, the first corrected image is obtained by performing a planar projection change on the pixels in the first image using the first stereo correction matrix. The second corrected image is obtained by performing a planar projection change on the pixels in the second image using the first stereo correction matrix.

[0049] In this embodiment, when the first image is subjected to stereoscopic correction, if a pixel in the first image can be mapped to the first corrected image, the pixel is marked, and if it cannot be mapped, the pixel is not marked. It can be understood that only when a pixel in the first image is mapped to the first corrected image, the pixel value of the pixel is retained in the first corrected image, otherwise the pixel value of the pixel is cleared so that the pixel becomes a completely black pixel or a completely white pixel.

[0050] S120 . Determine a non-blank area in the first corrected image according to a parallax range between the first corrected image and the second corrected image.

[0051] Among them, the parallax range between the first corrected image and the second corrected image is the parallax range composed of the maximum parallax and the minimum parallax between the first image and the second image. The image parallax is related to the image depth. The maximum parallax and the minimum parallax of the first image and the second image can be determined based on the image depth, and then the parallax range is determined. In this embodiment, the parallax range between the first corrected image and the second corrected image is determined according to the maximum depth and the minimum depth of two adjacent aerial images collected by the radar. Exemplarily, when the unmanned device shoots the image frame 11, the distance from the ground is collected by the ground radar, and the maximum depth and the minimum depth of the image frame 11 are determined according to the distance from the ground. When the unmanned device shoots the image frame 12, the distance from the ground is collected by the ground radar, and the maximum depth and the minimum depth of the image frame 12 are determined according to the distance from the ground. According to the maximum depth and the minimum depth of the image frame 12 and the maximum depth and the minimum depth of the image frame 11, the minimum parallax and the maximum parallax between the image frame 11 and the image frame 12 are determined, and the minimum parallax is used as the lower limit threshold of the parallax range and the maximum parallax is used as the upper limit threshold of the parallax range to obtain the parallax range.

[0052] The non-blank area can be understood as an area where the first corrected image can find corresponding matching pixels in the second corrected image. Figure 4 It is a schematic diagram of the first corrected image provided in an embodiment of the present application. Figure 5 is a schematic diagram of a second corrected image provided in an embodiment of the present application. Figure 4 and Figure 5 As shown, the pixels in the black area of ​​the first corrected image and the second corrected image have no pixel values, and the black area is the blank area generated after the first image and the second image are stereoscopically corrected. It can be understood that the completely black pixels in the first corrected image are pixels without pixel values, and the pixel points will not find matching pixels in the second corrected image, so the pixels without pixel values ​​in the first corrected image are pixels in the blank area. Correspondingly, if the pixel points in the second corrected image have no pixel values, then the pixel points matching the pixel points will not be found in the first corrected image. Since the first corrected image and the second corrected image only have parallax in the horizontal direction, that is, the horizontal coordinates of the pixel points in the first corrected image and the matching pixel points in the second corrected image are the same. Therefore, only when the pixel points in the first corrected image have pixel values, and the pixel points with pixel values ​​can be found in the corresponding parallax range in the second corrected image, it can be determined that there are pixel points in the second corrected image that match the pixel points in the first corrected image, that is, the pixel points in the first corrected image are determined to be pixels in the non-blank area.

[0053] In one embodiment, Figure 6 is a flow chart of determining a non-blank area in a first corrected image provided by an embodiment of the present application. Figure 6 As shown, the step of determining the non-blank area in the first corrected image specifically includes S1201-S1202:

[0054] S1201. Determine a mapping area in the first corrected image according to a first pixel point mapped from the first image to the first corrected image.

[0055] The first pixel is the pixel that is mapped to the first corrected image after stereo correction of the first image, that is, the pixel with a pixel value retained in the first corrected image. The mapping area is the area composed of the first pixel in the first corrected image. Figure 4 If a pixel of the first image cannot be mapped to the first corrected image, then after stereo correction, the pixel will become Figure 4 The pixel point in the first corrected image shown is completely black. On the contrary, if the pixel point of the first image can be mapped to the first corrected image, the pixel point will become Figure 4The pixel points with pixel values ​​in the first corrected image are shown. Therefore, in the process of stereo correction of the first image, the pixel points in the first corrected image that are successfully mapped can be marked, and after the stereo correction is completed, the pixel coordinates of the marked pixel points in the first corrected image are integrated to obtain the coordinate range of the mapping area. Figure 7 It is a schematic diagram of the mapping area of ​​the first corrected image provided in an embodiment of the present application. Figure 7 The white area in the figure is based on Figure 4 The mapping area is generated by the coordinates of the marked pixels.

[0056] S1202: Determine a non-blank area in the mapping area according to a disparity range between the first corrected image and the second corrected image.

[0057] refer to Figure 4 and Figure 7 , the pixel points in the mapping area are the pixel points with pixel values ​​in the first corrected image. Therefore, the first pixel point in the mapping area satisfies the prerequisite that the pixel points in the non-blank area have pixel values, and the non-blank area of ​​the first corrected image can be determined from the mapping area to improve the recognition efficiency of the non-blank area.

[0058] In this embodiment, Figure 8 FIG. 1 is a flowchart of determining a non-blank area in a mapping area provided by an embodiment of the present application. Figure 8 As shown, the step of determining the non-blank area in the mapping area specifically includes S12021-S12023:

[0059] S12021. Determine whether there is a second pixel point corresponding to the first pixel point in the second corrected image according to the disparity range between the first corrected image and the second corrected image and the position information of the first pixel point in the mapping area.

[0060] Among them, the second pixel point can be understood as a pixel point in the second corrected image that may match the first pixel point. Exemplarily, the area range where the pixel point corresponding to the first pixel point is located can be determined based on the pixel coordinates of the first pixel point in the mapping area and the parallax range between the first corrected image and the second corrected image. If there are pixels with pixel values ​​in the area range, it can be determined that there is a second pixel point corresponding to the first pixel point in the second corrected image. If there are no pixels with pixel values ​​in the area range, it can be determined that there is no second pixel point corresponding to the first pixel point in the second corrected image.

[0061] In this embodiment, the corresponding pixel row can be determined in the second corrected image according to the row coordinates of the first pixel in the mapping area; multiple third pixel points can be determined in the pixel row according to the column coordinates of the first pixel and the parallax range between the first corrected image and the second corrected image; when the pixel value of at least one third pixel satisfies the preset pixel range, it is determined that there is a second pixel point corresponding to the first pixel point in the second corrected image. Exemplarily, the pixel coordinates of the first pixel in the mapping area are (x, y), where x is the row coordinate of the first pixel point and y is the column coordinate of the first pixel point. Since there is only horizontal parallax between the first corrected image and the second corrected image, the row coordinates of the first pixel point corresponding to the matching pixel point in the second corrected image are the same. Therefore, the pixel point with row coordinate x in the second corrected image can be obtained first. Assuming the parallax range is [y1, y2], the pixel point with column coordinate [y+y1, y+y2] can be obtained from the pixel point with row coordinate x in the second corrected image as the third pixel point. Among them, the preset pixel range is the range of valid pixel values, that is, 0 to 255. It can be understood that when there is at least one pixel point whose pixel value is within the preset pixel range among the acquired third pixel points, it indicates that the pixel point is likely to be the pixel point that matches the first pixel point in the second corrected image, that is, it is determined that there is a second pixel point corresponding to the first pixel point in the second corrected image. When there is no pixel point whose pixel value is within the preset pixel range among the acquired third pixel points, it indicates that there is no pixel point that matches the first pixel point in the second corrected image, that is, it is determined that there is no second pixel point that matches the first pixel point in the second corrected image.

[0062] S12022. When there is a second pixel point corresponding to the first pixel point in the second corrected image, determine that the first pixel point is a target pixel point in the non-blank area.

[0063] Among them, the target pixel point is the first pixel point in the mapping area that is located in the non-blank area. Exemplarily, when there is a pixel point matching the first pixel point in the second corrected image, the first pixel point can be used as a pixel point in the non-blank area for subsequent stereo matching. When there is no pixel point matching the first pixel point in the second corrected image, the introduction of the first pixel point into the subsequent stereo matching will increase redundant calculations and introduce parallax noise, so the first pixel point can be used as a pixel point in the blank area. For example, most of the pixels in the lower half of the second corrected image have no pixel values, then the lower half of the first corrected image cannot find matching pixels in the lower half of the second corrected image, so the lower half of the first corrected image can also be regarded as a blank area.

[0064] S12023. Determine the position range of the non-blank area according to the position information of each target pixel point in the non-blank area.

[0065] For example, Fig. 9 It is a schematic diagram of the non-blank area provided in an embodiment of the present application. Fig. 9 The white area in the figure is based on Figure 7 The mapping area and Figure 5 The non-blank area is determined by the second corrected image in the non-blank area. The column coordinate range of the corresponding pixel row can be determined according to the column coordinates of the leftmost pixel point and the rightmost pixel point of each pixel row in the non-blank area, and the position range of the non-blank area is generated according to the column coordinate range of each pixel row.

[0066] S130 , generating a disparity map of two adjacent aerial images according to the non-blank area and the semi-global matching algorithm.

[0067] Exemplarily, the SAD-BT cost of each pixel in the non-blank area and the pixel in the corresponding disparity range in the second corrected image can be calculated by the SGBM algorithm, and the disparity map of image frame 11 and image frame 12 can be solved according to the SAD-BT cost.

[0068] In one embodiment, the processing parameters in the SGBM algorithm can be adjusted according to the position range of the non-blank area, that is, the non-blank area in the first corrected image and the second corrected image can be stereo matched by the adjusted SGBM algorithm to obtain the disparity map of the image frame 11 and the image frame 12. Exemplarily, Fig.10 : is a flow chart of generating a disparity map of two adjacent aerial images provided by an embodiment of the present application. Fig.10 As shown, the step of generating a disparity map of two adjacent aerial images specifically includes S1301-S1302:

[0069] S1301. Determine the calculation range of the cost space and the cost aggregation according to the position range of the non-blank area.

[0070] Exemplarily, the calculation range of the cost space in the original SGBM algorithm is x∈[0,W], y∈[0,H], where W and H are the width and height of the image frame, respectively. The cost space can be adjusted to the position range of the non-blank area, so that when the SGMB algorithm is subsequently run, the cost space of the non-blank area is established instead of the cost space of the first corrected image, avoiding the introduction of cost calculations in blank areas. Similarly, the calculation range of the cost aggregation in the original SGBM algorithm is x∈[0,W], y∈[0,H], and the cost aggregation can be adjusted to the position range of the non-blank area, so that when the SGMB algorithm is subsequently run, the cost aggregation of the non-blank area is calculated instead of the cost aggregation of the first corrected image, avoiding the introduction of cost calculations in blank areas.

[0071] S1302 , using the calculation range as a processing parameter of a semi-global matching algorithm, and performing stereo matching on the first corrected image and the second corrected image according to the semi-global matching algorithm to obtain a disparity map of two adjacent aerial images.

[0072] Exemplarily, after adjusting the calculation range of the cost space and the cost fitting, the SGBM algorithm calculates the SAD-BT cost in the first corrected image and the second corrected image within the calculation range, and solves the disparity maps of image frame 11 and image frame 12 based on the SAD-BT cost.

[0073] It should be noted that since the SGBM algorithm only calculates the SAD-BT cost of pixels in non-blank areas, it can save a lot of time in the cost calculation process, thereby improving the efficiency of disparity map generation. After experimental comparison, the processing time for stereo matching can be reduced by 10% after identifying the non-blank area, greatly improving the image processing efficiency. In terms of accuracy, compared with the disparity map generated directly based on the corrected image, the edges of the disparity map generated based on the non-blank area are obviously more concentrated and more accurate.

[0074] In summary, the image processing method provided by the embodiment of the present application determines the first image and the second image through two adjacent aerial images, stereoscopically corrects the first image and the second image to obtain the first corrected image and the second corrected image; determines the non-blank area in the first corrected image according to the disparity range between the first corrected image and the second corrected image; generates the disparity map of the two adjacent aerial images according to the non-blank area and the semi-global matching algorithm. Through the above-mentioned technical means, after stereoscopic correction of the left and right binocular images, the non-blank area in the corrected first image is identified based on the disparity range between the left and right binocular images. Stereo matching is performed between the non-blank area in the corrected first image and the corrected second image through the SGBM algorithm, so as to avoid introducing the blank area without disparity into the calculation process of stereo matching, solve the problem of disparity calculation of blank areas by unmanned equipment in the prior art, avoid redundant calculation and reduce disparity noise, and improve the quality and generation efficiency of the disparity map.

[0075] Based on the above embodiments, Fig.11 This is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. Fig.11 The image processing device provided in this embodiment specifically includes: a stereo correction module 21, a region determination module 22 and a disparity map construction module 23.

[0076] The stereo correction module 21 is configured to determine the first image and the second image according to two adjacent aerial images, and perform stereo correction on the first image and the second image to obtain a first corrected image and a second corrected image;

[0077] The area determination module 22 is configured to determine a non-blank area in the first corrected image according to a parallax range between the first corrected image and the second corrected image;

[0078] The disparity map construction module 23 is configured to generate a disparity map of two adjacent aerial images according to the non-blank area and the semi-global matching algorithm.

[0079] On the basis of the above embodiment, the stereo correction module 21 includes: a correction matrix determination submodule, which is configured to determine a first stereo correction matrix for the first image and a second stereo correction matrix for the second image according to the shooting posture of the first image and the second image; a stereo correction submodule, which is configured to map the first image through the first stereo correction matrix to obtain a first corrected image, and map the second image through the second stereo correction matrix to obtain a second corrected image.

[0080] Based on the above embodiment, the area determination module 22 includes: a mapping area determination submodule, which is configured to determine the mapping area in the first corrected image according to the first pixel point mapped from the first image to the first corrected image; and a non-blank area determination submodule, which is configured to determine the non-blank area in the mapping area according to the parallax range between the first corrected image and the second corrected image.

[0081] Based on the above embodiment, the non-blank area determination submodule includes: a judgment unit, configured to determine whether there is a second pixel point corresponding to the first pixel point in the second corrected image according to the disparity range between the first corrected image and the second corrected image, and the position information of the first pixel point in the mapping area; a target pixel point determination unit, configured to determine that the first pixel point is a target pixel point of the non-blank area when there is a second pixel point corresponding to the first pixel point in the second corrected image; and a non-blank area determination unit, configured to determine the position range of the non-blank area according to the position information of each target pixel point in the non-blank area.

[0082] On the basis of the above embodiment, the judgment unit includes: a pixel row determination subunit, configured to determine the corresponding pixel row in the second corrected image according to the row coordinates of the first pixel in the mapping area; a pixel point determination unit, configured to determine a plurality of third pixel points in the pixel row according to the column coordinates of the first pixel and the parallax range between the first corrected image and the second corrected image; and a judgment unit, configured to determine the existence of a second pixel corresponding to the first pixel in the second corrected image when the pixel value of at least one third pixel satisfies a preset pixel range.

[0083] On the basis of the above embodiment, the disparity map construction module 23 includes: a calculation range determination submodule, which is configured to determine the calculation range of the cost space and the cost aggregation according to the position range of the non-blank area; a disparity map generation module, which is configured to use the calculation range as a processing parameter of the semi-global matching algorithm, and stereo match the first corrected image and the second corrected image according to the semi-global matching algorithm to obtain a disparity map of two adjacent aerial images.

[0084] Based on the above embodiment, the area determination module 22 also includes: a disparity range determination submodule, which is configured to determine the disparity range between the first corrected image and the second corrected image according to the maximum depth and minimum depth of two adjacent aerial images collected by the radar before determining the non-blank area in the first corrected image according to the disparity range between the first corrected image and the second corrected image.

[0085] As mentioned above, the image processing device provided by the embodiment of the present application determines the first image and the second image through two adjacent aerial images, performs stereo correction on the first image and the second image to obtain the first corrected image and the second corrected image; determines the non-blank area in the first corrected image according to the disparity range between the first corrected image and the second corrected image; generates the disparity map of the two adjacent aerial images according to the non-blank area and the semi-global matching algorithm. Through the above technical means, after stereo correction of the left and right binocular images, the non-blank area in the corrected first image is identified based on the disparity range between the left and right binocular images. Stereo matching is performed between the non-blank area in the corrected first image and the corrected second image through the SGBM algorithm, so as to avoid introducing the blank area without disparity into the calculation process of stereo matching, solve the problem of disparity calculation of blank areas by unmanned equipment in the prior art, avoid redundant calculation and reduce disparity noise, and improve the quality and generation efficiency of the disparity map.

[0086] The image processing device provided in the embodiment of the present application can be used to execute the image processing method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0087] Fig.12 This is a schematic diagram of the structure of an unmanned device provided in an embodiment of the present application, with reference to Fig.12 The unmanned device includes: a processor 31, a memory 32, a communication device 33, an input device 34 and an output device 35. The number of processors 31 in the unmanned device can be one or more, and the number of memories 32 in the unmanned device can be one or more. The processor 31, memory 32, communication device 33, input device 34 and output device 35 of the unmanned device can be connected through a bus or other methods.

[0088] The memory 32, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the image processing method of any embodiment of the present application (for example, the stereo correction module 21, the region determination module 22 and the disparity map construction module 23 in the image processing device). The memory 32 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the memory 32 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0089] The communication device 33 is used for data transmission.

[0090] The processor 31 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 32, that is, implements the above-mentioned image processing method.

[0091] The input device 34 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 35 may include a display device such as a display screen.

[0092] The unmanned equipment provided above can be used to execute the image processing method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0093] An embodiment of the present application also provides a storage medium containing computer executable instructions, which, when executed by a computer processor, are used to execute an image processing method, the image processing method comprising: determining a first image and a second image based on two adjacent aerial image frames, performing stereo correction on the first image and the second image to obtain a first corrected image and a second corrected image; determining a non-blank area in the first corrected image based on a disparity range between the first corrected image and the second corrected image; and generating a disparity map of two adjacent aerial image frames based on the non-blank area and a semi-global matching algorithm.

[0094] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system, which is connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media residing in different locations (for example, in different computer systems connected by a network). The storage medium may store program instructions (for example, embodied as a computer program) that can be executed by one or more processors.

[0095] Of course, the storage medium containing computer executable instructions provided in the embodiment of the present application, whose computer executable instructions are not limited to the above image processing method, can also execute related operations in the image processing method provided in any embodiment of the present application.

[0096] The image processing device, storage medium and unmanned equipment provided in the above embodiments can execute the image processing method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the image processing method provided in any embodiment of the present application.

[0097] The above are only preferred embodiments of the present application and the technical principles used. The present application is not limited to the specific embodiments herein, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not deviate from the protection scope of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may also include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. An image processing method, characterized in that: include: Determine a first image and a second image according to two adjacent aerial image frames, and perform stereo correction on the first image and the second image to obtain a first corrected image and a second corrected image; determining a non-blank area in the first corrected image according to a parallax range between the first corrected image and the second corrected image; A disparity map of the two adjacent aerial images is generated according to the non-blank area and a semi-global matching algorithm.

2. The image processing method according to claim 1, characterized in that: The performing stereoscopic correction on the first image and the second image to obtain a first corrected image and a second corrected image includes: Determine a first stereo correction matrix for the first image and a second stereo correction matrix for the second image according to shooting postures of the first image and the second image; The first image is mapped using the first stereo correction matrix to obtain a first corrected image, and the second image is mapped using the second stereo correction matrix to obtain a second corrected image.

3. The image processing method according to claim 1, characterized in that: The determining, according to the parallax range between the first corrected image and the second corrected image, a non-blank area in the first corrected image comprises: Determining a mapping area in the first corrected image according to a first pixel point mapped from the first image to the first corrected image; A non-blank area is determined in the mapping area according to a disparity range between the first corrected image and the second corrected image.

4. The image processing method according to claim 3, characterized in that: The determining a non-blank area in the mapping area according to the parallax range between the first corrected image and the second corrected image includes: Determining whether there is a second pixel point corresponding to the first pixel point in the second corrected image according to the parallax range between the first corrected image and the second corrected image and the position information of the first pixel point in the mapping area; In a case where there is a second pixel point corresponding to the first pixel point in the second corrected image, determining the first pixel point as a target pixel point in a non-blank area; The position range of the non-blank area is determined according to the position information of each target pixel point in the non-blank area.

5. The image processing method according to claim 4, characterized in that: The determining, according to the parallax range between the first corrected image and the second corrected image and the position information of the first pixel in the mapping area, whether there is a second pixel corresponding to the first pixel in the second corrected image comprises: Determining a corresponding pixel row in the second corrected image according to the row coordinates of the first pixel point in the mapping area; Determining a plurality of third pixel points in the pixel row according to the column coordinates of the first pixel point and a parallax range between the first corrected image and the second corrected image; When a pixel value of at least one of the third pixel points satisfies a preset pixel range, it is determined that there is a second pixel point corresponding to the first pixel point in the second corrected image.

6. The image processing method according to claim 1, characterized in that: The step of generating the disparity map of the two adjacent aerial images according to the non-blank area and the semi-global matching algorithm includes: Determining a calculation range of a cost space and a cost aggregation according to a position range of the non-blank area; The calculation range is used as a processing parameter of the semi-global matching algorithm, and stereo matching is performed on the first corrected image and the second corrected image according to the semi-global matching algorithm to obtain a disparity map of the two adjacent aerial images.

7. The image processing method according to claim 1, characterized in that: Before determining the non-blank area in the first corrected image according to the parallax range between the first corrected image and the second corrected image, the method further includes: The parallax range between the first corrected image and the second corrected image is determined according to the maximum depth and the minimum depth of the two adjacent aerial images collected by the radar.

8. An image processing device, characterized in that: include: A stereo correction module is configured to determine a first image and a second image according to two adjacent aerial images, and perform stereo correction on the first image and the second image to obtain a first corrected image and a second corrected image; an area determination module, configured to determine a non-blank area in the first corrected image according to a parallax range between the first corrected image and the second corrected image; The disparity map construction module is configured to generate a disparity map of the two adjacent aerial images according to the non-blank area and a semi-global matching algorithm.

9. An unmanned device, characterized in that: include: one or more processors; A memory stores one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any one of claims 1-7.

10. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to perform the image processing method according to any one of claims 1 to 7 when executed by a computer processor.

Images