Disparity map generation method and device, unmanned equipment and storage medium

Abstract

The invention discloses a disparity map generation method and device, unmanned equipment and a storage medium, and relates to the technical field of unmanned equipment, and the method comprises the steps: obtaining a first image and a second image, and carrying out the stereo correction of the first image and the second image, and obtaining a first corrected image and a second corrected image; according to a preset window size, acquiring a neighborhood window corresponding to each first pixel point in the first corrected image, and determining a cost weight of each second pixel point in the neighborhood window; the first pixel point is a central pixel point corresponding to the neighborhood window; determining a first matching cost of each second pixel point according to the neighborhood window and the second correction image, and performing weighted summation on the first matching cost according to a cost weight to obtain a second matching cost of the first pixel point; and determining an optimal parallax value of each first pixel point according to the second matching cost of each first pixel point, and generating a target parallax image according to the optimal parallax value of each first pixel point.

Description

Technical Field

[0001] The present application relates to the technical field of unmanned equipment, and in particular to a disparity map generation method, device, unmanned equipment and storage medium. Background Art

[0002] SGBM (Semi-Global Block Matching) is a semi-global matching algorithm for calculating disparity in binocular vision. It is a compromise algorithm proposed based on the advantages and disadvantages of global matching algorithms and local matching algorithms to ensure the quality of the disparity map of binocular images and reduce the computational complexity. In the field of unmanned equipment technology, the disparity map of binocular images can be used to reconstruct dense point clouds or three-dimensional maps.

[0003] In the prior art, unmanned equipment can perform stereo correction on aerial images according to the shooting postures of two adjacent aerial images, and determine the disparity map of two adjacent aerial images through the corrected aerial images and the semi-global matching algorithm. However, the size of the neighborhood window for calculating the matching cost in the semi-global matching algorithm is limited, resulting in the inability of unmanned equipment to restore high-quality disparity maps in any scene, and the applicability of unmanned equipment is poor. Summary of the invention

[0004] The present application provides a disparity map generation method, device, unmanned equipment and storage medium, which determine the matching cost through adaptive weights to apply to neighborhood windows of various sizes while ensuring the disparity map generation effect, solves the problem in the prior art that unmanned equipment cannot restore high-quality disparity maps in any scenario, and improves the applicability of unmanned equipment.

[0005] In a first aspect, the present application provides a disparity map generation method, comprising:

[0006] Acquire a first image and a second image, and perform stereoscopic correction on the first image and the second image to obtain a first corrected image and a second corrected image;

[0007] According to a preset window size, a neighborhood window corresponding to each first pixel point is obtained in the first corrected image, and a cost weight of each second pixel point in the neighborhood window is determined; the first pixel point is a central pixel point of the corresponding neighborhood window;

[0008] Determine a first matching cost of each of the second pixel points according to the neighborhood window and the second corrected image, and perform weighted summation of the first matching costs according to the cost weight to obtain a second matching cost of the first pixel point;

[0009] An optimal disparity value for each of the first pixels is determined according to the second matching cost of each of the first pixels, and a target disparity map is generated according to the optimal disparity value for each of the first pixels.

[0010] In a second aspect, the present application provides a disparity map generating device, comprising:

[0011] A stereo correction module is configured to acquire a first image and a second image, and perform stereo correction on the first image and the second image to obtain a first corrected image and a second corrected image;

[0012] A weight determination module is configured to obtain a neighborhood window corresponding to each first pixel point in the first corrected image according to a preset window size, and determine a cost weight of each second pixel point in the neighborhood window; the first pixel point is a central pixel point of the corresponding neighborhood window;

[0013] a cost determination module, configured to determine a first matching cost of each of the second pixel points according to the neighborhood window and the second corrected image, and to perform weighted summation of the first matching costs according to the cost weights to obtain a second matching cost of the first pixel point;

[0014] The disparity map generation module is configured to determine the optimal disparity value of each of the first pixels according to the second matching cost of each of the first pixels, and generate a target disparity map according to the optimal disparity value of each of the first pixels.

[0015] In a third aspect, the present application provides an unmanned device, including:

[0016] One or more processors; a memory storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the disparity map generation method as described in the first aspect.

[0017] In a fourth aspect, the present application provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the disparity map generation method as described in the first aspect.

[0018] In the present application, by acquiring a first image and a second image, stereo correction is performed on the first image and the second image to obtain a first corrected image and a second corrected image; according to a preset window size, a neighborhood window corresponding to each first pixel is obtained in the first corrected image, and the cost weight of each second pixel in the neighborhood window is determined; the first pixel is the central pixel of the corresponding neighborhood window; according to the neighborhood window and the second corrected image, the first matching cost of each second pixel is determined, and the first matching cost is weighted and summed according to the cost weight to obtain the second matching cost of the first pixel; according to the second matching cost of each first pixel, the optimal disparity value of each first pixel is determined, and the target disparity map is generated according to the optimal disparity value of each first pixel. Through the above technical means, when the neighborhood window of the first pixel is increased, the cost contribution degree of each second pixel in the neighborhood window can be automatically adjusted by adaptive weights to avoid the overflow of the cost value of the first pixel, and more local feature information around the first pixel is obtained, which is conducive to improving the disparity estimation effect. The neighborhood window can be increased or reduced according to actual needs, and is no longer limited to the existing window size, which solves the problem that unmanned equipment in the prior art cannot restore high-quality disparity maps in any scene, and improves the applicability of unmanned equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of a disparity map generation method provided in an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of a neighborhood window provided in an embodiment of the present application;

[0021] Figure 3 is a flowchart of determining a first matching cost of a second pixel point based on a disparity range provided by an embodiment of the present application;

[0022] Figure 4 is a flow chart of calculating the cost of a second pixel point and a corresponding third pixel point provided by an embodiment of the present application;

[0023] Figure 5 is a flow chart of determining an optimal disparity value of a first pixel point provided by an embodiment of the present application;

[0024] Figure 6 is a schematic diagram of a cost space provided in an embodiment of the present application;

[0025] Figure 7 is a structural schematic diagram of a disparity map generating device provided in an embodiment of the present application;

[0026] Figure 8 It is a structural schematic diagram of an unmanned device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0028] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0029] The disparity map generation method provided in this embodiment can be executed by an unmanned device, which can be implemented by software and / or hardware, and can be composed of two or more physical entities or one physical entity. The unmanned device refers to a flying device or ground platform that operates according to remote control instructions or preset instructions, such as a drone and an unmanned vehicle.

[0030] The unmanned device is installed with at least one type of operating system, and the unmanned device can install at least one application based on the operating system, and the application can be an application that comes with the operating system, or an application downloaded from a third-party device or server. In this embodiment, the unmanned device has at least an application that can execute the disparity map generation method.

[0031] For ease of understanding, this embodiment is described by taking an unmanned device as an example of a subject that executes the disparity map generation method.

[0032] In one embodiment, the unmanned device can perform stereo correction on the aerial image according to the shooting posture of two adjacent frames of aerial images, and determine the disparity map of the two adjacent frames of aerial images through the corrected aerial image and the semi-global matching algorithm. When the unmanned device runs the semi-global matching algorithm, the matching cost of the pixel point in the aerial image can be calculated by other pixels in the neighborhood window of the pixel point. When the size of the neighborhood window is too large, the matching cost of the feature-rich area will be greater, resulting in the accumulation of the matching cost of other pixels in the neighborhood window close to the maximum value of the short type, and the subsequent cost aggregation will further accumulate the aggregated cost, eventually resulting in the overflow of the cost value. For slender objects in aerial images, there is always a circle of continuous erroneous matching values ​​at their edge positions, which causes the edge of the slender object to expand by a circle. This edge fattening problem cannot be eliminated by consistency checks and other means, so the estimated error of the slender object will be fatter than the actual one. If the size of the neighborhood window is simply increased, the edge fattening problem of the slender object will continue to be aggravated, resulting in a more serious edge hypertrophy phenomenon. Therefore, the size of the neighborhood window used by existing unmanned devices to calculate the matching cost is limited, resulting in the inability of unmanned devices to restore high-quality disparity maps in all scenarios, and the applicability of unmanned devices is poor. For example, in a scene with a cylindrical surface, increasing the size of the neighborhood window can better restore the cylindrical surface with a slight disparity transition, but unmanned devices do not support increasing the size of the neighborhood window.

[0033] In order to solve the problems existing in the above-mentioned prior art, this embodiment provides a disparity map generation method, which determines the matching cost through adaptive weights to apply to neighborhood windows of various sizes while ensuring the disparity map generation effect, thereby improving the applicability of unmanned equipment.

[0034] Figure 1 A flow chart of a disparity map generation method provided by an embodiment of the present application is given. Figure 1 , the disparity map generation method specifically includes:

[0035] S110 , acquiring a first image and a second image, and performing stereoscopic correction on the first image and the second image to obtain a first corrected image and a second corrected image.

[0036] The first image and the second image are two frames of aerial images continuously captured by the unmanned equipment through the camera. The two frames of aerial images can be regarded as left and right binocular images. Accordingly, the first image can be regarded as the left image in the left and right binocular images, and the second image can be regarded as the right image in the left and right binocular images. Generally, the aerial image captured first is regarded as the first image, and the aerial image captured later is regarded as the second image.

[0037] The first image and the second image are subjected to distortion correction by using the distortion parameters of the camera carried by the unmanned device, and the first image and the second image after the distortion correction are subjected to stereo correction to obtain the first corrected image and the second corrected image. Among them, the first correction U-shape is the first image after stereo correction, and the second correction image is the second image after stereo correction. Since the first image and the second image are both taken by the camera of the unmanned device, the distortion parameters for the distortion correction of the first image and the second image are the same.

[0038] The unmanned device can perform stereo correction on the dedistorted first image and the second image according to the shooting posture of the camera when shooting the first image and the shooting posture of the camera when shooting the second image, so as to adjust the epipolar lines of the first image and the second image to the same level and map the antipodes to infinity, so that the corrected first image and the second image only have parallax in the horizontal direction. At this time, the stereo matching problem of the first image and the second image is reduced from two dimensions to one dimension, thereby improving the subsequent stereo matching speed.

[0039] In this embodiment, illustratively, according to the shooting postures of the first image and the second image, the rotation matrix R and the translation matrix T between the shooting postures of the first image and the second image are determined. In order to minimize the image reprojection distortion, the rotation matrix R for rotating the camera plane of the second image to the camera plane of the first image is divided into two parts, namely, two synthetic rotation matrices r1 and r2 of the first image and the second image. Each camera plane is rotated by half. Such a rotation can make the two camera planes coplanar but not aligned. In order to transform the pole corresponding to the first image to infinity and align the pole lines horizontally, a matrix R starting from the pole direction can be created. rect . Calculate e1 = T / ||T||, where e1 is the pole in the same direction of the translation matrix T. Select the cross product of the principal optical axis direction (0,0,1) and e1 and perform normalization to obtain e2. e2 is orthogonal to e1. ​​The calculation formula of e2 is as follows:

[0040]

[0041] T=[T x T y T z ] T , T x , T y and T z are the translation vectors in the x, y and z directions respectively, and e2 is the vector in the direction of the camera plane.

[0042] Calculate e3 = e1 * e2, and transform the pole corresponding to the first image to the matrix R at infinity rect =[e1 T e2 T e3 T] T , the matrix R rect The first stereo correction matrix for stereo correction of the first image can be obtained by multiplying the rotation matrix r1 on the left. rect The second stereo correction matrix for stereo correction of the second image can be obtained by left multiplying the rotation matrix r2. The first stereo correction matrix is ​​used to perform a plane projection change on the pixels in the first image to obtain the first corrected image. The first stereo correction matrix is ​​used to perform a plane projection change on the pixels in the second image to obtain the second corrected image.

[0043] S120. According to a preset window size, obtain a neighborhood window corresponding to each first pixel in the first corrected image, and determine a cost weight of each second pixel in the neighborhood window; the first pixel is the central pixel of the corresponding neighborhood window.

[0044] The window size is the size of the neighborhood window used to calculate the matching cost. In the existing semi-global matching algorithm, the size of the neighborhood window is generally 3 or 5, and this embodiment supports setting a larger window size, for example, setting the window size to 11. This embodiment is described by taking the preset window size equal to 11 as an example. The first pixel point is any pixel point in the first corrected image, and the second pixel point is any pixel point in the neighborhood window corresponding to the first pixel point. Figure 2 is a schematic diagram of a neighborhood window provided in an embodiment of the present application. Figure 2 As shown, the first pixel 12 is taken as the central pixel of the corresponding neighborhood window 14, and 11 pixels are taken as the length and width of the neighborhood window 14, and the pixels around the first pixel 12 in the first corrected image 10 that fall within the neighborhood window 14 are taken as the second pixel 13 in the neighborhood window 14 corresponding to the first pixel 11. If the pixel coordinates of the first pixel 12 in the i-th row and j-th column in the first corrected image 10 are (i, j), then the neighborhood window 13 of the first pixel 12 is a rectangular area surrounded by four pixels with pixel coordinates of (i-5, j-5), (i-5, j+5), (i+5, j-5) and (i+5, j+5) in the first corrected image 10, and then the pixel coordinates of each second pixel 13 in the neighborhood window 13 in the first corrected image 10 can be determined.

[0045] The cost weight of the second pixel is a parameter used to adjust the contribution of the matching cost of the second pixel to the matching cost of the corresponding first pixel. It can be understood that in the existing semi-global matching algorithm, the cost weight of the second pixel is 1, that is, the matching cost of the first pixel is equal to the sum of the matching costs of all second pixels in the neighborhood window. In this embodiment, when the size of the neighborhood window is increased, the contribution of the matching cost of each second pixel in the neighborhood window can be adjusted by the cost weight, rather than adding the matching costs of the second pixels, thereby reducing the matching cost of the first pixel and avoiding the risk of overflow of the cost value to a certain extent. Since the size of the neighborhood window is increased, the feature information of more pixels around the first pixel can be obtained. When there is a cylindrical surface in the shooting scene, the cylindrical surface with a slight parallax transition can be better restored.

[0046] In one embodiment, the cost weight of the second pixel point can be a pre-set value less than one, and the cost weight of the second pixel point in the neighborhood window is the same, or as the pixel distance between the second pixel point and the first pixel point increases, the cost weight of the second cost weight decreases accordingly.

[0047] In another embodiment, the cost weight of each second pixel can be determined based on the grayscale value of the center pixel of the neighborhood window and the grayscale value of each second pixel. It can be understood that the grayscale value of the center pixel of the neighborhood window is the grayscale value of the first pixel, and the L1 distance (block distance) between the first pixel and the second pixel in the RGB space can be determined based on the grayscale value of the first pixel and the grayscale value of any second pixel in the corresponding neighborhood window. The smaller the L1 distance, the greater the possibility that the first pixel and the second pixel are located on the same plane, that is, the higher the correlation between the first pixel and the second pixel, the higher the cost weight of the second pixel. The larger the L1 distance, the smaller the possibility that the first pixel and the second pixel are located on the same plane, that is, the lower the correlation between the first pixel and the second pixel, the smaller the cost weight of the second pixel. In this embodiment, the cost weight of the second pixel can be determined by a preset weight calculation formula and the grayscale value of the center pixel of the neighborhood window and the grayscale value of the second pixel; the preset weight calculation formula is:

[0048]

[0049] Among them, p is the center pixel of the neighborhood window, q is the second pixel of the neighborhood window, and I p is the gray value of the center pixel, I q is the gray value of the second pixel, γ is the weight parameter, and w(p,q) is the cost weight of the second pixel. ||I p -I q|| can represent the L1 distance between the first pixel and the corresponding second pixel in the RGB space. From the above weight calculation formula, it can be seen that the smaller the grayscale difference between the first pixel and the second pixel, that is, the closer the colors of the first pixel and the second pixel are, the smaller the L1 distance is, the higher the correlation between the first pixel and the second pixel, and the higher the cost weight of the second pixel. The larger the grayscale difference between the first pixel and the second pixel, that is, the less similar the colors of the first pixel and the second pixel are, the larger the L1 distance is, the smaller the correlation between the first pixel and the second pixel, and the lower the cost weight of the second pixel.

[0050] It should be noted that p is both the central pixel point of the neighborhood window and the second pixel point of the neighborhood window. When calculating the cost weight corresponding to p, ||I p -I p ||=0, then w(p, p)=1, that is, the cost weight of the center pixel is 1, which is the largest weight among all the cost weights of the second pixel points. Due to the constraint of γ, the calculated cost weights of the second pixel points except the center pixel point are all less than 1 or equal to 1.

[0051] S130 , determining a first matching cost of each second pixel point according to the neighborhood window and the second corrected image, and performing weighted summation on the first matching costs according to the cost weights to obtain a second matching cost of the first pixel point.

[0052] Among them, the first matching cost is the cost value of matching between the second pixel point and the pixel point in the second corrected image at the corresponding disparity value. The second matching cost is the cost value of matching between the first pixel point and the pixel point in the second corrected image at the corresponding disparity value. Exemplarily, according to the pixel row coordinates of each second pixel point in the neighborhood window, a third pixel point that may match the second pixel point is determined in the pixel row corresponding to the second corrected image, and according to the gray value of the second pixel point and the gray value of the corresponding third pixel point, the first matching cost of the second pixel point and the corresponding third pixel point is calculated, so as to characterize the cost of matching the second pixel point and the corresponding third pixel point at the corresponding disparity value through the first matching cost. Furthermore, according to the cost weight of each second pixel point, the first matching cost of each second pixel point at the same disparity value is weighted and summed to obtain the second matching cost of the first pixel point at the corresponding disparity value.

[0053] In another embodiment, the disparity range between the first image and the second image can be determined based on the maximum disparity value and the minimum disparity value between the first image and the second image, wherein the maximum disparity value is the upper threshold of the disparity range, and the minimum disparity value is the lower threshold of the disparity range. Image disparity is related to image depth, that is, the disparity range has been determined when the camera captures the first image and the second image, and the first matching cost between the second pixel and the third pixel at the corresponding disparity value can be determined based on the disparity range, so as to subsequently quickly determine the optimal disparity value of each first pixel in the first corrected image. Exemplarily, Figure 3 is a flowchart of determining a first matching cost of a second pixel point based on a disparity range provided by an embodiment of the present application. Figure 3 As shown, the step of determining the first matching cost of the second pixel point based on the disparity range specifically includes S1301-S1302:

[0054] S1301. Determine, according to a disparity range between a first image and a second image, a third pixel point in a second corrected image corresponding to each second pixel point in a neighborhood window.

[0055] Among them, the third pixel point is a pixel point that may match the second pixel point. Exemplarily, the pixel row corresponding to the second corrected image is determined according to the pixel row coordinates of the second pixel point, and the third pixel point corresponding to the second pixel point is determined in the pixel row according to the parallax range between the first image and the second image. For example, the pixel coordinates of the second pixel point are (x, y), where x is the pixel row coordinate and y is the pixel column coordinate. The parallax range is [y1, y2], then the pixel point with column coordinates [y+y1, y+y2] can be obtained from the pixel point with row coordinate x in the second corrected image as the third pixel point of the second pixel point.

[0056] S1302 , calculating the cost of the second pixel and the corresponding third pixel to obtain a first matching cost of the second pixel.

[0057] The first matching cost is the cost of matching the second pixel with the third pixel under the corresponding disparity. For example, the first matching cost between the second pixel and each third pixel within the disparity range can be calculated by using the grayscale absolute difference algorithm (AD, Absolute Differences). For example, the grayscale value of the second pixel q is I q , the third pixel n d The gray value is I nd , d represents the second pixel q and the third pixel n d The disparity between them is d. Calculate |I q -I nd | is the first matching cost of the second pixel point q when the disparity value is d.

[0058] In another embodiment, Figure 4 : is a flowchart of calculating the cost of a second pixel and a corresponding third pixel provided by an embodiment of the present application. Figure 4 As shown, the step of calculating the cost of the second pixel point and the corresponding third pixel point specifically includes S13021-S13023:

[0059] S13021. Determine the grayscale value of the first sub-pixel between the third pixel and the corresponding adjacent pixel based on the adjacent pixel points of the third pixel, determine the first difference between the grayscale value of the second pixel and the first sub-pixel based on the minimum grayscale value and the maximum grayscale value between the third pixel and the third pixel, and determine the third matching cost based on the first difference.

[0060] For example, for the third pixel n d , according to the pixel coordinates of the third pixel, obtain the left adjacent pixel and the right adjacent pixel of the third pixel in the second corrected image. Calculate the average of the grayscale value of the third pixel and the grayscale value of the left adjacent pixel, and use the average as the grayscale value of the first sub-pixel between the third pixel and the left adjacent pixel. Calculate the average of the grayscale value of the third pixel and the grayscale value of the right adjacent pixel, and use the average as the grayscale value of the first sub-pixel between the third pixel and the right adjacent pixel. Compare the grayscale value of the first sub-pixel between the third pixel and the left adjacent pixel, the grayscale value of the first sub-pixel between the third pixel and the right adjacent pixel, and the grayscale value of the third pixel, and determine the maximum grayscale value and the minimum grayscale value among the three. Subtract the maximum grayscale value from the grayscale value of the second pixel to obtain a first difference, and subtract the minimum grayscale value from the grayscale value of the second pixel to obtain another first difference. Compare the two first differences and zero, and use the maximum value as the third matching cost.

[0061] S13022. Determine the grayscale value of the second sub-pixel between the second pixel and the corresponding adjacent pixel based on the adjacent pixel points of the second pixel, determine the second difference with the grayscale value of the third pixel based on the minimum grayscale value and the maximum grayscale value between the second sub-pixel and the second pixel, and determine the fourth matching cost based on the second difference.

[0062] Exemplarily, for the second pixel q, according to the pixel coordinates of the second pixel, the left adjacent pixel and the right adjacent pixel of the second pixel are obtained in the first corrected image. The average value of the grayscale value of the second pixel and the grayscale value of the left adjacent pixel is calculated, and the average value is used as the grayscale value of the second sub-pixel between the second pixel and the left adjacent pixel. The average value of the grayscale value of the second pixel and the grayscale value of the right adjacent pixel is calculated, and the average value is used as the grayscale value of the second sub-pixel between the second pixel and the left adjacent pixel, the grayscale value of the second sub-pixel between the second pixel and the right adjacent pixel, and the grayscale value of the second pixel are compared to determine the maximum grayscale value and the minimum grayscale value among the three. The grayscale value of the third pixel is subtracted from the maximum grayscale value to obtain a second difference, and the grayscale value of the third pixel is subtracted from the minimum grayscale value to obtain another second difference. The two second differences are compared with zero, and the maximum value is used as the fourth matching cost.

[0063] S13023. Add the third matching cost and the fourth matching cost to obtain a first matching cost of the second pixel point.

[0064] Exemplarily, the sum of the third matching cost and the fourth matching cost is determined as the first matching cost of the second pixel point q when the disparity value is k.

[0065] Furthermore, the product of the cost weight of each second pixel in the neighborhood window and the first matching cost is added to obtain the second matching cost of the first pixel. Exemplarily, the cost weight of the second pixel is multiplied by the second matching cost of the second pixel at each disparity value within the disparity range. The cost weight of each second pixel in the neighborhood window at the same disparity value is added to the product of the second matching cost to obtain the first matching cost of the first pixel corresponding to the neighborhood window at the corresponding disparity value. For example, the cost weight of the second pixel is multiplied by the first matching cost of each second pixel when the disparity value is d, and the sum of the products is determined as the second matching cost corresponding to the first pixel when the disparity value is d. The calculation formula for the first matching cost of the first pixel when the disparity value is d is as follows:

[0066]

[0067] Wherein, m(p) is the first matching cost of the first pixel point p when the disparity value is d. p is a neighborhood window with the first pixel as the center pixel, w(p, q) is the cost weight of the second pixel, and ρ(q, qd) represents the second matching cost of the second pixel when the disparity value is d.

[0068] It should be noted that after increasing the neighborhood window, the cost contribution of the corresponding pixel is automatically adjusted according to the color difference between the first pixel and other pixels in the neighborhood window. This can avoid cost value overflow and effectively obtain feature information of pixels with greater correlation. This makes the second matching cost of the first pixel calculated by the cost weight more consistent with the actual cost of the corresponding pixel, thereby effectively solving the problem of edge fattening of slender objects and improving the boundary effect of the disparity map.

[0069] S140 , determining an optimal disparity value for each first pixel according to the second matching cost of each first pixel, and generating a target disparity map according to the optimal disparity value for each first pixel.

[0070] For example, the second matching cost of the first pixel when the disparity value is d can measure the cost of matching it with the corresponding pixel in the second corrected image. The smaller the second matching cost, the greater the correlation between the first pixel and the corresponding pixel in the second corrected image, and the greater the probability that the two pixels are homonymous points. Therefore, the optimal disparity value can be understood as the maximum probability that the first pixel and the corresponding pixel are homonymous points.

[0071] In one embodiment, the second matching costs of each first pixel under various disparity values ​​within the disparity range may be compared, and the disparity value with the smallest second matching cost is used as the optimal disparity value corresponding to the first pixel. However, the target disparity map generated in this way only considers local information, which is easily affected by image noise, weak texture or repeated texture area. The cost value calculated by the local information may not accurately reflect the correlation between pixels, so the generated target disparity map will be very inaccurate.

[0072] In another embodiment, the connection between adjacent pixels can be established through cost aggregation, and the first matching cost of the first pixel can be optimized based on certain criteria, such as adjacent pixels should have continuous disparity values. This optimization is often global, and the first matching cost of each first pixel at a certain disparity value will be recalculated based on the first matching cost of its adjacent pixels at the same disparity value or nearby disparity values. The cost distance is similar to a disparity propagation step, and the matching effect is good in areas with high signal-to-noise ratios. The initial cost can reflect the correlation well, and the optimal disparity value can be obtained more accurately. It is propagated to areas with low signal-to-noise ratios and poor matching effects through cost aggregation, and ultimately the cost values ​​of all first pixels can accurately reflect the true correlation, which is beneficial to improve the accuracy of the subsequently generated target disparity map. Exemplarily, Figure 5 is a flow chart of determining the optimal disparity value of the first pixel provided by an embodiment of the present application. Figure 5 As shown, the step of determining the optimal disparity value of the first pixel point specifically includes S1401-S1402:

[0073] S1401 . Generate a cost space according to the second matching cost of each first pixel.

[0074] Figure 6 Schematic diagram of the cost space provided by the embodiment of the present application. Figure 6 As shown, based on the second matching cost of each pixel at a certain disparity value d within the disparity range, a corresponding cost image 15 is generated. The cost images corresponding to all disparity values ​​within the disparity range [y1, y2] are stacked from the upper layer to the lower layer in ascending order of the disparity values ​​to obtain a cost space. Among them, any generation value in the cost space can be expressed as C(p, D p ), which is equal to the first pixel point p at the disparity value D p The second matching cost when .

[0075] S1402 , cost aggregation is performed on the cost space to obtain an aggregated cost matrix, and a disparity value corresponding to a minimum aggregated cost value of a first pixel is determined according to the aggregated cost matrix as an optimal disparity value of the first pixel.

[0076] Exemplarily, a global energy function E(D) is set, and the set global energy function is minimized to solve the optimal disparity value of each first pixel. The global energy function E(D) is as follows:

[0077]

[0078] Among them, p and k are the first pixel points, N p is the pixel point adjacent to the first pixel point p. p ) is the first pixel point p at the disparity value D p P1 is a penalty coefficient used to calculate the pixel whose disparity value among the neighboring pixels of the first pixel p differs from the disparity value of the first pixel p by 1; P2 is another penalty coefficient used to calculate the pixel whose disparity value among the neighboring pixels of the first pixel p differs from the disparity value of the first pixel p by more than 1. The I[] function indicates that the return value is 1 if the parameter in the function is true, otherwise it returns 0.

[0079] The minimum cost value of the first pixel p is calculated using the minimum cost value calculation formula. The minimum cost value calculation formula is as follows:

[0080]

[0081] Where r is the adjacent pixel to the left of the first pixel p; L r(p, d) represents the adjusted minimum cost value corresponding to the first pixel point p when the disparity value of the first pixel point p is d along the current direction. When the disparity value of the left adjacent pixel point is d, the adjusted minimum cost value corresponding to the first pixel point p is its minimum cost value itself. When the disparity value of the left adjacent pixel point is d-1, the adjusted minimum cost value corresponding to the first pixel point p is its minimum cost value plus the penalty coefficient P1. When the disparity value of the left adjacent pixel point is d+1, the adjusted minimum cost value corresponding to the first pixel point p is its minimum cost value plus the penalty coefficient P1. When the disparity value of the left adjacent pixel point is other, the adjusted minimum cost value corresponding to the first pixel point p is its minimum cost value plus the penalty coefficient P2.

[0082] Furthermore, the cost value of the first pixel point p is calculated minus the minimum cost value C(p, d) when the previous pixel point takes different disparity values. The calculation formula is as follows:

[0083] C(p, d) = min(d(p, pd, I L , I R ),d(pd,I R ,I L ))

[0084] Among them, d(pd,I R ,I L )=min p-d-0.5≤p-d+0.5 |(I L (p)-I L (q)

[0085] Calculate multiple cost values ​​of the first pixel point p along different directions, accumulate the multiple cost values ​​calculated in each direction, and select the disparity value in the direction with the smallest accumulated cost value as the optimal disparity value of the first pixel point.

[0086] After determining the optimal disparity value of each first pixel, a disparity map can be generated based on the optimal disparity value of each first pixel, and then the disparity map is optimized by a left-right consistency check algorithm to eliminate erroneous disparity caused by occlusion and noise, and isolated abnormal points in the disparity map are eliminated by eliminating small connected area algorithms. Since the optimal disparity value of the first pixel currently determined is integer pixel accuracy, in order to obtain higher sub-pixel accuracy, the cost value under the optimal disparity value and the cost value under the left and right disparities can be fitted to a univariate quadratic curve by a univariate quadratic curve fitting method, and the disparity value represented by the minimum point of the curve is taken as the optimized disparity value, and the target disparity map is generated based on the optimized disparity value of each first pixel.

[0087] In summary, the disparity map generation method provided by the embodiment of the present application obtains a first image and a second image, performs stereo correction on the first image and the second image, and obtains a first corrected image and a second corrected image; obtains a neighborhood window corresponding to each first pixel in the first corrected image according to a preset window size, and determines the cost weight of each second pixel in the neighborhood window; the first pixel is the central pixel of the corresponding neighborhood window; determines the first matching cost of each second pixel according to the neighborhood window and the second corrected image, and performs weighted summation of the first matching cost according to the cost weight to obtain the second matching cost of the first pixel; determines the optimal disparity value of each first pixel according to the second matching cost of each first pixel, and generates a target disparity map according to the optimal disparity value of each first pixel. Through the above technical means, when the neighborhood window of the first pixel is increased, the cost contribution degree of each second pixel in the neighborhood window can be automatically adjusted by adaptive weights to avoid overflow of the cost value of the first pixel while obtaining more local feature information around the first pixel, which is conducive to improving the disparity estimation effect. The neighborhood window can be increased or decreased according to actual needs and is no longer limited to the existing window size. This solves the problem in the prior art that unmanned equipment cannot restore high-quality disparity maps in any scenario, and improves the applicability of unmanned equipment.

[0088] Based on the above embodiments, Figure 7 A schematic diagram of the structure of a disparity map generation device provided in an embodiment of the present application. Figure 7 The disparity map generating device provided in this embodiment specifically includes: a stereo correction module 21, a weight determination module 22, a cost determination module 23 and a disparity map generating module 24.

[0089] The stereo correction module 21 is configured to obtain a first image and a second image, and perform stereo correction on the first image and the second image to obtain a first corrected image and a second corrected image;

[0090] The weight determination module 22 is configured to obtain a neighborhood window corresponding to each first pixel point in the first corrected image according to a preset window size, and determine a cost weight of each second pixel point in the neighborhood window; the first pixel point is a central pixel point of the corresponding neighborhood window;

[0091] a cost determination module 23, configured to determine a first matching cost of each of the second pixel points according to the neighborhood window and the second corrected image, and to perform weighted summation of the first matching costs according to the cost weights to obtain a second matching cost of the first pixel point;

[0092] The disparity map generating module 24 is configured to determine an optimal disparity value of each of the first pixels according to the second matching cost of each of the first pixels, and generate a target disparity map according to the optimal disparity value of each of the first pixels.

[0093] Based on the above embodiment, the weight determination module 22 includes: a weight determination unit configured to determine the cost weight of each second pixel point according to the grayscale value of the central pixel point of the neighborhood window and the grayscale value of each second pixel point.

[0094] On the basis of the above embodiment, the weight determination unit includes: a weight determination subunit, configured to determine the cost weight of the second pixel by a preset weight calculation formula and the grayscale value of the central pixel of the neighborhood window and the grayscale value of the second pixel; the preset weight calculation formula is:

[0095]

[0096] Among them, I p is the gray value of the center pixel, I q is the gray value of the second pixel, γ is the weight parameter, and w(p,q) is the cost weight of the second pixel.

[0097] Based on the above embodiment, the cost determination module 23 includes: a third pixel determination unit, configured to determine the third pixel corresponding to each second pixel in the neighborhood window in the second corrected image according to the disparity range between the first image and the second image; a first cost determination unit, configured to perform cost calculation on the second pixel and the corresponding third pixel to obtain a first matching cost for the second pixel.

[0098] On the basis of the above embodiment, the first cost determination unit includes: a first cost determination subunit, configured to determine the grayscale value of the first sub-pixel between the third pixel point and the corresponding adjacent pixel point according to the adjacent pixel points of the third pixel point, determine the first difference with the grayscale value of the second pixel point according to the minimum grayscale value and the maximum grayscale value between the first sub-pixel and the third pixel point, and determine the third matching cost according to the first difference; a second cost determination subunit, configured to determine the grayscale value of the second sub-pixel between the second pixel point and the corresponding adjacent pixel point according to the adjacent pixel points of the second pixel point, determine the second difference with the grayscale value of the third pixel point according to the minimum grayscale value and the maximum grayscale value between the second sub-pixel and the second pixel point, and determine the fourth matching cost according to the second difference; a third cost determination subunit, configured to add the third matching cost and the fourth matching cost to obtain the first matching cost of the second pixel point.

[0099] Based on the above embodiment, the cost determination module 23 includes: a second cost determination unit, configured to add the product of the cost weight of each second pixel point in the neighborhood window and the first matching cost to obtain the second matching cost of the first pixel point.

[0100] Based on the above embodiment, the disparity map generation module 24 includes: a cost space generation unit, configured to generate a cost space according to the second matching cost of each of the first pixel points; a disparity value determination unit, configured to perform cost aggregation on the cost space to obtain an aggregated cost matrix, and determine, according to the aggregated cost matrix, that the disparity value corresponding to the minimum aggregated cost value of the first pixel point is the optimal disparity value of the first pixel point.

[0101] As mentioned above, the disparity map generation device provided by the embodiment of the present application obtains the first image and the second image, performs stereo correction on the first image and the second image, and obtains the first corrected image and the second corrected image; obtains the neighborhood window corresponding to each first pixel in the first corrected image according to the preset window size, and determines the cost weight of each second pixel in the neighborhood window; the first pixel is the central pixel of the corresponding neighborhood window; determines the first matching cost of each second pixel according to the neighborhood window and the second corrected image, and performs weighted summation of the first matching cost according to the cost weight to obtain the second matching cost of the first pixel; determines the optimal disparity value of each first pixel according to the second matching cost of each first pixel, and generates the target disparity map according to the optimal disparity value of each first pixel. Through the above technical means, when the neighborhood window of the first pixel is increased, the cost contribution degree of each second pixel in the neighborhood window can be automatically adjusted by adaptive weights to avoid the overflow of the cost value of the first pixel while obtaining more local feature information around the first pixel, which is conducive to improving the disparity estimation effect. The neighborhood window can be increased or decreased according to actual needs and is no longer limited to the existing window size. This solves the problem in the prior art that unmanned equipment cannot restore high-quality disparity maps in any scenario, and improves the applicability of unmanned equipment.

[0102] The disparity map generation device provided in the embodiment of the present application can be used to execute the disparity map generation method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0103] Figure 8 This is a schematic diagram of the structure of an unmanned device provided in an embodiment of the present application, with reference to Figure 8The unmanned device includes: a processor 31, a memory 32, a communication device 33, an input device 34 and an output device 35. The number of processors 31 in the unmanned device can be one or more, and the number of memories 32 in the unmanned device can be one or more. The processor 31, memory 32, communication device 33, input device 34 and output device 35 of the unmanned device can be connected through a bus or other methods.

[0104] The memory 32, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the disparity map generation method of any embodiment of the present application (for example, the stereo correction module 21, the weight determination module 22, the cost determination module 23 and the disparity map generation module 24 in the disparity map generation device). The memory 32 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the memory 32 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0105] The communication device 33 is used for data transmission.

[0106] The processor 31 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 32, that is, implements the above-mentioned disparity map generation method.

[0107] The input device 34 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 35 may include a display device such as a display screen.

[0108] The unmanned equipment provided above can be used to execute the disparity map generation method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0109] An embodiment of the present application also provides a storage medium containing computer executable instructions, which are used to execute a disparity map generation method when executed by a computer processor. The disparity map generation method includes: acquiring a first image and a second image, performing stereo correction on the first image and the second image to obtain a first corrected image and a second corrected image; according to a preset window size, obtaining a neighborhood window corresponding to each first pixel in the first corrected image, and determining a cost weight of each second pixel in the neighborhood window; the first pixel is the center pixel of the corresponding neighborhood window; according to the neighborhood window and the second corrected image, determining a first matching cost for each second pixel, performing weighted summation of the first matching cost according to the cost weight, and obtaining a second matching cost for the first pixel; according to the second matching cost of each first pixel, determining an optimal disparity value for each first pixel, and generating a target disparity map according to the optimal disparity value for each first pixel.

[0110] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system, which is connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media residing in different locations (for example, in different computer systems connected by a network). The storage medium may store program instructions (for example, embodied as a computer program) that can be executed by one or more processors.

[0111] Of course, the storage medium containing computer executable instructions provided in the embodiment of the present application is not limited to the above disparity map generation method, and the computer executable instructions can also execute related operations in the disparity map generation method provided in any embodiment of the present application.

[0112] The disparity map generation device, storage medium and unmanned equipment provided in the above embodiments can execute the disparity map generation method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the disparity map generation method provided in any embodiment of the present application.

[0113] The above are only preferred embodiments of the present application and the technical principles used. The present application is not limited to the specific embodiments herein, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not deviate from the protection scope of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A disparity map generation method, characterized in that: include: Acquire a first image and a second image, and perform stereoscopic correction on the first image and the second image to obtain a first corrected image and a second corrected image; According to a preset window size, a neighborhood window corresponding to each first pixel point is obtained in the first corrected image, and a cost weight of each second pixel point in the neighborhood window is determined; the first pixel point is a central pixel point of the corresponding neighborhood window; Determine a first matching cost of each of the second pixel points according to the neighborhood window and the second corrected image, and perform weighted summation of the first matching costs according to the cost weight to obtain a second matching cost of the first pixel point; An optimal disparity value for each of the first pixels is determined according to the second matching cost of each of the first pixels, and a target disparity map is generated according to the optimal disparity value for each of the first pixels.

2. The disparity map generation method according to claim 1, characterized in that: The determining the cost weight of each second pixel point in the neighborhood window includes: The cost weight of each of the second pixels is determined according to the grayscale value of the central pixel of the neighborhood window and the grayscale value of each of the second pixels.

3. The disparity map generation method according to claim 2, characterized in that: The step of determining the cost weight of each second pixel point according to the grayscale value of the central pixel point of the neighborhood window and the grayscale value of each second pixel point includes: The cost weight of the second pixel is determined by a preset weight calculation formula, the grayscale value of the central pixel of the neighborhood window, and the grayscale value of the second pixel; the preset weight calculation formula is: Among them, I p is the gray value of the center pixel, I q is the gray value of the second pixel, γ is the weight parameter, and w(p,q) is the cost weight of the second pixel.

4. The disparity map generation method according to claim 1, characterized in that: The determining, according to the neighborhood window and the second corrected image, a first matching cost of each of the second pixels includes: Determining, according to a disparity range between the first image and the second image, a third pixel point in the second corrected image corresponding to each second pixel point in the neighborhood window; Cost calculation is performed on the second pixel point and the corresponding third pixel point to obtain a first matching cost of the second pixel point.

5. The disparity map generation method according to claim 4, characterized in that: The performing cost calculation on the second pixel point and the corresponding third pixel point to obtain a first matching cost of the second pixel point includes: Determine, according to the neighboring pixels of the third pixel, a gray value of a first sub-pixel between the third pixel and the corresponding neighboring pixel, determine, according to the minimum gray value and the maximum gray value between the first sub-pixel and the third pixel, a first difference with the gray value of the second pixel, and determine a third matching cost according to the first difference; Determine, according to the adjacent pixels of the second pixel, a gray value of a second sub-pixel between the second pixel and the corresponding adjacent pixel, determine, according to the minimum gray value and the maximum gray value between the second sub-pixel and the second pixel, a second difference with the gray value of the third pixel, and determine a fourth matching cost according to the second difference; The third matching cost and the fourth matching cost are added to obtain a first matching cost of the second pixel.

6. The disparity map generation method according to claim 2 or 3, characterized in that: The step of performing weighted summation on the first matching cost according to the cost weight to obtain a second matching cost of the first pixel point includes: The product of the cost weight of each second pixel point in the neighborhood window and the first matching cost is added to obtain the second matching cost of the first pixel point.

7. The disparity map generation method according to claim 1, characterized in that: The determining the optimal disparity value of each of the first pixels according to the second matching cost of each of the first pixels includes: generating a cost space according to the second matching cost of each of the first pixels; Cost aggregation is performed on the cost space to obtain an aggregated cost matrix, and a disparity value corresponding to a minimum aggregated cost value of the first pixel is determined according to the aggregated cost matrix as an optimal disparity value of the first pixel.

8. A disparity map generating device, characterized in that: include: A stereo correction module is configured to acquire a first image and a second image, and perform stereo correction on the first image and the second image to obtain a first corrected image and a second corrected image; A weight determination module is configured to obtain a neighborhood window corresponding to each first pixel point in the first corrected image according to a preset window size, and determine a cost weight of each second pixel point in the neighborhood window; the first pixel point is a central pixel point of the corresponding neighborhood window; a cost determination module, configured to determine a first matching cost of each of the second pixel points according to the neighborhood window and the second corrected image, and to perform weighted summation of the first matching costs according to the cost weights to obtain a second matching cost of the first pixel point; The disparity map generation module is configured to determine the optimal disparity value of each of the first pixels according to the second matching cost of each of the first pixels, and generate a target disparity map according to the optimal disparity value of each of the first pixels.

9. An unmanned device, characterized in that: include: one or more processors; A memory storing one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the disparity map generation method as described in any one of claims 1-7.

10. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to execute the disparity map generation method according to any one of claims 1 to 7 when executed by a computer processor.

Images