Stereo matching method, system, device and medium based on high-precision parallax refinement

Abstract

The present invention discloses a stereo matching method, system, device, and medium based on high-precision disparity refinement. The method comprises: obtaining a binocular image after epipolar correction, the binocular image comprising a first image and a second image; dividing the binocular image into multiple texture regions based on texture segmentation; establishing a first adaptive window centered on each pixel within the multiple texture regions, and then calculating a cost set for the first image and the second image; obtaining a first initial disparity map and a second initial disparity map based on the cost set of the first image and the cost set of the second image, respectively; performing occlusion recovery on a third initial disparity map based on the first and second initial disparity maps according to consistency detection, invalid point classification, and fitness-based disparity reconstruction; and performing edge correction on the occlusion-recovered third initial disparity map to obtain a final disparity map. The present invention can effectively improve the capability of disparity refinement without increasing computational complexity.

Description

Technical Field

[0001] The present invention relates to a stereo matching method, system, device and medium based on high-precision parallax refinement, and belongs to the technical field of image processing. Background Art

[0002] Stereo matching methods consist of two major steps: disparity calculation and disparity refinement. The former, as the foundation of stereo matching, has long attracted considerable attention. Stereo matching is categorized into global and local approaches based on the disparity calculation method. Research on global stereo matching focuses on developing better energy functions, which yield good matching results but are time-consuming. Local stereo matching is based on epipolar constraints, where a single match only searches along the same epipolar line, significantly improving computational efficiency and becoming the preferred method for real-time stereo matching.

[0003] Disparity calculation, both global and local methods, has been extensively researched and is relatively mature. In contrast, disparity refinement, as a post-processing component of stereo matching, has developed relatively slowly. Disparity map post-processing optimization can significantly improve the quality of the initial disparity map, raising the upper limit of stereo matching. Since binocular cameras inevitably encounter occlusions between cameras when capturing the same target, information is lacking in these occluded areas, requiring occlusion recovery within disparity refinement to repair them. Furthermore, high mismatches in disparity discontinuities have long been a bottleneck in stereo matching and should be further optimized within disparity refinement. High-precision stereo matching is arguably impossible without disparity refinement. In recent years, some researchers have dedicated themselves to developing advanced disparity refinement methods, but overall, occlusion recovery methods remain unsatisfactory. This is reflected in the high proportionality between computational complexity and recovery performance, resulting in high computational time associated with even the most advanced occlusion recovery methods. Furthermore, due to the limitations of disparity layering techniques, few researchers have directly optimized disparity edges during the disparity refinement phase. Instead, they choose to incorporate gradient information from the original image into the disparity calculation phase to protect edges. This approach, which uses secondary attributes to protect edges, is still prone to high mismatch rates at disparity edges.

[0004] In summary, binocular stereo matching still has the problem of insufficient effectiveness in the disparity refinement part, especially in the occlusion area and the disparity discontinuity area, where there is still a high mismatch rate. This is a difficulty that must be overcome for high-precision stereo matching. Summary of the Invention

[0005] In view of this, the present invention provides a stereo matching method, system, computer device and storage medium based on high-precision disparity refinement, which introduces texture segmentation to improve disparity calculation, calculates fitness in the disparity refinement stage, completes occlusion recovery, and uses secondary guided filtering based on the disparity map to correct disparity discontinuities, thereby effectively improving the disparity refinement capability without increasing the computational complexity as much as possible.

[0006] A first object of the present invention is to provide a stereo matching method based on high-precision disparity refinement.

[0007] A second object of the present invention is to provide a stereo matching system based on high-precision disparity refinement.

[0008] A third object of the present invention is to provide a computer device.

[0009] A fourth object of the present invention is to provide a storage medium.

[0010] The first object of the present invention can be achieved by adopting the following technical solutions:

[0011] A stereo matching method based on high-precision disparity refinement, the method comprising:

[0012] Acquire an epipolar-corrected binocular image, the binocular image comprising a first image and a second image;

[0013] Based on texture segmentation, the binocular image is divided into multiple texture regions;

[0014] In the plurality of texture regions, a first adaptive window is established with each pixel as the center, and then a cost set of the first image and a cost set of the second image are calculated;

[0015] Obtaining a first initial disparity map and a second initial disparity map according to the cost set of the first image and the cost set of the second image, respectively;

[0016] Based on the first and second initial disparity maps, performing occlusion recovery on a third initial disparity map according to consistency detection, invalid point classification, and fitness-based disparity reconstruction, where the third initial disparity map is one of the first and second initial disparity maps;

[0017] Edge correction is performed on the third initial disparity map after occlusion recovery to obtain a final disparity map.

[0018] Furthermore, the texture region includes one of a high texture region, a second repeated texture region, and a weak texture region.

[0019] Furthermore, the binocular image is divided into a plurality of texture regions according to texture segmentation, specifically including:

[0020] In the binocular image, a fixed window is established with each pixel as the center;

[0021] Calculate the color variance within a fixed window;

[0022] According to the color variance and bilateral filtering in all fixed windows, the binocular image is segmented once to obtain the high texture area, weak texture area and the first repeated texture area;

[0023] The first repeated texture region is divided twice to obtain a second repeated texture region.

[0024] Furthermore, establishing a first adaptive window with each pixel as the center in the multiple texture areas, and then calculating the cost set of the first image and the cost set of the second image, specifically includes:

[0025] Using successively larger window sizes for pixels in high texture areas, second repeated texture areas, and weak texture areas;

[0026] Within the disparity search range, calculate the cost of any window in the first image and all windows in the second image, thereby obtaining a cost set of the first image;

[0027] The central pixel point in any one window and the central pixel points in all windows are on the same epipolar line;

[0028] Among them, the cost is calculated by SAD cost calculation, Census cost calculation and gradient cost calculation;

[0029] Similarly, the cost set of the second image is obtained.

[0030] Furthermore, the method of performing occlusion recovery on the third initial disparity map based on the first initial disparity map and the second initial disparity map according to consistency detection, invalid point classification, and disparity reconstruction based on fitness specifically includes:

[0031] Performing consistency detection using the first initial disparity map and the second initial disparity map, thereby obtaining a plurality of invalid points, wherein the invalid points include occlusion points and mismatched points;

[0032] Classify invalid points according to their location characteristics;

[0033] Based on the classified invalid point, a second adaptive window is established with the invalid point as the center, wherein the length of the second adaptive window is equal to twice the distance between the nearest valid points on the left and right of the invalid point, and the width of the second adaptive window is equal to twice the distance between the nearest valid points above and below the invalid point;

[0034] Calculating the valid points in the second adaptive window to obtain the fitness;

[0035] According to the valid points with the maximum fitness, occlusion recovery is performed on the invalid points in the third initial disparity map.

[0036] Furthermore, the consistency detection is performed using the first initial disparity map and the second initial disparity map, as shown in the following formula:

[0037] D L (i,j)-D R [i,jD L (i,j)]≤d

[0038] Among them, D L (i, j) represents the left point disparity, D R (i, j) represents the disparity of the right point, and d represents the threshold for judging whether the consistent detection is passed;

[0039] The invalid points are classified according to their position characteristics, as shown in the following formula:

[0040]

[0041] Among them, 0 represents the leftmost point, 1 represents the rightmost point, 2 represents the center point, 3 represents the left edge point, and 4 represents the right edge point. The invalid point obtains five consecutive valid points to the left, and the average disparity of the second, third, and fourth valid points is Get five consecutive valid points to the right of the invalid point, and the average disparity of the second, third and fourth valid points is Δd represents the threshold for judging the disparity step;

[0042] The effective points in the second adaptive window are calculated to obtain the fitness, which is as follows:

[0043]

[0044] Wherein, ΔI(x,y) represents the color difference, Δl(x,y) represents the distance difference, d(x,y) represents the parallax value of the effective point, δ r , δ s , δ d Respectively represent the scale factors of color difference, distance difference, and parallax value of the effective point; λ r ,λ s ,λ d In the leftmost area: 1, 1, 0; λ r ,λ s ,λ d In the rightmost area: 0, 1, 1; λ r ,λ s ,λ d In the center area: 1, 0, 0; λ r ,λ s ,λ d In the left side of the edge: 1, 1, 1; λ r ,λ s ,λ d In the area to the right of the edge: 1, 0, 1.

[0045] Furthermore, performing edge correction on the third initial disparity map after occlusion recovery to obtain a final disparity map specifically includes:

[0046] Performing color segmentation on the original image of the third initial disparity map after occlusion recovery, and using the original image after color segmentation as a guide image;

[0047] Based on the guide map and the third initial disparity map after occlusion recovery, generating a first pseudo disparity map according to a guided filtering formula, wherein the guided filtering formula includes a penalty function ε(x, y), where ε(x, y)=p, where p is a constant;

[0048] Based on the first pseudo-disparity map, a second pseudo-disparity map is generated according to the guided filtering formula. In this case, ε(x, y) = 0 is assigned to the points on the disparity edge of the first pseudo-disparity map, and ε(x, y) = 1 is assigned to other areas.

[0049] enhancing the second pseudo disparity map according to an adaptive image enhancement function to obtain a third pseudo disparity map;

[0050] The third pseudo disparity map is used to correct the edge of the third initial disparity map after occlusion restoration to obtain a final disparity map.

[0051] The second object of the present invention can be achieved by adopting the following technical solutions:

[0052] A stereo matching system based on high-precision disparity refinement, the system comprising:

[0053] an acquisition unit, configured to acquire a binocular image after epipolar correction, the binocular image comprising a first image and a second image;

[0054] A texture segmentation unit, configured to divide the binocular image into a plurality of texture regions according to texture segmentation;

[0055] A cost calculation unit, configured to establish a first adaptive window with each pixel as a center within the plurality of texture regions, and further calculate a cost set of the first image and a cost set of the second image;

[0056] a cost aggregation and disparity calculation unit, configured to obtain a first initial disparity map and a second initial disparity map according to a cost set of the first image and a cost set of the second image, respectively;

[0057] an occlusion recovery unit, configured to perform occlusion recovery on a third initial disparity map based on the first and second initial disparity maps, according to consistency detection, invalid point classification, and fitness-based disparity reconstruction, wherein the third initial disparity map is one of the first and second initial disparity maps;

[0058] The edge correction unit is used to perform edge correction on the third initial disparity map after occlusion recovery to obtain a final disparity map.

[0059] The third object of the present invention can be achieved by adopting the following technical solutions:

[0060] A computer device includes a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the above-mentioned stereo matching method is implemented.

[0061] The fourth object of the present invention can be achieved by adopting the following technical solutions:

[0062] A storage medium stores a program, and when the program is executed by a processor, the above-mentioned stereo matching method is implemented.

[0063] The present invention has the following beneficial effects compared to the prior art:

[0064] 1. The initial disparity calculation is faster and the mismatch rate of the initial disparity map is lower: This invention introduces texture segmentation in the image preprocessing stage, making the window in the cost calculation stage adaptive, avoiding the calculation of each pixel in a large window manner, thereby speeding up the calculation while avoiding some errors caused by fixed window calculations;

[0065] 2. More accurate occlusion recovery and lower computational complexity: Compared to existing technologies, the occlusion recovery method proposed in this paper utilizes all information related to restored disparity, resulting in more accurate occlusion recovery. Furthermore, this method integrates this information into a single formula and adaptively adapts this formula using invalid point classification. This avoids the numerous fill-in formulas required by multi-step disparity refinement, reduces computational complexity, and facilitates parallelization.

[0066] 3. Directly and accurately correct parallax edges: The edge protection method proposed in this invention differs from the existing method of using a gradient compensation operator to protect edges. Instead, it directly and accurately corrects the parallax edges. This method is not only more effective but also has lower time complexity.

[0067] 4. The cost aggregation stage in the embodiment of the present invention improves the problem of the number of iterations, avoids the introduction of more errors by multiple iterations in high-texture areas, and strengthens the iterations in weak-texture areas and repeated-texture areas, so that they obtain more stable information. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0069] Figure 1 This is a specific flow chart of the stereo matching method based on high-precision disparity refinement according to Example 1 of the present invention.

[0070] Figure 2 This is a brief flowchart of the stereo matching method based on high-precision disparity refinement according to Example 1 of the present invention.

[0071] Figure 3(a) to Figure 6(a) Multiple indoor left images obtained from the Middbury standard dataset in Example 1 of the present invention;

[0072] Figure 3(b) to Figure 6(b) The multiple right indoor images obtained from the Middbury standard dataset in Example 1 of the present invention;

[0073] Figures 7(a) to 10(a) These are multiple indoor left images taken by the user in Example 1 of the present invention;

[0074] Figure 7(b) to Figure 10(b) These are multiple right indoor images taken by the user in Example 1 of the present invention;

[0075] Figure 11 This is a flowchart of occlusion area recovery according to embodiment 1 of the present invention.

[0076] Figure 12 Flowchart of parallax discontinuity correction according to embodiment 1 of the present invention.

[0077] Figure 13 This is a structural block diagram of a stereo matching system based on high-precision disparity refinement according to embodiment 2 of the present invention.

[0078] Figure 14 This is a structural block diagram of a computer device according to embodiment 3 of the present invention. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0080] Example 1:

[0081] Binocular vision technology has become a crucial component of computer vision, with local stereo matching becoming increasingly mainstream. Because its matching space lies confined to a single epipolar line, local stereo matching is faster than global stereo matching and more readily meets real-time requirements. However, local stereo matching suffers from relatively low accuracy, particularly difficulties recovering occluded areas and fitting parallax edges. Therefore, addressing these two challenges is crucial for achieving high-precision and fast stereo matching.

[0082] In response to the above two questions, Figure 1 and Figure 2 As shown, this embodiment provides a stereo matching method based on high-precision disparity refinement, which uses bilateral filtering and image variance characteristics to segment image texture, thereby making the window in the cost calculation stage adaptive; by combining SAD, improved Census and gradient accumulation cost arrays, the cost aggregation of the improved iterative problem is then completed; in the disparity refinement part, the recovery of the occluded area is completed using the fitness formula, and a new secondary guided filtering and image enhancement are developed to achieve edge correction.

[0083] Specifically, the method comprises the following steps:

[0084] S101 : Acquire a binocular image after epipolar correction, where the binocular image includes a first image and a second image.

[0085] Before step S101, the following steps are included:

[0086] S1011. Capture a binocular image using a parallel binocular vision camera.

[0087] In this embodiment, the binocular image is an indoor binocular image. Figure 3(a) to Figure 3(b) 、 Figure 4(a) to Figure 4(b) 、 Figure 5(a) to Figure 5(b) 、 Figure 6(a) to Figure 6(b) 、 Figure 7(a) to Figure 7(b) 、 Figure 8(a) to Figure 8(b) 、 Figure 9(a) to Figure 9(b) or Figure 10(a) to Figure 10(b) ; Where: a represents the left image, b represents the right image; Figures 3 to 6 are obtained from the Middbury standard dataset, and Figures 7 to 10 are self-photographed indoor binocular images.

[0088] S1012: Perform epipolar correction on the captured binocular image to obtain an epipolar-corrected binocular image, where the epipolar-corrected binocular image includes a left image and a right image, where the left image is a first image and the right image is a second image.

[0089] In another embodiment, the right image is the first image and the left image is the second image.

[0090] It should be noted that the left image and the right image obtained in step S101 must be images taken by a parallel binocular vision camera and must undergo strict epipolar correction in relevant software; otherwise, a large number of matching errors will occur in the output disparity map.

[0091] S102 : Divide the binocular image into multiple texture regions according to texture segmentation.

[0092] The texture region in this embodiment includes one of a high texture region, a second repeated texture region, and a weak texture region.

[0093] The idea of steps S1021 to S1023 is specifically as follows: when the bilateral filtering has a certain effect and the area with obvious texture changes can be distinguished, the variances of the left image, the right image, the left filtered image, and the right filtered image are calculated point by point and window by window, and then the difference in variances between the left image and the left filtered image, and the difference in variances between the right image and the right filtered image are calculated point by point and window by window, to generate two matrices carrying the difference in variances; finally, the OTSU threshold segmentation method is used to binarize the two above matrices, and the high-value part is classified as high texture, and the low-value part is classified as weak texture, thereby obtaining a high texture area, a first repeated texture area, and a weak texture area.

[0094] S1021. In the binocular image, a fixed window is established with each pixel as the center.

[0095] Step S1021 specifically includes traversing each pixel point in the left image and the right image, and establishing a 5×5 fixed window with the pixel point as the center.

[0096] It should be noted that a fixed window that is too large does not bring greater judgment benefits, but instead introduces greater computational complexity.

[0097] S1022: Calculate the color variance within the fixed window.

[0098] Calculate the grayscale variance within a fixed window. Since color variance can reflect the degree of grayscale dispersion within a fixed window, in most cases, a fixed window with rich texture will have a larger variance; conversely, a fixed window with less texture will have a smaller variance.

[0099] In this embodiment, the color variance within a fixed window is calculated as follows:

[0100]

[0101] Among them, p represents the center pixel, q represents the area pixel within its fixed window, W represents the 5×5 fixed window, i indicates the color space, s 2 (i, j) represents the color variance, that is, the image color variance, Ii (q) represents the color value of the pixel in a certain color space, Represents the average color value within a fixed window, and n represents the total number of pixels within the fixed window.

[0102] S1023 , segmenting the binocular image based on the color variance and bilateral filtering in all fixed windows to obtain a high texture area, a weak texture area, and a first repeated texture area.

[0103] To avoid unexpected errors in judgment, step S1023 uses bilateral filtering to further improve the accuracy of texture segmentation. For an input image, such as the left or right image, due to the smoothing effect, the difference in variance before and after bilateral filtering will be larger in high-texture areas, while the opposite is true for weak-texture areas.

[0104] In this embodiment, the high texture area after segmentation is set to black, and the weak texture area after segmentation is set to white, as shown in the following formula:

[0105]

[0106] Among them, a represents the threshold value of the intermittent high texture area and weak texture area, which is generated by the OTSU threshold segmentation method, 255 represents the white weak texture area, 0 represents the black high texture area, S ref (i,j) and S ref (i, j) represents the variance value of the reference image and the filtered image in the window centered at point (i, j).

[0107] In practical applications, the parameters of bilateral filtering and the thresholds of discontinuous high texture areas and weak texture areas can be manually adjusted.

[0108] It should be noted that the first repeated texture refers to the distribution of the same color features appearing multiple times on the same polar line. This is also an area that is extremely prone to mismatching for stereo matching. It exists in the high-texture area, and it is necessary to continue looking for the second repeated texture (further segmentation, i.e., secondary segmentation, refer to step S1024). Specifically, each high-texture point is looped through and points are searched on its polar line. For each pair of points established, a pair of fixed-size windows are created for them. The difference in the total grayscale values within a pair of windows reflects the degree of similarity between them. The size of this fixed window is 3×3. After marking a pair of second repeated texture points, the two points will no longer participate in the judgment of the second repeated texture, thereby avoiding multiple repeated operations that introduce greater computational complexity.

[0109] It should be noted that bilateral filtering is an edge-preserving filter that uses the spatial distance and color difference between pixels in the image to make a comprehensive judgment, achieving the effect of smoothing the image without affecting the edges.

[0110] S1024 , performing secondary segmentation on the first repeated texture region to obtain a second repeated texture region.

[0111] Step S1024 is as follows:

[0112]

[0113] f(i,j)=f(i′,j′)=125,ifΔI(i,j)<b

[0114] Among them, I i (q) represents a color value of a high texture point, I' i (q′) represents a color value of a point on the disparity search extreme line, ΔI(i,j) represents the difference in the total grayscale value between the two points, representing the degree of similarity between the two points, and b represents the threshold for judging the second repeated texture. If it is lower than the threshold, the point is the second repeated texture point and is set to gray, with a value of 125.

[0115] It should be noted that high texture points refer to pixel points in high texture areas, and the same applies to weak texture points and second repeated texture points.

[0116] The texture area generated so far includes one of the black area, white area, and gray area. The black area corresponds to the high texture area, the white area corresponds to the weak texture area, and the gray area corresponds to the second repeated texture area. Among them, due to the point-by-point calculation characteristics, some unexpected small connected areas will generally be generated. Therefore, the area is intelligently found in the texture area and the small connected area is eliminated, and then a simple processing is performed using the median filter.

[0117] S103 : Establish a first adaptive window with each pixel as the center in the multiple texture areas, and then calculate the cost set of the first image and the cost set of the second image.

[0118] S1031 . Use successively larger window sizes for pixel points in the high texture area, the second repeated texture area, and the weak texture area.

[0119] Step S1031 specifically comprises: setting the first image as the reference image and the second image as the matching image, traversing each pixel point of the first image and the second image, and establishing a first adaptive window with the pixel point as the center, wherein the first adaptive window includes a window for high texture points, a window for weak texture points, and a window for second repeated texture points; setting the window size of high texture points to 5×5, the window size of weak texture points to 9×9, and the window size of second repeated texture points to 7×7.

[0120] S1032. Within the disparity search range, calculate the cost of any window in the first image and all windows in the second image to obtain a cost set of the first image, where the central pixel point in any window and the central pixel points in all windows are on the same epipolar line.

[0121] It should be noted that in the parallax search range d min to d max Get the total number of pixels in d max -d min The cost represents the similarity between two points; the disparity search range is determined by the resolution of the input image and the distance between the camera and the target object, and can be manually modified before the cost calculation.

[0122] Among them, this embodiment adopts the combined cost calculation method and improves the AD-Census method into SAD+improved Census+gradient factor, namely SAD cost calculation, Census cost calculation and gradient cost calculation, and uses the texture area to achieve window adaptation in the cost calculation stage; the three cost calculations are as follows:

[0123] A. SAD cost calculation is as follows:

[0124]

[0125] Among them, q represents the point in the left window, d represents the disparity search value, I l with I r Represents the grayscale value in the left and right images, and W represents the calculation window.

[0126] It is worth noting that the SAD cost calculation is used to replace the AD cost calculation. AD is the average of the color difference between a single pixel in the left image and its corresponding pixel in the right image in the R, G, and B space. Since the color information considered by a single pixel is not comprehensive enough, SAD is used to replace AD. SAD is specifically as follows: construct windows of the same size for the left and right points, calculate the absolute grayscale difference of each pair of pixels between the two windows, and then sum all the differences. SAD is easy to fit areas that are in the same disparity plane and have rich textures, but is not easy to fit edges.

[0127] B. Census cost calculation.

[0128] The core idea of Census is to replace specific grayscale operations by comparing sizes. Traditional Census compares the grayscale values of the area pixels within the window with the center pixel, setting the grayscale values of the pixels above the center pixel to 1 and vice versa, forming a binary string within the window. The binary strings corresponding to the left and right points are obtained, and the Hamming distance between the strings is obtained by XOR. Census enables the stereo matching algorithm to have better resistance to light interference, but a single binary number in the window often does not express enough information, that is, it has strong anti-noise ability but weak ability to obtain image information. This embodiment continues the idea of improving Census by specifying a threshold, so that positions with grayscale values sufficiently smaller than the center are set to 00, and grayscale values with larger values are set to 01, 10, and 11, respectively. The calculation formula is as follows:

[0129]

[0130] Among them, Census l With Census r Represents the binary strings calculated by the left and right windows respectively.

[0131] C. Gradient cost calculation

[0132] The introduced gradient factor also depends on the window execution and only changes the grayscale difference in the SAD to the gradient difference. Therefore, the gradient information can be used in the disparity calculation stage, which is beneficial for the initial fitting of the disparity discontinuity area. The gradient factor is calculated by the positive gradient in the x-direction and the positive gradient in the y-direction, as shown in the following formula:

[0133]

[0134] in, Represent the gradient of the left image in the X and Y directions respectively, Represent the gradient of the right image in the X and Y directions respectively.

[0135] Furthermore, the normalization functions of Census cost calculation, SAD cost calculation, and gradient cost calculation are as follows:

[0136] C(p,d)=ρ(C cen (p,d),λ cen )+ρ(C SAD (p,d),λ SAD )+ρ(C gra (p,d),λ gra )

[0137] Among them, C, C cen 、C SAD 、C graRepresent the final cost, improved Census cost, SAD cost and gradient cost respectively. ρ(c,λ) comes from the AD-Census normalization function, and λ represents the proportional factor that controls the size of the cost. In the combined cost calculation, the proportional factor λ assigned to the three formulas is cen ,λ SAD and λ gra Can be adjusted arbitrarily.

[0138] Finally, the result of the cost calculation is d max -d min matrices (cost sets), each matrix includes the costs of all points in the left image generated under a disparity search distance; similarly, the cost set of the second image is obtained.

[0139] S104 , obtaining a first initial disparity map and a second initial disparity map according to the cost set of the first image and the cost set of the second image, respectively.

[0140] S1041. Aggregate the costs of the multiple matrices obtained in step S103 in sequence. Specifically, first determine a center point, and stretch horizontal arms from this point to the left and right, with the maximum stretch not exceeding 10 pixels. If there is a large gradient step during the stretching process, stop stretching in that direction. Then, stretch each pixel on the horizontal arm in the same way in the vertical direction, and finally construct a cross domain of the center point. The cost of the center pixel is aggregated as follows:

[0141]

[0142] Where ω represents the cross-domain of pixels located in the i-th row and j-th column, q represents a pixel in it, and n represents the total number of pixels in ω.

[0143] It is worth noting that cost aggregation is the key to optimizing the initial disparity map, which can greatly improve the quality of the initial cost. This part also continues the cross-domain cost aggregation improved in AD-Census. Under the premise of the longest arm span, the grayscale gradient is used as the standard for truncating the arm length. Among them, in order to obtain stable results, the first aggregation is often completed by first constructing the horizontal arm and then the vertical arm, and then the second aggregation is completed by first constructing the vertical arm and then the horizontal arm. A total of four aggregations are completed, the third aggregation is the same as the first, and the fourth is the same as the second, thereby further reducing the error.

[0144] S1042. After the first polymerization, change the arm length extension order, polymerize the weak texture area and the second repeated texture area once, and then polymerize the weak texture area twice.

[0145] After aggregation, the cost of the high-texture area is usually much lower than the cost of the second repeated texture and the cost of the weak texture area, but the multiple costs of each point are on the same order of magnitude, so there is no need to normalize the costs.

[0146] S1043. Traverse the cost matrix and use the Winner-Take-All (WTA) method to find the minimum cost among all costs contained in each point, that is, the corresponding disparity value.

[0147] In this embodiment, the WTA formula is as follows:

[0148]

[0149] Among them, C i (i, j) represents all the costs contained in a pixel.

[0150] S1044. Select the costs before and after the minimum cost to fit a quadratic curve, and then use the mathematical principle that the derivative equals 0 to the extreme point of the curve to find the optimal disparity value of the floating point number.

[0151] S1045 : Add each optimal disparity value to corresponding pixel points of the left image and the right image to generate a first initial disparity map and a second initial disparity map.

[0152] S105 , based on the first initial disparity map and the second initial disparity map, perform occlusion recovery on the third initial disparity map according to consistency detection, invalid point classification, and disparity reconstruction based on fitness.

[0153] like Figure 11 As shown, step S105 specifically includes:

[0154] S1051 : Perform consistency detection using the first initial disparity map and the second initial disparity map, thereby obtaining a plurality of invalid points, where the invalid points include occlusion points and mismatched points.

[0155] Furthermore, the consistency check is performed using the first initial disparity map and the second initial disparity map, as shown in the following formula:

[0156] D L (i,j)-D R [i,jD L (i,j)]≤d

[0157] Among them, D L (i, j) represents the left point disparity, D R (i, j) represents the disparity of the right point, and d represents the threshold for determining whether the consistent detection is passed.

[0158] In this embodiment, d is set to 1, pixels that fail to pass are invalid and set to 0, and pixels that pass are valid and set to 255.

[0159] S1052. Classify the invalid points according to their positional features.

[0160] Furthermore, the invalid points are classified according to their position characteristics, as follows:

[0161]

[0162] Among them, 0 represents the leftmost point, 1 represents the rightmost point, 2 represents the center point, 3 represents the left edge point, and 4 represents the right edge point. The invalid point obtains five consecutive valid points to the left, and the average disparity of the second, third, and fourth valid points is Get five consecutive valid points to the right of the invalid point, and the average disparity of the second, third and fourth valid points is Δd represents the threshold for judging the parallax step.

[0163] In this embodiment, Δd is set to 1.

[0164] S1053. Based on the classified invalid point, establish a second adaptive window with the invalid point as the center, wherein the length of the second adaptive window is equal to twice the distance between the nearest valid points on the left and right of the invalid point, and the width of the second adaptive window is equal to twice the distance between the nearest valid points above and below the invalid point.

[0165] The purpose of step S1053 is to ensure that the second adaptive window established with the invalid point as the center contains enough valid points around it.

[0166] S1054: Calculate the valid points in the second adaptive window to obtain the fitness.

[0167] Fitness indicates whether a valid point is suitable for filling the disparity of an invalid point. It is determined by the color difference between the valid and invalid points in the original image, the Euler distance, and the disparity of the valid point. Since these three factors cover all information related to disparity reconstruction, disparity reconstruction based on fitness will appear more accurate.

[0168] Furthermore, the valid points in the second adaptive window are calculated to obtain the fitness, as shown in the following formula:

[0169]

[0170] Wherein, ΔI(x,y) represents the color difference, Δl(x,y) represents the distance difference, d(x,y) represents the parallax value of the effective point, δ r , δ s , δ dRespectively represent the scale factors of color difference, distance difference, and parallax value of the effective point to prevent a value from being too small or too large.

[0171] To further reduce the computational complexity, the three parameters will be represented only by binary numbers, namely: r ,λ s ,λ d In the leftmost area: 1, 1, 0; λ r ,λ s ,λ d In the rightmost area: 0, 1, 1; λ r ,λ s ,λ d In the center area: 1, 0, 0; λ r ,λ s ,λ d In the left side of the edge: 1, 1, 1; λ r ,λ s ,λ d In the area to the right of the edge: 1, 0, 1.

[0172] It is worth noting that in step S105, the first step is to perform a traditional consistency check to obtain the unmatched points in the first initial disparity map and the second initial disparity map, set them to 0, and the rest to 255; among them, the points set to 0 are divided into occlusion points and mismatched points, but this embodiment is not committed to distinguishing occlusion points from mismatched points. This is because accurately separating the two is likely to introduce a large amount of calculation, and the misclassified points will definitely be filled with errors and have no fault tolerance. Therefore, this embodiment classifies the points scattered in five areas based on position features, creates a second adaptive window for each invalid point, and calculates the fitness. The point with the highest fitness is used to reconstruct the central invalid point disparity, which has been explained in this section. The specific process of occlusion recovery is as follows. Figure 11 As shown, the dashed box represents the steps that each invalid point undergoes. This method of recovering occlusions and compensating for mismatches allows for faster classification and more accurate position-based classification. Furthermore, the fitness formula used when reconstructing disparity always provides a guaranteed baseline for filling. For example, even when color errors occur, valid points with closer distances and smaller disparity are still filled in. Valid points with very small disparity are also considered based on their color and distance before being considered for filling.

[0173] S1055 . Perform occlusion recovery on invalid points in the third initial disparity map according to the valid points with the maximum fitness.

[0174] In this embodiment, the third initial disparity map is one of the first initial disparity map and the second initial disparity map.

[0175] S106 : Perform edge correction on the third initial disparity map after occlusion recovery to obtain a final disparity map.

[0176] like Figure 12 As shown, step S106 specifically includes:

[0177] S1061 : Perform color segmentation on the original image of the third initial disparity map after occlusion recovery, and use the original image after color segmentation as a guide image.

[0178] In step S1061 , a mean-shift algorithm is used to perform color segmentation on the original image of the third initial disparity map after occlusion restoration.

[0179] S1062 : Based on the guide map and the third initial disparity map after occlusion recovery, generate a first pseudo disparity map according to a guided filtering formula, wherein the guided filtering formula includes a penalty function ε(x, y), where ε(x, y)=p, where p is a constant.

[0180] In this embodiment, the guiding filtering formula is as follows:

[0181]

[0182]

[0183]

[0184] Where q represents the output image, I represents the guide image, a and b represent the parameters in the linear formula, which control the edge protection ability and image smoothing ability respectively, |ω| represents the total number of pixels in the calculation window, and p i and is the grayscale value of the input image pixel and its grayscale mean in the window, μ k and They represent the grayscale mean and grayscale variance within the guide image pixel window respectively.

[0185] It should be noted that the key lies in the penalty function ε(x,y) with a value between 0 and 1. The larger its value, the smoother the image will be, and the smaller its value, the more prominent the edge will be.

[0186] S1063 , based on the first pseudo-disparity map, generate a second pseudo-disparity map according to a guided filtering formula, assigning ε(x, y)=0 to points on the disparity edge in the first pseudo-disparity map, and assigning ε(x, y)=1 to other areas.

[0187] S1064 : Enhance the second pseudo disparity map according to the adaptive image enhancement function to obtain a third pseudo disparity map.

[0188] In this embodiment, the adaptive image enhancement function is as follows:

[0189]

[0190]

[0191] Among them, Factor(x,y) represents the enhancement factor of adaptive change, and δ represents the proportional factor of the control gradient.

[0192] According to the adaptive image enhancement function, the edge with a larger disparity gradient will cause its scale factor to change more, thereby increasing the contrast on both sides of the edge and achieving the purpose of strengthening the edge.

[0193] S1065 , using the third pseudo disparity map to correct edges of the third initial disparity map after occlusion restoration, to obtain a final disparity map.

[0194] It should be noted that the edge correction process is completed in two iterations. First, the adaptive Canny operator is used to detect the edge between the pseudo-disparity image and the true disparity image. Then, in the first iteration, the edge points of the true disparity image are traversed, searching for pseudo-disparity image edges that are close to their epipolar lines, and the space between the two edge points is re-painted. The second iteration expands the search range to four directions: up, down, left, and right. The same method is used to process disparity discontinuities to complete edge correction.

[0195] Before obtaining the final disparity map, weighted median filtering is used to filter out some burrs generated during the edge correction process, and the process image is pseudo-colored to output the final disparity map. This concludes this embodiment.

[0196] Those skilled in the art will appreciate that all or part of the steps in the method for implementing the above embodiments may be completed by instructing related hardware through a program, and the corresponding program may be stored in a computer-readable storage medium.

[0197] It should be noted that although the method operations of the above embodiments are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all of the illustrated operations must be performed to achieve the desired results. Rather, the depicted steps may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into a single step, and / or a single step may be broken down into multiple steps.

[0198] Example 2:

[0199] like Figure 13 As shown, this embodiment provides a stereo matching system based on high-precision disparity refinement. The system includes an acquisition unit 1301, a texture segmentation unit 1302, a cost calculation unit 1303, a cost aggregation and disparity calculation unit 1304, an occlusion recovery unit 1305, and an edge correction unit 1306. The specific functions of each unit are as follows:

[0200] An acquiring unit 1301 is configured to acquire a binocular image after epipolar correction, where the binocular image includes a first image and a second image.

[0201] The texture segmentation unit 1302 is used to divide the binocular image into multiple texture regions according to texture segmentation;

[0202] The cost calculation unit 1303 is configured to establish a first adaptive window with each pixel as the center in the plurality of texture regions, and then calculate a cost set of the first image and a cost set of the second image;

[0203] a cost aggregation and disparity calculation unit 1304 for obtaining a first initial disparity map and a second initial disparity map according to a cost set of the first image and a cost set of the second image, respectively;

[0204] an occlusion recovery unit 1305 for performing occlusion recovery on a third initial disparity map based on the first and second initial disparity maps, according to consistency detection, invalid point classification, and fitness-based disparity reconstruction, where the third initial disparity map is one of the first and second initial disparity maps;

[0205] The edge correction unit 1306 is configured to perform edge correction on the third initial disparity map after occlusion recovery to obtain a final disparity map.

[0206] Example 3:

[0207] like Figure 14 As shown, this embodiment provides a computer device, which includes a processor 1402, a memory, an input device 1403, a display device 1404, and a network interface 1405 connected via a system bus 1401. The processor 1402 is used to provide computing and control capabilities, and the memory includes a non-volatile storage medium 1406 and an internal memory 1407. The non-volatile storage medium 1406 stores an operating system, a computer program, and a database. The internal memory 1407 provides an environment for the operation of the operating system and computer program in the non-volatile storage medium 1406. When the computer program is executed by the processor 1402, the stereo matching method of the above-mentioned embodiment 1 is implemented as follows:

[0208] Acquire an epipolar-corrected binocular image, the binocular image comprising a first image and a second image;

[0209] Based on texture segmentation, the binocular image is divided into multiple texture regions;

[0210] In the plurality of texture regions, a first adaptive window is established with each pixel as the center, and then a cost set of the first image and a cost set of the second image are calculated;

[0211] Obtaining a first initial disparity map and a second initial disparity map according to the cost set of the first image and the cost set of the second image, respectively;

[0212] Based on the first and second initial disparity maps, performing occlusion recovery on a third initial disparity map according to consistency detection, invalid point classification, and fitness-based disparity reconstruction, where the third initial disparity map is one of the first and second initial disparity maps;

[0213] Edge correction is performed on the third initial disparity map after occlusion recovery to obtain a final disparity map.

[0214] Example 4:

[0215] This embodiment provides a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, the stereo matching method of the above embodiment 1 is implemented as follows:

[0216] Acquire an epipolar-corrected binocular image, the binocular image comprising a first image and a second image;

[0217] Based on texture segmentation, the binocular image is divided into multiple texture regions;

[0218] In the plurality of texture regions, a first adaptive window is established with each pixel as the center, and then a cost set of the first image and a cost set of the second image are calculated;

[0219] Obtaining a first initial disparity map and a second initial disparity map according to the cost set of the first image and the cost set of the second image, respectively;

[0220] Based on the first and second initial disparity maps, performing occlusion recovery on a third initial disparity map according to consistency detection, invalid point classification, and fitness-based disparity reconstruction, where the third initial disparity map is one of the first and second initial disparity maps;

[0221] Edge correction is performed on the third initial disparity map after occlusion recovery to obtain a final disparity map.

[0222] It should be noted that the computer-readable storage medium of the present embodiment may be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0223] In this embodiment, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. Furthermore, in this embodiment, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0224] The computer readable storage medium can be written in one or more programming languages or a combination thereof to execute the computer program for performing the present embodiment, including object-oriented programming languages such as Java, Python, C++, and conventional procedural programming languages such as C or similar programming languages. The program can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect via the Internet).

[0225] In summary, the present invention introduces texture segmentation to improve disparity calculation, calculates fitness in the disparity refinement stage to complete occlusion recovery, and uses secondary guided filtering based on the disparity map to correct disparity discontinuities, thereby effectively improving the disparity refinement capability without increasing the computational complexity as much as possible.

[0226] The above is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes based on the technical solution and inventive concept of the present invention within the scope disclosed by the present invention, which falls within the scope of protection of the present invention.

Claims

1. A stereo matching method based on high-precision disparity refinement, characterized in that: The method comprises: Acquire an epipolar-corrected binocular image, the binocular image comprising a first image and a second image; Based on texture segmentation, the binocular image is divided into multiple texture regions; In the plurality of texture regions, a first adaptive window is established with each pixel as the center, and then a cost set of the first image and a cost set of the second image are calculated; Obtaining a first initial disparity map and a second initial disparity map according to the cost set of the first image and the cost set of the second image, respectively; Based on the first and second initial disparity maps, performing occlusion recovery on a third initial disparity map according to consistency detection, invalid point classification, and fitness-based disparity reconstruction, where the third initial disparity map is one of the first and second initial disparity maps; Performing edge correction on the third initial disparity map after occlusion recovery to obtain a final disparity map; The method performs occlusion recovery on the third initial disparity map based on the first initial disparity map and the second initial disparity map according to consistency detection, invalid point classification, and disparity reconstruction based on fitness, specifically including: Performing consistency detection using the first initial disparity map and the second initial disparity map, thereby obtaining a plurality of invalid points, wherein the invalid points include occlusion points and mismatched points; Classify invalid points according to their location characteristics; Based on the classified invalid point, a second adaptive window is established with the invalid point as the center, wherein the length of the second adaptive window is equal to twice the distance between the nearest valid points on the left and right of the invalid point, and the width of the second adaptive window is equal to twice the distance between the nearest valid points above and below the invalid point; Calculating the valid points in the second adaptive window to obtain the fitness; According to the valid points with the maximum fitness, occlusion recovery is performed on the invalid points in the third initial disparity map.

2. The stereo matching method according to claim 1, wherein: The texture region includes one of a high texture region, a second repeating texture region, and a weak texture region.

3. The stereo matching method according to claim 1, wherein: The binocular image is divided into a plurality of texture regions according to texture segmentation, specifically including: In the binocular image, a fixed window is established with each pixel as the center; Calculate the color variance within a fixed window; According to the color variance and bilateral filtering in all fixed windows, the binocular image is segmented once to obtain the high texture area, weak texture area and the first repeated texture area; The first repeated texture region is divided twice to obtain a second repeated texture region.

4. The stereo matching method according to claim 2, wherein: The step of establishing a first adaptive window with each pixel as the center within the plurality of texture regions and then calculating a cost set of the first image and a cost set of the second image specifically includes: Using successively larger window sizes for pixels in high texture areas, second repeated texture areas, and weak texture areas; Within the disparity search range, calculate the cost of any window in the first image and all windows in the second image, thereby obtaining a cost set of the first image; The central pixel point in any one window and the central pixel points in all windows are on the same epipolar line; Among them, the cost is calculated by SAD cost calculation, Census cost calculation and gradient cost calculation; Similarly, the cost set of the second image is obtained.

5. The stereo matching method according to claim 1, wherein: The consistency detection is performed using the first initial disparity map and the second initial disparity map, as shown in the following formula: ； in, D L ( i , j ) represents the left point parallax, D R ( i , j ) represents the right point parallax, d Indicates the threshold for judging whether the consistency detection is passed; The invalid points are classified according to their position characteristics, as shown in the following formula: ； Among them, 0 represents the leftmost point, 1 represents the rightmost point, 2 represents the center point, 3 represents the left edge point, and 4 represents the right edge point. The invalid point obtains five consecutive valid points to the left, and the average disparity of the second, third, and fourth valid points is ; Get five consecutive valid points to the right of the invalid point, and the average disparity of the second, third, and fourth valid points is ; Indicates the threshold for judging disparity steps; The effective points in the second adaptive window are calculated to obtain the fitness, which is as follows: ； Among them, △ I ( x , y ) indicates color difference, △ l ( x , y ) represents the distance difference, d ( x , y ) represents the disparity value of the effective point, δ r 、 δ s 、 δ d Respectively represent the scale factors of color difference, distance difference, and parallax value of the effective point; λ r 、 λ s 、 λ d In the leftmost area: 1, 1, 0; λ r 、 λ s 、 λ d In the rightmost area: 0, 1, 1; λ r 、 λ s 、 λ d In the center area: 1, 0, 0; λ r 、 λ s 、 λ d In the left area of the edge: 1, 1, 1; λ r 、 λ s 、 λ d In the area to the right of the edge: 1, 0, 1.

6. The stereo matching method according to claim 1, wherein: The edge correction is performed on the third initial disparity map after occlusion recovery to obtain a final disparity map, specifically comprising: Performing color segmentation on the original image of the third initial disparity map after occlusion recovery, and using the original image after color segmentation as a guide image; Based on the guide map and the third initial disparity map after occlusion recovery, a first pseudo disparity map is generated according to a guide filtering formula, wherein the guide filtering formula includes a penalty function ε ( x , y ),at this time ε ( x , y )=p, p is a constant; Based on the first pseudo-disparity map, a second pseudo-disparity map is generated according to the guided filtering formula. At this time, the points on the disparity edge of the first pseudo-disparity map are assigned ε ( x , y )=0, other areas are given ε ( x , y )=1; enhancing the second pseudo disparity map according to an adaptive image enhancement function to obtain a third pseudo disparity map; The third pseudo disparity map is used to correct the edge of the third initial disparity map after occlusion restoration to obtain a final disparity map.

7. A stereo matching system based on high-precision disparity refinement, characterized in that: The system comprises: an acquisition unit, configured to acquire a binocular image after epipolar correction, the binocular image comprising a first image and a second image; A texture segmentation unit, configured to divide the binocular image into a plurality of texture regions according to texture segmentation; A cost calculation unit, configured to establish a first adaptive window with each pixel as a center within the plurality of texture regions, and further calculate a cost set of the first image and a cost set of the second image; a cost aggregation and disparity calculation unit, configured to obtain a first initial disparity map and a second initial disparity map according to a cost set of the first image and a cost set of the second image, respectively; an occlusion recovery unit, configured to perform occlusion recovery on a third initial disparity map based on the first and second initial disparity maps, according to consistency detection, invalid point classification, and fitness-based disparity reconstruction, wherein the third initial disparity map is one of the first and second initial disparity maps; an edge correction unit, configured to perform edge correction on the third initial disparity map after occlusion recovery to obtain a final disparity map; The method performs occlusion recovery on the third initial disparity map based on the first initial disparity map and the second initial disparity map according to consistency detection, invalid point classification, and disparity reconstruction based on fitness, specifically including: Performing consistency detection using the first initial disparity map and the second initial disparity map, thereby obtaining a plurality of invalid points, wherein the invalid points include occlusion points and mismatched points; Classify invalid points according to their location characteristics; Based on the classified invalid point, a second adaptive window is established with the invalid point as the center, wherein the length of the second adaptive window is equal to twice the distance between the nearest valid points on the left and right of the invalid point, and the width of the second adaptive window is equal to twice the distance between the nearest valid points above and below the invalid point; Calculating the valid points in the second adaptive window to obtain the fitness; According to the valid points with the maximum fitness, occlusion recovery is performed on the invalid points in the third initial disparity map.

8. A computer device comprising a processor and a memory for storing a program executable by the processor, characterized in that: When the processor executes the program stored in the memory, the stereo matching method according to any one of claims 1 to 6 is implemented.

9. A storage medium storing a program, characterized in that: When the program is executed by a processor, the stereo matching method according to any one of claims 1 to 6 is implemented.

Images