Speckle structured light binocular stereo matching method and system based on multi-cost fusion, computer readable storage medium and computer program product

Abstract

The invention relates to the technical field of image processing, and discloses a speckle structured light binocular stereo matching method and system based on multi-cost fusion, a computer readable storage medium and a computer program product. The method comprises the following steps: respectively carrying out epipolar correction on an obtained left camera image and an obtained right camera image; performing boundary segmentation on the corrected left camera image to obtain a corresponding boundary image; calculating a Census cost and a BT cost by using the corrected left camera image and right camera image, and carrying out weighted fusion to obtain a matched cost body; performing cost aggregation on the obtained cost body by using the boundary image as prior information; and calculating a parallax image corresponding to the left camera image through the aggregated cost body. According to the invention, abnormal matching points in a speckle image stereo matching process are reduced, and the real-time performance and robustness of speckle image stereo matching are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a method, a system, a computer-readable storage medium, and a computer program product for binocular stereo matching of speckle structured light based on multi-cost fusion. Background Art

[0002] Binocular speckle structured light is an active binocular vision system composed of a speckle projector and two cameras, which has advantages such as non-contact, high real-time performance, simple structure, and low hardware cost, and is widely used in the field of industrial inspection. Compared with traditional passive binocular vision, binocular speckle structured light projects a speckle pattern onto the surface of the object to be measured to enhance the texture features of the object surface, overcoming the problem of difficult matching in traditional binocular vision in weak texture, repetitive texture, and low-light environments.

[0003] The performance of the binocular speckle structured light system largely depends on the accuracy of the stereo matching algorithm. Traditional stereo matching algorithms mainly use semi-global stereo matching, such as the SGM algorithm (Semi-Global Matching). This method uses the Census transform to calculate the cost and approximates the global optimization energy function through dynamic programming. However, there are some problems with this algorithm. Firstly, the single Census transform has poor stability. Secondly, the two smoothing constraints introduced by the algorithm cannot be applied to all scenarios, resulting in possible stratification, boundary anomalies, or over-smoothing problems in the output depth. Summary of the Invention

[0004] The purpose of the present invention is to provide a method, a system, a computer-readable storage medium, and a computer program product for binocular stereo matching of speckle structured light based on multi-cost fusion, so as to solve or at least partially solve the technical problems mentioned in the above background art.

[0005] To achieve this purpose, the present invention adopts the following technical solutions: In the first aspect, the present invention provides a method for binocular stereo matching of speckle structured light based on multi-cost fusion, including: Obtaining a left camera image and a right camera image captured by a binocular speckle camera; Performing epipolar correction on the obtained left camera image and right camera image respectively; Performing boundary segmentation on the corrected left camera image to obtain a corresponding boundary image; Calculating the Census cost and the BT cost by using the corrected left camera image and right camera image, and weighted-fusing them to obtain a matching cost volume; Using the boundary image as prior information to perform cost aggregation on the obtained cost volume; Calculating a disparity image corresponding to the left camera image through the aggregated cost volume; Use a marker - based small connected component filtering method to remove abnormal disparity points in the disparity image.

[0006] Optionally, the boundary segmentation of the corrected left - camera image to obtain the corresponding boundary image specifically includes: Perform grid processing on the corrected left - camera image to obtain a grid image, and then perform median filtering and mean filtering on the grid image successively; Calculate the weighted variance in the grid image; Using the calculated weighted variance, perform maximum - value search on the grid image in the row direction and column direction respectively to obtain a row - direction extreme - value map and a column - direction extreme - value map; Merge the row - direction extreme - value map and the column - direction extreme - value map through the direction characteristics of the boundary to obtain an extreme - value image containing boundary information; Remove the outliers in the extreme - value image, and upsample the extreme - value image after removing the outliers to obtain the finally segmented boundary image.

[0007] Optionally, the method for performing grid processing on the corrected left - camera image is as follows: Assume the grid size is m×m. First, obtain the coordinates of the maximum - value pixel point in each grid, calculate the pixel mean within the 3×3 neighborhood centered on this coordinate as the output value of this grid, and obtain an image of size M / m×N / m, where M and N respectively represent the width and height of the input image; The calculation formula for the weighted variance is as follows: ; In the above calculation formula, V(u,v) represents the original variance of a certain point to be calculated, k represents the neighborhood size, f(x,y) and respectively represent the pixel value of a point within the neighborhood and the pixel mean within the neighborhood, represents the weighted variance, and grayMin and grayMax respectively represent the minimum pixel value and the maximum pixel value within the neighborhood.

[0008] Optionally, the calculation method of the Census cost is as follows: First, perform Census transformation on the corrected left - and right - camera images respectively: ; Where C S (p) represents the Census transformation value corresponding to the pixel point p, represents the bit - by - bit connection operation, q represents a pixel point within the neighborhood of p, I(p) and I(q) represent the pixel values of the corresponding pixel points, The operation is defined as follows: ; The Hamming distance between the left and right camera images is calculated by the following formula, i.e., the Census cost: ; where CT(p, d) represents the Census cost of pixel point p in the left camera image at disparity d, and C sl (p) represents the Census transform value corresponding to pixel point p in the left camera image, and C sr (p, d) represents the Census transform value corresponding to pixel point p - d in the right camera image; The BT cost uses the BT based on the xsobel image xsobel and the BT based on the grayscale image gray , and is obtained through weighted fusion, expressed as: ; The calculation process of the BT cost is as follows: ; where BT(p, d) represents the BT cost value of pixel point p in the left camera image at disparity d, and respectively represent the BT cost calculated using sub-pixel interpolation constructed from the left camera image and the BT cost calculated using sub-pixel interpolation constructed from the right camera image. The construction of the BT cost using sub-pixel interpolation is shown as follows: ; where I L (p) represents the pixel value corresponding to pixel point p in the left camera image, I L - represents the sub-pixel value from pixel point p - 1 to pixel point p in the left camera image, I L + represents the sub-pixel value from pixel point p to pixel point p + 1 in the left camera image, I R (p, d) represents the pixel value corresponding to pixel point p + d in the right camera image; ; where I R (p) represents the pixel value corresponding to pixel point p in the right camera image, I R - represents the sub-pixel value from pixel point p - 1 to pixel point p in the right camera image, I R + represents the sub-pixel value from pixel point p to pixel point p + 1 in the right camera image, I L (p, d) represents the pixel value corresponding to pixel point p + d in the left camera image; The calculation formula for the weighted fusion of the Census cost and the BT cost is as follows: ; Among them, C(p, d) is the merged cost, BT(p, d) and CT(p, d) respectively represent the calculated BT cost and Census cost, w is the window size selected when calculating the Census cost, and c1 and c2 are proportionality coefficients.

[0009] Optionally, using the boundary image as prior information to perform cost aggregation on the obtained cost volume specifically includes: Performing cost aggregation on the obtained cost volume through the dynamic programming method, and the aggregation formula is as follows: ; Among them, L r (p, d) represents the aggregated cost of pixel point p when the disparity is d, r represents the aggregation path, and L r (p - r, i) represents the aggregated cost of the previous path point of pixel point p under the disparity i, and C(p, d) represents the initial cost, and P1 and P2 are penalty coefficients; Dynamically adjusting the penalty coefficient P2 using the boundary image, and the calculation formula is as follows: ; Among them, P 2- Init represents the cost initial value, and gray represents the gray value of the corresponding point obtained from the boundary image.

[0010] Optionally, calculating the disparity image corresponding to the left camera image through the aggregated cost volume specifically includes: Selecting the minimum cost value from the aggregated cost volume, obtaining the disparity corresponding to the minimum cost value and the disparities adjacent to the current disparity on the left and right, and then calculating the disparity through the quadratic curve fitting method as the binocular matching disparity of the current pixel point. As shown in the following formula: ; Among them, d sub represents the fitted disparity, d min is the disparity corresponding to the minimum cost, c0 is the minimum cost value, and c1 and c2 respectively represent the cost values corresponding to the disparities adjacent to the left and right.

[0011] Optionally, using the method of filtering out abnormal disparity points in the disparity image based on labeled small connected components specifically includes: Finding the root node of each pixel point in the disparity image through a recursive method, classifying the pixel points belonging to the same root node into the same sub-region, and using the index of the root node as the serial number of the sub-region where the root node is located; Traversing the pixel points at the junctions of all sub-regions, and merging all related sub-regions into one region; Statistically calculate the area of each region, and filter out the same positions in the disparity maps corresponding to the regions with areas smaller than a predetermined area threshold to obtain the final disparity map.

[0012] Optionally, the method for finding the root node of each pixel point in the disparity image through a recursive method is as follows: Within the 3×3 neighborhood of the pixel point, compare the disparity value of the pixel point with the disparity values of the four adjacent points, namely the upper left, upper, upper right, and right adjacent points, in sequence; If the absolute value of the difference between the disparity value of the adjacent point and the disparity value of the pixel point is less than a preset difference threshold, then the adjacent point is the parent node; if none of the four adjacent points is the parent node, then the pixel point is the root node.

[0013] Optionally, the method for traversing the pixel points at the junctions of all sub-regions and merging all the associated sub-regions into one region is as follows: Let the sets of pixel points at the junctions of two sub-regions be A and B respectively. A is expressed as {a1, a2, a3,......, a n}, and B is expressed as {b1, b2, b3,......, b n}. Pixel point a i and pixel point b i are adjacent pixel points of the two sub-regions, and i is a natural number less than or equal to n; Determine whether the absolute value of a i - b i is less than a preset difference threshold; if so, then pixel point a i is associated with pixel point b i ; if not, then record that pixel point a i is not associated with pixel point b i ; If there are k consecutive pixel points in A that are associated with B, then A and B are two associated sub-regions; k is a preset associated point number threshold.

[0014] Optionally, before using the method of removing abnormal disparity points in the disparity image based on marker-based small connected component filtering, it further includes: Remove the noise in the disparity image through uniqueness detection, left-right consistency detection, and median filtering.

[0015] In a second aspect, the present invention provides a system for binocular stereo matching of speckle structured light based on multi-cost fusion, including: A binocular speckle camera for capturing a left camera image and a right camera image containing a speckle pattern; An image correction module electrically connected to the binocular speckle camera for respectively performing epipolar correction on the captured left camera image and right camera image; A boundary segmentation module, electrically connected to the image correction module, for performing boundary segmentation on the corrected left camera image to obtain a corresponding boundary image; A cost calculation module, electrically connected to the image correction module, for calculating the Census cost and the BT cost using the corrected left camera image and the right camera image, and weighted fusing to obtain a matching cost volume; A cost aggregation module, electrically connected to the boundary segmentation module and the cost calculation module respectively, for using the boundary image as prior information to perform cost aggregation on the obtained cost volume; A disparity image generation module, electrically connected to the cost aggregation module and the image correction module respectively, for calculating the disparity image corresponding to the left camera image through the aggregated cost volume; An image optimization module, electrically connected to the disparity image generation module, for removing abnormal disparity points in the disparity image using a marker-based small connected component filtering method.

[0016] Optionally, the boundary segmentation module is specifically configured to: Perform grid processing on the corrected left camera image to obtain a grid image, and perform median filtering and mean filtering on the grid image successively; Calculate the weighted variance in the grid image; Use the calculated weighted variance to perform maximum value search on the grid image in the row direction and the column direction respectively to obtain a row direction extreme value map and a column direction extreme value map; Merge the row direction extreme value map and the column direction extreme value map through the direction characteristics of the boundary to obtain an extreme value image containing boundary information; Remove the outliers in the extreme value image, and upsample the extreme value image after removing the outliers to obtain the finally segmented boundary image.

[0017] Optionally, the method for performing grid processing on the corrected left camera image is specifically: Set the grid size to m×m. First, obtain the coordinates of the maximum pixel point in each grid, calculate the pixel mean in the 3×3 neighborhood centered on this coordinate as the output value of this grid, and obtain an image with a size of M / m×N / m, where M and N respectively represent the width and height of the input image; The calculation formula for the weighted variance is as follows: ; In the above calculation formula, V(u,v) represents the original variance of a certain point to be calculated, k represents the neighborhood size, f(x,y) and respectively represent the pixel value of a point in the neighborhood and the pixel mean in the neighborhood, represents the weighted variance, and grayMin and grayMax respectively represent the minimum pixel value and the maximum pixel value in the neighborhood.

[0018] Optionally, the calculation method of the Census cost is as follows: First, perform the Census transform on the corrected left and right camera images respectively: ; where C S (p) represents the Census transform value corresponding to the pixel point p, represents the bit-by-bit concatenation operation of bits, q represents a pixel point in the neighborhood of point p, and I(p) and I(q) represent the pixel values of the corresponding pixel points, The operation is defined as follows: ; Calculate the Hamming distance between the left and right camera images through the following formula, that is, the Census cost: ; where CT(p,d) represents the Census cost of the pixel point p in the left camera image at the disparity d, C sl (p) represents the Census transform value corresponding to the pixel point p in the left camera image, and C sr (p,d) represents the Census transform value corresponding to the pixel point p - d in the right camera image; The BT cost uses the BT based on the xsobel image xsobel and the BT based on the grayscale image gray , and then obtains it through weighted fusion, expressed as: ; The calculation process of the BT cost is as follows: ; where BT(p,d) represents the BT cost value of the pixel point p in the left camera image at the disparity d, and respectively represent the BT cost calculated by constructing sub-pixel interpolation using the left camera image and the BT cost calculated by constructing sub-pixel interpolation using the right camera image. The construction of the BT cost using sub-pixel interpolation is shown in the following formula: ; where, I L (p) represents the pixel value corresponding to the pixel point p in the left camera image, I L - represents the sub-pixel value from the pixel point p - 1 to the pixel point p in the left camera image, I L + represents the sub-pixel value from the pixel point p to the pixel point p + 1 in the left camera image, I R(p, d) represents the pixel value corresponding to pixel point p + d in the right camera image; ; where I R (p) represents the pixel value corresponding to pixel point p in the right camera image, and I R - represents the sub-pixel value from pixel point p - 1 to pixel point p in the right camera image, and I R + represents the sub-pixel value from pixel point p to pixel point p + 1 in the right camera image, and I L (p, d) represents the pixel value corresponding to pixel point p + d in the left camera image; The calculation formula for weighted fusion of Census cost and BT cost is as follows: ; where C(p, d) is the fused cost, BT(p, d) and CT(p, d) respectively represent the calculated BT cost and Census cost, w is the window size selected when calculating the Census cost, and c1 and c2 are proportionality coefficients.

[0019] Optionally, the cost aggregation module is specifically configured to: Perform cost aggregation on the obtained cost volume through the dynamic programming method, and the aggregation formula is as follows: ; where L r (p, d) represents the aggregated cost of pixel point p when the disparity is d, r represents the aggregation path, and L r (p - r, i) represents the aggregated cost of the previous path point of pixel point p under the disparity i, and C(p, d) represents the initial cost, and P1 and P2 are penalty coefficients; Dynamically adjust the penalty coefficient P2 using the boundary image, and the calculation formula is as follows: ; where P 2- Init represents the cost initial value, and gray represents the gray value of the corresponding point obtained from the boundary image.

[0020] Optionally, the disparity image generation module is specifically configured to: Select the minimum cost value from the aggregated cost volume, obtain the disparity corresponding to the minimum cost value and the disparities adjacent to the current disparity on the left and right, and then calculate the disparity through the quadratic curve fitting method as the binocular matching disparity of the current pixel point. As shown in the following formula: ; where d sub represents the fitted disparity, and d minThe disparity corresponding to the minimum cost, c0 is the minimum cost value, and c1 and c2 respectively represent the cost values corresponding to the left and right adjacent disparities.

[0021] Optionally, the disparity anomaly point filtering module is specifically configured to: Find the root node of each pixel point in the disparity image through a recursive method, classify the pixel points belonging to the same root node into the same sub-region, and use the index of the root node as the serial number of the sub-region where the root node is located; Traverse the pixel points at the junctions of all sub-regions, and merge all the associated sub-regions into one region; Count the area of each region, filter out the same positions in the disparity map corresponding to the regions with an area smaller than a predetermined area threshold, and obtain the final disparity map.

[0022] Optionally, the method for specifically finding the root node of each pixel point in the disparity image through a recursive method is: Within the 3×3 neighborhood of the pixel point, compare the disparity value of the pixel point with the disparity values of the four adjacent points of the upper left, upper, upper right, and right in turn; If the absolute value of the difference between the disparity value of the adjacent point and the disparity value of the pixel point is less than a preset difference threshold, then the adjacent point is the parent node; if none of the four adjacent points is the parent node, then the pixel point is the root node.

[0023] Optionally, the method for specifically traversing the pixel points at the junctions of all sub-regions and merging all the associated sub-regions into one region is: Let the sets of pixel points at the junctions of two sub-regions be A and B respectively. A is expressed as {a1, a2, a3,......, a n}, B is expressed as {b1, b2, b3,......, b n}, the pixel point a i and the pixel point b i are the adjacent pixel points of the two sub-regions, and i is a natural number less than or equal to n; Judge whether the absolute value of a i - b i is less than a preset difference threshold; if so, the pixel point a i is associated with the pixel point b i ; if not, record that the pixel point a i is not associated with the pixel point b i ; If there are k consecutive pixel points in A that are associated with B, then A and B are two associated sub-regions; k is a preset associated point number threshold.

[0024] Optionally, the image optimization module is further configured to remove the noise in the disparity image through uniqueness detection, left-right consistency detection, and median filtering.

[0025] In a third aspect, the present invention further provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the method for speckle structured light binocular stereo matching based on multi-cost fusion as described above.

[0026] In a fourth aspect, the present invention further provides a computer program product, including a computer program / instructions, characterized in that when the computer program / instructions are executed by a processor, the method for speckle structured light binocular stereo matching based on multi-cost fusion as described above is implemented.

[0027] Compared with the prior art, the beneficial effects of the present invention are as follows: The method for speckle structured light binocular stereo matching based on multi-cost fusion provided by the present invention reduces abnormal matching points and effectively improves the real-time performance and robustness of stereo matching of speckle images. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0029] Figure 1 It is a flowchart of the method for speckle structured light binocular stereo matching based on multi-cost fusion provided by an embodiment of the present invention.

[0030] Figure 2 It is a schematic architecture diagram of the system for speckle structured light binocular stereo matching based on multi-cost fusion provided by an embodiment of the present invention.

[0031] Figure 3 It is a corrected left camera image provided by an embodiment of the present invention.

[0032] Figure 4 For Figure 3 The boundary image obtained by performing boundary segmentation.

[0033] Figure 5 It is an image obtained after classifying the pixel points belonging to the same root node in the disparity image into the same sub-region.

[0034] Figure 6 For Figure 5 The image obtained after merging all the associated sub-regions in

[0035] Figure 7The disparity map is obtained after removing the abnormal disparity points in the disparity image.

[0036] In the figure: 10. Binocular speckle camera; 20. Image correction module; 30. Boundary segmentation module; 40. Stereo calculation module; 50. Cost aggregation module; 60. Disparity image generation module; 70. Image optimization module. Detailed implementation manners

[0037] In order to make the invention objectives, features, and advantages of the present invention more obvious and understandable, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the following described embodiments are only a part of the embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0038] Embodiment 1: Please refer to Figure 1 , Figure 1 , which is a flowchart of a method for speckle structured light binocular stereo matching based on multi-cost fusion provided by an embodiment of the present invention. The method specifically includes: Step 110: Obtain the left camera image and the right camera image captured by the binocular speckle camera.

[0039] Use the binocular speckle camera to capture the target object to obtain the left camera image and the right camera image containing the speckle pattern; Among them, the speckle pattern is a pattern formed by the laser projector on the speckle camera emitting laser light onto the object surface, and then an image containing the speckle pattern can be obtained by capturing with the camera.

[0040] Step 120: Perform epipolar correction on the obtained left camera image and right camera image respectively.

[0041] Perform correction on the left camera image and the right camera image through known calibration parameters to obtain the corresponding corrected images of the left camera and the right camera.

[0042] Exemplarily, as Figure 3 shown, Figure 3 is a corrected left camera image provided by an embodiment of the present invention, and the image size is 1904×1728.

[0043] Step 130: Perform boundary segmentation on the corrected left camera image to obtain the corresponding boundary image.

[0044] Step 130 specifically includes: Step 131: Perform grid processing on the corrected left camera image to obtain a grid image, and then perform median filtering and mean filtering on the grid image successively; Step 132: Calculate the weighted variance in the grid image; Step 133: Use the calculated weighted variance to perform maximum value search on the grid image in the row direction and column direction respectively to obtain a row direction extreme value map and a column direction extreme value map; Step 134: Merge the row direction extreme value map and the column direction extreme value map through the direction characteristics of the boundary to obtain an extreme value image containing boundary information; Step 135: Remove the outliers in the extreme value image, and upsample the extreme value image after removing the outliers to obtain the finally segmented boundary image.

[0045] Exemplarily, as Figure 4 shown, Figure 4 for Figure 3 is the boundary image obtained by performing boundary segmentation.

[0046] More specifically, in Step 131, the method for performing grid processing on the corrected left camera image is as follows: Assume the grid size is m×m. First, obtain the coordinates of the maximum pixel point in each grid, calculate the pixel mean value in the 3×3 neighborhood centered on this coordinate as the output value of this grid, and obtain an image with a size of M / m×N / m, where M and N respectively represent the width and height of the input image; The calculation formula for the weighted variance is as follows: ; In the above calculation formula, V(u,v) represents the original variance of a certain point to be calculated, k represents the neighborhood size, f(x,y) and respectively represent the pixel value of a point in the neighborhood and the pixel mean value in the neighborhood, represents the weighted variance, and grayMin and grayMax respectively represent the minimum pixel value and the maximum pixel value in the neighborhood.

[0047] Step 140: Use the corrected left camera image and the right camera image to calculate the Census cost and the BT cost, and perform weighted fusion to obtain a matching cost volume.

[0048] Specifically, the calculation method for the Census cost is as follows: First, perform Census transformation on the corrected left and right camera images respectively: ; where C S (p) represents the Census transformation value corresponding to the pixel point p, Denotes the bitwise concatenation operation of bit positions. Let q be a pixel point in the neighborhood of point p. I(p) and I(q) represent the pixel values of the corresponding pixel points. The operation is defined as follows: ; The Hamming distance between the left and right camera images is calculated by the following formula, which is the Census cost: ; where CT(p, d) represents the Census cost of pixel point p in the left camera image at disparity d, and C sl (p) represents the Census transform value corresponding to pixel point p in the left camera image, and C sr (p, d) represents the Census transform value corresponding to pixel point p - d in the right camera image; The BT cost uses the BT based on the xsobel image xsobel and the BT based on the grayscale image gray , and is obtained through weighted fusion, expressed as: ; The calculation process of the BT cost is as follows: ; where BT(p, d) represents the BT cost value of pixel point p in the left camera image at disparity d, and respectively represent the BT cost calculated using sub-pixel interpolation constructed from the left camera image and the BT cost calculated using sub-pixel interpolation constructed from the right camera image. The construction of the BT cost using sub-pixel interpolation is shown in the following formula: ; where, I L (p) represents the pixel value corresponding to pixel point p in the left camera image, I L - represents the sub-pixel value from pixel point p - 1 to pixel point p in the left camera image, I L + represents the sub-pixel value from pixel point p to pixel point p + 1 in the left camera image, I R (p, d) represents the pixel value corresponding to pixel point p + d in the right camera image; ; where, I R (p) represents the pixel value corresponding to pixel point p in the right camera image, I R - represents the sub-pixel value from pixel point p - 1 to pixel point p in the right camera image, I R +Denote the sub-pixel value from pixel point p to pixel point p + 1 in the right camera image as I L (p, d) represents the pixel value corresponding to pixel point p + d in the left camera image; The calculation formula for weighted fusion of Census cost and BT cost is as follows: ; Where C(p, d) is the fused cost, BT(p, d) and CT(p, d) respectively represent the calculated BT cost and Census cost, w is the window size selected when calculating the Census cost, and c1 and c2 are proportionality coefficients.

[0049] Step 150: Use the boundary image as prior information to perform cost aggregation on the obtained cost volume.

[0050] Specifically, step 150 is implemented as: Perform cost aggregation on the obtained cost volume through the dynamic programming method, and the aggregation formula is as follows: ; Where, L r (p, d) represents the aggregated cost of pixel point p at disparity d, r represents the aggregation path, L r (p - r, i) represents the aggregated cost of the previous path point of pixel point p at disparity i, and C(p, d) represents the initial cost, and P1 and P2 are penalty coefficients; Dynamically adjust the penalty coefficient P2 using the boundary image, and the calculation formula is as follows: ; Where, P 2- Init represents the cost initial value, and gray represents the gray value of the corresponding point obtained from the boundary image.

[0051] Step 160: Calculate the disparity image corresponding to the left camera image through the aggregated cost volume.

[0052] Specifically, step 160 is implemented as: Select the minimum cost value from the aggregated cost volume to obtain the disparity corresponding to the minimum cost value and the disparities adjacent to the current disparity on the left and right, and then calculate the disparity through the quadratic curve fitting method as the binocular matching disparity of the current pixel point. As shown in the following formula: ; Where d sub represents the fitted disparity, d min is the disparity corresponding to the minimum cost, c0 is the minimum cost value, and c1 and c2 respectively represent the cost values corresponding to the disparities adjacent to the left and right.

[0053] Step 170. Use the small connected component filtering method based on markers to remove the abnormal disparity points in the disparity image.

[0054] Specifically, Step 170 is implemented as follows: Step 171. Find the root node of each pixel point in the disparity image through a recursive method, classify the pixel points belonging to the same root node into the same sub-region, and use the index of the root node as the serial number of the sub-region where the root node is located.

[0055] Exemplarily, as Figure 5 shown, Figure 5 is the image obtained after classifying the pixel points belonging to the same root node in the disparity image into the same sub-region.

[0056] Step 172. Traverse the pixel points at the junctions of all sub-regions, and merge all the associated sub-regions into one region.

[0057] Exemplarily, as Figure 6 shown, Figure 6 is the image obtained after merging all the associated sub-regions in Figure 5 .

[0058] Step 173. Count the area of each region, filter out the same positions in the disparity map corresponding to the regions with an area smaller than a predetermined area threshold, and obtain the final disparity map.

[0059] More specifically, in Step 171, the method for finding the root node is as follows: Within the 3×3 neighborhood of the pixel point, compare the disparity value of the pixel point with the disparity values of the four adjacent points, namely the upper left, upper, upper right, and right adjacent points, in turn; If the absolute value of the difference between the disparity value of the adjacent point and the disparity value of the pixel point is less than a preset difference threshold, then the adjacent point is the parent node; if none of the four adjacent points is the parent node, then the pixel point is the root node.

[0060] In Step 172, the method for determining the associated sub-regions is specifically as follows: Let the sets of pixel points at the junction of two sub-regions be A and B respectively. A is expressed as {a1, a2, a3,......, a n}, B is expressed as {b1, b2, b3,......, b n}, the pixel point a i and the pixel point b i are the adjacent pixel points of the two sub-regions, and i is a natural number less than or equal to n; Judge whether the absolute value of a i - b i is less than the preset difference threshold; if so, then the pixel point a i and the pixel point b iAssociate; otherwise, mark pixel point a i with pixel point b i not associated; If there are k consecutive pixel points in A that are associated with B, then A and B are two sub-regions with an association; k is a preset threshold for the number of associated points. Exemplarily, when k is 4, if a2, a3, a4, and a5 are respectively associated with b2, b3, b4, and b5, then A and B are two sub-regions with an association.

[0061] Further, before step 170, it further includes: Optimize the disparity image, and the specific optimization method is: Remove the noise in the disparity image through uniqueness detection, left-right consistency detection, and median filtering.

[0062] Exemplarily, after optimizing the disparity image, remove the abnormal disparity points in the disparity image to obtain an image as Figure 7 shown.

[0063] In this embodiment, the method of boundary segmentation is used to improve the detection effect at the object boundary and enhance the detection ability of small objects; by fusing multiple cost calculation methods, the system robustness is improved; an efficient small connected component filtering algorithm is adopted. Compared with the traditional region growing method, the filtering algorithm of this embodiment can be highly parallelized; In summary, a speckle structured light binocular stereo matching method based on multi-cost fusion provided by this embodiment effectively improves the effect and robustness of stereo matching of speckle images and reduces abnormal matching points.

[0064] Embodiment 2: Please refer to Figure 2 , Figure 2 which is the architecture schematic diagram of a speckle structured light binocular stereo matching system provided by an embodiment of the present invention.

[0065] This system specifically includes: A binocular speckle camera 10, configured to capture a left camera image and a right camera image containing a speckle pattern; An image correction module 20, electrically connected to the binocular speckle camera 10, configured to perform epipolar correction on the captured left camera image and right camera image respectively; A boundary segmentation module 30, electrically connected to the image correction module 20, configured to perform boundary segmentation on the corrected left camera image to obtain a corresponding boundary image; A cost calculation module 40, electrically connected to the image correction module 20, configured to calculate the Census cost and the BT cost by using the corrected left camera image and right camera image, and weighted-fuse them to obtain a matching cost volume; The cost aggregation module 50 is electrically connected to the boundary segmentation module 30 and the cost calculation module 40 respectively, and is used to perform cost aggregation on the obtained cost volume by using the boundary image as prior information; The disparity image generation module 60 is electrically connected to the cost aggregation module 50 and the image correction module 20 respectively, and is used to calculate the disparity image corresponding to the left camera image through the aggregated cost volume; The image optimization module 70 is electrically connected to the disparity image generation module 60, and is used to remove the abnormal disparity points in the disparity image by using the small connected domain filtering method based on markers. Further, the image optimization module 70 is also used to remove the noise in the disparity image through uniqueness detection, left-right consistency detection, and median filtering.

[0066] Since a method for speckle structured light binocular stereo matching based on multi-cost fusion has been described in detail in Embodiment 1, it will not be elaborated herein in this embodiment.

[0067] Embodiment 3: This embodiment also provides a computer-readable storage medium, in which at least one instruction is stored, and the instruction is loaded and executed by a processor to implement a method for speckle structured light binocular stereo matching based on multi-cost fusion as described in Embodiment 1.

[0068] Since a method for speckle structured light binocular stereo matching based on multi-cost fusion has been described in detail in Embodiment 1, it will not be elaborated herein in this embodiment.

[0069] Embodiment 4: The present invention also provides a computer program product, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, a method for speckle structured light binocular stereo matching based on multi-cost fusion as described in Embodiment 1 is implemented.

[0070] Since a method for speckle structured light binocular stereo matching based on multi-cost fusion has been described in detail in Embodiment 1, it will not be elaborated herein in this embodiment.

[0071] Those of ordinary skill in the art can understand that all or part of the steps to implement the above embodiments can be completed by hardware, or can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disc, etc. Based on such an understanding, the technical solution of the present invention, in essence, or the part that makes a contribution to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. And the aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0072] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of various embodiments of the present invention.

Claims

1. A method for speckle structured light binocular stereo matching based on multi-cost fusion, characterized in that Including: Obtain the left camera image and the right camera image captured by the binocular speckle camera; Perform epipolar rectification on the obtained left camera image and right camera image respectively; Perform boundary segmentation on the rectified left camera image to obtain the corresponding boundary image; Utilize the rectified left camera image and right camera image to calculate the Census cost and the BT cost, and perform weighted fusion to obtain the matching cost volume; Use the boundary image as prior information to perform cost aggregation on the obtained cost volume; Calculate the disparity image corresponding to the left camera image through the aggregated cost volume; Use the small connected component filtering method based on markers to remove the abnormal disparity points in the disparity image.

2. The method for speckle structured light binocular stereo matching based on multi-cost fusion according to claim 1, wherein The performing boundary segmentation on the rectified left camera image to obtain the corresponding boundary image specifically includes: Perform grid processing on the rectified left camera image to obtain a grid image, and perform median filtering and mean filtering on the grid image successively; Calculate the weighted variance in the grid image; Utilize the calculated weighted variance to perform maximum value search on the grid image in the row direction and column direction respectively to obtain the row direction extreme value map and the column direction extreme value map; Merge the row direction extreme value map and the column direction extreme value map through the direction characteristics of the boundary to obtain an extreme value image containing boundary information; Remove the outliers in the extreme value image, and perform upsampling on the extreme value image after removing the outliers to obtain the finally segmented boundary image.

3. A method for speckle structured light binocular stereo matching based on multi-cost fusion according to claim 2, characterized in that The method for performing grid processing on the rectified left camera image specifically is: Set the grid size as m×m. First, obtain the coordinates of the maximum pixel point in each grid, calculate the pixel mean value within the 3×3 neighborhood centered on this coordinate as the output value of this grid, and obtain an image with a size of M / m×N / m, where M and N respectively represent the width and height of the input image; The calculation formula for the weighted variance is as follows: ； In the above calculation formula, V(u, v) represents the original variance of a certain point to be calculated, k represents the neighborhood size, f(x, y) and represent the pixel value of a point in the neighborhood and the average pixel value in the neighborhood respectively, represents the variance after weighting, and grayMin and grayMax represent the minimum pixel value and the maximum pixel value in the neighborhood respectively.

4. A method for speckle structured light binocular stereo matching based on multi-cost fusion according to claim 3, characterized in that The calculation method for the Census cost is as follows: First, perform Census transformation on the rectified left and right camera images respectively: ； Among them, C S (p) represents the Census transform value corresponding to pixel point p, represents the bit-by-bit concatenation operation of bits, q represents a pixel point within the neighborhood of point p, and I(p) and I(q) represent the pixel values of the corresponding pixel points, The operation is defined as follows: ； Calculate the Hamming distance between the left and right camera images through the following formula, that is, the Census cost: ； Where CT(p, d) represents the Census cost of pixel point p in the left camera image at disparity d, and C sl (p) represents the Census transform value corresponding to pixel point p in the left camera image, and C sr (p, d) represents the Census transform value corresponding to pixel point p - d in the right camera image; The BT cost uses the BT based on the xsobel image xsobel and the BT based on the grayscale image gray , and is obtained through weighted fusion, expressed as: ； The calculation process for the BT cost is as follows: ； where \(BT(p, d)\) represents the BT cost of pixel point \(p\) in the left camera image at disparity \(d\). and respectively represent the BT cost constructed by sub-pixel interpolation using the left camera image and the BT cost constructed by sub-pixel interpolation using the right camera image. The construction of the BT cost using sub-pixel interpolation is shown as follows: ； Among them, I L (p) represents the pixel value corresponding to pixel point p in the left camera image, I L - represents the sub-pixel value from pixel point p - 1 to pixel point p in the left camera image, I L + represents the sub-pixel value from pixel point p to pixel point p + 1 in the left camera image, I R (p, d) represents the pixel value corresponding to pixel point p + d in the right camera image; ； Among them, I R (p) represents the pixel value corresponding to pixel point p in the right camera image, I R - represents the sub-pixel value from pixel point p - 1 to pixel point p in the right camera image, I R + represents the sub-pixel value from pixel point p to pixel point p + 1 in the right camera image, I L (p, d) represents the pixel value corresponding to pixel point p + d in the left camera image; The calculation formula for the weighted fusion of the Census cost and the BT cost is as follows: ； Where C(p,d) is the fused cost, BT(p,d) and CT(p,d) respectively represent the calculated BT cost and Census cost, w is the window size selected when calculating the Census cost, and c1 and c2 are proportionality coefficients.

5. A method for speckle structured light binocular stereo matching based on multi-cost fusion according to claim 4, characterized in that, The using the boundary image as prior information to perform cost aggregation on the obtained cost volume specifically includes: Perform cost aggregation on the obtained cost volume through the dynamic programming method, and the aggregation formula is as follows: ； where L r (p, d) represents the aggregation cost of pixel p at disparity d, r represents the aggregation path, and L r (p - r, i) represents the aggregation cost of the previous path point of pixel p at disparity i, C(p, d) represents the initial cost, and P1 and P2 are penalty coefficients; Dynamically adjust the penalty coefficient P2 using the boundary image, and the calculation formula is as follows: ； Among them, P 2- Init represents the initial cost value, and gray represents the gray value of the corresponding point obtained from the boundary image.

6. A method for speckle structured light binocular stereo matching based on multi-cost fusion according to claim 5, characterized in that, The calculating the disparity image corresponding to the left camera image through the aggregated cost volume specifically includes: Select the minimum cost value from the aggregated cost volume to obtain the disparity corresponding to the minimum cost value and the disparities adjacent to the current disparity on the left and right, and then calculate the disparity through the quadratic curve fitting method as the binocular matching disparity of the current pixel point. As shown in the following formula: ； where d sub represents the fitted parallax, d min is the parallax corresponding to the minimum cost, c0 is the minimum cost value, and c1 and c2 respectively represent the cost values corresponding to the left and right adjacent parallaxes.

7. A method for speckle structured light binocular stereo matching based on multi-cost fusion according to claim 5, characterized in that The using the small connected component filtering method based on markers to remove the abnormal disparity points in the disparity image specifically includes: Find the root node of each pixel in the disparity image through a recursive method, classify the pixels belonging to the same root node into the same sub-region, and use the index of the root node as the serial number of the sub-region where the root node is located; Traverse the pixels at the junctions of all sub-regions, and merge all related sub-regions into one region; Count the area of each region, filter out the same positions in the disparity map corresponding to the regions with an area smaller than the predetermined area threshold, and obtain the final disparity map.

8. A method for speckle structured light binocular stereo matching based on multi-cost fusion according to claim 7, characterized in that The method of finding the root node of each pixel in the disparity image through a recursive method is specifically as follows: Within the 3×3 neighborhood of the pixel, compare the disparity value of the pixel with the disparity values of the four adjacent points, namely the upper left, upper, upper right, and right adjacent points, in turn; If the absolute value of the difference between the disparity value of the adjacent point and the disparity value of the pixel is less than the preset difference threshold, then the adjacent point is the parent node; If none of the four adjacent points is the parent node, then the pixel is the root node.

9. A method for speckle structured light binocular stereo matching based on multi-cost fusion according to claim 7, characterized in that, The method of traversing the pixels at the junctions of all sub-regions and merging all related sub-regions into one region is specifically as follows: Let the pixel point sets at the junction of the two sub-regions be A and B respectively. A is expressed as {a1, a2, a3,......, a n}, and B is expressed as {b1, b2, b3,......, b n}. The pixel point a i and the pixel point b i are adjacent pixel points of the two sub-regions, and i is a natural number less than or equal to n; Determine a i -b i Whether the absolute value of is less than a preset difference threshold; if so, then pixel point a i Is associated with pixel point b i If not, then record pixel point a i Is not associated with pixel point b i Not associated; If there are k consecutive pixels in A that are related to B, then A and B are two related sub-regions; k is the preset threshold of the number of related points.

10. A method for speckle structured light binocular stereo matching based on multi-cost fusion according to claim 7, characterized in that Before using the marker-based small connected component filtering method to remove the abnormal disparity points in the disparity image, it also includes: Remove the noise in the disparity image through uniqueness detection, left-right consistency detection, and median filtering.

11. A system for speckle structured light binocular stereo matching based on multi-cost fusion, characterized in that, It includes: A binocular speckle camera for taking a left camera image and a right camera image containing a speckle pattern; An image correction module electrically connected to the binocular speckle camera for performing epipolar correction on the taken left camera image and right camera image respectively; A boundary segmentation module electrically connected to the image correction module for performing boundary segmentation on the corrected left camera image to obtain a corresponding boundary image; A cost calculation module electrically connected to the image correction module for calculating the Census cost and the BT cost using the corrected left camera image and right camera image, and weighted fusing to obtain a matching cost volume; A cost aggregation module electrically connected to the boundary segmentation module and the cost calculation module respectively for using the boundary image as prior information to perform cost aggregation on the obtained cost volume; A disparity image generation module electrically connected to the cost aggregation module and the image correction module respectively for calculating the disparity image corresponding to the left camera image through the aggregated cost volume; An image optimization module electrically connected to the disparity image generation module for using the marker-based small connected component filtering method to remove the abnormal disparity points in the disparity image.

12. A system for speckle structured light binocular stereo matching based on multi-cost fusion according to claim 11, characterized in that, The boundary segmentation module is specifically used for: Perform grid processing on the corrected left camera image to obtain a grid image, and perform median filtering and mean filtering on the grid image successively; Calculate the weighted variance in the grid image; Use the calculated weighted variance to perform maximum value search on the grid image in the row direction and column direction respectively to obtain a row direction extreme value map and a column direction extreme value map; Merge the row direction extreme value map and the column direction extreme value map through the direction characteristics of the boundary to obtain an extreme value image containing boundary information; Remove the outliers in the extreme value image, and upsample the extreme value image after removing the outliers to obtain the finally segmented boundary image.

13. A system for speckle structured light binocular stereo matching based on multi-cost fusion according to claim 12, characterized in that, The method for grid processing the corrected left camera image is as follows: Let the grid size be m×m. First, obtain the coordinates of the maximum pixel point in each grid, and calculate the pixel mean value in the 3×3 neighborhood centered on this coordinate as the output value of this grid, obtaining an image of size M / m×N / m, where M and N respectively represent the width and height of the input image. The calculation formula for the weighted variance is as follows: ； In the above calculation formula, V(u, v) represents the original variance of a certain point to be calculated, k represents the neighborhood size, f(x, y) and respectively represent the pixel value of a point in the neighborhood and the average pixel value in the neighborhood, represents the variance after weighting, and grayMin and grayMax respectively represent the minimum pixel value and the maximum pixel value in the neighborhood.

14. A system for speckle structured light binocular stereo matching based on multi-cost fusion according to claim 13, characterized in that, The calculation method for the Census cost is as follows: First, perform Census transformation on the corrected left and right camera images respectively: ； Among them, C S (p) represents the Census transform value corresponding to pixel point p, represents the bit-by-bit concatenation operation of bits. q represents a pixel point within the neighborhood of point p, and I(p) and I(q) represent the pixel values of the corresponding pixel points, The operation is defined as follows: ； Calculate the Hamming distance between the left and right camera images through the following formula, that is, the Census cost: ； Where CT(p, d) represents the Census cost of pixel point p in the left camera image at disparity d, and C sl (p) represents the Census transform value corresponding to pixel point p in the left camera image, and C sr (p, d) represents the Census transform value corresponding to pixel point p - d in the right camera image; The BT cost uses BT based on the xsobel image xsobel and BT based on the grayscale image gray , and is obtained through weighted fusion, expressed as: ； The calculation process of the BT cost is as follows: ； where \(BT(p, d)\) represents the BT cost of pixel point \(p\) in the left camera image at disparity \(d\), and respectively represent the BT cost calculated by sub-pixel interpolation using the left camera image and the BT cost calculated by sub-pixel interpolation using the right camera image. The construction of the BT cost using sub-pixel interpolation is shown as follows: ； Among them, I L (p) represents the pixel value corresponding to pixel point p in the left camera image, I L - represents the sub-pixel value from pixel point p - 1 to pixel point p in the left camera image, I L + represents the sub-pixel value from pixel point p to pixel point p + 1 in the left camera image, I R (p, d) represents the pixel value corresponding to pixel point p + d in the right camera image; ； Wherein, I R (p) represents the pixel value corresponding to pixel point p in the right camera image, I R - represents the sub-pixel value from pixel point p - 1 to pixel point p in the right camera image, I R + represents the sub-pixel value from pixel point p to pixel point p + 1 in the right camera image, I L (p, d) represents the pixel value corresponding to pixel point p + d in the left camera image; The calculation formula for weighted fusion of the Census cost and the BT cost is as follows: ； Among them, C(p,d) is the cost after fusion, BT(p,d) and CT(p,d) respectively represent the calculated BT cost and Census cost, w is the window size selected when calculating the Census cost, and c1 and c2 are proportionality coefficients.

15. A system for speckle structured light binocular stereo matching based on multi-cost fusion according to claim 14, characterized in that, The cost aggregation module is specifically used for: Perform cost aggregation on the obtained cost volume through the dynamic programming method, and the aggregation formula is as follows: ； Among them, L r (p, d) represents the aggregation cost of pixel point p at a parallax of d, r represents the aggregation path, and L r (p - r, i) represents the aggregation cost of the previous path point of pixel point p at a parallax of i, C(p, d) represents the initial cost, and P1 and P2 are penalty coefficients; Dynamically adjust the penalty coefficient P2 using the boundary image, and the calculation formula is as follows: ； Among them, P 2- Init represents the initial cost value, and gray represents the gray value of the corresponding point obtained from the boundary image.

16. A system for speckle structured light binocular stereo matching based on multi-cost fusion according to claim 15, characterized in that, The disparity image generation module is specifically used for: Select the minimum cost value from the aggregated cost volume to obtain the disparity corresponding to the minimum cost value and the disparities adjacent to the current disparity on the left and right, and then calculate the disparity as the binocular matching disparity of the current pixel point through the quadratic curve fitting method. As shown in the following formula: ； where d sub represents the fitted parallax, d min is the parallax corresponding to the minimum cost, c0 is the minimum cost value, and c1 and c2 respectively represent the cost values corresponding to the left and right adjacent parallaxes.

17. A system for speckle structured light binocular stereo matching based on multi-cost fusion according to claim 15, characterized in that, The disparity abnormal point filtering module is specifically used for: Find the root node of each pixel point in the disparity image through a recursive method, classify the pixel points belonging to the same root node into the same sub-region, and use the index of the root node as the serial number of the sub-region where the root node is located; Traverse the pixel points at the junctions of all sub-regions, and merge all sub-regions with associations into one region; Count the area of each region, and filter out the same positions in the disparity map corresponding to the regions with an area smaller than the predetermined area threshold to obtain the final disparity map.

18. A system for speckle structured light binocular stereo matching based on multi-cost fusion according to claim 17, characterized in that, The method for finding the root node of each pixel point in the disparity image through a recursive method is as follows: In the 3×3 neighborhood of the pixel point, compare the disparity value of the pixel point with the disparity values of the four adjacent points, namely the upper left, upper, upper right, and right adjacent points, in turn; If the absolute value of the difference between the disparity value of the adjacent point and the disparity value of the pixel point is less than the preset difference threshold, then the adjacent point is the parent node; If none of the four adjacent points is the parent node, then the pixel point is the root node.

19. A system for speckle structured light binocular stereo matching based on multi-cost fusion according to claim 17, characterized in that, The method for traversing the pixel points at the junctions of all sub-regions and merging all sub-regions with associations into one region is as follows: Let the pixel point sets at the junction of the two sub-regions be A and B respectively. A is expressed as {a1, a2, a3,......, a n}, and B is expressed as {b1, b2, b3,......, b n}. The pixel point a i and the pixel point b i are adjacent pixel points of the two sub-regions, and i is a natural number less than or equal to n; Determine a i -b i whether the absolute value is less than a preset difference threshold; if so, then pixel point a i is associated with pixel point b i ; if not, then record that pixel point a i is not associated with pixel point b i ; If there are k consecutive pixel points in A that are associated with B, then A and B are two sub-regions with associations; k is the preset association point number threshold.

20. A system for speckle structured light binocular stereoscopic matching based on multi-cost fusion according to claim 17, characterized in that, The image optimization module is also used to remove the noise in the disparity image through uniqueness detection, left-right consistency detection, and median filtering.

21. A computer-readable storage medium storing at least one instruction, characterized in that, The instructions are loaded and executed by a processor to implement a method for speckle structured light binocular stereo matching based on multi-cost fusion as described in any one of claims 1-10.

22. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by a processor, a method for speckle structured light binocular stereo matching based on multi-cost fusion as described in any one of claims 1-10 is implemented.

Images