A multi-scale stereo matching method and system based on superpixel segmentation

Abstract

The present invention relates to the technical field of multi-scale stereo matching, and discloses a multi-scale stereo matching method and system based on superpixel segmentation, the method comprising the following steps: obtaining the illumination intensity of left and right images and a shooting environment; performing image segmentation on the left image; detecting small target objects in the left image to determine the number of Gaussian pyramid layers and constructing Gaussian pyramids for the left and right images; calculating improved Census cost, SAD cost and gradient cost and the environmental weight coefficient of each cost, and calculating the total fusion cost of the Gaussian pyramid; performing cost aggregation; calculating the disparity values ​​of all pixel points in the left image and performing disparity optimization. The present invention can not only effectively solve the problem that the number of superpixels in the existing superpixel segmentation algorithm needs to be manually set and the result image has a serious over-segmentation phenomenon, but also can effectively solve the problem that the matching accuracy is low due to a single matching cost and the matching cost is insensitive to illumination changes.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-scale stereo matching, and in particular to a multi-scale stereo matching method and system based on superpixel segmentation. Background Art

[0002] Binocular stereo vision is a technology that obtains three-dimensional spatial information by simulating the binocular vision principle of the human eye. Its principle is to use two cameras to capture images of the same scene from different angles, and infer the depth information of the object by calculating the disparity value between corresponding points in the two images, thereby achieving accurate reconstruction of the spatial structure. Binocular stereo vision technology can provide accurate three-dimensional spatial information in industrial production and is an important part of the field of machine vision.

[0003] Stereo matching is a key link and technical difficulty of binocular stereo vision. In order to improve the accuracy of stereo matching algorithms, domestic and foreign scholars have proposed a large number of stereo matching algorithms, among which the most widely used algorithms include Census algorithm, Sum of Absolute Errors (SAD) algorithm, Normalized Cross Correlation Algorithm (NCC), etc. However, they are easily affected by factors such as noise, external lighting, and weak texture areas, and the adaptability of a single algorithm in complex scenes is low, which may seriously affect the accuracy of stereo matching in practical applications.

[0004] Although the prior art provides a matching strategy combining multiple stereo matching algorithms to improve robustness in complex scenes, fixed weights are used for different costs of color differences during cost calculation, making it difficult to maximize the advantages of each cost function. In addition, the prior art fails to take into account the different scale characteristics of the visual signal received by the human eye binocular vision system, and fails to compensate for the impact of different lighting conditions in the shooting scene.

[0005] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0006] In view of the problems in the related art, the present invention proposes a multi-scale stereo matching method and system based on superpixel segmentation to overcome the above-mentioned technical problems existing in the existing related art.

[0007] To this end, the specific technical solution adopted by the present invention is as follows:

[0008] According to one aspect of the present invention, a multi-scale stereo matching method based on superpixel segmentation is provided, comprising the following steps:

[0009] S1. Use an illuminance meter to detect the light intensity in the environment, and use a binocular camera to obtain left and right images, perform binocular positioning, distortion correction and epipolar correction processing;

[0010] S2, based on the superpixel segmentation method of the SLIC algorithm, the processed left image is segmented to obtain a superpixel label map of the segmented image;

[0011] S3, using the YOLOv5 network to detect small target objects in the left image, combining the detection results and segmentation results to determine the number of Gaussian pyramid layers, and constructing the Gaussian pyramids of the left and right images;

[0012] S4. Calculate the improved Census cost, SAD cost and gradient cost according to each layer of the Gaussian pyramid, determine the environmental weight coefficient of each cost in combination with the illumination intensity and the segmentation result, take the normalized weighted average of the improved Census cost, SAD cost and gradient cost as the cost of each layer, and calculate the total fusion cost of the Gaussian pyramid by upsampling and averaging;

[0013] S5, taking the disparity value that minimizes the energy function at each pixel as the result disparity value of cost aggregation, performing cost aggregation based on the eight-path dynamic programming cost aggregation algorithm and combining the superpixel label map to obtain the final aggregation result;

[0014] S6. Calculate the disparity values ​​of all pixels in the left image based on the aggregation cost, and perform disparity optimization through left-right consistency detection, disparity filling, and median filtering in sequence.

[0015] Furthermore, the superpixel segmentation method based on the SLIC algorithm performs image segmentation on the processed left image to obtain a superpixel label map of the segmented image, including the following steps:

[0016] S21, respectively calculating the one-dimensional information entropy and variance of the left image, and calculating the number of adaptive superpixels according to the one-dimensional information entropy and variance; based on the number of superpixels, using the SLIC superpixel segmentation algorithm to segment the left image to obtain a segmentation label map of the image;

[0017] S22, searching for all small connected areas and outer contour points of each superpixel within a preset range around the cluster center, counting the number of pixels with different superpixel labels in the four neighborhoods of the outer contour point, and assigning all pixels in the small connected area to the superpixel label with the highest frequency;

[0018] S23, extract the pixel points of the outer contour of the current superpixel and find all the superpixel labels adjacent to the outer contour, calculate the average color value and one-dimensional information entropy of the adjacent superpixels under the R, G, and B channels respectively, and calculate the difference in one-dimensional information entropy and color difference between the adjacent superpixels and the current superpixel;

[0019] S24. When the difference in one-dimensional information entropy between the current superpixel and the adjacent superpixel is less than the preset threshold of the difference in one-dimensional information entropy, and the color difference between the current superpixel and the adjacent superpixel is less than the preset color difference threshold, the adjacent superpixel is determined to be an over-segmented superpixel, and the superpixel labels of all pixels in the over-segmented superpixel are replaced with the current superpixel labels, otherwise they remain unchanged.

[0020] Furthermore, the calculation formula of one-dimensional information entropy is: ; ;

[0021] The calculation formula for the number of adaptive superpixels is: ;

[0022] In the formula, E is the one-dimensional information entropy of the left image, N is the number of pixels in the left image, The gray value is i The number of pixels, V is the left image variance, The left image j The gray value of a pixel, is the average gray value of the pixels in the left image, K is the number of adaptive superpixels, w is the width of the left image, h is the height of the left image, is the adjustment coefficient.

[0023] Furthermore, the calculation formula for the average color value is: ;

[0024] The formula for calculating color difference is: ; ;

[0025] In the formula, For the color channel m The average color value under Superpixel S The number of pixels in For the color channel m Lower superpixel inliers i The pixel value of Superpixel and The color difference between For the color channel m Lower Super Pixel The average color value of For the color channel m Lower Super Pixel The average color value.

[0026] Furthermore, the method of using the YOLOv5 network to detect the small target object in the left image, combining the detection result and the segmentation result to determine the number of Gaussian pyramid layers, and constructing the Gaussian pyramids of the left and right images includes the following steps:

[0027] S31, use the pre-built YOLOv5 network to detect the target object in the left image, and filter the small target objects in the detection results. , the current detection result is taken as a small target object, and the number of all small target objects, the width and height of each small target frame are counted, where is the width of the target box, is the height of the target frame, w is the width of the left image, h is the height of the left image, is the detection threshold;

[0028] S32, calculating the number of Gaussian pyramid layers according to the detection results and the segmentation results, and constructing Gaussian pyramids of the left and right images in combination with the left and right images obtained by the binocular camera.

[0029] Furthermore, the calculation formula for the number of Gaussian pyramid layers is: ;

[0030] In the formula, O is the number of adaptive Gaussian pyramid layers, is the maximum number of layers of the Gaussian pyramid, is the number of superpixels in the image after being processed by the superpixel segmentation algorithm, n is the number of small target objects detected by the YOLOv5 network, For the i The width of the small target box, For the i The height of the small target box.

[0031] Furthermore, the calculation formula of the improved Census cost is: ; ; ;

[0032] The calculation formula of SAD cost is: ;

[0033] The calculation formula of the gradient cost is: ;

[0034] The calculation formula of the environmental weight coefficient is: ; ; ;

[0035] The fusion cost calculation formula is: ; ;

[0036] In the formula, is the first i Pixels in the left image of the layer p In the parallax value d The cost value is is the first i Pixels in the left image of the layer p Census transformation code, is the first i Layer right image corresponding to the pixel point Census transformation code, S ( p ) is a pixel p Census transformation code, q Remove pixels from the window p Pixels outside of N p Pixel p Field window pixel set, To encode the concatenation operator, is the encoding function, is the pixel in the window p The gray value of Remove pixels from the window p Pixels outside q The gray value of is the first i Pixels in the left image of the layer p When the parallax value is d The SAD cost under is the first i Pixels in the left image of the layer p The gray value of is the first i The pixel in the right image of the layer p The corresponding disparity value is d The grayscale value of the pixel point, L is the left image, R is the right image, is the first i Pixels in the left image of the layer p When the parallax value is d The gradient cost under for x The gradient value in the direction, for y The gradient value in the direction, To improve the environmental cost weights of Census costs, wis the width of the left image, h is the height of the left image, is the environmental cost weight of the gradient cost, is the environmental cost weight of the SAD cost, To improve the weight factor of Census cost, is the weight factor of the gradient cost, is the light intensity, K is the number of adaptive superpixels, For the Gaussian pyramid i Pixels in the left image of the layer p In the parallax value d The fusion cost under is the normalized cost value, To improve the normalization parameters of the Census cost, is the normalization parameter of SAD cost, is the normalization parameter of the gradient cost, c To match the cost.

[0037] Furthermore, the expression of the energy function is: ; ; ;

[0038] The calculation formula of aggregation cost is: ; ;

[0039] In the formula, Pixel p In the parallax value d The energy value below Pixel p In the parallax value The cost value is P 1. P 2 are two different penalty items, For penalty items P The initial value is 2, Pixel p The gray value of Remove pixels from the window p Pixels outside q The gray value of is the penalty control coefficient, 1. 2 are two different penalty coefficients, and 1< 2, T is the indicator function, Pixel p The disparity value of Remove pixels from the window p Pixels outside q The disparity value of Pixel p In the parallax value d The aggregation cost under r is the number of aggregation paths, Pixel p Along the Path r The one-way aggregation cost is Pixel p In the parallax value d The cost value is Remove pixels from the path p The remaining pixels are in the disparity value d The cost value is Remove pixels from the path p The remaining pixels are in the disparity value i The next cost value.

[0040] Further, the parallax optimization by parallax filling includes: determining the pixels in the occluded area and the mismatched pixels: assuming that the pixel point in the right image q is the pixel in the left image p By parallax value d If there is another pixel point in the left image, the same point is found. Also with pixels q are points of the same name, and the pixel The parallax value is greater than the parallax value d Large, then the pixel p Invalid pixels that belong to the occluded area but not to the occluded area are classified as mismatched points;

[0041] Perform invalid disparity filling: For pixels in the occluded area, select the second smallest disparity value among the surrounding valid pixels as the disparity value of the pixel; for invalid pixels, select the median of the disparity values ​​among the surrounding pixels as the disparity value of the pixel.

[0042] According to another aspect of the present invention, a multi-scale stereo matching system based on superpixel segmentation is provided, the multi-scale stereo matching system based on superpixel segmentation comprises an image acquisition and detection module, an image segmentation module, a Gaussian pyramid construction module, a cost calculation module, a cost aggregation module and a disparity optimization module;

[0043] The image acquisition and detection module is used to detect the light intensity in the environment using an illuminometer, and to acquire left and right images through a binocular camera to perform binocular positioning, distortion correction and epipolar correction processing;

[0044] The image segmentation module is used to perform image segmentation on the processed left image based on the superpixel segmentation method of the SLIC algorithm to obtain a superpixel label map of the segmented image;

[0045] The Gaussian pyramid construction module is used to detect small target objects in the left image using the YOLOv5 network, determine the number of Gaussian pyramid layers based on the detection results and the segmentation results, and construct Gaussian pyramids for the left and right images;

[0046] The cost calculation module is used to calculate the improved Census cost, SAD cost and gradient cost according to each layer image of the Gaussian pyramid, determine the environmental weight coefficient of each cost in combination with the illumination intensity and the segmentation result, take the normalized weighted average of the improved Census cost, SAD cost and gradient cost as the cost of each layer, and calculate the total fusion cost of the Gaussian pyramid by upsampling and averaging;

[0047] The cost aggregation module is used to take the disparity value that minimizes the energy function at each pixel as the result disparity value of cost aggregation, perform cost aggregation based on the eight-path dynamic programming cost aggregation algorithm and the superpixel label map to obtain the final aggregation result;

[0048] The disparity optimization module is used to calculate the disparity values ​​of all pixels in the left image based on the aggregation cost, and perform disparity optimization through left-right consistency detection, disparity filling, and median filtering in sequence.

[0049] The beneficial effects of the present invention are:

[0050] 1) The present invention firstly determines the number of superpixels adaptively according to the image information entropy and variance, and merges the over-segmented superpixels, so that the proposed superpixel segmentation algorithm can cope with stereo matching in complex scenes. Secondly, superpixel segmentation and YOLOv5 are used to process and detect the captured images and determine the number of adaptive Gaussian pyramid layers. Multi-scale calculation of matching cost is realized by constructing Gaussian pyramid and cost fusion, and a penalty term is applied to the energy function according to the superpixel segmentation result in the subsequent cost aggregation stage, so that the algorithm proposed in the present invention can ensure robustness in diversified scenes and high matching accuracy in complex environments.

[0051] 2) The present invention can not only calculate the number of super-pixel segmentation blocks based on the image size, image variance, one-dimensional information entropy and other factors taken by the left and right cameras, but also calculate the number of super-pixel segmentation blocks based on the image size, image variance, one-dimensional information entropy and other factors. K ,accomplish KThe algorithm can not only adapt the value of the image to the desired value, but also judge the over-segmented superpixels by setting the threshold according to the information entropy difference and the color difference, and eliminate them by merging them into adjacent superpixels. This can effectively reduce the over-segmentation of superpixels in the image and reduce the impact of over-segmentation on subsequent cost calculations. It can also effectively solve the problems in the existing superpixel segmentation algorithm that the number of superpixels needs to be set manually and the result image has serious over-segmentation phenomenon.

[0052] 3) The present invention can not only determine the pixel points in the Census window that are in the same superpixel as the central pixel according to the segmentation result, and replace the Census reference pixel value with the grayscale mean of the above pixel points, thereby effectively improving the noise resistance of the Census cost, but also introduce YOLOv5 to detect small targets in the captured image, calculate the number of adaptive Gaussian pyramid layers according to the results of small target detection and superpixel segmentation, and calculate the improved Census cost, SAD cost and gradient cost in the cost calculation process of each layer of the Gaussian pyramid respectively, determine the environmental weight coefficient of the cost in combination with the light intensity and the segmentation result, and take the weighted mean of the normalized three costs as the cost of each layer, thereby effectively solving the problems of low matching accuracy caused by a single matching cost and the insensitivity of the matching cost to illumination changes. In addition, different penalty items can be applied to different superpixel labels of pixel points in the aggregation path according to the superpixel segmentation result of the captured image, thereby significantly improving the accuracy and robustness of the matching algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 is a flowchart of a multi-scale stereo matching method based on superpixel segmentation according to an embodiment of the present invention;

[0055] Figure 2 is a schematic diagram of the principle of a multi-scale stereo matching method based on superpixel segmentation according to an embodiment of the present invention;

[0056] Figure 3 is a schematic diagram of a Gaussian pyramid in a multi-scale stereo matching method based on superpixel segmentation according to an embodiment of the present invention;

[0057] Figure 4 is a schematic diagram of eight-path cost aggregation in a multi-scale stereo matching method based on superpixel segmentation according to an embodiment of the present invention;

[0058] Figure 5 4 is a structural block diagram of a multi-scale stereo matching system based on superpixel segmentation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] To further illustrate each embodiment, the present invention provides drawings, which are part of the disclosure of the present invention and are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in the field should be able to understand other possible implementations and advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0060] According to an embodiment of the present invention, a multi-scale stereo matching method and system based on superpixel segmentation are provided.

[0061] The present invention is further described with reference to the accompanying drawings and specific embodiments. Figure 1-Figure 4 As shown, according to an embodiment of the present invention, a multi-scale stereo matching method based on superpixel segmentation is provided, comprising the following steps:

[0062] S1. Use an illuminance meter to detect the light intensity in the environment, and use a binocular camera to obtain left and right images for binocular positioning, distortion correction, and epipolar correction.

[0063] Specifically, the binocular positioning, distortion correction and epipolar correction processing in this embodiment are all prior arts, which are briefly described below:

[0064] Perform dual-target calibration processing: take multiple images of the checkerboard calibration plate at different positions and angles, then use the image processing algorithm to detect the corner features in each image, obtain the pixel coordinates of the corner points, and finally use the homography matrix to calculate the camera's intrinsic parameters, extrinsic parameters, and distortion parameters based on the known size and corner coordinates of the calibration plate.

[0065] Perform distortion correction processing: Construct a polynomial model for distortion correction based on the distortion parameters obtained during the binocular positioning process, then calculate the corrected coordinates of each pixel in the image under the distortion correction model, that is, the ideal position after eliminating the distortion effect, and finally move the pixel to the corrected position through interpolation and other methods to obtain the corrected image.

[0066] Perform epipolar correction: calculate the rotation matrix and translation vector of the binocular camera, adjust the imaging planes of the two cameras to be parallel and vertically aligned; then correct the image through reprojection transformation so that the projection points of the same scene point in the left and right images are on the same horizontal line, that is, the epipolar line.

[0067] S2. Based on the superpixel segmentation method of the SLIC algorithm, the processed left image is segmented to obtain a superpixel label map of the segmented image.

[0068] The existing SLIC superpixel segmentation algorithm has the disadvantages that the number of superpixels needs to be manually set and the over-segmentation phenomenon is serious, and it cannot be adapted to the stereo matching algorithm mentioned in this embodiment. In view of the above problems, this embodiment provides a superpixel image segmentation algorithm based on the SLIC algorithm according to the technical characteristics of the stereo matching algorithm. Specifically, the superpixel segmentation method based on the SLIC algorithm performs image segmentation on the processed left image, and obtains the superpixel label map of the segmented image, including the following steps:

[0069] S21. Calculate the one-dimensional information entropy and variance of the left image respectively, and calculate the number of adaptive superpixels according to the one-dimensional information entropy and variance; based on the number of superpixels, use the SLIC superpixel segmentation algorithm to segment the left image to obtain a segmentation label map of the image.

[0070] Specifically, the calculation formula of one-dimensional information entropy is: ;

[0071] The formula for calculating variance is: ;

[0072] The calculation formula for the number of adaptive superpixels is: ;

[0073] In the formula, E is the one-dimensional information entropy of the left image, N is the number of pixels in the left image, The gray value is i The number of pixels; V is the left image variance, The left image j The gray value of a pixel, is the average gray value of pixels in the image; K is the number of adaptive superpixels, w is the width of the left image, h is the height of the left image, is the adjustment coefficient, generally [0.05, 0.2].

[0074] S22. Search for all small connected areas and outer contour points of each superpixel within a preset range around the cluster center, count the number of pixels with different superpixel labels in the four areas of the outer contour points, and assign all pixels in the small connected areas to the superpixel label with the highest frequency.

[0075] Specifically, small connected areas in the results are removed: for each superpixel, first find all small connected areas and their outer contour points within a 2s×2s range around the cluster center, then count the number of pixels with different superpixel labels in the four neighborhoods of the outer contour point, and finally assign all pixels in the small connected area to the superpixel label with the highest frequency.

[0076] S23, extract the pixel points of the outer contour of the current superpixel and find all the superpixel labels adjacent to the outer contour, calculate the average color value and one-dimensional information entropy of the adjacent superpixels in the R, G, and B channels respectively, and calculate the difference in one-dimensional information entropy and color difference between the adjacent superpixels and the current superpixel.

[0077] Specifically, determine and eliminate the under-segmentation phenomenon: extract the outer contour pixel points of the current superpixel and find all the superpixel labels adjacent to the outer contour, and then calculate the average color value and one-dimensional information entropy of the adjacent superpixels in the R, G, and B channels respectively. The average color value calculation method in the superpixel S is: ;

[0078] In the formula, For the color channel m ( m Take the average color value under R, G, B), Superpixel S The number of pixels in For the color channel m Lower superpixel inliers i The pixel value of .

[0079] Then calculate the information entropy difference and color difference between the current superpixel. For any two superpixels and , the color difference between the two is calculated as: ;

[0080] In the formula, Superpixel and The color difference between For the color channel m Lower Super Pixel The average color value of For the color channel m Lower Super Pixel The average color value.

[0081] S24, the difference in one-dimensional information entropy between the current superpixel and the adjacent superpixel is less than a preset threshold value of the difference in one-dimensional information entropy , and the color difference between the current superpixel and the adjacent superpixel is less than the preset color difference threshold When , the adjacent superpixel is determined to be an over-segmented superpixel, and the superpixel labels of all pixels in the over-segmented superpixel are replaced with the current superpixel labels, otherwise they remain unchanged.

[0082] Specifically, it is determined whether the difference in one-dimensional information entropy and color difference between the current superpixel and the adjacent superpixel is less than a preset threshold value of the difference in one-dimensional information entropy. and color difference threshold If yes, the adjacent superpixel is determined as an over-segmented superpixel, and the superpixel labels of all pixels in the over-segmented superpixel are replaced with the current superpixel labels. If no, they remain unchanged.

[0083] S3. Use the YOLOv5 network to detect small target objects in the left image, determine the number of Gaussian pyramid layers based on the detection results and segmentation results, and construct the Gaussian pyramids of the left and right images.

[0084] YOLOv5 is a fast and accurate target detection model. Through its excellent network architecture and advanced feature extraction technology, it can quickly identify and locate the target object in an image. In the training stage, the labeled data set is input into the YOLOv5 network, the network model is trained with the criterion of minimizing the loss function, and the weight file containing the characteristic parameters of the target object is output; in the detection stage, the input image is feature extracted, followed by feature fusion and multi-scale processing, and the target box and category probability are generated on the feature map according to the trained weights. Finally, non-maximum suppression is used to remove overlapping target boxes. Compared with other target detection algorithms, YOLOv5 combines the advantages of high speed and high precision, and can handle multi-scale target detection tasks in complex scenes in a very short time while maintaining a high accuracy rate. In addition, YOLOv5 provides an easy-to-use API interface for easy training and deployment, and supports models of different sizes to meet different detection speed and accuracy requirements.

[0085] Specifically, the method of using the YOLOv5 network to detect the small target object in the left image, combining the detection result and the segmentation result to determine the number of Gaussian pyramid layers, and constructing the Gaussian pyramids of the left and right images includes the following steps:

[0086] S31. Use the COCO dataset to train the YOLOv5 network model weights, and use the trained YOLOv5 network to detect the target objects in the left image, and filter the small target objects in the detection results. When the target box of the detection result meets , the current detection result is taken as a small target object, and then the number of all small target objects, the width and height of each small target frame are counted, where is the width of the target box, is the height of the target frame, w is the width of the left image,h is the height of the left image, is the detection threshold, which is 0.015 in this embodiment.

[0087] S32, calculating the number of Gaussian pyramid layers according to the detection results and the segmentation results, and constructing Gaussian pyramids of the left and right images in combination with the left and right images obtained by the binocular camera.

[0088] Specifically, the calculation formula for the number of Gaussian pyramid layers is: ;

[0089] In the formula, O is the number of adaptive Gaussian pyramid layers, is the maximum number of layers of the Gaussian pyramid, which is set to seven in this embodiment. is the number of superpixels in the image after being processed by the superpixel segmentation algorithm, n is the number of small target objects detected by the YOLOv5 network, For the i The width of the small target box, For the i The height of the small target box.

[0090] Construct a Gaussian pyramid based on the images taken by the left and right cameras: Take the images taken by the left and right cameras as Level 0 images, and perform Gaussian downsampling on them, that is, perform convolution operation on the images using a 3×3 Gaussian convolution kernel, and then delete all even rows and columns in the original image to obtain two images of the Level 1 layer. The 3×3 Gaussian convolution kernel is as follows: ;

[0091] Perform Gaussian downsampling on the left and right images of the Level 1 layer to obtain two downsampled result images of the Level 2 layer; then repeat the above operation until the Level 0 layer result image is obtained, and combine the original image with the Level 1~0 layer images into a Gaussian pyramid, such as Figure 3 shown.

[0092] S4. Calculate the improved Census cost, SAD cost and gradient cost according to each layer of the Gaussian pyramid, determine the environmental weight coefficient of each cost based on the light intensity and segmentation result, take the normalized weighted average of the improved Census cost, SAD cost and gradient cost as the cost of each layer, and calculate the total fusion cost of the Gaussian pyramid by upsampling and averaging.

[0093] Specifically, the improved Census cost of each layer of the Gaussian pyramid is calculated: for the left and right images of any layer in the Gaussian pyramid, according to the result label map of the aforementioned superpixel segmentation algorithm, in the 9×9 neighborhood of each pixel of the left image, find the pixels with the same superpixel label as the central pixel, and use the average grayscale value of these pixels as the Census reference pixel value in the window. Subsequently, a square window of size 9×9 is used to traverse the left and right images respectively, and the pixels in the window are compared with the reference pixel values ​​in turn. The pixels less than or equal to the reference pixel value are marked as 0, and the pixels greater than the reference pixel value are marked as 1. The calculation process is as follows: ;

[0094] In the formula, is the encoding function, is the pixel in the window p The gray value of Remove pixels from the window p Pixels outside q Grayscale value; then, calculate the Census transform code in each window, the calculation process is as follows: ;

[0095] In the formula, Pixel p Census transformation code, q Remove pixels from the window p Pixels outside of N p Pixel p Field window pixel set, is the encoding connection operator; i Pixels in the left image of the layer p In the parallax value d The cost value The calculation method is as follows: ;

[0096] In the formula, is the first i Pixels in the left image of the layer p The Census transformation code of the point, is the first i Layer right image corresponding to the pixel point The Census transform code of the two can be obtained by calculating the Hamming distance between them. p In the parallax value d Improved Census cost under .

[0097] Calculate the SAD cost and gradient cost of each layer of the left image in the Gaussian pyramid: i For the left image of the layer, the SAD cost is calculated as: ;

[0098] The calculation formula of the gradient cost is: ;

[0099] In the formula, is the first i Pixels in the left image of the layer p When the parallax value is d The SAD cost under is the first i Pixels in the left image of the layer p The gray value of is the first i The pixel in the right image of the layer p The corresponding disparity value is d Gray value of the pixel point, L is the left image, R is the right image; is the first i Pixels in the left image of the layer p When the parallax value is d The gradient cost under

[0100] for x The gradient value in the direction, for y The gradient value in the direction.

[0101] Determine the environmental weight coefficients for improving the Census cost, gradient cost (GRD cost), and SAD cost: for the light intensity in the shooting environment , the calculation formula of the environmental weight coefficient of the improved Census cost is: ;

[0102] The calculation formula of the environmental weight coefficient of the gradient cost is: ;

[0103] The environmental weight coefficient calculation formula of SAD cost is: ;

[0104] In the formula, To improve the environmental cost weights of Census costs, is the environmental cost weight of the gradient cost, is the environmental cost weight of the SAD cost, To improve the weight factor of the Census cost, 0.3 is taken in this embodiment. is the weight factor of the gradient cost, which is 0.15 in this embodiment.

[0105] According to the above three cost results combined with the corresponding environment weight coefficient, the cost of each layer of the Gaussian pyramid and the total fusion cost of the original image are calculated. In the calculation process of the matching cost of each layer of the Gaussian pyramid, the above single costs are first normalized, and then the results are fused and calculated. i The calculation formula of layer fusion cost is: ; ;

[0106] In the formula, For the Gaussian pyramid i Pixels in the left image of the layer p In the parallax value d The fusion cost under is the normalized cost value, ranging from [0,1], To improve the normalization parameters of the Census cost, is the normalization parameter of SAD cost, is the normalization parameter of the gradient cost, c To match the cost.

[0107] Then the total fusion cost of the Gaussian pyramid is calculated: First, the matching cost of each layer of the image in the Gaussian pyramid is calculated by bilinear interpolation. i times upsampling ( i is the number of pyramid layers where the image is located), so that its resolution is consistent with the original image; then, the image costs of each layer are fused, that is, the average cost of the image costs of Level0~O layers of the Gaussian pyramid is obtained, and this value is used as the total fusion cost of the cost calculation step.

[0108] S5. The disparity value that minimizes the energy function at each pixel point is used as the result disparity value of cost aggregation. Based on the eight-path dynamic programming cost aggregation algorithm and combined with the superpixel label map, cost aggregation is performed to obtain the final aggregation result.

[0109] Specifically, the expression of the energy function is: ; ; ;

[0110] In the formula, Pixel p In the parallax value d The energy value below Pixel p In the parallax value The cost value is P 1. P 2 are two different penalty items, For penalty items P The initial value is 2, Pixel p The gray value of Remove pixels from the window p Pixels outside q The gray value of is the penalty control coefficient. If the central pixel in the original aggregation range is p With the rest of the pixels q Belong to the same superpixel, a smaller penalty coefficient is imposed 1. If the center pixel and the other pixels belong to the same superpixel, a larger penalty coefficient is applied 2, T is the indicator function, Pixel p The disparity value of Remove pixels from the window p Pixels outside q To solve the above problem more efficiently, this embodiment adopts an eight-path dynamic programming cost aggregation algorithm. First, the aggregation cost of a single pixel under different paths is calculated according to a fixed path, and then the aggregation cost under all paths is summed to obtain the final aggregation result. p Along the Path r The one-way aggregation cost The calculation method is as follows: ;

[0111] In the formula, Pixel p In the parallax value d The cost value is Remove pixels from the path p The remaining pixels are in the disparity value d The cost value is Remove pixels from the path p The remaining pixels are in the disparity value i The cost value of the pixel p The total path cost is calculated by the following formula: ;

[0112] In the formula, Pixel p In the parallax value d The aggregation cost under r is the number of aggregated paths. In this embodiment, eight-path cost aggregation is adopted, such as Figure 4 shown.

[0113] S6. Calculate the disparity values ​​of all pixels in the left image based on the aggregation cost, and perform disparity optimization through left-right consistency detection, disparity filling, and median filtering in sequence.

[0114] Specifically, the WTA (winner takes all) algorithm is used for disparity calculation: for the aggregation cost obtained in the previous step, the WTA principle is used to perform a preliminary calculation of the disparity value, that is, the aggregation cost under all disparity values ​​of a single pixel is compared, and the disparity value corresponding to the minimum aggregation cost is taken as the result disparity value of the disparity calculation. This method is used to calculate the disparity values ​​of all pixels in the left image.

[0115] Perform multi-step disparity optimization: For the disparity value calculated in the previous step, the left-right consistency detection, disparity filling, and median filtering algorithms are used for optimization in turn. The algorithm uses left-right consistency detection to eliminate incorrect matching points, calculates the disparity value of the right image according to the above steps, and calculates the absolute value of the difference between the disparity value of each pixel in the left image and the same-name point in the right image in turn. If it is greater than the threshold (generally 1.0), the pixel in the left image is added as an invalid pixel and the disparity result is erased. For invalid pixels, the disparity filling algorithm is used to fill the pixel disparity according to the valid disparity values ​​of the pixels around the invalid pixel. The specific algorithm steps are as follows:

[0116] 1) Determine the pixels in the occluded area and the mismatched pixels: Suppose the pixel point in the right image q is the pixel in the left image p By parallax value d If there is another pixel point in the left image, the same point is found. Also with pixels q are points of the same name, and the pixel The parallax value is greater than the parallax value d If you want it to be bigger, then the pixels p Invalid pixels that belong to the occluded area but not to the occluded area are classified as mismatched points.

[0117] 2) Perform invalid disparity filling: For pixels in the occluded area, select the second smallest disparity value among the surrounding valid pixels as the disparity value of the pixel; for invalid pixels, select the median of the disparity values ​​among the surrounding pixels as the disparity value of the pixel. p The parallax filling process is expressed by the formula as follows: ;

[0118] In the formula, Pixels p The parallax value after parallax filling, Pixel p The surrounding disparity values, represents the median operation, Represents pixel pFor the case of invalid pixels, It means taking the next minimum value. Represents pixel p In the case of being in the obstruction area, In other cases, represents . Finally, in order to remove isolated outliers in the disparity map, the median filtering algorithm is used to smooth the disparity map, that is, the disparity value of each pixel is set to the median of the disparity values ​​in the neighborhood window of the point.

[0119] According to another embodiment of the present invention, Figure 5 As shown, a multi-scale stereo matching system based on superpixel segmentation is provided, which includes an image acquisition and detection module 1, an image segmentation module 2, a Gaussian pyramid construction module 3, a cost calculation module 4, a cost aggregation module 5 and a disparity optimization module 6.

[0120] The image acquisition and detection module 1 is used to detect the light intensity in the environment using an illuminometer, and to acquire left and right images through a binocular camera to perform binocular positioning, distortion correction and epipolar correction processing.

[0121] The image segmentation module 2 is used to perform image segmentation on the processed left image based on the superpixel segmentation method of the SLIC algorithm to obtain a superpixel label map of the segmented image.

[0122] The Gaussian pyramid construction module 3 is used to detect small target objects in the left image using the YOLOv5 network, determine the number of Gaussian pyramid layers based on the detection results and the segmentation results, and construct Gaussian pyramids for the left and right images.

[0123] The cost calculation module 4 is used to calculate the improved Census cost, SAD cost and gradient cost according to each layer image of the Gaussian pyramid, determine the environmental weight coefficient of each cost in combination with the light intensity and the segmentation result, take the normalized weighted average of the improved Census cost, SAD cost and gradient cost as the cost of each layer, and calculate the total fusion cost of the Gaussian pyramid by upsampling and averaging.

[0124] The cost aggregation module 5 is used to take the disparity value that minimizes the energy function at each pixel point as the result disparity value of cost aggregation, perform cost aggregation based on the eight-path dynamic programming cost aggregation algorithm and the superpixel label map to obtain the final aggregation result.

[0125] The disparity optimization module 6 is used to calculate the disparity values ​​of all pixels in the left image based on the aggregation cost, and perform disparity optimization through left-right consistency detection, disparity filling, and median filtering in sequence.

[0126] In summary, with the aid of the above technical solution of the present invention, the present invention firstly determines the number of superpixels adaptively according to the image information entropy and variance, and merges the over-segmented superpixels, so that the proposed superpixel segmentation algorithm can cope with stereo matching in complex scenes, and then uses superpixel segmentation and YOLOv5 to process and detect the captured images and determine the number of adaptive Gaussian pyramid layers, and realizes multi-scale calculation of matching costs by constructing a Gaussian pyramid and fusion of adaptive weight costs according to light intensity, and applies a penalty term to the energy function according to the superpixel segmentation results in the subsequent cost aggregation stage, so that the algorithm proposed by the present invention can ensure robustness in diversified scenes and higher matching accuracy in complex environments. At the same time, the present invention can not only calculate the number of superpixel segmentation blocks according to factors such as the size of the images captured by the left and right cameras, the image variance, and the one-dimensional information entropy, etc. K ,accomplish K The algorithm can not only adapt the value of the image to the desired value, but also judge the over-segmented superpixels by setting the threshold according to the information entropy difference and the color difference, and eliminate them by merging them into adjacent superpixels. This can effectively reduce the over-segmentation of superpixels in the image and reduce the impact of over-segmentation on subsequent cost calculations. It can also effectively solve the problems in the existing superpixel segmentation algorithm that the number of superpixels needs to be set manually and the result image has serious over-segmentation phenomenon. At the same time, the present invention can not only determine the pixel points in the Census window that are in the same superpixel as the central pixel according to the segmentation result, and replace the Census reference pixel value with the grayscale mean of the above pixel points, thereby effectively improving the anti-noise ability of the Census cost, but also introduce YOLOv5 to detect small targets on the captured images, calculate the number of adaptive Gaussian pyramid layers according to the results of small target detection and superpixel segmentation, and respectively calculate the improved Census cost, SAD cost and gradient cost in the cost calculation process of each layer of the Gaussian pyramid, determine the environmental weight coefficient of the cost in combination with the illumination intensity and the segmentation result, and take the weighted mean of the normalized three costs as the cost of each layer, thereby effectively solving the problems of low matching accuracy caused by a single matching cost and insensitivity of the matching cost to illumination changes. In addition, different penalty items can be applied to different superpixel labels of pixel points in the aggregation path according to the superpixel segmentation result of the captured image, thereby significantly improving the accuracy and robustness of the matching algorithm.

[0127] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-scale stereo matching method based on superpixel segmentation, characterized in that: The following steps are involved: S 1. Use an illuminance meter to detect the light intensity in the environment, and use a binocular camera to obtain left and right images, perform binocular positioning, distortion correction and epipolar correction processing; S2, based on the superpixel segmentation method of the SLIC algorithm, the processed left image is segmented to obtain a superpixel label map of the segmented image; S3, using the YOLOv5 network to detect small target objects in the left image, combining the detection results and segmentation results to determine the number of Gaussian pyramid layers, and constructing the Gaussian pyramids of the left and right images; S4. Calculate the improved Census cost, SAD cost and gradient cost according to each layer of the Gaussian pyramid, determine the environmental weight coefficient of each cost in combination with the illumination intensity and the segmentation result, take the normalized weighted average of the improved Census cost, SAD cost and gradient cost as the cost of each layer, and calculate the total fusion cost of the Gaussian pyramid by upsampling and averaging; Among them, the calculation formula of the improved Census cost is: C census (p,d,i)=Hamming[S i，L (p),S i，R (p-d)] The calculation formula of SAD cost is: The calculation formula of the gradient cost is: The calculation formula of the environmental weight coefficient is: c GRD =d GRD in E ill c SAD =3-(γ census +g GRD ) The fusion cost calculation formula is: C Calc (p, d, i)=γ census p(C census (p, d, i),λ census ,i)+γ SAD p(C SAD (p, d, i),λ SAD ,i)+γ GRD p(C GRD (p, d, i),λ GRD ,i) In the formula, C census (p, d, i) is the cost value of pixel p in the left image of the i-th layer in the Gaussian pyramid under the disparity value d, S i，L (p) is the Census transform code of pixel p in the left image of the i-th layer in the Gaussian pyramid, S i，R (pd) is the Census transform code of the pixel (pd) corresponding to the right image in the i-th layer of the Gaussian pyramid, S(p) is the Census transform code of the pixel p, q is the pixel in the window except the pixel p, N p is the pixel set of the window in the area of ​​pixel point p, is the encoding connection operator, ξ[, *] is the encoding function, I(p) is the gray value of pixel p in the window, I(q) is the gray value of pixel q in the window except pixel p, C SAD (p, a, i) is the SAD cost of pixel p in the left image at the i-th layer in the Gaussian pyramid under the disparity value d, I i，L (p) is the gray value of pixel p in the left image of the i-th layer in the Gaussian pyramid, I i，R (pa) is the gray value of the pixel with disparity value d corresponding to pixel p in the right image of the i-th layer in the Gaussian pyramid, L is the left image, R is the right image, C GRD (p, a, i) is the gradient cost of the pixel p in the left image of the i-th layer in the Gaussian pyramid under the disparity value d, is the gradient value in the x direction, is the gradient value in the y direction, γ census To improve the environmental cost weight of the Census cost, w is the width of the left image, h is the height of the left image, and γ GRD is the environmental cost weight of the gradient cost, γ SAD is the environmental cost weight of the SAD cost, δ census To improve the weight factor of Census cost, O GRD is the weight factor of the gradient cost, E ill is the illumination intensity, K is the number of adaptive superpixels, C calc (p, d, i) is the fusion cost of pixel p in the left image of the i-th layer of the Gaussian pyramid under the disparity value d, ρ(c, λ, i) is the normalized cost value, λ census To improve the normalization parameter of Census cost, λ SAD is the normalization parameter of SAD cost, λ GRD is the normalization parameter of the gradient cost, and c is the matching cost; S5, taking the disparity value that minimizes the energy function at each pixel as the result disparity value of cost aggregation, performing cost aggregation based on the eight-path dynamic programming cost aggregation algorithm and combining the superpixel label map to obtain the final aggregation result; S6. Calculate the disparity values ​​of all pixels in the left image based on the aggregation cost, and perform disparity optimization through left-right consistency detection, disparity filling, and median filtering in sequence.

2. The multi-scale stereo matching method based on superpixel segmentation according to claim 1, characterized in that: The superpixel segmentation method based on the SLIC algorithm performs image segmentation on the processed left image to obtain a superpixel label map of the segmented image, including the following steps: S21, respectively calculating the one-dimensional information entropy and variance of the left image, and calculating the number of adaptive superpixels according to the one-dimensional information entropy and variance; based on the number of superpixels, using the SLIC superpixel segmentation algorithm to segment the left image to obtain a segmentation label map of the image; S22, searching for all small connected areas and outer contour points of each superpixel within a preset range around the cluster center, counting the number of pixels with different superpixel labels in the four neighborhoods of the outer contour point, and assigning all pixels in the small connected area to the superpixel label with the highest frequency; S23, extract the pixel points of the outer contour of the current superpixel and find all the superpixel labels adjacent to the outer contour, calculate the average color value and one-dimensional information entropy of the adjacent superpixels under the R, G, and B channels respectively, and calculate the difference in one-dimensional information entropy and color difference between the adjacent superpixels and the current superpixel; S24. When the difference in one-dimensional information entropy between the current superpixel and the adjacent superpixel is less than the preset threshold of the difference in one-dimensional information entropy, and the color difference between the current superpixel and the adjacent superpixel is less than the preset color difference threshold, the adjacent superpixel is determined to be an over-segmented superpixel, and the superpixel labels of all pixels in the over-segmented superpixel are replaced with the current superpixel labels, otherwise they remain unchanged.

3. The multi-scale stereo matching method based on superpixel segmentation according to claim 2, characterized in that: The calculation formula of one-dimensional information entropy is: The formula for calculating variance is: The calculation formula for the number of adaptive superpixels is: K=(w+h)·θE·lnV Where E is the one-dimensional information entropy of the left image, N is the number of pixels in the left image, and N i is the number of pixels with gray value i, V is the variance of the left image, I j is the gray value of the jth pixel in the left image, is the average gray value of the pixels in the left image, K is the number of adaptive superpixels, w is the width of the left image, h is the height of the left image, and θ is the adjustment coefficient.

4. The multi-scale stereo matching method based on superpixel segmentation according to claim 2, characterized in that: The average color value is calculated as: The formula for calculating color difference is: In the formula, is the average color value under color channel m, N S is the number of pixels in the superpixel S, I m，i is the pixel value of point i in the superpixel under color channel m, D c (α, β) is the color difference between superpixels α and β, is the average color value of superpixel α in color channel m, is the average color value of superpixel β in color channel m.

5. The multi-scale stereo matching method based on superpixel segmentation according to claim 1, characterized in that: The method of using the YOLOv5 network to detect the small target object in the left image, combining the detection result and the segmentation result to determine the number of Gaussian pyramid layers, and constructing the Gaussian pyramids of the left and right images includes the following steps: S31, use the pre-built YOLOv5 network to detect the target object in the left image, and filter the small target objects in the detection results. When the target box of the detection result meets w t h t When <ηwh, the current detection result is taken as a small target object, and the number of all small target objects, the width and height of each small target frame are counted, where w t is the width of the target box, h t is the height of the target box, w is the width of the left image, h is the height of the left image, and η is the detection threshold; S32, calculating the number of Gaussian pyramid layers according to the detection results and the segmentation results, and constructing Gaussian pyramids of the left and right images in combination with the left and right images obtained by the binocular camera.

6. The multi-scale stereo matching method based on superpixel segmentation according to claim 5, characterized in that: The calculation formula for the number of Gaussian pyramid layers is: Where O is the number of layers of the adaptive Gaussian pyramid, O max is the maximum number of layers of the Gaussian pyramid, k is the number of superpixels in the image after being processed by the superpixel segmentation algorithm, n is the number of small target objects detected by the YOLOv5 network, and w i is the width of the i-th small target box, h i is the height of the i-th small target box.

7. The multi-scale stereo matching method based on superpixel segmentation according to claim 1, characterized in that: The expression of the energy function is: The calculation formula of aggregation cost is: Where E(d) is the energy value of pixel p at disparity value d, C(p, d p ) is the disparity value d of pixel p p The cost value under this condition, P1 and P2 are two different penalty terms, P′2 is the initial value of the penalty term P2, I p is the gray value of pixel p, I q is the gray value of pixel q except pixel p in the window, μ is the penalty control coefficient, μ1 and μ2 are two different penalty coefficients, and μ1<μ2, T is the indicator function, a p is the disparity value of pixel p, a q is the disparity value of pixel q in the window except pixel p, C aggr (p, d) is the aggregation cost of pixel p under disparity value d, r is the number of aggregation paths, L r (p, d) is the unidirectional aggregation cost of pixel point p along path r, C(p, d) is the cost value of pixel point p under disparity value d, L r (pr, d) is the cost value of the remaining pixels on the path except pixel p under the disparity value d, L r (pr, i) is the cost value of the remaining pixels on the path except pixel p at disparity value i.

8. The multi-scale stereo matching method based on superpixel segmentation according to claim 1, characterized in that: Parallax optimization via parallax fill includes: Determine the pixels in the occlusion area and the mismatched pixels: suppose the pixel q in the right image is the same name point found by the pixel p in the left image through the disparity value d. If there is another pixel p' in the left image that is also the same name point as the pixel q, and the disparity value of the pixel p' is larger than the disparity value d, then the pixel p belongs to the occlusion area pixel, and the invalid pixels that do not belong to the occlusion area pixel are classified as mismatched points; Perform invalid disparity filling: For pixels in the occluded area, select the second smallest disparity value among the surrounding valid pixels as the disparity value of the pixel; for invalid pixels, select the median of the disparity values ​​among the surrounding pixels as the disparity value of the pixel.

9. A multi-scale stereo matching system based on superpixel segmentation, used to implement the steps of the multi-scale stereo matching method based on superpixel segmentation according to any one of claims 1 to 8, characterized in that: The multi-scale stereo matching system based on superpixel segmentation includes an image acquisition and detection module, an image segmentation module, a Gaussian pyramid construction module, a cost calculation module, a cost aggregation module and a disparity optimization module; The image acquisition and detection module is used to detect the light intensity in the environment using an illuminometer, and to acquire left and right images through a binocular camera to perform binocular positioning, distortion correction and epipolar correction processing; The image segmentation module is used to perform image segmentation on the processed left image based on the superpixel segmentation method of the SLIC algorithm to obtain a superpixel label map of the segmented image; The Gaussian pyramid construction module is used to detect small target objects in the left image using the YOLOv5 network, determine the number of Gaussian pyramid layers based on the detection results and the segmentation results, and construct Gaussian pyramids for the left and right images; The cost calculation module is used to calculate the improved Census cost, SAD cost and gradient cost according to each layer image of the Gaussian pyramid, determine the environmental weight coefficient of each cost in combination with the illumination intensity and the segmentation result, take the normalized weighted average of the improved Census cost, SAD cost and gradient cost as the cost of each layer, and calculate the total fusion cost of the Gaussian pyramid by upsampling and averaging; The cost aggregation module is used to take the disparity value that minimizes the energy function at each pixel as the result disparity value of cost aggregation, perform cost aggregation based on the eight-path dynamic programming cost aggregation algorithm and the superpixel label map to obtain the final aggregation result; The disparity optimization module is used to calculate the disparity values ​​of all pixels in the left image based on the aggregation cost, and perform disparity optimization through left-right consistency detection, disparity filling, and median filtering in sequence.

Images