A deep learning-based DPS-SGBM binocular vision ranging method

Abstract

The application provides a DPS-SGBM binocular vision ranging method based on deep learning, and relates to the technical field of computer vision and three-dimensional measurement. The application uses a YOLOv8 target detection network to obtain semantic information and image scene features of a scene as prior input, dynamically calculates optimal stereo matching parameters for each target in the image, and outputs a disparity map with confidence measurement. A probability model is introduced to optimize and evaluate the confidence of the matching result. The application effectively balances the precision, efficiency and practicability of binocular vision ranging by deeply fusing deep learning semantic information and a dynamic probability self-adaptive semi-global matching algorithm, and provides a more robust, intelligent and efficient solution for three-dimensional perception applications such as automatic driving and robots.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and 3D measurement technology, and in particular to a deep learning-based DPS-SGBM binocular vision ranging method. Background Technology

[0002] In recent years, deep learning-based object detection algorithms such as YOLOv8 have demonstrated excellent performance in both accuracy and speed. However, current technologies have not fully utilized the semantic information provided by object detection to optimize the performance of traditional stereo matching algorithms. Binocular stereo vision is an important 3D environment perception technology in computer vision. It uses two cameras to capture the same scene from different perspectives and calculates the depth information of objects using the principle of triangulation. Semi-Global Block Matching (SGBM) is a classic stereo matching algorithm that is widely used due to its good balance between accuracy and efficiency.

[0003] However, the traditional SGBM algorithm has the following technical drawbacks: SGBM requires pre-setting parameters such as disparity search range and penalty coefficient. These parameters are usually set globally based on the entire image scene, which cannot adapt to the characteristics of targets at different distances and sizes; to ensure the matching of small targets at a distance, a large disparity search range needs to be set, but this increases the computational complexity and wastes resources for large targets at close range; a fixed penalty coefficient cannot adapt to targets with different texture characteristics, resulting in matching errors in weak texture areas or areas with repeated textures. Summary of the Invention

[0004] Purpose of the invention: To provide a deep learning-based DPS-SGBM binocular vision ranging method, which aims to solve the problems of fixed parameters in traditional SGBM algorithms, inability to adapt to multi-scale targets and complex scenes, and disparity output as a single-point estimate lacking reliability measurement.

[0005] To achieve the above objectives, this invention proposes a deep learning-based DPS-SGBM binocular vision ranging method. It utilizes the YOLOv8 object detection network to obtain semantic information of the scene and image scene features as prior input. It dynamically calculates the optimal stereo matching parameters for each target in the image and outputs a disparity map with confidence metrics. A probabilistic model is introduced to optimize the matching results and evaluate confidence. Specifically, the core technical solution of this invention includes the following steps:

[0006] S1. Obtain the left and right views captured by the binocular camera, perform target detection on the left view to obtain target information, and extract scene features from the left and right views;

[0007] S2. Based on the scene features, dynamically calculate a unique DPS-SGBM parameter set for each target region corresponding to the target information;

[0008] S3. Based on the DPS-SGBM parameter set, perform zonal matching in the corresponding target area, optimize and evaluate the matching results based on the probability model, and output the optimized disparity map and the corresponding confidence map.

[0009] S4. Calculate the three-dimensional distance of the target area based on the calibration parameters of the binocular camera, the optimized disparity map, and the confidence map.

[0010] As a preferred embodiment, before acquiring the left and right views captured by the stereo camera, the following steps are also included:

[0011] Obtain the intrinsic parameter matrices, distortion coefficients, and extrinsic parameter matrices describing the relative positional relationship between the left and right cameras for the left and right cameras;

[0012] Based on the obtained calibration parameter results, the stereo correction mapping of the binocular image is calculated, and the original left and right views acquired in real time are remapped and transformed to generate the corrected left and right views.

[0013] The corrected left and right views are converted into grayscale images and used as input images for subsequent object detection and feature extraction.

[0014] As a preferred embodiment, target detection is performed on the left view to obtain target information, and scene features of the left and right views are extracted, specifically including:

[0015] The left view is input into a pre-trained deep learning model for real-time object detection to obtain object information, which includes bounding box information, category labels, and confidence scores.

[0016] Low-level feature analysis is performed on the left and right views to calculate scene features, which include texture complexity, depth difference level, and motion degree.

[0017] As a preferred embodiment, in step S2, the DPS-SGBM parameter set includes: matching window size, dynamic penalty coefficient, and adaptive disparity search range; wherein:

[0018] The matching window size is adaptively adjusted based on the texture complexity.

[0019] The dynamic penalty coefficient is obtained by dynamically adjusting the category label and the confidence score in the left view.

[0020] The adaptive disparity search range is calculated based on the bounding box width, image width, and depth difference level in the bounding box information.

[0021] As a preferred embodiment, the dynamic penalty coefficient includes a first penalty coefficient and a second penalty coefficient;

[0022] The first penalty coefficient and the second penalty coefficient are dynamically adjusted based on the texture richness and structural consistency of the local region. The adjustment process is calculated by combining the gradient standard deviation of the pixel within its local window and the texture confidence of the pixel.

[0023] As a preferred option, the penalty coefficient The adjustment is made dynamically based on the texture richness and structural consistency of local areas, as shown in the following mathematical formula:

[0024]

[0025]

[0026] in, Based on pixels The standard deviation of the gradient within a local window centered on the target; It is the texture confidence of that point, calculated using the following formula: ,in and These are the gradients in the horizontal and vertical directions, respectively. It is a minimum value to prevent division by zero.

[0027] As a preferred embodiment, when calculating the adaptive disparity search range, an adjustment coefficient and the maximum disparity value allowed by the system are introduced;

[0028] Based on the width of the target bounding box With image width The proportion of the scene depth difference level d is used to calculate a compact and efficient disparity search range for this target, avoiding the computational waste caused by a large-scale global search.

[0029]

[0030] in, is the ratio of the target bounding box width to the image width, reflecting the size of the target in the image; d is the scene depth difference level; k is an adjustment coefficient that controls the range of change in the search area. This is the maximum disparity value allowed by the system.

[0031] As a preferred embodiment, based on the DPS-SGBM parameter set, zonal matching is performed within the corresponding target area, specifically including:

[0032] The region of interest is defined using the bounding box information in the target information. Within the region of interest of each target, the semi-global block matching (SGBM) algorithm is executed using the corresponding DPS-SGBM parameter set to calculate the local disparity map.

[0033] For background regions in the image that are not covered by any target, a semi-global block matching algorithm is performed using preset default parameters to obtain a disparity map of the background region.

[0034] The local disparity maps of all target areas are stitched together with the disparity map of the background area to form a complete initial disparity map.

[0035] As a preferred approach, the matching results are optimized and evaluated based on a probabilistic model, outputting an optimized disparity map and a corresponding confidence map, specifically including:

[0036] A Gaussian mixture model is introduced to model the disparity distribution of the initial disparity map for each pixel;

[0037] The mean, variance, and weight parameters of the Gaussian components of the Gaussian mixture model are iteratively updated using the Expectation Maximization (EM) algorithm to maximize the log-likelihood function.

[0038] The iteration stops when the log-likelihood change is less than the threshold. The updated disparity estimate is output as the optimized disparity map, and the corresponding disparity confidence is output as the confidence map.

[0039] As a preferred approach, a hierarchical optimization strategy is employed to improve computational efficiency when optimizing and evaluating the matching results. This includes:

[0040] The first layer performs local cost aggregation based on an adaptive window within the region of interest;

[0041] The second layer performs dynamic path optimization only on motion regions with a degree of motion or low-confidence regions.

[0042] As a preferred embodiment, step S3 specifically includes:

[0043] S3-1. Perform stereo matching in partitions, instead of using uniform SGBM parameters for the entire image. For each detected target, within its region of interest (ROI) defined by its bounding box, execute the SGBM algorithm using the dynamic specific parameters calculated in step two.

[0044] S3-2. To address the issues of traditional algorithms outputting disparity as a single-point estimate and lacking confidence assessment, this invention introduces a Gaussian mixture model to model the disparity distribution of each pixel:

[0045]

[0046] Where K is the number of Gaussian components; The weight of the k-th Gaussian component; and These are the mean and variance of the k-th Gaussian component, respectively. The parameters are updated using the Expectation-Maximization (EM) algorithm, ultimately outputting a more robust disparity estimate:

[0047]

[0048] Simultaneously output disparity confidence:

[0049]

[0050] S3-3. To improve computational efficiency, a hierarchical optimization strategy is adopted. The first layer performs local cost aggregation based on an adaptive window within the target ROI; the second layer performs dynamic path optimization only for moving regions or low-confidence regions to reduce the global computational load.

[0051] As a preferred embodiment, step S4 specifically includes:

[0052] Locate the center pixel coordinates of the target region, read the disparity value of the center pixel coordinates from the optimized disparity map, and read the confidence from the confidence map;

[0053] Determine whether the confidence level is greater than or equal to a preset effective confidence level threshold;

[0054] If the confidence level is greater than or equal to the effective confidence threshold, the disparity value is used directly, combined with the camera focal length and baseline in the binocular camera calibration parameters, and the three-dimensional distance is calculated using the triangulation principle.

[0055] If the confidence level is less than the effective confidence threshold, then a pixel that meets the confidence level requirement is searched within the target area, and the pixel with the closest disparity value is used as a substitute for calculating the three-dimensional distance.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] (1) By using a dynamic penalty coefficient based on local texture confidence and an adaptive disparity search mechanism, the matching defects of traditional fixed parameters on weak textures, edges and multi-scale targets are effectively overcome, and the disparity calculation and depth estimation accuracy of key targets are significantly improved.

[0058] (2) The method of the present invention can dynamically adjust the core parameters in real time based on multi-dimensional features such as target category, scene texture, and depth difference, realizing the leap from manual parameter adjustment to scene self-adaptation, and has stronger robustness and practicality in complex dynamic environments.

[0059] (3) By allocating a suitable compact parallax search range to the target area and combining it with a hierarchical optimization strategy, the global redundant calculation is greatly reduced while ensuring the accuracy of key targets. This balances the contradiction between high precision and real-time performance and is more suitable for resource-constrained platforms such as vehicle-mounted and embedded systems.

[0060] In summary, the DPS-SGBM method proposed in this invention achieves an effective balance between accuracy, efficiency, and practicality in binocular vision ranging by deeply fusing deep learning semantic information with a dynamic probabilistic adaptive semi-global matching algorithm, providing a more robust, intelligent, and efficient solution for 3D perception applications such as autonomous driving and robotics. Attached Figure Description

[0061] Figure 1 This is the overall system flowchart of the DPS-SGBM algorithm of this invention.

[0062] Figure 2 This is a depth map (grayscale image) in Embodiment 1 of the present invention.

[0063] Figure 3 This is a schematic diagram of the distance measurement results in Embodiment 1 of the present invention. Detailed Implementation

[0064] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0065] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0066] The following describes an implementation example using an in-vehicle binocular vision system. The specific process is as follows:

[0067] Step 1: Binocular positioning and image correction

[0068] This step aims to establish an accurate mapping from pixel coordinates to world coordinates and provide a standard input image for stereo matching.

[0069] 1. Using a checkerboard calibration board, obtain the intrinsic parameter matrices K (including focal length) of the left and right cameras respectively through Zhang Zhengyou's calibration method or OpenCV's stereoCalibrate function. and the main point ), distortion coefficient D, and extrinsic matrix (rotation matrix R and translation vector T) describing the relative positional relationship between the left and right cameras.

[0070] 2. Based on the calibration results, calculate the stereo correction mapping cv2.stereoRectify for the binocular images, and perform a remapping transformation cv2.remap on the original left and right views acquired in real time to generate row-aligned corrected image pairs. and This eliminates lens distortion and ensures that the epipolar lines are horizontally aligned.

[0071] 3. Convert the corrected color image to a grayscale image, denoted as . and This reduces data dimensions and color interference, thereby improving subsequent processing efficiency.

[0072] Step 2: Target Detection and Scene Feature Extraction

[0073] This step obtains the semantic information and underlying features of the scene, providing a basis for parameter dynamics.

[0074] 1. Input the corrected left view into the pre-trained YOLOv8 deep learning model to perform real-time processing of the left view acquired by the binocular vision system. Object detection. The model identifies all objects of interest in an image and obtains the bounding box information of the i-th object. and category tags And confidence score.

[0075] 2. For grayscale images Analyze the images of the preceding and following frames to quantify three key scene features:

[0076] Calculate texture complexity t: Calculate the left grayscale image average gradient magnitude and will Normalize to the [0,1] interval.

[0077]

[0078]

[0079] Depth difference level d: Calculates the pixel intensity difference between the left and right views, and after normalization, reflects the magnitude of scene depth change.

[0080]

[0081]

[0082] Motion level m: Calculates the current frame With the previous frame The differences, after normalization, reflect the dynamic changes in the scene.

[0083]

[0084]

[0085] Step 3: Calculation of dynamic parameters

[0086] For each detected target By combining its own attributes and global scene features, a set of exclusive stereo matching parameters is calculated.

[0087] 1. Calculate the matching window size. Adaptively adjust based on texture complexity. Use a small window to preserve details when the texture is complex, and use a large window to suppress noise when the texture is simple.

[0088]

[0089] in, This indicates rounding down to the nearest integer, ensuring the result is an odd number.

[0090] 2. Calculate the dynamic penalty coefficient According to the target category and texture confidence By looking up the table and making dynamic adjustments, the dynamic penalty coefficient specific to this region can be obtained. :

[0091]

[0092]

[0093] in, This is a global maximum value function used to find the maximum value within the target data range.

[0094] 3. Calculate the adaptive disparity search range Set the target box width Image width Substituting the depth difference level d into the formula, calculate its disparity search range. :

[0095]

[0096] In this embodiment, a reference disparity value is taken. Adjustment coefficient Maximum disparity value The calculation result needs to be adjusted to an integer multiple of 16.

[0097] Step 4: Partitioned Stereo Matching and Probabilistic Optimization

[0098] 1. Partition DPS-SGBM Matching

[0099] Traverse each target bounding box The bounding box is used as the region of interest (ROI). Within this ROI, the specific parameter tuples calculated in step three are used. Initialize and execute the core process of the DPS-SGBM algorithm to calculate the local disparity map. .

[0100] For background areas in the image that are not covered by any target, a set of preset, conservative default parameters are used. The standard SGBM algorithm is executed to balance overall computational efficiency.

[0101] Disparity map stitching: stitching together local disparity maps of all target regions. Seamlessly stitch the initial disparity map with the disparity map of the background area to form a complete disparity map. .

[0102] 2. Probabilistic parallax optimization

[0103] In the target area Inside, for the initial disparity value Perform K-Means clustering ( Cluster centers are used as the mean of Gaussian components. initial value, variance Initialized as the variance of similar disparities, weights Initialize to the proportion of each cluster point.

[0104] Apply the expectation-maximization algorithm to iteratively optimize the parameter set. To maximize the log-likelihood function:

[0105]

[0106] The iteration stops when the log-likelihood change is less than the threshold.

[0107] Final optimized disparity map and the corresponding confidence plot for:

[0108]

[0109]

[0110] Step 5: 3D Distance Calculation

[0111] For each target bounding box Locate the pixel coordinates of the center point of its bounding box. From the optimized disparity map Read the disparity value of this point And from the confidence plot Read confidence level .

[0112] like ( The effective confidence threshold (which can be set to 0.65 in this embodiment) is then used directly. Perform distance calculation. Otherwise, within the target bounding box. Internal search satisfies The pixels are selected, and the one with the closest disparity value is chosen. The point is used as a substitute.

[0113] Using the principle of triangulation, combined with the camera focal length f and baseline B, the depth distance of the target is calculated. :

[0114]

[0115] Step Six: Output Results

[0116] The system ultimately outputs structured results: {target category} Bounding box coordinates 3D distance Distance confidence This is for use by the upper-level autonomous driving decision-making system.

[0117] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0119] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0120] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0121] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A deep learning-based DPS-SGBM binocular vision ranging method, characterized in that, Includes the following steps: S1. Obtain the left and right views captured by the binocular camera, perform target detection on the left view to obtain target information, and extract scene features from the left and right views; S2. Based on the scene features, dynamically calculate a unique DPS-SGBM parameter set for each target region corresponding to the target information; S3. Based on the DPS-SGBM parameter set, perform zonal matching in the corresponding target area, optimize and evaluate the matching results based on the probability model, and output the optimized disparity map and the corresponding confidence map. S4. Calculate the three-dimensional distance of the target area based on the calibration parameters of the binocular camera, the optimized disparity map, and the confidence map.

2. The deep learning-based DPS-SGBM binocular vision ranging method according to claim 1, characterized in that, Before acquiring the left and right views captured by the stereo camera, the process also includes: Obtain the intrinsic parameter matrices, distortion coefficients, and extrinsic parameter matrices describing the relative positional relationship between the left and right cameras for the left and right cameras; Based on the obtained calibration parameter results, the stereo correction mapping of the binocular image is calculated, and the original left and right views acquired in real time are remapped and transformed to generate the corrected left and right views. The corrected left and right views are converted into grayscale images and used as input images for subsequent object detection and feature extraction.

3. The deep learning-based DPS-SGBM binocular vision ranging method according to claim 1, characterized in that, Target detection is performed on the left view to obtain target information, and scene features of the left and right views are extracted, specifically including: The left view is input into a pre-trained deep learning model for real-time object detection to obtain object information, which includes bounding box information, category labels, and confidence scores. Low-level feature analysis is performed on the left and right views to calculate scene features, which include texture complexity, depth difference level, and motion degree.

4. The deep learning-based DPS-SGBM binocular vision ranging method according to claim 3, characterized in that, In step S2, the DPS-SGBM parameter set includes: matching window size, dynamic penalty coefficient, and adaptive disparity search range; wherein: The matching window size is adaptively adjusted based on the texture complexity. The dynamic penalty coefficient is obtained by dynamically adjusting the category label and the confidence score in the left view. The adaptive disparity search range is calculated based on the bounding box width, image width, and depth difference level in the bounding box information.

5. The deep learning-based DPS-SGBM binocular vision ranging method according to claim 4, characterized in that, The dynamic penalty coefficient includes a first penalty coefficient and a second penalty coefficient; The first penalty coefficient and the second penalty coefficient are dynamically adjusted based on the texture richness and structural consistency of the local region. The adjustment process is calculated by combining the gradient standard deviation of the pixel within its local window and the texture confidence of the pixel.

6. The deep learning-based DPS-SGBM binocular vision ranging method according to claim 4, characterized in that, When calculating the adaptive disparity search range, an adjustment factor and the maximum disparity value allowed by the system are introduced; The adaptive disparity search range is calculated by combining the ratio of the bounding box width to the image width, the depth difference level, and the baseline disparity value, thereby avoiding a large-scale global search.

7. The deep learning-based DPS-SGBM binocular vision ranging method according to claim 1, characterized in that, Based on the DPS-SGBM parameter set, perform zonal matching within the corresponding target area, specifically including: The region of interest is defined using the bounding box information in the target information. Within the region of interest of each target, a semi-global block matching algorithm is executed using the corresponding DPS-SGBM parameter set to calculate a local disparity map. For background regions in the image that are not covered by any target, a semi-global block matching algorithm is performed using preset default parameters to obtain a disparity map of the background region. The local disparity maps of all target areas are stitched together with the disparity map of the background area to form a complete initial disparity map.

8. The deep learning-based DPS-SGBM binocular vision ranging method according to claim 7, characterized in that, The matching results are optimized and evaluated based on a probabilistic model, outputting an optimized disparity map and a corresponding confidence map, specifically including: A Gaussian mixture model is introduced to model the disparity distribution of the initial disparity map for each pixel; The mean, variance, and weight parameters of the Gaussian components of the Gaussian mixture model are iteratively updated using the expectation-maximization algorithm to maximize the log-likelihood function. The iteration stops when the log-likelihood change is less than the threshold. The updated disparity estimate is output as the optimized disparity map, and the corresponding disparity confidence is output as the confidence map.

9. The deep learning-based DPS-SGBM binocular vision ranging method according to claim 8, characterized in that, When optimizing and evaluating the matching results, a hierarchical optimization strategy is adopted to improve computational efficiency, specifically including: The first layer performs local cost aggregation based on an adaptive window within the region of interest; The second layer performs dynamic path optimization only on motion regions with a degree of motion or low-confidence regions.

10. The deep learning-based DPS-SGBM binocular vision ranging method according to claim 1, characterized in that, Step S4 specifically includes: Locate the center pixel coordinates of the target region, read the disparity value of the center pixel coordinates from the optimized disparity map, and read the confidence from the confidence map; Determine whether the confidence level is greater than or equal to a preset effective confidence level threshold; If the confidence level is greater than or equal to the effective confidence threshold, the disparity value is used directly, combined with the camera focal length and baseline in the binocular camera calibration parameters, and the three-dimensional distance is calculated using the triangulation principle. If the confidence level is less than the effective confidence threshold, then a pixel that meets the confidence level requirement is searched within the target area, and the pixel with the closest disparity value is used as a substitute for calculating the three-dimensional distance.

Images