Three-dimensional road scene generation method based on binocular depth estimation

Abstract

The invention discloses a three-dimensional road scene generation method based on binocular depth estimation, and the method is based on left and right eye depth camera sensors, a binocular depth estimation neural network model, an edge constraint road segmentation module and a point cloud data optimization module, and comprises the steps: 1, dynamically capturing a road scene image through the left and right eye depth camera sensors; 2, calculating the left and right eye depth images through a binocular depth estimation neural network model; and step 3, generating point cloud data according to the predicted depth map and the camera internal reference matrix. According to the method, binocular depth estimation is combined with an edge constraint algorithm, so that the precision of three-dimensional reconstruction in a complex road scene is improved; meanwhile, the real-time performance and the visualization are realized by utilizing the efficient point cloud processing and rendering capability of Open3D; finally, a low-cost binocular camera is adopted to replace a laser radar, the hardware cost is remarkably reduced, and feasibility is provided for large-scale application.

Description

Technical Field

[0001] The present invention relates to the field of automobile automatic driving system display technology and road scene reconstruction, and in particular to a three-dimensional road scene generation method based on binocular depth estimation. Background Art

[0002] In the field of autonomous driving and intelligent transportation, three-dimensional reconstruction of road environment is a crucial research direction. Currently, many technical solutions have been applied in this field, but they all have certain limitations. For example, patent CN110796728B uses laser radar to obtain three-dimensional point cloud data and reconstructs the target's size, structure, position and posture through a greedy projection algorithm. This type of traditional three-dimensional reconstruction method relies on high-precision laser radar. Although it has high accuracy, the equipment is expensive and it is difficult to achieve economical efficiency in large-scale application scenarios.

[0003] Relatively speaking, as a cost-effective sensor, cameras have gradually replaced lidar in some scenarios. For example, patent CN116091695A uses hierarchical reinforcement learning technology for 3D reconstruction. Although this method can achieve high accuracy, it is mainly aimed at single objects rather than complex road scenes, so its applicability is limited. Another patent, CN116091703A, uses real-time 3D reconstruction technology based on multi-view stereo matching to build a 3D model by shooting one by one with a monocular camera. Although this solution can improve the reconstruction accuracy to a certain extent, the shooting efficiency is low, and the accuracy performance in dynamic environments is not ideal.

[0004] In contrast, binocular cameras use the parallax information of their left and right perspectives to achieve depth estimation, which is a relatively low-cost sensor solution with low hardware requirements. However, the current binocular depth estimation algorithm still faces challenges such as insufficient accuracy and high computational complexity when facing complex road scenes. Therefore, a new technical solution is urgently needed that can effectively reduce hardware costs and improve reconstruction accuracy and efficiency in complex scenes to overcome the limitations of existing technologies. Summary of the invention

[0005] The present invention proposes a three-dimensional road scene generation method based on binocular depth estimation, which uses a binocular camera to obtain left and right view images of the road environment, combines a depth estimation algorithm to generate depth information for each pixel, and uses an edge constraint algorithm to highlight the key features of the road scene. Finally, Open3D is used to achieve efficient three-dimensional road scene generation. The present invention can significantly improve the reconstruction accuracy and efficiency of complex road scenes, which is helpful for the application of autonomous driving and intelligent transportation systems, especially for achieving real-time perception in a low-cost hardware environment.

[0006] The present invention discloses a method for generating a three-dimensional road scene based on binocular depth estimation. The method is based on left and right depth camera sensors, a binocular depth estimation neural network model, an edge-constrained road segmentation module, and a point cloud data optimization module. The method comprises the following steps: Step 1: Dynamically capture the road scene through the left and right depth camera sensors; Step 2: Calculate the left and right eye depth images through a binocular depth estimation neural network model; Step 3: Generate point cloud data based on the predicted depth map and camera intrinsic parameter matrix.

[0007] Furthermore, in step 1, the following steps are also included: Step 1.1, configure the left and right depth cameras according to the sensor's intrinsic parameter matrix, make them work synchronously, and set the capture frame rate and resolution to meet real-time processing requirements; Step 1.2: Collect left and right images of the road scene through left and right vision sensors installed on the vehicle and perform preprocessing; including: Step 1.2.1: Use the camera to calibrate the intrinsic parameter matrix and distortion coefficients to perform geometric correction on the image: , ; , ; Among them, (x, y, z) is the pixel coordinate, (x′, y′) is the standard coordinate, ( , ) is the corrected coordinate, , , is the distortion coefficient, ; Step 1.2.2: Extract the road region of interest from the rectified image, use the edge detection algorithm combined with the set region mask to retain the road surface and important edge features.

[0008] Furthermore, in step 2, the following steps are also included: Step 2.1, input the preprocessed left and right eye images into the neural network as the basic data of the model; Step 2.2: Extract and fuse the multi-layer features of the left and right images through the encoder; Step 2.3, calculating the initial disparity map through the disparity estimation module; ; in, is the pixel coordinate The disparity value on is used to indicate the displacement difference between the left and right images at the pixel position. is the pixel coordinate, and are the horizontal coordinates of the corresponding points in the left and right images respectively; Step 2.4: Use the spatial attention mechanism to increase the saliency of key areas of the scene; Step 2.5, enhancing the boundary information in the depth map through the depth edge constraint module; ; in, Represents the depth value after optimization by the depth edge constraint module. is the original depth value, is the edge detection value, λ is the weight parameter; Step 2.6: weighted fusion of the depth maps output by the disparity estimation module, the spatial attention module, and the depth edge constraint module; The depth maps output by the disparity estimation module, spatial attention module, and depth edge constraint module are weighted fused: ; in, is the final depth map obtained after weighted processing of the three modules. is the depth map for disparity estimation, Enhance the depth map for spatial attention, Enhance the depth map for depth edges, , , is the fusion weight.

[0009] Furthermore, in step 2.1, the following steps are also included: Step 2.2.1, multi-level feature extraction, including: Low-level feature extraction: Edge and texture features are extracted from the left and right images through the initial convolution layer: ; in, is the input image pixel value, is the width of the convolution kernel, is the height of the convolution kernel, is the low-level convolution kernel, Output for low-level features; Mid-level feature extraction: Use convolutional modules with pooling layers to extract geometric and texture information in the scene and reduce redundant data: ; in, The pixel coordinate range of the pooling window in the horizontal direction, is the pixel coordinate range of the pooling window in the vertical direction, is the pixel coordinate index within the pooling window, Output for mid-level features; High-level feature extraction: Use stacked convolutional layers to extract scene semantic features: ; in: Output for high-level features; It is the feature map output of the previous layer, in channel The eigenvalues ​​on is the feature position within the local window; is the convolution kernel, and its size is , the input channel is , the output channel is ; is the bias term of output channel c; is the activation function, used to introduce nonlinearity; By stacking multiple convolutional layers, the receptive field is gradually expanded and the semantic features are captured. The formula is iterated until the final result is ; The low, medium and high level features are combined through skip connections to form hierarchical features: ; , , They respectively represent the weights of low-level, medium-level, and high-level features in the fusion process.

[0010] Furthermore, in step 2.4, the following steps are also included: Step 2.4.1, multi-scale feature generation; Input feature map Features of different scales are extracted through different convolution kernels: ; in, The kernel size is The convolution operation, It means that the convolution kernel size is After the convolution operation, at the pixel coordinates The generated Scale feature map; Step 2.4.2: Superimpose feature maps of different scales according to weights: ; in, is the scale quantity, Make sure the weights are normalized, Represented in pixel coordinates Above, the final eigenvalue after integrating all scale features; Step 2.4.3: Superimpose the fused feature map onto the input feature map to obtain the final enhanced feature: ; Represented in pixel coordinates The original hierarchical features are combined and multi-scale fusion features The enhanced feature map value.

[0011] Furthermore, in step 3, the following steps are also included: Step 3.1: Depth map Combined with the camera intrinsic parameter matrix, the initial point cloud data is generated: ; in, is the pixel coordinate; ( ) is the optical center position of the camera; , are the focal lengths of the camera in the horizontal and vertical directions respectively; is the depth value; X, Y represent the horizontal and vertical positions of the three-dimensional point in the camera coordinate system respectively, and Z is the depth in the direction of the camera optical axis; Step 3.2: Downsample and denoise the initial point cloud to improve data quality; including the following steps: Step 3.2.1, divide the initial point cloud into cubic grids of size s by voxel grid method, and use the centroid of each point in the grid to represent the point of the grid: ; Where N is the number of points in the grid, is the centroid coordinate of the grid, is the coordinates of each point in the grid; Step 3.2.2, calculate each point and its The average distance of the nearest neighbors, eliminating outliers that do not meet the following conditions: ; in, is the average distance, is the standard deviation, Set thresholds for users, For each point The average distance of the nearest neighbors; Step 3.3: Input the optimized point cloud data into the Open3D library, generate and render a 3D road scene, add color information to the point cloud data, and map the depth value to color: ; in, For point The depth value of and The beneficial effects achieved by the present invention are: The present invention improves the accuracy of 3D reconstruction in complex road scenes by combining binocular depth estimation with edge constraint algorithm; The present invention utilizes the efficient point cloud processing and rendering capabilities of Open3D to achieve real-time performance and visualization; The present invention adopts a low-cost binocular camera to replace the laser radar, which significantly reduces the hardware cost and provides feasibility for large-scale application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a process framework of a 3D road scene generation method based on binocular depth estimation; Figure 2 It is a binocular depth estimation network model for a three-dimensional road scene generation method based on binocular depth estimation; Figure 3 It is a point cloud data processing flow of a three-dimensional road scene generation method based on binocular depth estimation. DETAILED DESCRIPTION

[0013] The present invention will be further described below in conjunction with specific embodiments, and the advantages and features of the present invention will become clearer as the description proceeds. However, these embodiments are exemplary only and do not constitute any limitation to the scope of the present invention. It should be understood by those skilled in the art that the details and forms of the technical solution of the present invention may be modified or replaced without departing from the spirit and scope of the present invention, but these modifications and replacements all fall within the scope of protection of the present invention.

[0014] like Figure 1 As shown, this embodiment is a process framework of a three-dimensional road scene generation method based on binocular depth estimation. The framework includes the following modules: Left and right depth camera sensors: used to capture road scene images in real time, equipped with the sensor's internal parameter matrix; Stereo depth estimation neural network model: used to generate depth maps of road scenes; Edge-constrained road segmentation module: optimizes the features of road edges and scene boundaries; Point cloud data optimization module: downsampling, denoising and rendering optimization processing are performed based on the generated point cloud data.

[0015] The method specifically comprises the following steps: Step 1: Capture and preprocess left and right eye images.

[0016] The road scene is captured dynamically by the left and right depth camera sensors. This step includes the following: Step 1.1, initialize the left and right visual sensors; Configure the left and right depth cameras according to the sensor's intrinsic parameter matrix to work synchronously, and set the capture frame rate and resolution to meet real-time processing requirements.

[0017] Step 1.2, capturing road scene images and preprocessing them; The left and right images of the road scene are collected by the left and right vision sensors installed on the vehicle and preprocessed; specifically, the following steps are included: Step 1.2.1, image distortion correction; The image is geometrically corrected through the camera calibration parameters (intrinsic matrix and distortion coefficient). The specific formula is as follows: , ; , ; Among them, (x, y, z) is the pixel coordinate, (x′, y′) is the standard coordinate, , , is the distortion coefficient, .

[0018] Step 1.2.2, region of interest (ROI) extraction; The road area of ​​interest is extracted from the rectified image, and the road surface and important edge features are retained using an edge detection algorithm (such as Canny) combined with a set area mask.

[0019] Step 2: Input and calculation of the binocular depth estimation neural network model; like Figure 2 As shown, this embodiment shows the architecture of the binocular depth estimation network model, which specifically includes the following modules: Disparity estimation module: generates the initial disparity map; Spatial attention module: highlights key areas in the scene and enhances the accuracy of depth estimation; Deep edge constraint module: Improves boundary clarity through semantic edge enhancement.

[0020] The specific steps are as follows: Step 2.1, input left and right eye images; The preprocessed left and right eye images are input into the neural network as the basic data of the model.

[0021] Step 2.2, feature encoding; The encoder extracts three types of features from the left and right images: low-level features (such as edge information), mid-level features (such as texture information), and high-level features (such as scene semantic information).

[0022] The specific steps are as follows: Step 2.2.1, multi-level feature extraction; (1) Low-level feature extraction: Edge and texture features of the left and right images are extracted through the initial convolutional layer.

[0023] Use multiple 3×3 convolution kernels to slide the image. The calculation formula is as follows: ; in, is the input image pixel value, is the low-level convolution kernel, It is the low-level feature output.

[0024] (2) Mid-level feature extraction: Use a convolutional module with a pooling layer to extract geometric and texture information in the scene and reduce redundant data.

[0025] Pooling operation formula: ; in, , is the window range, It is the mid-level feature output.

[0026] (3) High-level feature extraction: Use stacked convolutional layers to extract scene semantic features, such as roads, obstacles, etc.

[0027] The high-level features are represented as: ; in: Output for high-level features; It is the feature map output of the previous layer, and the channel index is i; is the convolution kernel, and its size is , the input channel is C, and the output channel is ; is the bias term of the convolution; is an activation function used to introduce nonlinearity.

[0028] By stacking multiple convolutional layers, the receptive field is gradually expanded and the semantic features are captured. The formula is iterated until the final result is .

[0029] Step 2.2.2, feature fusion; The low, medium and high level features are combined through skip connections to form hierarchical features: ; , , They respectively represent the weights of low-level, medium-level, and high-level features in the fusion process.

[0030] Step 2.3, disparity estimation; After feature extraction and fusion, this step aims to generate an initial disparity map through the disparity estimation module as the basis for depth estimation.

[0031] The disparity map reflects the displacement difference between the left and right images, which is directly related to the depth information and provides input data for subsequent depth edge optimization. The calculation formula of the disparity estimation module is as follows: ; in, is the pixel coordinate, and are the horizontal coordinates of the corresponding points in the left and right images respectively.

[0032] Step 2.4: Enhance spatial attention; In order to further optimize the accuracy of depth estimation and make up for the perspective phenomenon that parallax estimation cannot solve, this step introduces a spatial attention mechanism, which is used to improve the saliency of key areas of the scene (such as lane lines and road edges). The specific steps are as follows: Step 2.4.1, multi-scale feature generation; The feature map obtained by feature extraction in step 2.2.2 Features of different scales are extracted through different convolution kernels to generate multi-scale feature maps: ; in, The kernel size is The convolution operation.

[0033] Step 2.4.2, cross-scale fusion; After step 2.4.1, feature maps of different scales are obtained. Feature maps of different scales are superimposed according to weights. The formula is: ; in, is the scale quantity, Make sure the weights are normalized.

[0034] Step 2.4.3, enhance output; Overlay the fused feature map onto the feature map The final enhanced features are obtained: ; Step 2.5, depth edge constraint; After completing the spatial attention enhancement in step 2.4, in order to further improve the boundary clarity of the depth map and reduce the blurred area, this step introduces the depth edge constraint module. This module optimizes the boundary characteristics of the key area by fusing edge detection information and depth map data, providing more accurate input data for the final 3D reconstruction. The depth edge constraint module is optimized by the following formula: ; in, is the original depth value, is the edge detection value, and λ is the weight parameter.

[0035] Step 2.6, depth map fusion; The depth maps output by the disparity estimation module, spatial attention module, and depth edge constraint module are weighted fused: ;

[0036] in, is the depth map for disparity estimation, Enhance the depth map for spatial attention, Enhance the depth map for depth edges, , , is the fusion weight.

[0037] Step 3: Point cloud data generation and optimization; like Figure 3 As shown, this embodiment demonstrates the point cloud data processing flow of the method.

[0038] This step uses the optimized depth map output from step 2 And the camera intrinsic parameter matrix, specifically including the following steps: Step 3.1, initial point cloud generation; The depth map Combined with the camera intrinsic parameter matrix, the initial point cloud data is generated. The formula is as follows: ; in, is the pixel coordinate; ( , ) is the optical center position; , is the focal length; is the depth value; X, Y represent the horizontal and vertical positions of the three-dimensional point in the camera coordinate system, respectively, and Z is the depth in the direction of the camera optical axis.

[0039] Step 3.2, point cloud optimization; The initial point cloud is downsampled and denoised to improve data quality.

[0040] The specific implementation steps are as follows: Step 3.2.1, voxel grid downsampling; The initial point cloud is divided into a cubic grid of size s by the voxel grid method, and the centroid of each point in the grid is used to represent the point of the grid. The formula is as follows: ; Where N is the number of points in the grid, is the centroid coordinate of the grid, is the coordinate of each point in the grid.

[0041] Step 3.2.2, statistical filtering denoising; After downsampling by the voxel grid method, the relationship between each point and its The average distance of the nearest neighbors, eliminating outliers that do not meet the following conditions: ; Among them, μ is the average distance, σ is the standard deviation, and α is the threshold set by the user.

[0042] Step 3.3, 3D scene construction; The optimized point cloud data is input into the Open3D library to generate and render a 3D road scene. Color information is added to the point cloud data, and the depth value is mapped to color. The specific formula is as follows: ; in, The optimized point The depth value of , are the maximum and minimum depth values ​​respectively.

[0043] The above are only specific steps of the present invention and do not constitute any limitation to the protection scope of the present invention; any technical solutions formed by equivalent transformation or equivalent replacement shall fall within the protection scope of the present invention; some parts not elaborated in detail in the present invention belong to the known technologies of those skilled in the art.

Claims

1. A three-dimensional road scene generation method based on binocular depth estimation, based on left and right eye depth camera sensors, a binocular depth estimation neural network model, an edge constraint road segmentation module and a point cloud data optimization module, characterized in that: The three-dimensional road scene generation method based on binocular depth estimation comprises the following steps: Step 1: Dynamically capture the road scene through the left and right depth camera sensors; Step 2: Calculate the left and right eye depth images through a binocular depth estimation neural network model; Step 3: Generate point cloud data based on the predicted depth map and camera intrinsic parameter matrix.

2. The method for generating a three-dimensional road scene based on binocular depth estimation according to claim 1, characterized in that: In step 1, the following steps are also included: Step 1.1, configure the left and right depth cameras according to the sensor's intrinsic parameter matrix, make them work synchronously, and set the capture frame rate and resolution to meet real-time processing requirements; Step 1.2: Collect left and right images of the road scene through left and right vision sensors installed on the vehicle and perform preprocessing; including: Step 1.2.1: Use the camera to calibrate the intrinsic parameter matrix and distortion coefficients to perform geometric correction on the image: , ； , ； Among them, (x, y, z) is the pixel coordinate, (x′, y′) is the standard coordinate, ( , ) are the corrected coordinates, , , is the distortion coefficient, ; Step 1.2.2: Extract the road region of interest from the rectified image, use the edge detection algorithm combined with the set region mask to retain the road surface and important edge features.

3. The method for generating a three-dimensional road scene based on binocular depth estimation according to claim 1, characterized in that: In step 2, the following steps are also included: Step 2.1, input the preprocessed left and right eye images into the neural network as the basic data of the model; Step 2.2: Extract and fuse the multi-layer features of the left and right images through the encoder; Step 2.3, calculating the initial disparity map through the disparity estimation module; ; in, is the pixel coordinate The disparity value on is used to indicate the displacement difference between the left and right images at the pixel position. is the pixel coordinate, and are the horizontal coordinates of the corresponding points in the left and right images respectively; Step 2.4: Use the spatial attention mechanism to increase the saliency of key areas of the scene; Step 2.5, enhancing the boundary information in the depth map through the depth edge constraint module; ； in, Represents the depth value after optimization by the depth edge constraint module. is the original depth value, is the edge detection value, λ is the weight parameter; Step 2.6: weighted fusion of the depth maps output by the disparity estimation module, the spatial attention module, and the depth edge constraint module; The depth maps output by the disparity estimation module, spatial attention module, and depth edge constraint module are weighted fused: ； in, is the final depth map obtained after weighted processing of the three modules. is the depth map for disparity estimation, Enhance the depth map for spatial attention, Enhance the depth map for depth edges, , , is the fusion weight.

4. The method for generating a three-dimensional road scene based on binocular depth estimation according to claim 3, characterized in that: In step 2.1, the following steps are also included: Step 2.2.1, multi-level feature extraction, including: Low-level feature extraction: Edge and texture features are extracted from the left and right images through the initial convolution layer: ； in, is the input image pixel value, is the width of the convolution kernel, is the height of the convolution kernel, is the low-level convolution kernel, Output for low-level features; Mid-level feature extraction: Use convolutional modules with pooling layers to extract geometric and texture information in the scene and reduce redundant data: ； in, The pixel coordinate range of the pooling window in the horizontal direction, is the pixel coordinate range of the pooling window in the vertical direction, is the pixel coordinate index within the pooling window, Output for mid-level features; High-level feature extraction: Use stacked convolutional layers to extract scene semantic features: ； in: Output for high-level features; It is the feature map output of the previous layer, in channel The eigenvalues ​​on is the feature position within the local window; is the convolution kernel, and its size is , the input channel is , the output channel is ; is the bias term of output channel c; is the activation function, used to introduce nonlinearity; By stacking multiple convolutional layers, the receptive field is gradually expanded and the semantic features are captured. The formula is iterated until the final result is ; The low, medium and high level features are combined through skip connections to form hierarchical features: ; , , They respectively represent the weights of low-level, medium-level, and high-level features in the fusion process.

5. The method for generating a three-dimensional road scene based on binocular depth estimation according to claim 4, characterized in that: In step 2.4, the following steps are also included: Step 2.4.1, multi-scale feature generation; Input feature map Features of different scales are extracted through different convolution kernels: ； in, The kernel size is The convolution operation, It means that the convolution kernel size is After the convolution operation, at the pixel coordinates The generated Scale feature map; Step 2.4.2: Superimpose feature maps of different scales according to weights: ; in, is the scale quantity, Make sure the weights are normalized, Represented in pixel coordinates Above, the final eigenvalue after integrating all scale features; Step 2.4.3: Superimpose the fused feature map onto the input feature map to obtain the final enhanced feature: ; Represented in pixel coordinates The original hierarchical features are combined and multi-scale fusion features The enhanced feature map value.

6. The method for generating a three-dimensional road scene based on binocular depth estimation according to claim 1, characterized in that: In step 3, the following steps are also included: Step 3.1: Depth map Combined with the camera intrinsic parameter matrix, the initial point cloud data is generated: ; in, is the pixel coordinate; ( ) is the optical center position of the camera; , are the focal lengths of the camera in the horizontal and vertical directions respectively; is the depth value; X, Y represent the horizontal and vertical positions of the three-dimensional point in the camera coordinate system respectively, and Z is the depth in the direction of the camera optical axis; Step 3.2: Downsample and denoise the initial point cloud to improve data quality; including the following steps: Step 3.2.1, divide the initial point cloud into cubic grids of size s by voxel grid method, and use the centroid of each point in the grid to represent the point of the grid: ; Where N is the number of points in the grid, is the centroid coordinate of the grid, is the coordinates of each point in the grid; Step 3.2.2, calculate each point and its The average distance of the nearest neighbors, eliminating outliers that do not meet the following conditions: ; in, is the average distance, is the standard deviation, Set thresholds for users, For each point The average distance of the nearest neighbors; Step 3.3: Input the optimized point cloud data into the Open3D library, generate and render a 3D road scene, add color information to the point cloud data, and map the depth value to color: ; in, The optimized point The depth value of and are the maximum and minimum depth values ​​respectively.

Images