Depth completion method for laser radar and RGB (Red, Green, Blue) camera with intelligent perception function

Abstract

The invention discloses a depth completion method for a laser radar with intelligent perception and an RGB camera. The method comprises the following steps: receiving original data of the RGB camera and the laser radar, and performing preprocessing (synchronization, calibration and noise reduction); rGB image relative depth prediction is generated by using a pre-trained monocular depth model, laser radar sparse information is injected through pixel alignment and distance reweighting, and a pre-filled rough dense depth map is generated; inputting the pre-filled image and the relative depth prediction into a conditional MDE network, and generating a high-precision dense depth image in combination with an RGB texture semantic correction error; and outputting for follow-up use of the appliance body. According to the method, multi-sensor information is fused, the depth completion precision and integrity are improved, and reliable support is provided for intelligent perception of a tool body.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligent sensing technology, and more specifically to a depth completion method for embodied intelligent sensing using LiDAR and RGB camera. Background Technology

[0002] In the fields of embodied intelligence and robotics, accurate and real-time 3D perception of the environment is fundamental to achieving advanced functions such as autonomous navigation, obstacle avoidance, grasping, and human-computer interaction. Existing depth completion methods using LiDAR and RGB cameras suffer from the following technical challenges and limitations in embodied intelligence perception scenarios: 1. Sparsity of LiDAR point clouds: LiDAR provides accurate depth, but the point cloud is sparse, especially at long distances, in specific materials, or in occluded areas, resulting in incomplete depth maps, which makes it difficult to meet the needs of embodied intelligence for dense geometric structures.

[0003] 2. Insufficient robustness to dynamic environments and robot motion: Motion blur, lighting changes, low-texture areas and transparent / reflective objects affect the quality of RGB images and monocular depth estimation, which in turn affects the accuracy and stability of depth completion.

[0004] 3. Computing resource and power consumption limitations: Embodied intelligent devices (such as mobile robots) equipped with edge computing platforms have limited computing power, memory and power consumption, making it difficult to deploy high-performance algorithms efficiently.

[0005] 4. Challenges of hybrid depth prior processing and generalization ability: In real-world scenarios, depth measurement devices may provide depth priors with multiple incomplete modes. Existing methods often focus on specific types and lack generalization ability. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the present invention aims to provide a depth completion method for LiDAR and RGB camera for embodied intelligent perception. The core of this method lies in constructing a unified, coarse-to-fine progressive multi-sensor fusion framework. This framework efficiently combines the sparse but accurate depth measurement information from LiDAR with the dense but relative geometric and texture information from the RGB camera through explicit and implicit fusion strategies, generating a high-precision, high-real-time, robust, and edge-computing-compatible dense depth map.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a depth completion method for embodied intelligent perception using LiDAR and RGB camera, comprising the following steps: Step 1: Receive raw data from the RGB camera and LiDAR sensor, and then preprocess the raw data to perform time synchronization, spatial calibration, and noise filtering on the RGB image and LiDAR point cloud. Step 2: Use a pre-trained monocular depth estimation model to generate relative depth predictions for RGB images, and explicitly inject sparse depth information from the LiDAR through pixel-level metric alignment and distance-aware reweighting to generate a pre-filled coarse-dense depth map. Step 3: The pre-filled coarse depth map and relative depth prediction are used as conditional inputs and fed into the conditional MDE network. Combined with the texture semantic information of the RGB image, noise and errors are corrected to generate the final fine and high-precision dense depth map. Step four: Output the high-precision, high-completeness dense depth map generated in step three for use in subsequent embodied intelligence tasks.

[0008] As a further improvement of the present invention, the specific steps for preprocessing the raw data in step one are as follows: Step 1: Use hardware or software synchronization to ensure that the RGB image frames and LiDAR scan frames are highly consistent in time. Steps one and two: transform the LiDAR point cloud to the camera coordinate system through extrinsic parameter calibration, obtain camera parameters through intrinsic parameter calibration, and project the LiDAR point cloud onto the RGB image plane to generate a sparse depth map; Steps 1 and 3: Perform statistical filtering on the LiDAR point cloud; perform distortion correction and brightness / contrast adjustment on the RGB image; Step 14: Unify the preprocessed RGB image and sparse depth map into the network input format.

[0009] As a further improvement of the present invention, the specific steps for generating the pre-filled coarse density depth map in step two are as follows: Step 21: Use the frozen MDE base model to generate a dense relative depth prediction map from the RGB image to capture fine geometric structures; Step 22: For missing pixels in the sparse depth map, find the K nearest valid LiDAR depth points using K-nearest neighbor search. Based on these points and their corresponding... The optimal scaling factor s and offset t are calculated by minimizing the loss function: in, It's true depth. It is a relative depth prediction value. These are the distance-aware reweighting coefficients. Using the calculated optimal s and t, the missing pixels are... Linear transformation of values ​​( ), fill to In the middle, a pre-filled coarse density depth map is formed ( ).

[0010] As a further improvement of the present invention, the specific steps for generating the final fine and high-precision dense depth map in step three are as follows: Step 31: Receive the RGB image and pre-filled coarse-dense depth map using a conditional MDE model. ) and the original relative depth prediction ( ), where RGB images are used to provide rich texture, color, and semantic information, and a pre-filled coarse-dense depth map ( As a metric, the original relative depth prediction ( ) as geometric conditions; Step 32: Obtain the coarse density depth map. And the original relative depth prediction By using independent zero-initialized convolutional layers as additional channels to input into the conditional MDE model, the model inherits the pre-trained MDE capabilities and learns to utilize conditional information. Then, the conditional MDE model adopts an encoder-decoder structure and a multi-scale feature fusion mechanism to extract and fuse multi-scale features to generate a dense depth map. In this process, perturbation priors are introduced to enable the model to learn to handle sensor noise and errors.

[0011] As a further improvement of the present invention, step three also includes a step for optimizing embodied intelligence: Step 33: For the coarse dense depth map And the original relative depth prediction Normalization to the [0,1] range improves the model's generalization ability across diverse scenarios and different MDE models. It also enhances the depth perception capability of RGB image features by using a fusion strategy to provide accurate but sparse depth information from LiDAR.

[0012] The beneficial effects of this invention are: Provides high-precision, high-detail dense depth maps: Effectively utilizes precise measurement information from LiDAR and rich texture semantic information from RGB images to generate high-precision, highly complete dense depth maps.

[0013] Optimize algorithms to adapt to the resource constraints of edge computing platforms: Design a solution with high computational efficiency, low memory usage, and excellent energy efficiency to achieve real-time, high-precision depth perception on edge chips and meet the requirements of low-power deployment.

[0014] Enhance the generalization capability for mixed deep priors: Propose a unified framework to seamlessly handle various forms of deep priors such as sparse points, low resolution, and missing regions, thereby improving the system's versatility and adaptability.

[0015] Correcting inherent noise and errors in depth measurement: By intelligently fusing geometric structure information from RGB images, noise and errors in the original depth measurement are effectively corrected, generating a more accurate and detailed depth map. Attached Figure Description

[0016] Figure 1 This is a flowchart of the depth completion method for LiDAR and RGB camera used in embodied intelligent perception according to the present invention. Detailed Implementation

[0017] The present invention will now be described in further detail with reference to the embodiments shown in the accompanying drawings.

[0018] Reference Figure 1 As shown in this embodiment, a depth completion method for embodied intelligent perception using LiDAR and an RGB camera includes the following steps: Step 1: Receive raw data from the RGB camera and LiDAR sensor, and then preprocess the raw data. Perform time synchronization, spatial calibration, and noise filtering on the RGB image and LiDAR point cloud. This step solves the data inconsistency problem through synchronization calibration and improves data quality by filtering noise, thus addressing the issues of insufficient robustness in dynamic environments and sensor noise in the background technology. Step 2: Use a pre-trained monocular depth estimation model to generate relative depth predictions for RGB images, and explicitly inject sparse depth information from the LiDAR through pixel-level metric alignment and distance-aware reweighting to generate a pre-filled coarse-dense depth map. This step of explicitly injecting LiDAR information solves the problem of sparse point clouds, and pixel-level alignment improves completeness, thus solving the problem of incomplete depth maps caused by sparse point clouds in the background technology. Step 3: The pre-filled coarse depth map and relative depth prediction are used as conditional inputs and fed into the conditional MDE network. Combined with the texture semantic information of the RGB image, noise and errors are corrected to generate the final fine and high-precision dense depth map. This step corrects errors by fusing multiple features through conditional inputs and improves robustness by combining texture information, thus solving the problems of noise interference and insufficient adaptability to dynamic environments in the background technology. Step four: Output the high-precision, high-completeness dense depth map generated in step three for use in subsequent tasks of embodied intelligence. The depth map output in this step meets the needs of embodied intelligence for autonomous navigation, obstacle avoidance and other tasks, and solves the problem of insufficient accuracy and completeness of depth maps in the background technology.

[0019] Furthermore, the specific steps for preprocessing the raw data in step one are as follows: Step 1: Hardware or software synchronization is used to ensure that the RGB image frames and LiDAR scanning frames are highly consistent in time. This step ensures time synchronization to solve the problem of data misalignment in dynamic environments, improves depth estimation accuracy, and solves the problem of insufficient robustness in dynamic environments in the background technology. Steps one and two: transform the LiDAR point cloud to the camera coordinate system through extrinsic parameter calibration, obtain camera parameters through intrinsic parameter calibration, and project the LiDAR point cloud onto the RGB image plane to generate a sparse depth map. This step unifies the coordinate system to solve the spatial inconsistency problem and generates a sparse map as the basis for subsequent filling, thus solving the problems of point cloud sparsity and spatial inconsistency in the background technology. Steps 1 and 3 involve statistical filtering of the LiDAR point cloud and distortion correction and brightness / contrast adjustment of the RGB image. This step reduces sensor noise and image quality issues, improves robustness, and addresses the problem of decreased accuracy caused by sensor noise and illumination changes in the background technology. Step 14 unifies the preprocessed RGB image and sparse depth map into the network input format. This step adapts to the model input requirements, improves deployment efficiency, and solves the problem of resource constraints in the background technology that make it difficult to deploy efficiently.

[0020] Furthermore, the specific steps for generating the pre-filled coarse density depth map in step two are as follows: Step 21: Use the frozen MDE base model to generate a dense relative depth prediction map from the RGB image to capture fine geometric structures. This step uses a pre-trained model to reduce computational resource consumption and capture fine structures to solve the point cloud sparsity problem, thus addressing the resource constraints and point cloud sparsity issues in the background technique. Step 22: For missing pixels in the sparse depth map, find the K nearest valid LiDAR depth points using K-nearest neighbor search. Based on these points and their corresponding... The optimal scaling factor s and offset t are calculated by minimizing the loss function: in, It's true depth. It is a relative depth prediction value. These are the distance-aware reweighting coefficients. Using the calculated optimal s and t, the missing pixels are... Linear transformation of values ​​( ), fill to In the middle, a pre-filled coarse density depth map is formed ( This step achieves pixel-level missing region filling through optimal scale and offset calculation, effectively solving the problem of incomplete depth map caused by sparse LiDAR point clouds in the background technology. The generated coarse-dense map retains fine geometric structure, providing a high-quality foundation for subsequent fine completion and improving the integrity and measurement accuracy of the depth map.

[0021] Furthermore, the specific steps for generating the final, refined, and high-precision dense depth map in step three are as follows: Step 31: Receive the RGB image and pre-filled coarse-dense depth map using a conditional MDE model. ) and the original relative depth prediction ( ), where RGB images are used to provide rich texture, color, and semantic information, and a pre-filled coarse-dense depth map ( As a metric, the original relative depth prediction ( As a geometric condition, this step integrates texture, metric, and geometric information through multi-condition input to address the problem of insufficient adaptability to dynamic environments and low-texture areas in the background technology, thereby improving the robustness and generalization ability of the depth map. Step 32: Obtain the coarse density depth map. And the original relative depth prediction By using independent zero-initialized convolutional layers as additional input channels into the conditional MDE model, the model inherits the pre-trained MDE capabilities and learns to utilize conditional information. Then, the conditional MDE model employs an encoder-decoder structure and a multi-scale feature fusion mechanism to extract and fuse multi-scale features to generate a dense depth map. This step reduces computational resource consumption by inheriting pre-training capabilities, adapts to the resource limitations of edge computing platforms, and improves the detail accuracy of the depth map through multi-scale feature fusion, thus solving the problems of high resource consumption and insufficient depth map detail in the background technology. In particular, introducing perturbation priors during model training enables the model to learn to handle sensor noise and errors. This step gives the model the ability to resist noise interference, solves the problem of sensor noise causing a decrease in depth completion accuracy in the background technology, and improves the robustness of the model in dynamic environments.

[0022] Furthermore, step three also includes steps for optimizing embodied intelligence: Step 33: For the coarse dense depth map And the original relative depth prediction Normalization to the [0,1] range improves the model's generalization ability across diverse scenarios and different MDE models. For the precise but sparse depth information from LiDAR, a fusion strategy guides the depth perception capability of RGB image features. This normalization process enhances the model's generalization ability, solves the problem of insufficient generalization ability of mixed depth priors in the background technology, guides RGB features to use LiDAR information to solve the problem of point cloud sparsity, and further improves the integrity and accuracy of the depth map.

[0023] In summary, this invention provides a depth completion method for LiDAR and RGB camera for embodied intelligence perception. By preprocessing synchronous calibration data, pre-filling to generate a coarse depth map, and using a conditional MDE network to fuse multiple features to generate a fine depth map, it solves the technical problems of sparse LiDAR point clouds, insufficient robustness, high resource consumption, and weak generalization ability in existing methods. It achieves high-precision and high-completeness dense depth completion, is compatible with the edge computing platform of embodied intelligence devices, and improves the robustness and generalization ability of the model in various scenarios, meeting the needs of subsequent embodied intelligence tasks.

[0024] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A depth completion method using LiDAR and RGB camera for embodied intelligent perception, characterized in that: Includes the following steps: Step 1: Receive raw data from the RGB camera and LiDAR sensor, and then preprocess the raw data to perform time synchronization, spatial calibration, and noise filtering on the RGB image and LiDAR point cloud. Step 2: Use a pre-trained monocular depth estimation model to generate relative depth predictions for RGB images, and explicitly inject sparse depth information from the LiDAR through pixel-level metric alignment and distance-aware reweighting to generate a pre-filled coarse-dense depth map. Step 3: The pre-filled coarse depth map and relative depth prediction are used as conditional inputs and fed into the conditional MDE network. Combined with the texture semantic information of the RGB image, noise and errors are corrected to generate the final fine and high-precision dense depth map. Step four: Output the high-precision, high-completeness dense depth map generated in step three for use in subsequent embodied intelligence tasks.

2. The depth completion method for LiDAR and RGB camera used for embodied intelligent perception according to claim 1, characterized in that: The specific steps for preprocessing the raw data in step one are as follows: Step 1: Use hardware or software synchronization to ensure that the RGB image frames and LiDAR scan frames are highly consistent in time. Steps one and two: transform the LiDAR point cloud to the camera coordinate system through extrinsic parameter calibration, obtain camera parameters through intrinsic parameter calibration, and project the LiDAR point cloud onto the RGB image plane to generate a sparse depth map; Step 13: Perform statistical filtering on the lidar point cloud; Perform distortion correction and brightness / contrast adjustment on RGB images; Step 14: Unify the preprocessed RGB image and sparse depth map into the network input format.

3. The depth completion method for LiDAR and RGB camera used for embodied intelligent perception according to claim 1 or 2, characterized in that: The specific steps for generating the pre-filled coarse density depth map in step two are as follows: Step 21: Use the frozen MDE base model to generate a dense relative depth prediction map from the RGB image to capture fine geometric structures; Step 22: For missing pixels in the sparse depth map, find the K nearest valid LiDAR depth points using K-nearest neighbor search. Based on these points and their corresponding... The optimal scaling factor s and offset t are calculated by minimizing the loss function: in, It's true depth. It is a relative depth prediction value. These are the distance-aware reweighting coefficients. Using the calculated optimal s and t, the missing pixels are... Linear transformation of values ​​( ), fill to In the middle, a pre-filled coarse density depth map is formed. ).

4. The depth completion method for LiDAR and RGB camera used for embodied intelligent perception according to claim 3, characterized in that: The specific steps for generating the final, detailed, and high-precision dense depth map in step three are as follows: Step 31: Receive the RGB image and pre-filled coarse-dense depth map using a conditional MDE model. ) and the original relative depth prediction ( ), where RGB images are used to provide rich texture, color, and semantic information, and a pre-filled coarse-dense depth map ( As a metric, the original relative depth prediction ( ) as geometric conditions; Step 32: Obtain the coarse density depth map. And the original relative depth prediction By using independent zero-initialized convolutional layers as additional channels to input into the conditional MDE model, the model inherits the pre-trained MDE capabilities and learns to utilize conditional information. Then, the conditional MDE model adopts an encoder-decoder structure and a multi-scale feature fusion mechanism to extract and fuse multi-scale features to generate a dense depth map. In this process, perturbation priors are introduced to enable the model to learn to handle sensor noise and errors.

5. The depth completion method for LiDAR and RGB camera used for embodied intelligent perception according to claim 4, characterized in that: Step three also includes steps for optimizing embodied intelligence: Step 33: For the coarse dense depth map And the original relative depth prediction Normalization to the [0,1] range improves the model's generalization ability across diverse scenarios and different MDE models. It also enhances the depth perception capability of RGB image features by using a fusion strategy to provide accurate but sparse depth information from LiDAR.

Images