Depth completion method and system based on multi-modal modulation input

Abstract

The invention discloses a depth completion method and system based on multi-modal modulation input, and the method comprises the steps: S1, obtaining a first depth map and a second depth map, carrying out the depth fusion, obtaining a third depth map, and obtaining a fourth depth map; s2, processing the fourth depth map according to the distribution characteristics of the third depth map, fusing the third depth map and the fourth depth map into a fifth depth map, and segmenting the fifth depth map into a first near depth map smaller than a threshold and a first far depth map larger than the threshold; s3, generating a second near depth map and a second far depth map which are in pixel-level alignment; s4, inputting the second near depth map and the RGB image into a deep learning network, performing completion operation on the second near depth map, and outputting a dense third near depth map; and S5, splicing the third near depth map and the second far depth map to obtain a final depth map.

Description

Technical Field

[0001] This invention relates to the field of deep completion technology, and more specifically, to a deep completion method and system based on multimodal modulation input. Background Technology

[0002] Depth completion technology, as a core component of 3D visual perception, has significant application value in fields such as autonomous driving, robot navigation, and augmented reality. Traditional methods mainly rely on depth information acquired by a single sensor (such as sparse LiDAR or structured light cameras) to complete missing regions through interpolation or geometric inference. However, such methods exhibit significantly reduced robustness in weakly textured regions and complex occluded scenes, and are limited by sensor noise and sparsity, making it difficult to generate high-precision dense depth maps. In recent years, deep learning-based RGB-D fusion methods have improved completion results by jointly optimizing color images and sparse depth information, but the following key issues still exist:

[0003] Insufficient multimodal data modulation: Existing methods usually directly fuse RGB images with the original depth map without fully considering the differences in noise distribution and confidence characteristics of different sensors (such as ToF cameras and binocular vision), resulting in the loss of effective information or excessive amplification of noise during depth fusion.

[0004] Conflicts in near and far region characteristics: The sparsity and noise distribution of near and far regions in the depth map are significantly different (e.g., near structures are dense but easily occluded, while far structures are sparse but noise spreads severely), and a unified processing strategy is difficult to meet the optimization needs of both.

[0005] Imbalance in the credibility of completion results: Deep learning models are prone to over-smoothing or incorrect predictions for low-confidence regions (such as distant, weakly textured regions), while traditional threshold filtering may mistakenly delete effective details, affecting the integrity and reliability of the depth map.

[0006] To address the aforementioned issues, existing research has attempted to introduce multi-stage networks, regional optimization, or dynamic confidence weights. However, these methods often rely on complex model designs, suffer from low computational efficiency, and fail to fundamentally solve the problem of coupling multimodal modulation with regional processing. Furthermore, current techniques do not adequately preprocess the raw sensor data, failing to fully utilize complementary information in the multimodal data (such as filtered-out high-confidence depth values), further limiting the upper limit of depth completion accuracy.

[0007] The above background information is provided only to aid in understanding the inventive concept and technical solution of this invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above information was disclosed on the filing date of this patent application, the above background information should not be used to evaluate the novelty and inventiveness of this application. Summary of the Invention

[0008] To address this, the deep completion method based on multimodal modulation input proposed in this invention breaks through the limitations of the traditional framework through multimodal deep fusion and domain-specific progressive optimization.

[0009] In a first aspect, the present invention provides a depth completion method based on multimodal modulation input, characterized in that it includes:

[0010] Step S1: Using a multimodal modulation depth camera, acquire a first depth map and a second depth map, and perform depth fusion to obtain a third depth map. At the same time, acquire a fourth depth map composed of depth values ​​that were filtered out during the generation of the third depth map but whose confidence is greater than the depth threshold. The first depth map and the second depth map have different confidence levels in different distance ranges.

[0011] Step S2: Based on the distribution characteristics of the third depth map, process the fourth depth map, and merge the third depth map and the fourth depth map into a fifth depth map. According to the set depth threshold, divide the fifth depth map into a first near depth map that is less than the threshold and a first far depth map that is greater than the threshold.

[0012] Step S3: Obtain the corresponding RGB image, reproject the first near depth image and the first far depth image onto the RGB angle, and generate a second near depth image and a second far depth image with pixel-level alignment;

[0013] Step S4: Input the second near depth map and the RGB image into the deep learning network, perform a completion operation on the second near depth map, post-process the model output, remove pixel depths with low confidence and / or exceeding the threshold, and output a dense third near depth map;

[0014] Step S5: The third near depth map and the second far depth map are stitched together to obtain the final depth map.

[0015] Optionally, the depth completion method based on multimodal modulation input is characterized in that the depth fusion adopts a weighted fusion algorithm to fuse the first depth map and the second depth map.

[0016] Optionally, the depth completion method based on multimodal modulation input is characterized in that the weighted fusion algorithm assigns different weight values ​​to each pixel in the first depth map and the second depth map respectively.

[0017] Optionally, the depth completion method based on multimodal modulation input is characterized in that, when processing the fourth depth map, the depth values ​​of corresponding pixels in the third depth map and the fourth depth map are compared, and a set deviation threshold is used to determine whether a pixel in the fourth depth map is an erroneous point, and then the erroneous point is removed.

[0018] Optionally, the depth completion method based on multimodal modulation input is characterized in that the deep learning network is a convolutional neural network, which learns the depth relationship between the RGB image and the second near-depth map through training, thereby completing the depth of the second near-depth map.

[0019] Secondly, the present invention provides a deep completion system based on multimodal modulation input, characterized in that it includes:

[0020] The acquisition module is used to acquire a first depth map and a second depth map through a multimodal modulation depth camera, and perform depth fusion to obtain a third depth map. At the same time, it acquires a fourth depth map composed of depth values ​​that were filtered out during the generation of the third depth map but whose confidence is greater than the depth threshold.

[0021] The fusion module is used to process the fourth depth map according to the distribution characteristics of the third depth map, and fuse the third depth map and the fourth depth map into a fifth depth map. According to a set depth threshold, the fifth depth map is divided into a first near depth map less than the threshold and a first far depth map greater than the threshold.

[0022] The projection module is used to acquire the corresponding RGB image, reproject the first near depth image and the first far depth image onto the RGB angle, and generate a second near depth image and a second far depth image with pixel-level alignment.

[0023] The output module is used to input the second near-depth map and the RGB image into the deep learning network, perform a completion operation on the second near-depth map, perform post-processing on the model output, remove pixel depths with low confidence and / or exceeding the threshold, and output a dense third near-depth map.

[0024] The stitching module is used to stitch the third near-depth map and the second far-depth map together to obtain the final depth map.

[0025] Optionally, the depth completion system based on multimodal modulation input is characterized in that the acquisition module uses a weighted fusion algorithm to fuse the first depth map and the second depth map.

[0026] Optionally, the depth completion system based on multimodal modulation input is characterized in that the weighted fusion algorithm assigns different weight values ​​to each pixel in the first depth map and the second depth map respectively.

[0027] Optionally, the deep completion system based on multimodal modulation input is characterized in that the deep learning network is a convolutional neural network, and the post-processing operations include bilateral filtering and median filtering.

[0028] Optionally, the depth completion system based on multimodal modulation input is characterized in that, when processing the fourth depth map, the depth values ​​of corresponding pixels in the third depth map and the fourth depth map are compared, and a set deviation threshold is used to determine whether a pixel in the fourth depth map is an erroneous point, and then the erroneous point is removed.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] This invention significantly improves the effective information utilization rate of the original data by modulating multimodal depth input, separating high-confidence depth values ​​(fourth depth map) that are discarded by conventional filtering, and combining a dynamic reconstruction and fusion strategy based on distribution characteristics.

[0031] This invention solves the problem of conflict between near and far region optimization targets by using near-far reprojection segmentation and deep learning for targeted completion, while also combining post-processing mechanisms to suppress model prediction errors.

[0032] This invention utilizes detailed information from RGB images to guide depth map reprojection, mitigating the negative impact of spatial alignment deviations in multimodal data on completion accuracy. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort. Other features, objects, and advantages of the present invention will become more apparent by reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0034] Figure 1 This is a flowchart illustrating the steps of a depth completion method based on multimodal modulation input in an embodiment of the present invention.

[0035] Figure 2 This is a schematic diagram of a deep completion system based on multimodal modulation input in an embodiment of the present invention. Detailed Implementation

[0036] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0037] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0038] The present invention provides a deep completion method based on multimodal modulation input, which aims to solve the problems existing in the prior art.

[0039] The technical solutions of the present invention and how they solve the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0040] Figure 1 This is a flowchart illustrating the steps of a depth completion method based on multimodal modulation input in an embodiment of the present invention.

[0041] like Figure 1 As shown, the steps of a depth completion method based on multimodal modulation input in an embodiment of the present invention include:

[0042] Step S1: Using a multimodal modulation depth camera, acquire a first depth map and a second depth map, and perform depth fusion to obtain a third depth map. At the same time, acquire a fourth depth map composed of depth values ​​that were filtered out during the generation of the third depth map but whose confidence is greater than the depth threshold.

[0043] In this step, a multimodal modulation depth camera is a device capable of acquiring multiple depth information methods. It may combine different depth measurement principles, such as structured light and time-of-flight (ToF), to obtain more comprehensive and accurate depth data. The first depth map and the second depth map have different confidence levels at different distance ranges, meaning that the primary measurement ranges of the first and second depth maps are different.

[0044] The camera acquires a first depth map and a second depth map using different modalities or measurement methods. These two depth maps may differ in terms of accuracy, resolution, and measurement range. For example, one modality may be more suitable for high-precision close-range measurements, while the other modality may be better suited for coarse measurements at long distances.

[0045] The first and second depth maps are fused to combine their advantages and obtain a more accurate and complete third depth map. Fusion methods can include weighted averaging, confidence-based fusion, etc. For example, for each pixel, if the first depth map has a higher measurement confidence at that point, then the corresponding pixel value in the first depth map is given a higher weight.

[0046] During deep fusion, some depth values ​​are filtered out. Among these filtered-out depth values, some have a confidence level greater than a pre-set depth threshold. These filtered-out depth values ​​with confidence levels greater than the threshold are extracted to form a fourth depth map. This fourth depth map contains additional depth information that may be helpful for the final depth completion.

[0047] Step S2: Based on the distribution characteristics of the third depth map, process the fourth depth map, and merge the third depth map and the fourth depth map into a fifth depth map. According to the set depth threshold, divide the fifth depth map into a first near depth map that is less than the threshold and a first far depth map that is greater than the threshold.

[0048] In this step, the distribution characteristics of the third depth map can include the range of depth values ​​and the distribution density of depth values. Based on these characteristics, the fourth depth map is processed, for example, by performing interpolation, filtering, or other operations to make it more closely match the depth distribution of the third depth map.

[0049] The processed fourth depth map is then fused with the third depth map to obtain the fifth depth map. The purpose of this fusion is to integrate the additional depth information from the fourth depth map into the third depth map, thereby further improving the completeness of the depth map.

[0050] Based on a set depth threshold, the fifth depth map is divided into a first near depth map and a first far depth map. The first near depth map contains pixels with depth values ​​less than the threshold, corresponding to objects that are closer in the scene; the first far depth map contains pixels with depth values ​​greater than the threshold, corresponding to objects that are farther away in the scene. This segmentation helps in applying different processing strategies to objects at different distances.

[0051] Step S3: Obtain the corresponding RGB image, reproject the first near depth image and the first far depth image onto the RGB angle, and generate a second near depth image and a second far depth image with pixel-level alignment.

[0052] In this step, the RGB image is an image containing scene color information captured by a color camera. It is acquired at the same or similar time as the depth map and is used to provide texture and semantic information about the scene. The first near depth map and the first far depth map are reprojected onto the RGB image at different angles, so that the pixels of the depth map correspond one-to-one with the pixels of the RGB image, generating a pixel-aligned second near depth map and a second far depth map. The reprojection process requires knowledge of the relative positions and poses of the depth camera and the RGB camera, using these relationships to project the 3D points in the depth map onto the 2D plane of the RGB image.

[0053] Step S4: Input the second near-depth map and the RGB image into the deep learning network, perform a completion operation on the second near-depth map, post-process the model output, remove pixel depths with low confidence and / or exceeding the threshold, and output a dense third near-depth map.

[0054] In this step, the second near-depth map and the RGB image are fed into a pre-trained deep learning network. The deep learning network can be a convolutional neural network (CNN), a recurrent neural network (RNN), etc., which can learn the relationship between the depth map and the RGB image, thereby performing a completion operation on the second near-depth map.

[0055] Deep learning networks predict missing depth values ​​from the input second near-depth map and RGB image, thus completing the second near-depth map. By learning from a large number of depth map and RGB image pairs, the network can capture the structural and semantic information of the scene, thereby more accurately completing the depth map.

[0056] Post-processing is performed on the output of the deep learning network to remove pixel depths with low confidence and / or exceeding a threshold. Pixel depths with low confidence may be due to measurement errors or inaccurate network predictions; removing them can improve the quality of the depth map. Pixel depths exceeding the threshold may be outliers that do not conform to the actual scene and also need to be removed.

[0057] After post-processing, a dense third near-depth map is output. This depth map contains more complete and accurate depth information for near objects.

[0058] Step S5: The third near depth map and the second far depth map are stitched together to obtain the final depth map.

[0059] In this step, the completed third near depth map and second far depth map are stitched together to obtain the final depth map. The stitching process must ensure a smooth transition of depth values ​​between the third near depth map and the second far depth map at the stitching point, avoiding obvious stitching artifacts. The final depth map contains complete depth information for near and far objects in the scene and can be used for subsequent applications such as 3D reconstruction and object detection.

[0060] In some embodiments, the weighted fusion algorithm assigns different weight values ​​to each pixel in the first and second depth maps based on their respective confidence levels. The core idea of ​​the weighted fusion algorithm is to assign different weights to each corresponding pixel in the first and second depth maps according to their respective confidence levels; the higher the confidence level, the greater the proportion of that pixel in the fusion result. In this way, the advantages of the two depth maps can be combined to obtain a more accurate and reliable third depth map. First, it is necessary to determine the confidence level of each pixel in each depth map. The method for calculating the confidence level may vary depending on the depth measurement method; the following are some common calculation methods:

[0061] Signal strength is a key indicator of measurement reliability in some depth measurement techniques, such as time-of-flight methods. Higher signal strength generally indicates greater accuracy and confidence level.

[0062] Based on measurement error: If the measurement error for each pixel can be estimated, the smaller the error, the higher the confidence level. The confidence level can be calculated by taking the reciprocal of the measurement error.

[0063] To ensure the reasonableness of the weights, the calculated confidence scores need to be normalized so that the sum of the confidence scores of each pixel in the two depth maps is 1. Based on the normalized weights, the corresponding pixels in the first and second depth maps are weighted and summed to obtain the depth value of the corresponding pixel in the third depth map.

[0064] In some embodiments, when processing the fourth depth map, the depth values ​​of corresponding pixels in the third and fourth depth maps are compared. A set deviation threshold is used to determine whether a pixel in the fourth depth map is an error point, and these errors are removed. The third depth map is a relatively accurate and complete depth map obtained through depth fusion, while the fourth depth map includes depth values ​​that were filtered out during the fusion process but have a confidence level greater than the depth threshold. The deviation between corresponding pixels in the two depth maps is calculated by comparing their depth values. If the deviation exceeds a preset threshold, the pixel is considered an error point and removed from the fourth depth map to improve its quality. For each pixel in the fourth depth map, the corresponding pixel in the third depth map is found. It is assumed that the two depth maps have the same size and that the pixels are one-to-one. The deviation between the depth value of the pixel in the fourth depth map and the depth value of the corresponding pixel in the third depth map is calculated. The calculated deviation is compared with a preset deviation threshold. If the deviation is greater than the deviation threshold, the pixel is considered an error point. For pixels that are identified as erroneous, their depth values ​​are set to invalid values ​​(e.g., typically represented by 0 or a specific invalid value in a depth map).

[0065] In some embodiments, the deep learning network is a convolutional neural network (CNN). It learns the depth relationship between the RGB image and the second near-depth image through training, thereby completing the depth of the second near-depth image. CNNs have powerful feature extraction and pattern learning capabilities, making them well-suited for processing image data. In this depth completion method based on multimodal modulation input, the CNN is used to learn the depth relationship between the RGB image and the second near-depth image, thus completing the depth of the second near-depth image. The RGB image contains rich texture and semantic information, while the second near-depth image contains some depth information of nearby objects. The CNN can learn from a large amount of training data to uncover the intrinsic connection between these two images, and then predict the missing depth values ​​in the second near-depth image.

[0066] Collect a large number of RGB images and their corresponding second near-depth maps as training data. This data should cover different scenes, lighting conditions, and object types to improve the model's generalization ability.

[0067] The collected data is preprocessed, including image scaling and normalization. For example, the dimensions of the RGB image and the second near-depth image are adjusted to be the same, and the pixel values ​​are normalized to the [0,1] interval to speed up model training and improve training performance.

[0068] The preprocessed data is divided into training, validation, and test sets. The training set is used for training the model, the validation set is used to evaluate the model's performance and adjust its hyperparameters during training, and the test set is used to finally evaluate the model's generalization ability.

[0069] Network Architecture Design: Design a CNN model suitable for the depth completion task. Generally, a CNN model consists of convolutional layers, pooling layers, activation functions, and fully connected layers. In the depth completion task, an encoder-decoder structure is typically used. The encoder extracts features from the RGB image and the second near-depth map, and the decoder generates the completed depth map based on the extracted features.

[0070] Convolutional layers: Convolutional layers extract local features of an image by sliding convolutional kernels across the input image. Different sizes and numbers of convolutional kernels can be used to extract features at different scales.

[0071] Pooling layers: Pooling layers are used to reduce the size of feature maps, decrease the number of parameters in the model, and enhance the model's robustness. Commonly used pooling operations include max pooling and average pooling.

[0072] Activation functions: After convolutional and fully connected layers, activation functions are typically used to introduce non-linearity, enabling the model to learn more complex patterns. Commonly used activation functions include ReLU and Sigmoid.

[0073] Fully connected layer: The fully connected layer is used to map the extracted features to the final depth map space.

[0074] Define the loss function: Choose an appropriate loss function to measure the difference between the depth map predicted by the model and the true depth map. In depth completion tasks, commonly used loss functions include Mean Squared Error (MSE) loss and Mean Absolute Error (MAE) loss.

[0075] Optimizer Selection: Choose an appropriate optimizer to update the model's parameters to minimize the loss function. Commonly used optimizers include Stochastic Gradient Descent (SGD), Adam, and RMSProp.

[0076] Training process: Training data is input into the model in batches. The model's output is calculated through forward propagation, the loss value is calculated based on the loss function, the gradient is calculated through backpropagation, and finally, the optimizer is used to update the model's parameters. This process is repeated until the model's performance reaches a satisfactory level.

[0077] Evaluation using a validation set: During training, periodically evaluate the model's performance using a validation set. Based on the loss value and other evaluation metrics (such as depth error, accuracy, etc.) on the validation set, adjust the model's hyperparameters (such as learning rate, batch size, etc.) to prevent overfitting or underfitting.

[0078] Evaluation using the test set: After model training is complete, evaluate the model's final performance using the test set. Calculate the loss value and other evaluation metrics on the test set to assess the model's generalization ability.

[0079] Input data: Input the second near-depth map to be completed and the corresponding RGB image into the trained CNN model.

[0080] Forward propagation: The model calculates the output through forward propagation to obtain the completed depth map.

[0081] Post-processing: Post-process the depth map output by the model, such as removing pixel depths with low confidence and / or exceeding the threshold, to improve the quality of the depth map.

[0082] Figure 2 This is a schematic diagram of a deep completion system based on multimodal modulation input in an embodiment of the present invention.

[0083] like Figure 2 As shown, an embodiment of the present invention provides a deep completion system based on multimodal modulation input, comprising:

[0084] The acquisition module is used to acquire a first depth map and a second depth map through a multimodal modulation depth camera, and perform depth fusion to obtain a third depth map. At the same time, it acquires a fourth depth map composed of depth values ​​that were filtered out during the generation of the third depth map but whose confidence is greater than the depth threshold.

[0085] The fusion module is used to process the fourth depth map according to the distribution characteristics of the third depth map, and fuse the third depth map and the fourth depth map into a fifth depth map. According to a set depth threshold, the fifth depth map is divided into a first near depth map less than the threshold and a first far depth map greater than the threshold.

[0086] The projection module is used to acquire the corresponding RGB image, reproject the first near depth image and the first far depth image onto the RGB angle, and generate a second near depth image and a second far depth image with pixel-level alignment.

[0087] The output module is used to input the second near-depth map and the RGB image into the deep learning network, perform a completion operation on the second near-depth map, perform post-processing on the model output, remove pixel depths with low confidence and / or exceeding the threshold, and output a dense third near-depth map.

[0088] The stitching module is used to stitch the third near-depth map and the second far-depth map together to obtain the final depth map.

[0089] Specifically, the acquisition module is primarily responsible for acquiring depth data from a multimodal modulation depth camera, performing preliminary depth fusion processing, and extracting valuable additional depth information. It acquires a first depth map and a second depth map using different measurement modes of the multimodal modulation depth camera. Different measurement modes may have different accuracies, measurement ranges, and applicable scenarios; for example, one mode may be more suitable for close-range high-precision measurements, while another mode may have advantages for long-range measurements. The acquired first and second depth maps are fused to combine their advantages, resulting in a more accurate and complete third depth map. The fusion method can be based on a weighted average, assigning different weights according to the confidence level of each pixel; or it can employ more complex algorithms, such as confidence-based fusion or feature-matching-based fusion. During the depth fusion process, some depth values ​​are filtered out. This module selects those filtered-out depth values ​​whose confidence level is greater than a pre-set depth threshold and combines these depth values ​​into a fourth depth map. The fourth depth map contains additional depth information that may be helpful for the final depth completion.

[0090] The fusion module processes the fourth depth map based on the distribution characteristics of the third depth map, then merges the two and segments the fused depth map according to a depth threshold.

[0091] Analyze the distribution characteristics of the third depth map, such as the range and density of depth values. Based on these characteristics, process the fourth depth map, performing operations such as interpolation and filtering, to make the fourth depth map more closely match the third depth map in terms of depth distribution. Furthermore, by comparing the depth values ​​of corresponding pixels in the third and fourth depth maps, determine whether a pixel in the fourth depth map is an error point based on a set deviation threshold, and remove any erroneous pixels.

[0092] The processed fourth depth map is then fused with the third depth map to obtain the fifth depth map. The purpose of this fusion is to integrate the additional depth information from the fourth depth map into the third depth map, thereby further improving the completeness of the depth map.

[0093] The fifth depth map is segmented into a first near depth map and a first far depth map based on a pre-defined depth threshold. The first near depth map contains pixels with depth values ​​less than the threshold, corresponding to objects that are closer in the scene; the first far depth map contains pixels with depth values ​​greater than the threshold, corresponding to objects that are farther away in the scene. This segmentation helps in applying different processing strategies to objects at different distances.

[0094] The main task of the projection module is to acquire the corresponding RGB image and reproject the first near depth image and the first far depth image onto the RGB image at an angle to achieve pixel-level alignment.

[0095] RGB Image Acquisition: Acquire RGB images captured at the same or similar time points as the depth map. RGB images contain color information of the scene, providing texture and semantic cues for subsequent depth completion.

[0096] Reprojection operation: Utilizing the relative position and orientation of the depth camera and the RGB camera, the 3D points in the first near depth map and the first far depth map are projected onto the 2D plane of the RGB image, so that the pixels of the depth map correspond one-to-one with the pixels of the RGB image, generating a pixel-aligned second near depth map and a second far depth map.

[0097] The output module uses a deep learning network to complete the second near-depth map and performs post-processing on the model output to produce a high-quality third near-depth map.

[0098] Input to the deep learning network: The second near-depth map, aligned at the pixel level, and the corresponding RGB image are fed into a pre-trained deep learning network (such as a convolutional neural network, CNN). The deep learning network can learn the depth relationship between the RGB image and the second near-depth map, thereby predicting the missing depth values ​​in the second near-depth map.

[0099] Depth completion operation: Deep learning networks learn from a large amount of training data to uncover the intrinsic relationship between RGB images and the second near-depth map, and then complete the second near-depth map. The network output is the completed depth map.

[0100] Post-processing: The output of the deep learning network is post-processed to remove pixel depths with low confidence and / or exceeding a threshold. Pixel depths with low confidence may be due to measurement errors or inaccurate network predictions, while pixel depths exceeding the threshold may be outliers that do not conform to the actual scene. Removing these pixel depths can improve the quality of the depth map.

[0101] Output a third near depth map: After post-processing, a dense third near depth map is output, which contains more complete and accurate depth information of near objects.

[0102] The stitching module stitches together the completed third near-depth map and the second far-depth map to obtain the final depth map.

[0103] Depth map stitching: The third near depth map and the second far depth map are stitched together to ensure a natural transition in depth values ​​at the stitching point and avoid obvious stitching artifacts. Smoothing techniques such as linear interpolation and Gaussian filtering can be used to achieve a natural transition.

[0104] Output final depth map: After stitching, the final depth map containing complete depth information of near and far objects in the scene is output. This depth map can be used for subsequent 3D reconstruction, object detection and other applications.

[0105] In some embodiments, the acquisition module employs a weighted fusion algorithm to fuse the first depth map and the second depth map. The core idea behind using this algorithm is to assign different weights to pixels in each depth map based on their confidence level; the higher the confidence level, the greater the weight of that pixel in the fusion result. This allows for the combined use of the advantages of both depth maps, resulting in a more accurate and reliable third depth map.

[0106] In some embodiments, the weighted fusion algorithm assigns different weight values ​​to each pixel in the first and second depth maps based on their confidence levels. In depth map fusion scenarios, since the first and second depth maps may be obtained through different depth measurement methods or under different conditions, the reliability (i.e., confidence level) of each pixel varies. The weighted fusion algorithm allocates weights based on the confidence level of each pixel, fusing the two depth maps so that pixels with higher confidence levels contribute more to the final result, thereby improving the accuracy and reliability of the fused depth map.

[0107] In some embodiments, the deep learning network is a convolutional neural network (CNN), and post-processing operations include bilateral filtering and median filtering. CNNs possess powerful feature extraction and pattern learning capabilities, making them well-suited for processing image data. In the depth completion task, CNNs are used to learn the depth relationship between the RGB image and a second near-depth map, thereby completing the second near-depth map. The network structure typically employs an encoder-decoder architecture. The encoder extracts features from the RGB image and the second near-depth map through a series of convolutional and pooling layers, gradually reducing the dimensionality of the input data while extracting higher-level semantic features. The decoder then uses deconvolutional layers (or upsampling layers) to map the features extracted by the encoder back to the original depth map size, predicting the missing depth values.

[0108] During training, a large number of RGB images and their corresponding depth maps are used as training data. An appropriate loss function (such as mean squared error loss) is defined to measure the difference between the depth maps predicted by the model and the true depth maps. The model parameters are updated through the backpropagation algorithm, continuously optimizing the model so that it can accurately learn the relationship between RGB images and depth maps.

[0109] Bilateral filtering is a nonlinear filtering method that smooths an image while preserving its edge information. It combines information from both the spatial and value domains; for each pixel, it considers not only the spatial distance to its neighbors but also the differences in pixel values. Specifically, bilateral filtering performs a weighted average on each pixel using a bilateral filter, with the weights derived from the product of a spatial Gaussian kernel and a value-domain Gaussian kernel. The spatial kernel measures the spatial distance between pixels, while the value-domain kernel measures the differences in pixel values.

[0110] In depth completion, the depth map output by a CNN may contain some noise and discontinuous regions. Bilateral filtering can effectively smooth out this noise while preserving edge information in the depth map, making the transitions in the depth map more natural. For example, at the boundaries of objects, bilateral filtering can avoid boundary blurring caused by over-smoothing, thereby improving the quality of the depth map.

[0111] Median filtering is a non-linear filtering method that sorts the pixel values ​​in the neighborhood of each pixel and then replaces the value of that pixel with its median. Median filtering is very effective at suppressing impulse noise such as salt-and-pepper noise.

[0112] The depth map output by a CNN may contain isolated outliers, which could be due to measurement errors or inaccurate model predictions. Median filtering can effectively remove these outliers, making the depth map smoother and more continuous. For example, abrupt depth jumps may occur in the depth map; median filtering can replace these jump values ​​with the median of the neighborhood, thereby improving the quality of the depth map.

[0113] In some embodiments, when processing the fourth depth map, the depth values ​​of corresponding pixels in the third and fourth depth maps are compared. A set deviation threshold is used to determine whether a pixel in the fourth depth map is an error and removes it. The fourth depth map contains depth values ​​that were filtered out during the generation of the third depth map but have a confidence level greater than the depth threshold. However, these depth values ​​may be erroneous or inaccurate. By comparing the fourth depth map with the relatively accurate third depth map, the deviation of the corresponding pixel depth values ​​is calculated. Based on the set deviation threshold, error points can be identified and removed, thereby improving the quality of the fourth depth map. Because the depth values ​​of corresponding pixels in the two depth maps need to be compared, it is necessary to traverse every pixel in the fourth depth map. Assuming the third and fourth depth maps have the same size (i.e., the height and width are the same), the pixels are one-to-one. Two nested loops can be used to implement the traversal: the outer loop controls the rows, and the inner loop controls the columns. For each pixel in the fourth depth map, its depth value is obtained, and the corresponding pixel in the third depth map is found and its depth value is obtained. Then, the deviation between these two depth values ​​is calculated. The calculated deviation is compared with a pre-set deviation threshold. If the deviation is greater than the threshold, the pixel is considered an error. The deviation threshold needs to be determined based on the specific application scenario and the characteristics of the depth map; it is an empirical value, and different scenarios may require different thresholds. For pixels identified as errors, their depth values ​​are set to invalid values. In depth maps, common ways to represent invalid values ​​include setting the depth value to 0 or a specific value outside the normal depth range. This allows these invalid values ​​to be treated specially in subsequent processing, avoiding adverse effects on the results.

[0114] The various embodiments described in this specification are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0115] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A depth completion method based on multimodal modulation input, characterized in that, include: Step S1: Using a multimodal modulation depth camera, acquire a first depth map and a second depth map, and perform depth fusion to obtain a third depth map. At the same time, acquire a fourth depth map composed of depth values ​​that were filtered out during the generation of the third depth map but whose confidence is greater than the depth threshold. The first depth map and the second depth map have different confidence levels in different distance ranges. Step S2: Based on the distribution characteristics of the third depth map, process the fourth depth map, and merge the third depth map and the fourth depth map into a fifth depth map. According to the set depth threshold, divide the fifth depth map into a first near depth map that is less than the threshold and a first far depth map that is greater than the threshold. Step S3: Obtain the corresponding RGB image, reproject the first near depth image and the first far depth image onto the RGB angle, and generate a second near depth image and a second far depth image with pixel-level alignment; Step S4: Input the second near depth map and the RGB image into the deep learning network, perform a completion operation on the second near depth map, post-process the model output, remove pixel depths with low confidence and / or exceeding the threshold, and output a dense third near depth map; Step S5: The third near depth map and the second far depth map are stitched together to obtain the final depth map.

2. The depth completion method based on multimodal modulation input according to claim 1, characterized in that, The deep fusion uses a weighted fusion algorithm to fuse the first depth map and the second depth map.

3. The depth completion method based on multimodal modulation input according to claim 2, characterized in that, The weighted fusion algorithm assigns different weight values ​​to each pixel in the first depth map and the second depth map, respectively.

4. The depth completion method based on multimodal modulation input according to claim 1, characterized in that, When processing the fourth depth map, the depth values ​​of corresponding pixels in the third depth map and the fourth depth map are compared. Based on the set deviation threshold, it is determined whether the pixels in the fourth depth map are erroneous points and they are removed.

5. The depth completion method based on multimodal modulation input according to claim 1, characterized in that, The deep learning network is a convolutional neural network, which learns the depth relationship between the RGB image and the second near-depth map through training, thereby completing the depth of the second near-depth map.

6. A deep completion system based on multimodal modulation input, characterized in that, include: The acquisition module is used to acquire a first depth map and a second depth map through a multimodal modulation depth camera, and perform depth fusion to obtain a third depth map. At the same time, it acquires a fourth depth map composed of depth values ​​that were filtered out during the generation of the third depth map but whose confidence is greater than the depth threshold. The fusion module is used to process the fourth depth map according to the distribution characteristics of the third depth map, and fuse the third depth map and the fourth depth map into a fifth depth map. According to a set depth threshold, the fifth depth map is divided into a first near depth map less than the threshold and a first far depth map greater than the threshold. The projection module is used to acquire the corresponding RGB image, reproject the first near depth image and the first far depth image onto the RGB angle, and generate a second near depth image and a second far depth image with pixel-level alignment. The output module is used to input the second near-depth map and the RGB image into the deep learning network, perform a completion operation on the second near-depth map, perform post-processing on the model output, remove pixel depths with low confidence and / or exceeding the threshold, and output a dense third near-depth map. The stitching module is used to stitch the third near-depth map and the second far-depth map together to obtain the final depth map.

7. The depth completion system based on multimodal modulation input according to claim 6, characterized in that, The acquisition module uses a weighted fusion algorithm to fuse the first depth map and the second depth map.

8. The depth completion system based on multimodal modulation input according to claim 7, characterized in that, The weighted fusion algorithm assigns different weight values ​​to each pixel in the first depth map and the second depth map, respectively.

9. The depth completion system based on multimodal modulation input according to claim 6, characterized in that, The deep learning network is a convolutional neural network, and the post-processing operations include bilateral filtering and median filtering.

10. The depth completion system based on multimodal modulation input according to claim 6, characterized in that, When processing the fourth depth map, the depth values ​​of corresponding pixels in the third depth map and the fourth depth map are compared. Based on the set deviation threshold, it is determined whether the pixels in the fourth depth map are erroneous points and they are removed.

Images