Depth completion method, system and device based on multi-modal modulation input and storage medium

Abstract

The invention discloses a depth completion method, system and device based on multi-modal modulation input, and a storage medium. The method comprises the following steps: S1, generating a full-scale high-quality depth map; meanwhile, generating an auxiliary depth map; s2, establishing a geometric constraint model based on the depth distribution characteristics of the full-scale high-quality depth map, generating an optimized depth map, and segmenting the optimized depth map into a first short-distance sub-map and a first long-distance sub-map according to a preset depth threshold; s3, respectively carrying out view angle re-projection on the first short-distance sub-graph and the first long-distance sub-graph to generate a second short-distance sub-graph and a second long-distance sub-graph; s4, inputting the second short-distance sub-image and the RGB image into the depth completion network for depth completion, and outputting a dense depth image after confidence threshold filtering and preset depth threshold filtering; and S5, edge fusion based on depth continuity is carried out on the dense depth map and the second long-distance sub-map, and a final depth map is generated.

Description

Technical Field

[0001] This invention relates to the field of deep completion technology, and more specifically, to a deep completion method, system, device, and storage medium based on multimodal modulation input. Background Technology

[0002] In recent years, with the development of automation and intelligent technologies, depth cameras have been used in more and more scenarios, such as household robot vacuum cleaners, robots, autonomous driving and industrial inspection, while the requirements for depth quality are also getting higher and higher.

[0003] Common 3D depth cameras include binocular, structured light, Time-of-Flight (ToF), and LiDAR, but these cameras all have some limitations. For example:

[0004] Dual-lens RGB cameras are highly dependent on RGB quality, resulting in poor depth quality in scenes with weak textures, and also incurring high computational costs.

[0005] Active infrared cameras offer relatively good depth quality, but high-quality depth maps require significant computational resources.

[0006] Structured light is only suitable for close-range scenarios; LiDAR depth maps are relatively sparse and costly; traditional ToF cameras are more suitable for general scenarios, but in some scenarios their depth maps are of poor quality and suffer from problems such as missing data, multipath propagation, and depth errors.

[0007] A single camera can only generate one or more different types of depth maps, which often requires very high computing power and is costly when meeting the needs of navigation and obstacle avoidance.

[0008] The above background information is provided only to aid in understanding the inventive concept and technical solution of this invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above information was disclosed on the filing date of this patent application, the above background information should not be used to evaluate the novelty and inventiveness of this application. Summary of the Invention

[0009] To this end, the present invention utilizes several types of depth maps with different characteristics, and then combines them with RGB images. Using a deep learning network, these depth maps are combined and optimized to output high-quality, high-precision depth images that are dense at close range and sparse at long range. This satisfies the needs of navigation and obstacle avoidance while requiring only a small amount of computing power.

[0010] In a first aspect, the present invention provides a depth completion method based on multimodal modulation input, characterized in that it includes:

[0011] Step S1: Acquire a dense depth map at very close range, a sparse depth map at near-mid range, and a sparse depth map at far range using a multimodal modulation depth camera. Perform confidence weighting and spatial consistency fusion on the three depth maps to generate a high-quality depth map across the entire range. At the same time, extract the depth point set that was filtered out during the fusion process because its confidence was lower than the first threshold, but whose confidence was higher than the second threshold, to generate an auxiliary depth map.

[0012] Step S2: Based on the depth distribution characteristics of the full-range high-quality depth map, establish a geometric constraint model, remove outliers from the auxiliary depth map, and fuse it with the full-range high-quality depth map using multi-scale features to generate an optimized depth map. Then, according to a preset depth threshold, divide the optimized depth map into a first near-range sub-map and a first far-range sub-map.

[0013] Step S3: Based on the joint calibration parameters of the depth camera and the RGB camera, perform viewpoint reprojection on the first near sub-image and the first far sub-image respectively to generate a second near sub-image and a second far sub-image aligned with the pixels of the RGB image;

[0014] Step S4: Construct a depth completion network with an attention mechanism, input the second near sub-image and the RGB image into the depth completion network for depth completion, and output a dense depth map after confidence threshold filtering and preset depth threshold filtering;

[0015] Step S5: Perform edge fusion based on depth continuity on the dense depth map and the second distant sub-map to generate a final depth map containing near-distance high-density and far-distance low-density depth information.

[0016] Optionally, the depth completion method based on multimodal modulation input is characterized in that, during confidence-weighted fusion in step S1, the following is included:

[0017] Step S11: Assign dynamic weighting coefficients that are inversely correlated with measurement error to the extremely close-range dense depth map;

[0018] Step S12: Apply adaptive weight allocation based on signal-to-noise ratio to the near-mid-range sparse depth map;

[0019] Step S13: Establish a probability distribution model for the overlapping region of the extremely close-range dense depth map, the near-mid-range sparse depth map, and the far-range sparse depth map, and perform confidence-based iterative optimization.

[0020] Optionally, the depth completion method based on multimodal modulation input is characterized in that the geometric constraint model includes local plane constraints based on the surface normal estimation of the full-range high-quality depth map, physical consistency verification based on depth-intensity correlation, and geometric anomaly detection of the auxiliary depth map using a random sampling consistency algorithm.

[0021] Optionally, the depth completion method based on multimodal modulation input is characterized in that the viewpoint reprojection in step S3 includes:

[0022] Step S31: Generate an alignment error correction map using the edge information of the RGB image;

[0023] Step S32: Apply motion compensation based on optical flow estimation to adjust the reprojected depth value;

[0024] Step S33: Perform joint bilateral filtering on the first near-field sub-image and the first far-field sub-image, and optimize the edge alignment of the depth map guided by the RGB image.

[0025] Optionally, the depth completion method based on multimodal modulation input is characterized in that the depth completion network in step S4 comprises:

[0026] A cascaded residual encoder module is used to extract RGB-D multimodal features;

[0027] Deformable convolution-guided cross-modal attention fusion module;

[0028] A multi-scale decoder that includes depth uncertainty estimation.

[0029] Optionally, the aforementioned deep completion method based on multimodal modulation input is characterized in that the training process of the deep completion network employs a synthetic occlusion enhancement strategy, specifically including:

[0030] Step S41: Randomly inject deep missing regions simulating ToF noise patterns into the training data;

[0031] Step S42: Apply texture perturbation corresponding to the depth occlusion region to the RGB image;

[0032] Step S43: Generate realistic long-range depth artifact samples through adversarial training;

[0033] Step S44: Gradually increase the complexity of synthetic occlusion using a course learning strategy.

[0034] Optionally, the depth completion method based on multimodal modulation input is characterized in that the edge fusion in step S5 specifically includes:

[0035] Step S51: Establish a transition zone within the depth threshold range;

[0036] Step S52: Use the Poisson fusion algorithm to achieve a smooth transition of the depth gradient;

[0037] Step S53: Perform outlier correction on the fusion boundary based on the adjacent depth ratio.

[0038] Secondly, the present invention provides a depth completion system based on multimodal modulation input, used to implement the depth completion method based on multimodal modulation input described in any of the preceding claims, characterized in that it includes:

[0039] The initial generation module is used to acquire a dense depth map at very close range, a sparse depth map at near-mid range, and a sparse depth map at far range using a multimodal modulated depth camera. It performs confidence weighting and spatial consistency fusion on the three depth maps to generate a high-quality depth map across the entire range. At the same time, it extracts the depth point set that was filtered out during the fusion process because its confidence was lower than the first threshold, but whose confidence was higher than the second threshold, to generate an auxiliary depth map.

[0040] The sub-map generation module is used to establish a geometric constraint model based on the depth distribution characteristics of the full-range high-quality depth map, remove outliers from the auxiliary depth map and perform multi-scale feature fusion with the full-range high-quality depth map to generate an optimized depth map, and divide the optimized depth map into a first near sub-map and a first far sub-map according to a preset depth threshold.

[0041] The reprojection module is used to perform viewpoint reprojection on the first near sub-image and the first far sub-image respectively according to the joint calibration parameters of the depth camera and the RGB camera, and generate a second near sub-image and a second far sub-image that are aligned with the pixels of the RGB image.

[0042] A dense module is used to construct a depth completion network that includes an attention mechanism. The second near sub-image and the RGB image are input into the depth completion network for depth completion. After confidence threshold filtering and the preset depth threshold filtering, a dense depth map is output.

[0043] The fusion module is used to perform edge fusion based on depth continuity on the dense depth map and the second distant sub-map to generate a final depth map containing near-distance high-density and far-distance low-density depth information.

[0044] Thirdly, the present invention provides a depth completion device based on multimodal modulation input, characterized in that it includes:

[0045] processor;

[0046] A memory in which executable instructions of the processor are stored;

[0047] The processor is configured to perform the steps of any of the preceding deep completion methods based on multimodal modulation input by executing the executable instructions.

[0048] Fourthly, the present invention provides a computer-readable storage medium for storing a program, characterized in that, when the program is executed, it implements the steps of the depth completion method based on multimodal modulation input described in any of the preceding claims.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] This invention achieves sub-millimeter accuracy at extremely close distances by fusion of hierarchical depth acquisition (dense near distance + sparse mid-to-far distance) and confidence level. It provides high-density depth information at close range and low-density depth information at far distances, enabling the acquisition of corresponding depth value accuracy at different distances. This satisfies the needs of various navigation and obstacle avoidance while reducing the amount of data processing and the demand for computing power.

[0051] In this invention, the depth map and RGB image are aligned across modalities and attention-based completion, which enables the RGB texture information to repair the occlusion and loss of depth values, thus improving the success rate of depth recovery. At the same time, by integrating the physical measurement of depth values ​​with the semantic understanding of RGB, the depth integrity is significantly improved in strong light / low light scenes.

[0052] This invention optimizes the depth map by segmenting it using a preset depth threshold, and uses deep learning to complete the near-field region while retaining sparse point clouds in the far-field region, significantly improving computational efficiency compared to global densification; and restricts the propagation of long-distance errors to local regions through a physical constraint model. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort. Other features, objects, and advantages of the present invention will become more apparent by reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0054] Figure 1 This is a flowchart illustrating the steps of a depth completion method based on multimodal modulation input in an embodiment of the present invention.

[0055] Figure 2 This is a flowchart illustrating a step of confidence-weighted fusion in an embodiment of the present invention;

[0056] Figure 3This is a flowchart illustrating the steps of a viewpoint reprojection in an embodiment of the present invention;

[0057] Figure 4 This is a flowchart illustrating the steps of training a deep completion network in an embodiment of the present invention.

[0058] Figure 5 This is a flowchart illustrating the steps of edge blending in an embodiment of the present invention;

[0059] Figure 6 This is a schematic diagram of the structure of a deep completion system based on multimodal modulation input in an embodiment of the present invention;

[0060] Figure 7 This is a schematic diagram of the structure of a depth completion device based on multimodal modulation input according to an embodiment of the present invention; and

[0061] Figure 8 This is a schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Detailed Implementation

[0062] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0063] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0064] The present invention provides a deep completion method based on multimodal modulation input, which aims to solve the problems existing in the prior art.

[0065] The technical solutions of the present invention and how they solve the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0066] This invention utilizes several types of depth maps with different characteristics, and then combines them with RGB images. Using a deep learning network, these depth maps are combined and optimized to output high-quality, high-precision depth images that are dense at close range and sparse at long range. This satisfies the needs of navigation and obstacle avoidance while requiring only a small amount of computing power.

[0067] Figure 1 This is a flowchart illustrating the steps of a depth completion method based on multimodal modulation input in an embodiment of the present invention.

[0068] like Figure 1 As shown, the steps of a depth completion method based on multimodal modulation input in an embodiment of the present invention include:

[0069] Step S1: Acquire a dense depth map at very close range, a sparse depth map at near-mid range, and a sparse depth map at far range using a multimodal modulation depth camera. Perform confidence weighting and spatial consistency fusion on the three depth maps to generate a high-quality depth map across the entire range. At the same time, extract the depth point set that was filtered out during the fusion process because its confidence was lower than the first threshold, but whose confidence was higher than the second threshold, to generate an auxiliary depth map.

[0070] In this step, the depth maps acquired by the multimodal modulation depth camera at different distances possess different characteristics. The extremely close-range dense depth map provides rich depth information in the near-range region, while the near-mid-range and far-range sparse depth maps supplement the depth information at mid-to-long-range distances. By using confidence-weighted and spatial consistency fusion, the advantages of the three depth maps can be comprehensively utilized to generate a high-quality depth map across the entire measurement range, improving the accuracy and completeness of depth information throughout the entire range. Simultaneously, depth point sets that were filtered out due to low confidence but still possess a certain degree of reliability are extracted to generate auxiliary depth maps, providing additional information for subsequent optimization.

[0071] Multimodal modulation depth cameras employ different modulation methods or sensor configurations to acquire depth maps at extremely close, near-mid, and long distances, ensuring that the different depth maps are acquired simultaneously. This guarantees temporal alignment during post-processing, making subsequent fusion and other processing effective. For example, high-frequency modulated light sources (such as structured light) can generate dense depth maps at extremely close distances (high precision but limited range); low-frequency modulation or Time-of-Flight (ToF) technology can acquire sparse depth maps (covering farther distances but with lower resolution), including near-mid-range and long-range sparse depth maps.

[0072] A confidence value is assigned to each pixel in each depth map. The confidence value can be determined based on factors such as camera characteristics and measurement environment. During the fusion process, the corresponding pixels of the three depth maps are weighted and summed according to their confidence values. Spatial consistency is also considered, for example, by using filtering algorithms (such as bilateral filtering) to smooth the fusion result and avoid obvious boundary discontinuities.

[0073] During the fusion process, a first threshold and a second threshold are set. Depth points with a confidence level below the first threshold are considered unreliable and are filtered out, while depth points with a confidence level above the second threshold are retained. These points are then combined into a depth point set to generate an auxiliary depth map.

[0074] Spatial consistency fusion optimization is performed using bilateral filtering or Markov random fields (MRF) to preserve edge consistency and filter out isolated noise points.

[0075] Step S2: Based on the depth distribution characteristics of the full-range high-quality depth map, establish a geometric constraint model, remove outliers from the auxiliary depth map, and fuse it with the full-range high-quality depth map using multi-scale features to generate an optimized depth map. Then, based on a preset depth threshold, divide the optimized depth map into a first near-range sub-map and a first far-range sub-map.

[0076] In this step, a geometric constraint model is established based on the depth distribution features of the full-range high-quality depth map, which can model the geometric structure of the scene. This model is used to remove outliers from the auxiliary depth map, eliminating abnormal points that do not conform to the scene's geometry and improving the quality of the auxiliary depth map. Then, the processed auxiliary depth map is fused with the full-range high-quality depth map using multi-scale features to further optimize the detail and accuracy of the depth map. Finally, the optimized depth map is segmented into near-range and far-range sub-maps based on a preset depth threshold for separate processing in subsequent steps.

[0077] Analyze the depth distribution of a full-range high-quality depth map, such as calculating features like depth gradient and curvature. Based on these features, establish a geometric constraint model, which can be represented as a function or a set of rules to determine whether a depth point conforms to the geometric structure of the scene.

[0078] Each depth point in the auxiliary depth map is substituted into the geometric constraint model for judgment. If a depth point does not meet the constraints of the model, it is regarded as an outlier and removed.

[0079] The processed auxiliary depth map and the full-scale high-quality depth map are used to extract features at different scales, for example, by using different layers of a convolutional neural network (CNN) to extract features at different scales. These features are then fused, for example, by concatenation or weighted summation, to generate an optimized depth map.

[0080] Based on a preset depth threshold, pixels with depth values ​​less than the threshold in the optimized depth map are classified as the first near-field sub-map, and pixels with depth values ​​greater than the threshold are classified as the first far-field sub-map.

[0081] Step S3: Based on the joint calibration parameters of the depth camera and the RGB camera, perform viewpoint reprojection on the first near sub-image and the first far sub-image respectively to generate a second near sub-image and a second far sub-image aligned with the pixels of the RGB image.

[0082] In this step, the depth camera and the RGB camera have different viewpoints. By reprojecting the viewpoints of the first near-field sub-image and the first far-field sub-image, the viewpoints can be converted to the same viewpoints as the RGB image, achieving pixel alignment between the depth map and the RGB image. This allows for better utilization of the texture information of the RGB image to supplement the depth information during subsequent depth completion.

[0083] For RGBD cameras, the viewpoints of the depth map and RGB map are calibrated and can be directly converted using intrinsic and extrinsic parameters. If the depth camera and RGB camera are two independent cameras, joint calibration is required before using them to obtain their relative position and orientation relationships, i.e., joint calibration parameters. These parameters can be obtained using calibration objects such as checkerboard patterns. In this embodiment, those skilled in the art will understand that the depth camera and RGB camera can be two independent cameras or a single camera, both of which fall within the scope of this invention.

[0084] Based on the joint calibration parameters, each pixel in the first near sub-image and the first far sub-image is projected into the coordinate system of the RGB image using the projection transformation formula, generating a second near sub-image and a second far sub-image aligned with the pixels of the RGB image.

[0085] Step S4: Construct a depth completion network that includes an attention mechanism. Input the second near sub-image and the RGB image into the depth completion network for depth completion. After confidence threshold filtering and the preset depth threshold filtering, output a dense depth map.

[0086] In this step, a depth completion network incorporating an attention mechanism is constructed. This mechanism automatically focuses on important regions in the depth map and RGB image, improving the depth completion effect. The second nearest sub-image and the RGB image are input into this network for depth completion. Through the network's learning and inference capabilities, missing depth information is predicted. Finally, unreliable predictions are removed using confidence threshold filtering and a preset depth threshold, resulting in a dense depth map.

[0087] Design a deep completion network that incorporates an attention mechanism, such as adding an attention module to a convolutional neural network, like a channel attention module (SE module) or a spatial attention module (CBAM module). The network's input is a second nearest sub-image and an RGB image, and its output is the predicted depth map.

[0088] The second near-field sub-image and the RGB image are input into the depth completion network, and the predicted depth map is obtained through forward propagation.

[0089] Each pixel in the predicted depth map is assigned a confidence value. Pixels with confidence values ​​below the confidence threshold are filtered out. Simultaneously, pixels with depth values ​​that do not meet the requirements are removed based on a preset depth threshold, resulting in a dense depth map.

[0090] Step S5: Perform edge fusion based on depth continuity on the dense depth map and the second distant sub-map to generate a final depth map containing near-distance high-density and far-distance low-density depth information.

[0091] In this step, the dense depth map obtained from the previous steps mainly contains near-range depth information, while the second far-range sub-map contains far-range depth information. Through edge fusion based on depth continuity, these two parts of depth information can be smoothly transitioned to generate a final depth map containing near-range high-density and far-range low-density depth information, making the entire depth map more natural and continuous.

[0092] Analyze the depth value changes in the edge regions of the dense depth map and the second distant submap to determine the continuity of depth. The depth gradient of the edge regions can be calculated, and the continuity of depth can be determined based on the gradient value.

[0093] Based on the depth continuity analysis, a suitable fusion algorithm (such as weighted averaging) is used to fuse the edge regions of the dense depth map and the second distant sub-map. During the fusion process, regions with continuous depth can be fused with larger weights, while regions with discontinuous depth can be fused with smaller weights to avoid obvious boundary discontinuities. The final result is a depth map containing near-range high-density and far-range low-density depth information.

[0094] Figure 2 This is a flowchart illustrating a step in performing confidence-weighted fusion according to an embodiment of the present invention. Figure 2 As shown, a step in performing confidence-weighted fusion in an embodiment of the present invention includes:

[0095] Step S11: Assign a dynamic weighting coefficient that is inversely correlated with the measurement error to the extremely close-range dense depth map.

[0096] In this step, the extremely close-range dense depth map provides rich depth information in the close-range region. However, due to factors such as measurement principles and environmental interference, the measurement error may vary at different locations. By assigning a dynamic weighting coefficient that is inversely correlated with the measurement error, regions with smaller measurement errors can be given greater weight in the fusion process, thereby more effectively utilizing the reliable depth information of these regions and improving the accuracy of the fusion results.

[0097] First, it is necessary to evaluate the measurement error of each pixel in the dense depth map at extremely close range. The measurement error can be estimated in several ways, such as by establishing an error model based on the measurement principles and characteristics of the depth camera to calculate the theoretical error of each pixel; or by estimating the actual measurement error by taking the variance of multiple measurements.

[0098] Based on the measurement error obtained from the evaluation, a dynamic weight coefficient for each pixel is calculated. Typically, the weight coefficient is inversely proportional to the measurement error and can be calculated using the following formula:

[0099]

[0100] Among them, w i It is the weight coefficient of the i-th pixel, ∈ i This represents the measurement error of the pixel, and k is a constant used to adjust the range of the weighting coefficients.

[0101] Step S12: Apply adaptive weight allocation based on signal-to-noise ratio to the near-mid-range sparse depth map.

[0102] In this step, the depth information in the near-mid range sparse depth map is relatively sparse, and the reliability of its measurement is closely related to the signal-to-noise ratio (SNR). An adaptive weight allocation based on SNR can dynamically adjust the weight of each pixel in the fusion process according to its SNR. Pixels with high SNR indicate more reliable measurement results and should be assigned greater weight; pixels with low SNR indicate that the measurement results may contain significant noise and should be assigned less weight, thereby improving the quality of the fusion result.

[0103] For each pixel in the near-to-mid-range sparse depth map, calculate its signal-to-noise ratio (SNR). The SNR can be calculated as the ratio of signal strength to noise strength. The signal strength can be represented by the depth value of that pixel, and the noise strength can be estimated by statistically analyzing the depth value fluctuations of neighboring pixels.

[0104] Adaptive weight allocation: Based on the calculated signal-to-noise ratio, an adaptive weight is assigned to each pixel. The weight allocation can be performed using the following formula:

[0105] w j =f(SNR) j )

[0106] Among them, w j It is the weight of the j-th pixel, SNR j is the signal-to-noise ratio of the pixel, and f is a monotonically increasing function, such as the linear function f(x) = ax + b (a > 0) or a non-linear function (such as the sigmoid function), used to map the signal-to-noise ratio to a suitable weight range.

[0107] Step S13: Establish a probability distribution model for the overlapping region of the extremely close-range dense depth map, the near-mid-range sparse depth map, and the far-range sparse depth map, and perform confidence-based iterative optimization.

[0108] In this step, the three depth maps may overlap in certain areas. The depth information in these overlapping areas can complement and verify each other. By establishing a probability distribution model for the overlapping areas, the confidence level of each depth map in the overlapping areas can be quantitatively analyzed. The confidence level can be continuously adjusted through iterative optimization, making the fusion result more accurate and reliable.

[0109] Suppose the depth measurements from the three depth maps in the overlapping region are D1, D2, and D3, which can be viewed as random variables from different probability distributions. For example, assume that each depth measurement follows a Gaussian distribution. Where μ i It is the mean. It is the variance. By performing statistical analysis on the depth measurements of the overlapping regions, the parameters of each probability distribution can be estimated.

[0110] Assign an initial confidence score to each pixel in the overlapping region for each depth map; for example, the initial confidence scores can be set to equal values.

[0111] In each iteration, the likelihood function of each depth map in the overlapping region is calculated based on the probability distribution model. The likelihood function represents the degree of match between the measurements of that depth map and those of other depth maps. Then, the confidence score of each pixel is updated based on the likelihood function, for example, using Bayes' theorem.

[0112]

[0113] Among them, P(D) i |D -i ) is based on the known depth map measurement value D -i In the case of depth map i, the measured value D i The posterior probability, i.e., the updated confidence level; P(D) -i |D i P(D) is the likelihood function. i P(D) is the prior probability. -i ) is the marginal probability.

[0114] The iterative process is repeated until the confidence level converges or the preset number of iterations is reached. Finally, the three depth maps are weighted and fused based on the updated confidence level.

[0115] In some embodiments, the geometric constraint model includes local plane constraints based on the surface normal estimation of the full-range high-quality depth map, physical consistency verification based on depth-intensity correlation, and geometric anomaly detection of the auxiliary depth map using a random sampling consensus algorithm. These three parts are described below.

[0116] Local plane constraints based on surface normal estimation from full-range high-quality depth maps:

[0117] In a 3D scene, object surfaces typically possess a certain geometric structure, and local regions can be approximated as planes. By estimating the surface normals of a high-quality full-range depth map, the orientation information of the local plane containing each point can be obtained. Using this surface normal information to establish local plane constraints, it's possible to determine whether points in the auxiliary depth map conform to the geometry of that local plane. If a point significantly deviates from the local plane determined by the surface normals, it is likely an outlier. This constraint can effectively eliminate these outliers, improving the quality of the auxiliary depth map.

[0118] For each pixel in a full-scale, high-quality depth map, consider its neighborhood set of points. A plane can be fitted to this neighborhood set using the least squares method; the normal vector of the plane is the surface normal to that pixel. Specifically, assume the coordinates of the neighborhood points are (x... i ,y i ,z i (i = 1, 2, ..., n), the equation of the fitted plane is ax + by + cz + d = 0, which is achieved by minimizing the sum of squared distances from points to the plane. To solve for a, b, and c, we can then obtain the surface normal.

[0119] For each point in the auxiliary depth map, project it onto the corresponding position in the full-scale high-quality depth map and obtain the surface normal at that position. Calculate the distance from that point to the local plane determined by the surface normal and the corresponding point. If this distance exceeds a preset threshold, the point is considered to violate the local plane constraint and is identified as an outlier.

[0120] Physical consistency verification based on depth-strength correlation:

[0121] In real-world scenarios, there is often a physical correlation between depth information and intensity information (such as brightness in an RGB image). For example, under uniform lighting conditions, the depth and intensity variations on the same object's surface should be consistent. By establishing a depth-intensity correlation model, the physical consistency of points in an auxiliary depth map can be verified. If the relationship between the depth and intensity values ​​at a point clearly does not conform to the correlation model, then that point is likely a measurement error or an outlier. This verification can further improve the reliability of the auxiliary depth map.

[0122] First, by combining the full-range high-quality depth map with the corresponding RGB image, the depth and intensity values ​​of each pixel are extracted. Statistical analysis is then performed on this data; for example, linear or nonlinear regression methods can be used to establish a functional relationship between depth and intensity, I = f(D), where I is the intensity value and D is the depth value.

[0123] For each point in the auxiliary depth map, its corresponding intensity value is predicted based on its depth value using the correlation model established above. Simultaneously, the actual intensity value of that point in the RGB image is obtained. The difference between the predicted and actual intensity values ​​is calculated. If this difference exceeds a preset threshold, the point is considered physically inconsistent and is classified as an outlier.

[0124] The random sample consensus algorithm is used to detect geometric anomalies in the auxiliary depth map.

[0125] Random Sample Consensus (RANSAC) is an iterative algorithm used to estimate mathematical model parameters from a dataset containing outliers. In auxiliary depth maps, there may be outliers that significantly deviate from normal geometric structures, affecting subsequent processing and analysis. RANSAC can detect geometric outliers in auxiliary depth maps, retaining points that conform to a certain geometric model (such as planes or lines) as interiors and removing points that do not conform to the model as outliers, thereby improving the geometric consistency of the auxiliary depth map.

[0126] Choose an appropriate geometric model, such as a planar model, based on the characteristics and requirements of the scene. The equation of a planar model can be expressed as ax + by + cz + d = 0.

[0127] Random sampling: Randomly select a certain number of points from the auxiliary depth map (for planar models, usually select 3 non-collinear points) and estimate the parameters of the geometric model (such as a, b, c, d of the plane) based on these points.

[0128] Interior point determination: For each point in the auxiliary depth map, calculate the distance from that point to the estimated geometric model. If the distance is less than a preset threshold, the point is considered an interior point; otherwise, the point is considered an outlier.

[0129] Repeat the above random sampling and interior point determination process multiple times, recording the number of interior points each time. Select the geometric model estimated in the iteration with the largest number of interior points as the final model.

[0130] Based on the final geometric model, all points that are not interior points are removed from the auxiliary depth map.

[0131] Figure 3 This is a flowchart illustrating the steps of a viewpoint reprojection in an embodiment of the present invention. Figure 3 As shown, a step in a viewpoint reprojection embodiment of the present invention includes:

[0132] Step S31: Generate an alignment error correction map using the edge information of the RGB image.

[0133] In this step, when reprojecting the first near-range sub-image and the first far-range sub-image onto the RGB image viewpoint, alignment errors between the reprojected depth map and the RGB image may occur due to factors such as calibration errors of the depth camera and the RGB camera, and differences in imaging principles. The edge information of the RGB image clearly reflects the boundaries of objects in the scene. Using this edge information, the alignment deviation between the depth map and the RGB image at the edge positions can be detected, thereby generating an alignment error correction map. This map is then used to correct the reprojection results, improving the pixel alignment accuracy between the depth map and the RGB image.

[0134] Edge detection algorithms, such as the Canny edge detection algorithm, are applied to RGB images. This algorithm detects edge pixels in the image by calculating the gradient magnitude and direction of pixels in the image, thus obtaining the edge map of the RGB image.

[0135] The reprojected depth map is initially aligned with the RGB image, and then the positional differences between the edges of the depth map and the edges of the RGB image are compared. The alignment error can be measured by calculating the distance or offset of corresponding pixels. For example, for each pair of corresponding points on the edges of the depth map and the RGB image, their horizontal and vertical offsets on the image plane are calculated.

[0136] Based on the detected alignment errors, an alignment error correction map is generated. Each pixel value in this map represents the correction amount at the corresponding position, such as the number of pixels that need to be moved in the horizontal and vertical directions.

[0137] Step S32: Apply motion compensation based on optical flow estimation to adjust the depth value after reprojection.

[0138] In this step, during the depth map reprojection process, the reprojected depth values ​​may deviate due to the movement of objects in the scene or slight camera displacements at different times. Optical flow estimation can be used to calculate the motion information of objects in the image. Through motion compensation based on optical flow estimation, the reprojected depth values ​​can be adjusted to eliminate depth errors caused by motion, making the depth map more accurately reflect the actual depth information of the scene.

[0139] Optical flow estimation algorithms, such as the Lucas-Kanade optical flow algorithm or deep learning-based optical flow estimation algorithms (such as FlowNet), are used to calculate the optical flow field between adjacent frames or different regions in an RGB image. The optical flow field represents the motion velocity and direction of each pixel in the image.

[0140] Motion compensation is performed on the reprojected depth map based on the optical flow field information. For each pixel in the depth map, its new position after motion is calculated based on its corresponding optical flow vector, and the depth value at that position is adjusted accordingly. For example, if the optical flow vector of a pixel indicates that the pixel has moved one pixel to the right in the horizontal direction and one pixel downward in the vertical direction, then the depth value of that pixel is updated to its depth value at the new position (which can be obtained through interpolation).

[0141] Step S33: Perform joint bilateral filtering on the first near-field sub-image and the first far-field sub-image, and optimize the edge alignment of the depth map guided by the RGB image.

[0142] In this step, joint bilateral filtering is a nonlinear filtering method that combines information from both the spatial and value domains. After viewpoint reprojection, the edges of the depth map may not be sharp enough or accurately aligned with the edges of the RGB image. By applying joint bilateral filtering to the first near-field sub-image and the first far-field sub-image guided by the RGB image, the depth map can be smoothed while preserving its edge information. Furthermore, the edges of the depth map are optimized based on the edge information from the RGB image, resulting in better alignment between the depth map edges and the RGB image edges, thus improving the quality and visual effect of the depth map.

[0143] The output of the joint bilateral filter is a weighted average of the neighboring pixel values ​​in the input depth map. The weights are determined by spatial distance and differences in value range (such as color values ​​in an RGB image). For each pixel p in the depth map, its filtered output value D... out (p) can be calculated using the following formula:

[0144]

[0145] Where N(p) is the neighborhood of pixel p, and D in(q) is the input depth value of the neighboring pixel q, and w(p,q) is the weight, which is determined by the spatial distance weight w. s (p,q) and range weight w r Multiplying (p, q) yields:

[0146] w s (p,q)=w s (p,q)w r (p,q)

[0147] Spatial distance weight w s (p,q) is typically represented by a Gaussian function, reflecting the spatial distance between pixels p and q; the range weight w r (p,q) also uses the Gaussian function, calculated based on the color value difference of corresponding pixels in the RGB image, to preserve edge information.

[0148] Joint bilateral filtering is applied to the first near-range sub-image and the first far-range sub-image, guided by the RGB image. During the filtering process, value range weights are calculated based on the color information of corresponding pixels in the RGB image, thereby enabling the edges of the depth map to be better aligned with the edges of the RGB image. The final optimized depth map has sharper edges and more accurate alignment with the edges of the RGB image.

[0149] In some embodiments, the depth completion network in step S4 includes: a cascaded residual encoder module for extracting RGB-D multimodal features; a deformable convolution-guided cross-modal attention fusion module; and a multi-scale decoder containing depth uncertainty estimation. These three parts are described below.

[0150] Cascaded residual encoder modules are used to extract RGB-D multimodal features.

[0151] In the depth completion task, RGB images contain rich texture and semantic information, while depth maps directly provide geometric structure information of the scene. The main function of the cascaded residual encoder module is to extract features from the input RGB image and depth map (second near-field sub-image) and fuse their features to fully utilize the complementarity of RGB-D multimodal data, extracting more representative and discriminative features, and providing strong information support for subsequent depth completion.

[0152] The residual block is the basic building block of the residual encoder module. It alleviates the vanishing gradient problem in deep neural networks by introducing shortcut connections, enabling the network to train deeper models. A typical residual block contains multiple convolutional layers and non-linear activation functions (such as ReLU). After the input undergoes convolution and activation operations, it is added to the original input to form the residual output.

[0153] Multiple residual blocks are cascaded to form a residual encoder module. During the cascading process, the number of channels in the feature map gradually increases, while the spatial size gradually decreases to extract features at different scales and levels. For example, after each residual block, the number of channels in the feature map may double, while the spatial size is halved.

[0154] At different stages of the encoder, features from the RGB image and depth map are fused. A simple concatenation operation can be used to concatenate the RGB and depth features along the channel dimension, and then feature fusion and transformation are performed through convolutional layers to obtain RGB-D multimodal features.

[0155] Deformable convolution-guided cross-modal attention fusion module

[0156] In RGB-D multimodal data, the contributions of different modalities to depth completion may vary across different regions. The role of a deformable convolution-guided cross-modal attention fusion module is to adaptively focus on the regions most important for depth completion in both the RGB image and the depth map, and effectively fuse the features of these regions. Deformable convolution can automatically adjust the sampling position of the convolution kernel based on the input data, thereby better capturing the geometric shape and texture information of objects; the cross-modal attention mechanism can establish connections between different modalities, enhancing the transmission and fusion of useful information and improving the accuracy of depth completion.

[0157] Deformable convolution introduces an offset into traditional convolution. By using additional convolutional layers to learn the offset at each sampling position, the convolutional kernel can sample features at irregular locations. Specifically, for an input feature map x, the output y of deformable convolution can be expressed as:

[0158]

[0159] Where p0 is the pixel position of the output feature map, R is the sampling region of the convolution kernel, and w(p n ) represents the weights of the convolution kernel, Δp n It is the learned offset.

[0160] First, feature transformations are performed on the RGB and depth features respectively to obtain the query, key, and value. Then, the similarity between the query and key is calculated to obtain the attention weights. Finally, the values ​​are weighted and summed according to the attention weights to obtain the fused features. The specific formula is as follows:

[0161]

[0162] Where Q, K, and V are the query, key, and value matrices, respectively, and dk It is the dimension of the key.

[0163] The features obtained from deformable convolution and the fused features obtained from cross-modal attention mechanisms are further fused, for example, through concatenation and convolution operations, to obtain the final cross-modal fused features.

[0164] Multiscale decoder with depth uncertainty estimation

[0165] The primary function of the decoder is to upsample and restore the features extracted by the encoder, generating the final depth completion result. A multi-scale decoder that includes depth uncertainty estimation not only outputs the predicted depth map but also simultaneously estimates the depth uncertainty of each pixel. Depth uncertainty information reflects the reliability of the predicted depth value, allowing adjustments to the prediction result in subsequent processing (such as filtering and fusion) based on the uncertainty, thus improving the reliability and robustness of the depth completion. The multi-scale decoder, on the other hand, processes features at different scales, capturing depth information at different scales and improving the detail and accuracy of the depth map.

[0166] The decoder employs a multi-scale structure, comprising multiple upsampling and convolutional layers. At different stages of the decoder, features at different scales from the encoder output are upsampled and fused. For example, the spatial size of the feature map is gradually increased using methods such as deconvolution (transposed convolution) or bilinear interpolation, while features at different scales are concatenated and convolved to fuse information from different scales.

[0167] An additional branch is introduced into the decoder to estimate depth uncertainty. This branch can share some features with the depth prediction branch and outputs the depth uncertainty value for each pixel through convolutional layers and activation functions. Depth uncertainty can be represented by variance or a probability distribution; for example, a Gaussian distribution can be used to model depth values, and the depth uncertainty can be represented by predicting the variance of the Gaussian distribution.

[0168] The decoder's final output includes a predicted depth map and a corresponding depth uncertainty map. In subsequent processing, the predicted depth map can be filtered and adjusted based on the depth uncertainty, such as smoothing or re-estimating regions with high uncertainty.

[0169] Figure 4 This is a flowchart illustrating the steps of training a deep completion network according to an embodiment of the present invention. Figure 4 As shown, the steps for training a deep completion network in an embodiment of the present invention include:

[0170] Step S41: Randomly inject deep missing regions of simulated ToF noise patterns into the training data.

[0171] In this step, in practical applications, Time-of-Flight (ToF) cameras generate noise due to various factors (such as changes in reflectivity and ambient light interference), resulting in depth-missing regions in the depth map. By randomly injecting depth-missing regions simulating ToF noise patterns into the training data, the depth completion network can be exposed to more real-world scenarios during training, enhancing the network's robustness to depth-missing issues and enabling it to better handle noise and missing data in real-world data.

[0172] First, it is necessary to analyze the noise characteristics of the ToF camera to understand the causes and manifestations of its noise. For example, ToF noise may manifest as random depth value anomalies or blocky areas with missing depth.

[0173] Based on the Time-of-Flight (ToF) noise pattern obtained from the analysis, some regions are randomly selected in the depth map of the training data, and the depth values ​​of these regions are set to invalid values ​​(such as 0 or NaN) to simulate the situation of missing depth. The proportion and distribution of injected noise can be controlled to simulate different levels of noise interference.

[0174] Step S42: Apply texture perturbation corresponding to the depth occlusion region to the RGB image.

[0175] In this step, in real-world scenarios, depth occlusion regions are often accompanied by texture changes in RGB images because object occlusion affects light reflection and propagation. Applying texture perturbations corresponding to the depth occlusion regions to the RGB images allows the training data to more realistically simulate actual conditions, enabling the network to learn the correlation between depth information and RGB texture information, thereby improving the network's depth completion ability when handling occlusion situations.

[0176] Based on the depth-missing region injected in step S41, the corresponding region is found in the RGB image. Since the depth map and the RGB image are aligned at the pixel level, the corresponding region can be directly determined in the RGB image based on the position of the depth-missing region.

[0177] Texture perturbation is applied to a defined RGB image region. Common texture perturbation methods include adding Gaussian noise, salt-and-pepper noise, and blurring. These perturbation operations can simulate the texture changes in an RGB image when an object is occluded.

[0178] Step S43: Generate realistic long-range depth artifact samples through adversarial training.

[0179] In this step, we address the fact that in actual depth measurements, depth data from distant regions is often more susceptible to noise and measurement errors, resulting in depth artifacts. By generating realistic samples of distant depth artifacts through adversarial training, the depth completion network can learn how to handle these complex artifacts, thereby improving its depth completion performance in distant regions.

[0180] Construct a Generative Adversarial Network (GAN) consisting of a generator and a discriminator. The generator generates realistic long-range depth artifact samples, while the discriminator distinguishes the generated samples from real long-range depth data.

[0181] During training, the generator continuously attempts to generate more realistic depth artifact samples to deceive the discriminator; the discriminator, in turn, continuously learns and improves its discrimination ability to accurately distinguish between generated and real samples. Through this adversarial training process, the generator gradually becomes able to generate increasingly realistic long-distance depth artifact samples.

[0182] By incorporating the generated long-distance depth artifact samples into the training data, the depth completion network is exposed to these complex samples during training, thereby improving its ability to handle depth artifacts.

[0183] Step S44: Gradually increase the complexity of synthetic occlusion using a course learning strategy.

[0184] In this step, the learning strategy employs a progressively increasing difficulty training method. In training the deep completion network, the network initially learns simple synthetic occlusion scenarios, gradually increasing the complexity of the synthetic occlusions as training progresses. This allows the network to more easily learn basic deep completion skills in the early stages of training, and then gradually adapt to more complex scenarios, improving training efficiency and the network's generalization ability.

[0185] Define the complexity metrics for synthetic occlusion, such as the size, shape, and distribution density of the depth-deficient region, as well as the intensity of RGB image texture perturbation.

[0186] In the initial stages of training, a low complexity of synthetic occlusion is set, allowing the network to learn to handle simple occlusion scenarios. As training progresses, the complexity of synthetic occlusion is gradually increased according to preset rules, such as increasing the size of depth-deficient regions or the intensity of texture perturbations every certain number of training epochs. In this way, the network gradually adapts to more complex occlusion scenarios, improving its depth completion capabilities.

[0187] Figure 5 This is a flowchart illustrating the steps of edge blending in an embodiment of the present invention. Figure 5 As shown, an edge blending step in an embodiment of the present invention includes:

[0188] Step S51: Establish a transition zone within the depth threshold range.

[0189] In this step, in step 2, the optimized depth map is divided into a first near-range sub-map and a first far-range sub-map based on a preset depth threshold. When fusing the processed dense depth map (mainly containing near-range depth information) and the second far-range sub-map, a transition region needs to be established to ensure a natural and smooth transition between near-range and far-range depth information and avoid obvious boundary abrupt changes. By establishing a transition region within the depth threshold range, the fusion process can gradually adjust the depth values ​​within this region, making the final generated depth map more continuous and natural.

[0190] First, define the preset depth threshold D. thresh Then calculate the upper and lower limits of the transition region, i.e., the lower limit D. lower =D thresh ×(1-s), upper limit D upper =D thresh ×(1+s). Where s is the threshold range, which can be 10%, 9%, 8%, 7%, 6% or any other unlisted value.

[0191] Marking the transition region: Traverse each pixel in the depth map, for depth values ​​in [D]... lower , D upper Pixels within the specified range are marked as transition zone pixels. This completes the establishment of the transition zone, and subsequent blending operations will focus on this area.

[0192] Step S52: Use the Poisson fusion algorithm to achieve a smooth transition of the depth gradient.

[0193] In this step, the Poisson fusion algorithm is an image fusion method based on partial differential equations. It can achieve a smooth transition between different regions while preserving image gradient information. In edge fusion of depth maps, using the Poisson fusion algorithm allows the depth gradient in the transition region to smoothly transition from near depth values ​​to far depth values, avoiding sudden changes in depth values, thereby improving the overall quality and visual effect of the depth map.

[0194] The transition zone is used as the fusion region, and the depth values ​​of the near-field depth map and the far-field depth map at the boundary of the transition zone are used as boundary conditions.

[0195] Constructing the Poisson equation: The Poisson equation describes the variation of pixel values ​​within the fusion region. Its general form is Δu=f, where Δ is the Laplacian operator, u is the fused depth value to be solved, and f is the gradient information of the source image (such as a near-field depth map).

[0196] Numerical methods (such as iterative methods and matrix solving methods) can be used to solve the Poisson equation and obtain the fused depth value of each pixel in the transition region. During the solution process, the depth value is adjusted according to the boundary conditions and gradient information of the source image to achieve a smooth transition of the depth gradient.

[0197] Step S53: Perform outlier correction on the fusion boundary based on the adjacent depth ratio.

[0198] In this step, during edge fusion, various factors (such as noise, measurement errors, etc.) may cause abnormal depth values ​​at the fusion boundary. These abnormal values ​​can disrupt the continuity and accuracy of the depth map. An outlier correction method based on adjacent depth ratios can analyze the depth ratios of adjacent pixels to determine whether a pixel's depth value is abnormal and correct the outlier, thereby further improving the quality of the fused depth map.

[0199] For each pixel at the merging boundary, calculate its depth ratio to its neighboring pixels (e.g., the pixels above, below, left, and right). For example, for pixel p, its neighboring pixels are p1, p2, and p3. neighbor Then the depth ratio Where D(p) and D(p) neighbor ) are the depth values ​​of pixel p and its neighboring pixels, respectively.

[0200] Set a reasonable depth ratio threshold range [r] min ,r max If the depth ratio of a pixel exceeds this threshold range, the depth value of that pixel is considered an outlier.

[0201] For pixels identified as outliers, various methods can be used for correction. A simple method is to replace the outlier with the average depth value of its neighboring pixels. For example, if pixel p is identified as an outlier, its depth value is updated to... Where n is the number of adjacent pixels. It is the i-th adjacent pixel. In this way, abnormal depth values ​​at the fusion boundary can be effectively corrected, making the depth map more continuous and accurate.

[0202] Figure 6 This is a schematic diagram of a deep completion system based on multimodal modulation input in an embodiment of the present invention.

[0203] like Figure 6 As shown, an embodiment of the present invention provides a deep completion system based on multimodal modulation input, comprising:

[0204] The initial generation module is used to acquire a dense depth map at very close range, a sparse depth map at near-mid range, and a sparse depth map at far range using a multimodal modulated depth camera. It performs confidence weighting and spatial consistency fusion on the three depth maps to generate a high-quality depth map across the entire range. At the same time, it extracts the depth point set that was filtered out during the fusion process because its confidence was lower than the first threshold, but whose confidence was higher than the second threshold, to generate an auxiliary depth map.

[0205] The sub-map generation module is used to establish a geometric constraint model based on the depth distribution characteristics of the full-range high-quality depth map, remove outliers from the auxiliary depth map and perform multi-scale feature fusion with the full-range high-quality depth map to generate an optimized depth map, and divide the optimized depth map into a first near sub-map and a first far sub-map according to a preset depth threshold.

[0206] The reprojection module is used to perform viewpoint reprojection on the first near sub-image and the first far sub-image respectively according to the joint calibration parameters of the depth camera and the RGB camera, and generate a second near sub-image and a second far sub-image that are aligned with the pixels of the RGB image.

[0207] A dense module is used to construct a depth completion network that includes an attention mechanism. The second near sub-image and the RGB image are input into the depth completion network for depth completion. After confidence threshold filtering and the preset depth threshold filtering, a dense depth map is output.

[0208] The fusion module is used to perform edge fusion based on depth continuity on the dense depth map and the second distant sub-map to generate a final depth map containing near-distance high-density and far-distance low-density depth information.

[0209] Specifically, the core function of the initial generation module is to acquire depth maps at different distance ranges using a multimodal modulation depth camera: a dense depth map for very close distances, a sparse depth map for near-mid distances, and a sparse depth map for long distances. These three depth maps are then fused using a confidence-weighted and spatial consistency fusion method to generate a high-quality depth map covering the entire range. This combines the advantages of depth maps from different distance ranges, improving the accuracy and completeness of the depth information. Simultaneously, depth point sets that were filtered out during the fusion process due to confidence levels below the first threshold but still above the second threshold are extracted to generate auxiliary depth maps, providing additional information for subsequent optimization.

[0210] The sub-map generation module establishes a geometric constraint model based on the depth distribution features of the full-range high-quality depth map obtained by the initial generation module. This geometric constraint model is used to remove outliers from the auxiliary depth map, eliminating abnormal points that do not conform to the scene's geometric structure. The processed auxiliary depth map is then fused with the full-range high-quality depth map using multi-scale features to further optimize the depth map's details and accuracy, generating an optimized depth map. Based on a preset depth threshold, the optimized depth map is divided into a first near-range sub-map and a first far-range sub-map for subsequent processing.

[0211] The reprojection module performs viewpoint reprojection on the first near-field sub-image and the first far-field sub-image respectively, based on the joint calibration parameters of the depth camera and the RGB camera. The purpose is to convert the viewpoint of the depth map to the same viewpoint as the RGB image, realize pixel alignment between the depth map and the RGB image, and prepare for subsequent depth completion using the texture information of the RGB image.

[0212] A dense module constructs a depth completion network incorporating an attention mechanism, which automatically focuses on important regions in the depth map and RGB image, improving depth completion performance. The second near-field sub-image obtained from the reprojection module and the RGB image are input into the depth completion network for depth completion. The completed result is then subjected to confidence threshold filtering and preset depth threshold filtering to remove unreliable predictions, outputting a dense depth map.

[0213] The fusion module performs edge fusion based on depth continuity between the dense depth map output by the dense module (which mainly contains near-range depth information) and the second far-range sub-map obtained by the reprojection module. Through fusion, near-range and far-range depth information can be smoothly transitioned, generating a final depth map that contains near-range high-density and far-range low-density depth information.

[0214] The initial generation module is the starting point of the entire system. It provides the sub-image generation module with a full-range, high-quality depth map and auxiliary depth maps, serving as the foundational data source for subsequent processing. The sub-image generation module further processes the output of the initial generation module, generating a first near-range sub-image and a first far-range sub-image, which are then provided to the reprojection module. The reprojection module performs viewpoint reprojection based on the results from the sub-image generation module, obtaining a second near-range sub-image and a second far-range sub-image aligned with the RGB image pixels, providing suitable input for the density module. The density module uses the output of the reprojection module and the RGB image to perform depth completion, outputting a dense depth map, which serves as one of the inputs to the fusion module. The fusion module combines the dense depth map output by the density module and the second far-range sub-image obtained by the reprojection module to perform edge fusion, generating the final depth map and completing the entire depth completion process.

[0215] This embodiment utilizes several types of depth maps with different characteristics, and then combines them with RGB images. Using a deep learning network, these depth maps are combined and optimized to output high-quality, high-precision depth images that are dense at close range and sparse at long range. This satisfies the needs of navigation and obstacle avoidance while requiring only a small amount of computing power.

[0216] This invention also provides a depth completion device based on multimodal modulation input, including a processor and a memory storing executable instructions for the processor. The processor is configured to execute steps of a depth completion method based on multimodal modulation input by executing the executable instructions.

[0217] As shown above, this embodiment utilizes several types of depth maps with different characteristics, and then combines them with RGB images. Using a deep learning network, these depth maps are combined and optimized to output high-quality, high-precision depth images that are dense at close range and sparse at long range. This satisfies the needs of navigation and obstacle avoidance while requiring only a small amount of computing power.

[0218] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "platform."

[0219] Figure 7 This is a schematic diagram of a depth completion device based on multimodal modulation input according to an embodiment of the present invention. See below for reference. Figure 7 To describe an electronic device 600 according to this embodiment of the present invention. Figure 7 The electronic device 600 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0220] like Figure 7 As shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0221] The storage unit stores program code, which can be executed by the processing unit 610 to perform the steps described in the section on a depth completion method based on multimodal modulation input described in this specification, including various exemplary embodiments of the present invention. For example, the processing unit 610 can perform actions such as... Figure 1 The steps are shown in the figure.

[0222] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0223] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a grid environment.

[0224] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0225] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more meshes (e.g., local area network (LAN), wide area network (WAN), and / or public meshes, such as the Internet) via mesh adapter 660. Mesh adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although... Figure 7 As not shown in the diagram, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0226] This invention also provides a computer-readable storage medium for storing a program that, when executed, implements the steps of a depth completion method based on multimodal modulation input. In some possible implementations, various aspects of the invention can also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps described in the foregoing section on a depth completion method based on multimodal modulation input, according to various exemplary embodiments of the invention.

[0227] As shown above, this embodiment utilizes several types of depth maps with different characteristics, and then combines them with RGB images. Using a deep learning network, these depth maps are combined and optimized to output high-quality, high-precision depth images that are dense at close range and sparse at long range. This satisfies the needs of navigation and obstacle avoidance while requiring only a small amount of computing power.

[0228] Figure 8 This is a schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention. (Reference) Figure 8 As shown, a program product 800 for implementing the above-described method according to an embodiment of the present invention is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0229] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0230] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0231] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of mesh, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0232] This embodiment utilizes several types of depth maps with different characteristics, and then combines them with RGB images. Using a deep learning network, these depth maps are combined and optimized to output high-quality, high-precision depth images that are dense at close range and sparse at long range. This satisfies the needs of navigation and obstacle avoidance while requiring only a small amount of computing power.

[0233] The various embodiments described in this specification are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0234] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A depth completion method based on multimodal modulation input, characterized in that, include: Step S1: Acquire a dense depth map at very close range, a sparse depth map at near-mid range, and a sparse depth map at far range using a multimodal modulation depth camera. Perform confidence weighting and spatial consistency fusion on the three depth maps to generate a high-quality depth map across the entire range. At the same time, extract the depth point set that was filtered out during the fusion process because its confidence was lower than the first threshold, but whose confidence was higher than the second threshold, to generate an auxiliary depth map. Step S2: Based on the depth distribution characteristics of the full-range high-quality depth map, establish a geometric constraint model, remove outliers from the auxiliary depth map, and fuse it with the full-range high-quality depth map using multi-scale features to generate an optimized depth map. Then, according to a preset depth threshold, divide the optimized depth map into a first near-range sub-map and a first far-range sub-map. Step S3: Based on the joint calibration parameters of the depth camera and the RGB camera, perform viewpoint reprojection on the first near sub-image and the first far sub-image respectively to generate a second near sub-image and a second far sub-image aligned with the pixels of the RGB image; Step S4: Construct a depth completion network with an attention mechanism, input the second near sub-image and the RGB image into the depth completion network for depth completion, and output a dense depth map after confidence threshold filtering and preset depth threshold filtering; Step S5: Perform edge fusion based on depth continuity on the dense depth map and the second distant sub-map to generate a final depth map containing near-distance high-density and far-distance low-density depth information.

2. The depth completion method based on multimodal modulation input according to claim 1, characterized in that, The confidence-weighted fusion process in step S1 includes: Step S11: Assign dynamic weighting coefficients that are inversely correlated with measurement error to the extremely close-range dense depth map; Step S12: Apply adaptive weight allocation based on signal-to-noise ratio to the near-mid-range sparse depth map; Step S13: Establish a probability distribution model for the overlapping region of the extremely close-range dense depth map, the near-mid-range sparse depth map, and the far-range sparse depth map, and perform confidence-based iterative optimization.

3. The depth completion method based on multimodal modulation input according to claim 1, characterized in that, The geometric constraint model includes local plane constraints based on the surface normal estimation of the full-range high-quality depth map, physical consistency verification based on depth-intensity correlation, and geometric anomaly detection of the auxiliary depth map using a random sampling consistency algorithm.

4. The depth completion method based on multimodal modulation input according to claim 1, characterized in that, The view reprojection in step S3 includes: Step S31: Generate an alignment error correction map using the edge information of the RGB image; Step S32: Apply motion compensation based on optical flow estimation to adjust the reprojected depth value; Step S33: Perform joint bilateral filtering on the first near-field sub-image and the first far-field sub-image, and optimize the edge alignment of the depth map guided by the RGB image.

5. The depth completion method based on multimodal modulation input according to claim 1, characterized in that, The deep completion network mentioned in step S4 includes: A cascaded residual encoder module is used to extract RGB-D multimodal features; Deformable convolution-guided cross-modal attention fusion module; A multi-scale decoder that includes depth uncertainty estimation.

6. The depth completion method based on multimodal modulation input according to claim 5, characterized in that, The training process of the deep completion network employs a synthetic occlusion enhancement strategy, specifically including: Step S41: Randomly inject deep missing regions simulating ToF noise patterns into the training data; Step S42: Apply texture perturbation corresponding to the depth occlusion region to the RGB image; Step S43: Generate realistic long-range depth artifact samples through adversarial training; Step S44: Gradually increase the complexity of synthetic occlusion using a course learning strategy.

7. The depth completion method based on multimodal modulation input according to claim 1, characterized in that, The edge blending described in step S5 specifically includes: Step S51: Establish a transition zone within the depth threshold range; Step S52: Use the Poisson fusion algorithm to achieve a smooth transition of the depth gradient; Step S53: Perform outlier correction on the fusion boundary based on the adjacent depth ratio.

8. A depth completion system based on multimodal modulation input, used to implement the depth completion method based on multimodal modulation input as described in any one of claims 1 to 7, characterized in that, include: The initial generation module is used to acquire a dense depth map at very close range, a sparse depth map at near-mid range, and a sparse depth map at far range using a multimodal modulated depth camera. It performs confidence weighting and spatial consistency fusion on the three depth maps to generate a high-quality depth map across the entire range. At the same time, it extracts the depth point set that was filtered out during the fusion process because its confidence was lower than the first threshold, but whose confidence was higher than the second threshold, to generate an auxiliary depth map. The sub-map generation module is used to establish a geometric constraint model based on the depth distribution characteristics of the full-range high-quality depth map, remove outliers from the auxiliary depth map and perform multi-scale feature fusion with the full-range high-quality depth map to generate an optimized depth map, and divide the optimized depth map into a first near sub-map and a first far sub-map according to a preset depth threshold. The reprojection module is used to perform viewpoint reprojection on the first near sub-image and the first far sub-image respectively according to the joint calibration parameters of the depth camera and the RGB camera, and generate a second near sub-image and a second far sub-image that are aligned with the pixels of the RGB image. A dense module is used to construct a depth completion network that includes an attention mechanism. The second near sub-image and the RGB image are input into the depth completion network for depth completion. After confidence threshold filtering and the preset depth threshold filtering, a dense depth map is output. The fusion module is used to perform edge fusion based on depth continuity on the dense depth map and the second distant sub-map to generate a final depth map containing near-distance high-density and far-distance low-density depth information.

9. A depth completion device based on multimodal modulation input, characterized in that, include: processor; A memory in which executable instructions of the processor are stored; The processor is configured to perform the steps of the depth completion method based on multimodal modulation input as described in any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, characterized in that, When the program is executed, it implements the steps of the depth completion method based on multimodal modulation input as described in any one of claims 1 to 7.

Images