A general depth scan map completion method based on real-time enhanced structure information

Abstract

The application discloses a kind of based on real-time general depth scanning chart completion method of enhanced structure information, belong to computer vision technical field, the method includes: constructing depth completion network, it includes: for extracting the image branch of RGB image feature, for extracting the depth branch of depth map feature, distribution perception dynamic gating module and variance adaptive structure decoupling module;Wherein, double branch establishes feature interaction channel between corresponding scale layers, realizes cross-modal feature fusion by distribution perception dynamic gating module;Variance adaptive structure decoupling module realizes depth scanning chart completion based on fusion feature;Network is trained;Utilize the network of well-trained to realize depth scanning chart completion.The application can be through dynamic gating fusion mechanism and adaptive structure decoupling strategy, while maintaining real-time processing capability, significantly improve the geometric accuracy and boundary definition of depth map.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly to a general deep scan map completion method based on enhanced structural information in real time. BACKGROUND

[0002] With the rapid development of automatic driving, robot navigation and augmented reality technology, high-precision depth information map has become a core requirement for environmental perception. However, commercial depth sensors (such as laser radar and ToF radar) have significant sparsity in output depth map due to hardware limitations. For example, the effective pixels in the depth map generated by Velodyne HDL64E laser radar account for only about 5% of the corresponding RGB image, which is difficult to meet the high-resolution modeling requirements in complex scenarios (such as dense traffic and dynamic obstacle interaction). Depth completion technology, which restores dense information by fusing sparse depth map and RGB image, is a key means to solve this problem. However, existing methods still have significant bottlenecks in real-time performance, edge accuracy and multi-modal fusion efficiency. Although traditional convolutional neural networks (CNN) can directly predict dense depth maps, they have the problem of edge blurring. To this end, spatial propagation networks (such as CSPN) improve edge sharpness by iteratively optimizing affinity coefficients, but their framework needs to predict both the initial depth map and the propagation coefficient, which lack independent supervision signals, resulting in insufficient model generalization ability. In addition, complex propagation mechanisms (such as multiple iterations) result in a significant increase in computational load, making it difficult to meet real-time requirements. To reduce computational complexity, existing real-time methods often use simple feature superposition or channel splicing to fuse multi-modal data, ignoring the distribution differences between depth and RGB modalities (such as the sparsity of depth data and the richness of RGB texture), resulting in the loss of structural details, especially in the boundary between dynamic objects and static backgrounds, which is prone to artifacts, and the depth jump in the overlapping area of vehicles and pedestrians is difficult to accurately recover, directly affecting the safety of path planning.

[0003] The current technology still faces two major challenges: first, the homogenization of foreground and background areas leads to blurring of the depth map boundary, and existing frameworks lack explicit modeling mechanisms for scene structure. For example, although SLAM technology can achieve environmental perception through three-dimensional reconstruction, the NeRF algorithm it relies on is difficult to balance between real-time rendering speed and accuracy. Second, complex propagation networks (such as Transformers) can improve accuracy, but the computational load increases significantly, making it difficult to meet real-time requirements. For example, although the SLAM method based on 3DGaussian Splatting optimizes the rendering speed of NeRF, it requires high hardware resources and is difficult to deploy on embedded devices. In addition, existing depth completion methods still have deficiencies in dynamic data updating and adaptive fusion of multi-modal features. For example, although the automatic driving simulation image enhancement method improves image realism through a generative adversarial network (GAN), it relies on a large amount of labeled data and has high computational cost. SUMMARY

[0004] The application provides a general depth scan map completion method based on enhanced structure information in real time to solve the technical problems of distortion of multi-modal feature fusion, fuzzy scene structure and computational efficiency bottleneck in the prior art.

[0005] To solve the above technical problems, the application provides the following technical solutions:

[0006] In one aspect, the application provides a general depth scan map completion method based on enhanced structure information in real time, which comprises:

[0007] a depth completion network is constructed, wherein the depth completion network comprises an image branch, a depth branch, a distribution perception dynamic gating module and a variance adaptive structure decoupling module; the image branch is used to extract RGB image features; the depth branch is used to extract depth map features; the image branch and the depth branch establish a feature interaction channel between corresponding scale layers, realize cross-modal feature fusion of the RGB image features and the depth map features through the distribution perception dynamic gating module, and obtain fused features; the variance adaptive structure decoupling module realizes depth scan map completion based on the fused features;

[0008] The constructed depth completion network is trained;

[0009] The trained constructed depth completion network is used to realize depth scan map completion.

[0010] Further, the image branch adopts an improved residual structure, retains multi-scale texture features through cross-layer skip connection, and constructs a feature pyramid from the original resolution to the preset resolution in the multi-downsampling process; wherein each residual block fuses low-level detail features and high-level semantic features through cross-layer connection.

[0011] Further, the image branch extracts multi-scale RGB image features through an encoder comprising five residual blocks, wherein in the forward propagation process of the encoder, the feature map size output by each residual block is sequentially reduced to 1 / 2, 1 / 4, 1 / 8 and 1 / 16 of the original image, and the feature channel number of each layer is compressed to 1 and then fused through skip connection, so as to retain the edge positioning information in the original RGB image, and finally generate a multi-scale RGB image feature set, and the resolution of each scale feature is the original size, 1 / 2, 1 / 4, 1 / 8 and 1 / 16 respectively.

[0012] Further, the depth branch adopts a multi-level hourglass network stacking structure, inputs sparse depth maps of different scales into corresponding hourglass networks for depth feature extraction, each hourglass network processes a sparse depth map of a specific scale, and the spatial correlation of sparse measurement points is captured through multi-scale hourglass stacking, and the input sparse depth map is adaptively weighted down-sampled to generate multi-scale representation.

[0013] Further, each hourglass network in the depth branch comprises an encoder-decoder structure, the encoding stage gradually abstracts the depth distribution pattern, and the decoding stage recovers spatial details combined with skip connections.

[0014] Further, the cross-modal feature fusion of RGB image features and depth map features is realized through the distribution-aware dynamic gating module, and the process of obtaining the fused features includes:

[0015] The RGB image features X rgb and the depth map features X depth are preliminarily fused to obtain preliminary fusion features X mix .

[0016] The distribution-aware weight ω mix is generated through 3x3 convolution and ReLU activation function:

[0017] ω mix = ReLU(Conv 3×3 (X mix ))

[0018] The RGB image features X rgb and the depth map features X depth are respectively convolved and ReLU activated to generate adaptive weight maps; based on the adaptive weight maps, the features X output are obtained:

[0019] X output = ω mix ⊙ReLU(Conv 3×3 (X depth ))+ ω mix ⊙ReLU(Conv 3×3 (X rgb ))

[0020] The fusion features X final are obtained by stacking residual connections: X final = X output + X mix .

[0021] Further, the process of depth scan completion based on the fused features by the variance adaptive structure decoupling module includes:

[0022] Firstly, take the supervision information in KITTI depth completion dataset, i.e. the actual depth map D gt , as the basis for D gt Local variance analysis generates foreground and background segmentation masks, and completes foreground and background segmentation of the actual depth map D gt , and takes the background part D gt-back as the supervision information of the background enhancement branch in the subsequent two-way prediction.

[0023] Construct a two-way prediction branch; wherein the two-way prediction branch includes a background enhancement branch and a result prediction branch; the background enhancement branch uses dilated convolution to expand the receptive field for the input fusion feature x p , focuses on capturing large-scale structure features and depth mutation patterns, and finally generates a background enhancement depth map o s , which is supervised by the background part D gt-back of the actual depth map; the result prediction branch is for the fusion feature x p , which uses conventional convolution to maintain accurate reconstruction of local details, and finally generates a detail enhancement depth map o d ; o s and o d are added point by point to fuse the results D d , as the final output of the dense depth map. The dense depth map is supervised by D gt .

[0024] Further, the foreground and background segmentation masks are generated based on local variance analysis,

[0025] complete the foreground and background segmentation of the actual depth map D gt , and generate the background part D gt-back segmentation result, including:

[0026] For valid pixels in the depth map, calculate the normalized local variance V(i,j) in a preset size sliding window:

[0027]

[0028] Where μ i,j is the normalized mean value in the window; represents the sliding window; D(m,n) represents the depth value of the point with coordinates (m,n) in the window; max(D) represents the maximum depth value in the window; m and n represent the horizontal and vertical coordinates of the points in the window, respectively;

[0029] Take the variance median τ as the threshold, and divide the area with variance greater than τ into the background D gt-back .

[0030] Finally, the constructed depth completion network is trained, and the mean square error is calculated to obtain the final output dense depth map Dd loss between the predicted depth map D gt The loss is calculated as follows:

[0031]

[0032] where, and denote the predicted depth and the ground truth depth at pixel p at 1 / 2 i resolution, respectively, and V denotes the set of valid pixels. The overall loss function L is a weighted sum of the loss components at different resolutions, in the form of:

[0033] L = L 0 + δ1L 1 + δ2L 2 + δ3L b

[0034] where L b denotes the background loss. The background loss L b is computed from the background part of the ground truth depth map D gt-back and the background enhanced depth map o s by a variance adaptive structure decoupling module. In yet another aspect, the present application also provides an electronic device comprising a processor and a memory; wherein the memory has stored therein at least one instruction, which is loaded and executed by the processor to implement the above method.

[0035] In yet another aspect, the present application also provides a computer readable storage medium having stored therein at least one instruction, which is loaded and executed by a processor to implement the above method.

[0036] The technical solution provided by the present application has at least the following beneficial effects:

[0037] The present application can effectively improve the geometric precision and boundary consistency of depth completion while maintaining sub-millisecond processing speed. The dynamic gating mechanism significantly reduces the error in depth discontinuity regions by modeling the modal distribution difference, especially showing stronger robustness in the edges of dynamic objects and complex texture regions. The structure decoupling module significantly improves the depth transition effect at the junction of moving objects and static backgrounds through foreground and background differentiation processing, reducing the depth value diffusion phenomenon commonly seen in traditional methods. The multi-scale feature interaction mechanism effectively balances global structural consistency and local detail accuracy, and still maintains reasonable scene structure inference ability under extreme sparse input conditions. The system shows good adaptability in dynamic scenes and can effectively handle transient occlusion and depth discontinuity caused by moving objects, providing reliable technical support for real-time autonomous driving and other application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0039] Figure 1 is a structural schematic diagram of the deep completion network provided by the embodiments of the present application. DETAILED DESCRIPTION

[0040] In order to make the objects, technical solutions and advantages of the present application clearer, the following will further describe the embodiments of the present application in combination with the drawings.

[0041] First of all, it should be noted that in the embodiments of the present application, the words such as "exemplarily", "for example" and the like are used to represent as an example, illustration or description. Any embodiment or design scheme described as "exemplary" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "exemplarily" is intended to present the concept in a specific manner. In addition, in the embodiments of the present application, the meaning expressed by "and / or" can be both, or can be either one of the two.

[0042] The present embodiment provides a general depth scan completion method based on enhanced structure information in real time under a 3D visual scene, proposes a new depth completion framework as shown in Figure 1 , solves the inherent contradiction between edge blur and calculation efficiency through lightweight network design and structure perception optimization. And a dynamic updating mechanism is introduced in the depth completion to reflect the changes of the road environment (such as new obstacles, traffic signs) in real time, so as to improve the scene adaptability of the autonomous driving system. The method can be realized by an electronic device, which can be a terminal or a server. The execution process of the method includes the following steps:

[0043] S1, constructing a depth completion network; wherein the depth completion network comprises: an image branch, a depth branch, a distribution perception dynamic gating module and a variance adaptive structure decoupling module; wherein the image branch is used to extract RGB image features; the depth branch is used to extract depth map features; the image branch and the depth branch establish a feature interaction channel between corresponding scale layers, realize cross-modal feature fusion of the RGB image features and the depth map features through the distribution perception dynamic gating module, and obtain fused features; the variance adaptive structure decoupling module realizes depth scan completion based on the fused features;

[0044] S2, training the constructed depth completion network;

[0045] S3, using the trained depth completion network to realize depth scan completion.

[0046] Specifically, as shown in the embodiment, a dual-branch collaborative network architecture is first constructed, which includes a parallel processing system of an RGB feature extraction branch and a depth feature processing branch. The RGB branch adopts an improved residual structure, which retains multi-scale texture features through cross-layer jump connection, and constructs a feature pyramid from the original resolution to 1 / 16 resolution in four down-sampling processes. Each residual block fuses low-level detail features and high-level semantic features through cross-layer connection, effectively avoiding the loss of structural information in the traditional down-sampling process. Figure 1

[0047] The depth branch innovatively adopts a three-level hourglass network stacking structure, and each sub-network processes a sparse depth map of a specific scale. The input sparse depth map is adaptively weighted down-sampled to generate a multi-scale representation. This method dynamically adjusts the sampling weight according to the spatial distribution of the effective measurement points, while reducing the resolution and preserving the spatial topological relationship of the key depth information. The dual-branch establishes a feature interaction channel between the corresponding scale layers, realizes cross-modal feature fusion through a dynamic gating mechanism, and forms multi-level complementary features from coarse to fine.

[0048] To solve the fusion distortion problem caused by the distribution difference of multi-modal features, the embodiment proposes a distribution-aware dynamic gating module. This module models the spatial statistical feature difference between RGB images and depth maps to generate a pixel-level dynamic weight matrix. In specific implementation, first, the RGB features and depth features are preliminarily mixed, and a weight map reflecting the distribution difference is generated through a convolution layer and a Sigmoid activation function. This weight map encodes the complementary characteristics of the two modalities in the spatial dimension, for example, enhancing the RGB edge features in the depth mutation area and strengthening the depth continuity in the texture-rich area. Subsequently, the original features are nonlinearly transformed and element-wise modulated with the weight map, and finally the initial fusion information is retained through residual connection. This dynamic adjustment mechanism overcomes the feature conflict caused by traditional direct addition or splicing, maintains the integrity of the edge structure in complex scenes, and especially shows stronger adaptability at the junction of dynamic objects and static backgrounds.

[0049] ​To solve the foreground-background depth confusion problem, a variance adaptive structure decoupling module is developed. The module adaptively divides the high-variance background area and the low-variance foreground area by local variance analysis on the actual depth true value. The depth variance of each valid pixel is calculated in a sliding window, and the classification threshold is dynamically determined according to the scene characteristics, so that the area with dramatic depth changes is classified as background, and the area with stable depth is classified as foreground. Based on this, a two-way prediction branch is constructed, in which the background enhancement branch uses dilated convolution to expand the receptive field and focuses on capturing large-scale structure features and depth mutation patterns; the result prediction branch uses regular convolution to maintain the accurate reconstruction of local details. The two-way prediction results are added point by point for fusion. This decoupling learning mechanism effectively improves the depth prediction quality of the occluded area and the boundary of the moving object, and avoids the depth value ambiguity phenomenon produced by traditional methods in complex scenes.

[0050] In terms of network architecture design, a lightweight dual-branch structure is used to reconstruct the traditional hourglass network. The RGB branch preserves low-level texture information through cross-layer feature reuse. At each downsampling stage, low-order features of the original image are injected into high-level feature maps through a skip connection, forming a multi-scale feature enhancement mechanism. The depth branch captures the spatial correlation of sparse measurement points through multi-scale hourglass stacking. Each hourglass network contains an encoder-decoder structure. The encoding stage gradually abstracts the depth distribution pattern, and the decoding stage recovers the spatial details combined with the skip connection. Two branches establish feature interaction nodes between three scale layers, and realize cross-modal information complementation through a dynamic gating module: focusing on global structure alignment at the coarse scale layer, optimizing regional depth continuity at the medium scale layer, and enhancing boundary details at the fine scale layer. The final output layer introduces a structure decoupling module to achieve adaptive learning of scene structure without additional labeled data, and organically combines background structure enhancement and foreground detail optimization.

[0051] The steps of effectively fusing multi-modal features and enhancing structure through the above dual-branch architecture are as follows:

[0052] First, multi-modal data acquisition is performed. A depth sensor and a color camera are used to collect sparse depth information D s ∈R H×W and RGB image I ∈ R 3×H×W ; where H × W are the length and width of the image respectively. The subscript S of the depth information graph indicates that it is sparse.

[0053] Second, a multi-scale sparse depth graph pyramid is constructed, and an adaptive weighted downsampling method is used to generate multi-resolution input. Specifically, the original sparse depth graph is processed hierarchically: through the following formula:

[0054]

[0055] The input depth map is denoted by D, and the binary mask M represents the valid region. Data is sampled at each location (x, y) using adaptive weights. This indicates a downsampling operation using adaptive weights, ω i,j These represent the weighting coefficients for offsets i and j, and these weights are used to... and Perform a weighted average. A small constant ε is used to avoid division by zero and ensure numerical stability. Generate sparse depth map sequences at 1 / 4, 1 / 2, and original sizes.

[0056] The third step involves inputting the sparse depth map sequence and RGB images into a dual-branch perceptual network. The sparse depth map sequence is input to the depth branch according to its size, while the RGB images are fed into the RGB branch for feature extraction from both modalities. The RGB branch extracts multi-scale RGB features through an encoder containing five residual blocks, where shallow structural features are passed to deeper layers via cross-layer skip connections. Specifically, during the forward propagation of the RGB encoder, the feature map size output by each residual block is successively reduced to 1 / 2, 1 / 4, 1 / 8, and 1 / 16 of the original image. Simultaneously, the number of feature channels in each layer is compressed to 1 before cross-layer fusion is performed via skip connections, preserving low-level structural information such as edge localization in the original RGB image. This ultimately generates a multi-scale RGB feature set {I0, I1, I2, I3, I4} containing rich structural features, corresponding to resolutions of the original size, 1 / 2, 1 / 4, 1 / 8, and 1 / 16, respectively.

[0057] The depth branch employs a three-level cascaded hourglass network structure to process multi-scale sparse depth maps. Sparse depth maps of different scales are input into the corresponding hourglass networks for depth feature extraction. Before outputting features at each scale, they are fused with RGB features of the same scale in a distribution-aware dynamic gating module.

[0058] The fourth step involves the distribution-aware dynamic gating module achieving adaptive fusion by explicitly modeling the differences in cross-modal feature distributions. Specifically, the deep feature X... depth With RGB feature X rgb Initial fusion yields X mix Distribution-aware weights ω are generated through 3×3 convolution and Sigmoid activation. mix =ReLU(Conv 3×3 (X mix Simultaneously, each modality feature is convolved and ReLU activated separately to generate an adaptive weight map. The final formula for calculating the fused features can be expressed as: X output =ω mix ⊙ReLU(Conv 3×3 (X depth ))+ω mix ⊙ReLU(Conv3×3 (X rgb )) and superimposed residual connection to obtain X final = X output + X mix .

[0059] In the fifth step, the variance adaptive structure decoupling module is applied on the maximum scale feature after the three-level hourglass network processing. First, the foreground and background segmentation mask is generated based on local variance analysis: for the valid pixels in the depth map D∈R H×W , the normalized local variance is calculated in the k x k sliding window where μ i,j is the normalized mean value in the window, D(m, n) represents the depth value of the (m, n) point in the window; max(D) represents the maximum depth value in the window; m and n represent the horizontal and vertical coordinates of the points in the window. The region with variance greater than τ is divided into background D gt-back .

[0060] A double-branch prediction mechanism is constructed inside the variance adaptive structure decoupling module: the initial aggregated feature X p is obtained by dilated convolution to obtain large receptive field feature f r , and the background enhanced sharpened depth map o s is output by 3 x 3 convolution; at the same time, the optimized depth map o d is generated by the regular convolution branch, o s and o d are point-by-point added and fused as the final optimized dense depth prediction result. The dense depth map is supervised by D gt .

[0061] When the depth completion network of the embodiment is trained, the loss between the final output dense depth map D d and the actual depth map D gt is calculated by mean square error; the calculation method of the loss is as follows:

[0062]

[0063] wherein, and represent the predicted depth and the true depth at pixel p at 1 / 2 i resolution, respectively, and V represents the set of valid pixels; the overall loss function L is the weighted sum of the loss components at different resolutions, and the form is:

[0064] L = L0+ δ1L1+ δ2L2+ δ3Lb

[0065] wherein, L b represents the background loss; the background loss Lb by the variance adaptive structure decoupling module from the background part D gt-back and the background enhanced depth map o s is calculated.

[0066] In summary, the embodiment proposes a dual-branch structure perception network for real-time depth completion, which systematically innovates in the problems of multi-modal feature fusion distortion, scene structure ambiguity, and computational efficiency bottleneck in the prior art. The method uses a dynamic gating fusion mechanism and an adaptive structure decoupling strategy to significantly improve the geometric accuracy and boundary clarity of the depth map while maintaining real-time processing capability, providing reliable technical support for real-time perception scenarios such as autonomous driving and robot navigation.

[0067] The innovation of the present application is reflected in three aspects: the dynamic gating mechanism first introduces modal distribution differences into the real-time depth completion task, realizes adaptive optimization of the feature fusion process through differentiable weight generation, and solves the structural distortion caused by the traditional method due to the neglect of modal characteristic differences; the unsupervised scene segmentation method based on depth variance uses the characteristics of depth data to establish a differentiated learning mechanism for foreground and background, significantly improving the prediction quality of complex occluded areas and avoiding the generalization limitations caused by relying on artificial labeling; the dual-branch multi-scale architecture maintains the efficient characteristics of the hourglass network while enhancing the structure perception ability through cross-modal interaction, forming a progressive optimization process from global to local. The entire system uses full convolution operations to realize end-to-end inference, and the modules control the computational complexity through lightweight design, with good deployment characteristics on embedded platforms.

[0068] Finally, it should be noted that the above-described only is the preferred embodiment of the present application, it should be pointed out that although the preferred embodiments of the present application have been described, for those skilled in the art, once the basic creative concept of the present application is known, without departing from the principles of the present application, a number of improvements and refinements can be made, these improvements and refinements should also be considered as the protection scope of the present application. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.

Claims

1. A general-purpose deep scan map completion method based on real-time enhanced structural information, characterized in that, The application relates to a method for constructing a depth completion network. The depth completion network comprises an image branch, a depth branch, a distribution-aware dynamic gating module and a variance adaptive structure decoupling module; the image branch is used for extracting RGB image features; the depth branch is used for extracting depth map features; the image branch and the depth branch establish a feature interaction channel between corresponding scale layers, cross-modal feature fusion of the RGB image features and the depth map features is realized through the distribution-aware dynamic gating module, and fusion features are obtained; the variance adaptive structure decoupling module realizes depth scan map completion based on the fusion features. The constructed depth completion network is trained. The trained depth completion network is used to realize depth scan map completion. The process of realizing cross-modal feature fusion of the RGB image features and the depth map features through the distribution-aware dynamic gating module to obtain fusion features comprises the following steps: RGB image features X rgb and depth map features X depth are preliminarily fused to obtain preliminary fusion features X mix ; Generating distribution-aware weights ω by 3x3 convolution and ReLU activation function mix : ω mix = ReLU(Conv 3×3 (X mix )) RGB image features X rgb and depth map features X depth are respectively convolved and ReLU activated to generate adaptive weight maps; based on the adaptive weight maps, features X output are obtained: X output = ω mix ⊙ ReLU(Conv 3×3 (X depth ))+ ω mix ⊙ ReLU(Conv 3×3 (X rgb )) The superimposed residual connection obtains the fusion feature X final : X final = X output + X mix ; The process of realizing depth scan map completion by the variance adaptive structure decoupling module based on the fusion features comprises the following steps: Firstly, the supervision information in KITTI depth completion dataset, i.e., the actual depth map D gt , is taken gt Based on D gt Local variance analysis generates foreground and background segmentation masks, completes foreground and background segmentation of the actual depth map D gt , and takes the background part D gt-back As the supervision information of the background enhancement branch in the subsequent two-way prediction constructing a two-path prediction branch; wherein the two-path prediction branch comprises a background enhancement branch and a result prediction branch; the background enhancement branch is for input fusion feature x p , adopts dilated convolution to expand the receptive field, focuses on capturing large-scale structure features and depth mutation patterns, and finally generates a background enhancement depth map o s , uses the background part D gt-back of the actual depth map for supervision; the result prediction branch is for fusion feature x p , through regular convolution to keep accurate reconstruction of local details, and finally generates a detail enhancement depth map o d ; o s and o d are added point by point to fuse the results D d as the final output of the dense depth map, which is supervised by D gt ; Wherein, based on local variance analysis, foreground and background segmentation mask is generated, and foreground and background segmentation of actual depth map D gt is completed, and background part D gt-back segmentation result is generated, including: For the effective pixels in the depth map, the normalized local variance V(i,j) is calculated in a preset size sliding window. where μ i,j is the normalized mean within the window; represents a sliding window; D(m, n) represents the depth value of the point with coordinate (m, n) within the window; max(D) represents the maximum depth value within the window; m and n represent the horizontal and vertical coordinates of the point within the window, respectively; With the variance median τ as a threshold, the region with variance greater than τ is divided into background D gt-back .

2. The method of claim 1, wherein the method is based on enhanced structural information in real time for completing a general depth scan map. The image branch adopts an improved residual structure, retains multi-scale texture features through cross-layer jump connection, and constructs a feature pyramid from the original resolution to a preset resolution in the multi-downsampling process; wherein each residual block fuses low-level detail features and high-level semantic features through cross-layer connection.

3. The real-time general depth scan completion method based on enhanced structure information according to claim 2, wherein, The image branch extracts multi-scale RGB image features through an encoder comprising five residual blocks, wherein in the forward propagation process of the encoder, the feature map size output by each residual block is sequentially reduced to 1 / 2, 1 / 4, 1 / 8 and 1 / 16 of the original image, and after the channel number of each layer feature is compressed to 1, cross-layer fusion is performed through jump connection, the edge positioning information in the original RGB image is retained, and finally a multi-scale RGB image feature set is generated, and the resolution of each scale feature is the original size, 1 / 2, 1 / 4, 1 / 8 and 1 / 16 respectively.

4. The method of claim 1, wherein the method is based on enhanced structural information in real time for completing a general depth scan map, and The depth branch adopts a multi-level hourglass network stacking structure, inputs sparse depth maps of different scales into corresponding hourglass networks for depth feature extraction, each hourglass network processes a sparse depth map of a specific scale, and the spatial correlation of sparse measurement points is captured through multi-scale hourglass stacking, and the input sparse depth map is generated into a multi-scale representation through adaptive weighted downsampling.

5. The real-time general depth scan completion method based on enhanced structure information according to claim 4, wherein, Each hourglass network in the depth branch comprises an encoder-decoder structure, the encoding stage gradually abstracts the depth distribution pattern, and the decoding stage combines the jump connection to restore the spatial details.

6. The real-time general depth scan completion method based on enhanced structure information according to claim 1, wherein, When training the constructed deep completion network, the loss between the final output dense depth map D d and the actual depth map D gt is calculated by mean square error; the loss is calculated as follows: where, and respectively denote the predicted depth and the ground truth depth at pixel p at 1 / 2 i resolution, V denotes the set of valid pixels; the overall loss function L is a weighted sum of the loss components at different resolutions, in the form of: L = L 0 + δ1L 1 + δ2L 2 + δ3L b wherein, L b represents the background loss; the background loss L b is calculated by the variance adaptive structure decoupling module from the background part D gt-back and the background enhanced depth map o s is calculated.

Images