A depth completion method and device based on attention-based panoramic perception guidance

Abstract

This invention provides a depth completion method and apparatus based on attention-guided panoramic perception, belonging to the field of information fusion technology. It acquires RGB image data captured by a visible light camera and sparse depth image data scanned by a LiDAR scanner; processes the RGB image data into panoramic segmentation image data of the image to be processed using a panoramic segmentation network; constructs a 3U-DenseNet network, inputting the RGB image data, panoramic segmentation image data, and sparse depth image data into the 3U-DenseNet network, and outputs preliminary depth completion results after passing through a multimodal attention fusion module MMTSAFB; constructs a convolutional spatial propagation network CSPN++, inputting the preliminary depth completion results into CSPN++ to obtain the final depth completion result, thus completing the attention-guided panoramic perception depth completion. This invention achieves end-to-end depth completion, and the attention-guided panoramic perception depth completion method has high accuracy and good completion effect. By fusing panoramic segmentation images, the depth changes at object boundaries are more subtle.

Description

Technical Field

[0001] This invention relates to the field of information fusion technology, and in particular to a method and apparatus for depth completion based on attention-guided panoramic perception. Background Technology

[0002] In many 3D vision fields such as autonomous driving, 3D scene reconstruction, and VR, obtaining accurate and dense depth information is a fundamental and crucial perception technology. Currently, most depth sensing methods utilize two sensors: visible light cameras and LiDAR. Visible light cameras acquire image data that lacks depth information but possesses rich texture and color information; while LiDAR scans provide sparse depth information, but it is less affected by ambient light and can provide accurate depth information. Because the information from these two sensors complements each other well, current mainstream depth completion methods primarily utilize information acquired by these two sensors. However, although LiDAR provides accurate information, its measurement density is very low, with effective depth values ​​accounting for less than 6% of the generated sparse depth map. In the field of 3D vision, sparse depth maps cannot be directly applied; therefore, the challenge lies in how to complete sparse depth maps into more accurate and dense depth maps. Figure 1 This has always been a challenge in the field. Furthermore, how to better integrate information from these two types of sensors is also an important task in deep completion.

[0003] With the tremendous success of deep learning technology today, the mainstream approach involves inputting sparse depth maps and RGB images into a neural network to reconstruct a dense depth map. Deep learning-based neural network depth completion algorithms primarily rely on encoder-decoder convolutional neural networks for feature extraction and fusion to reconstruct a dense depth map. Recent methods focus on using color images as guide images to recover the depth of invalid pixels. However, color images alone are insufficient to provide the necessary understanding of a scene. Therefore, the depth completion task is affected by sudden changes in illumination (e.g., shadows) in the RGB image. Summary of the Invention

[0004] To address the issues of shadows, noise, and occlusion in sparse depth maps and RGB images in existing technologies, this invention proposes a depth completion method and apparatus based on attention-guided panoramic perception.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] On the one hand, a depth completion method based on attention-based panoramic perception guidance is provided, including:

[0007] S1: Acquire RGB image data captured by a visible light camera and sparse depth image data scanned by a lidar;

[0008] S2: The RGB image data is processed into panoramic segmentation image data of the image to be processed through a panoramic segmentation network;

[0009] S3: Construct a 3U-DenseNet network, input the RGB image data, panoramic segmentation image data and sparse depth image data into the 3U-DenseNet network, and output the preliminary depth completion result through the multimodal attention fusion module MMTSAFB;

[0010] S4: Construct a convolutional spatial propagation network CSPN++, input the preliminary depth completion result into CSPN++, obtain the depth completion result, and complete the depth completion based on attention-guided panoramic perception.

[0011] Optionally, in step S2, the RGB image data is processed into panoramic segmentation data of the image to be processed using a panoramic segmentation network, including:

[0012] RGB images are converted into panoramic segmentation semantic image data using a pre-trained Mask2Former model.

[0013] Optionally, in step S3, the 3U-DenseNet network includes three branches: a color-guided SC branch, a panoptic segmentation semantic-guided SP branch, and a depth-guided 3S branch.

[0014] Optionally, both the color-guided SC branch and the panorama segmentation semantic-guided SP branch adopt a densely skip-connected encoder-decoder network architecture.

[0015] The encoder includes a convolutional layer and five ResNet blocks, and six BT blocks are added after the convolutional layer and ResNet blocks to form another branch.

[0016] The decoder consists of one convolutional layer and five transposed convolutional layers, which upsample the input data and fuse the features of each decoder layer.

[0017] Optionally, in step S3, inputting the RGB image data, panoramic segmentation image data, and sparse depth image data into the 3U-DenseNet network includes:

[0018] The RGB image data and sparse depth image data are input into the color-guided SC branch to obtain color dense depth map SC-Depth data with scene color cues;

[0019] The predicted color dense depth map (SC-Depth) data of the SC branch, as well as the panoramic semantic image and sparse depth image data, are input into the panoramic segmentation semantic guidance (SP) branch to obtain panoramic semantic depth data.

[0020] Sparse depth image data, color dense depth map (SC-Depth) data, and panoramic semantic depth data are input into the depth-guided 3S branch to obtain dense depth map data.

[0021] Optionally, the RGB image data and sparse depth image data are input into the color-guided SC branch to obtain color-dense depth map (SC-Depth) data with scene color cues, including:

[0022] RGB image data and sparse depth image data are input into the color-guided SC branch in a concatenated manner, and color dense depth map SC-Depth data is output.

[0023] Optionally, the predicted dense depth map of the SC branch, along with the panoramic semantic image and sparse depth map, are input into the panoramic segmentation semantic guidance SP branch to obtain panoramic semantic depth data, including:

[0024] The decoder features of the SC branch are fused into the encoder features of the corresponding SP branch through MMTAFB. The SP branch takes the stitching of color dense depth map SC-Depth data, panoramic semantic image and sparse depth map as input, and outputs SP dense depth map, which is panoramic semantic depth data.

[0025] Optionally, sparse depth image data, color depth data, and panoramic semantic depth data are input into the 3S branch to obtain density depth map data, including:

[0026] The decoder features from the SC and SP branches are fused into the encoder features of the corresponding 3S branches through the multimodal attention fusion module MMTSAFB; in the 3S branches, MMTSAFB uses the feature maps of the three modes as input and output to refine the fused feature maps.

[0027] Optionally, in step S3, the preliminary depth completion result is output by the multimodal attention fusion module MMTAFB, including:

[0028] An attention fusion module MMTSAFB with different input resolutions is constructed to fuse features between the three branches and obtain preliminary depth completion results.

[0029] On the one hand, a depth completion device based on attention-based panoramic perception guidance is provided, which is applied to any of the above-mentioned methods; including:

[0030] The data acquisition module is used to acquire RGB image data captured by a visible light camera and sparse depth image data scanned by a lidar.

[0031] The RGB image processing module is used to process RGB image data into panoramic segmentation image data of the image to be processed through a panoramic segmentation network.

[0032] The preliminary depth completion module is used to construct a 3U-DenseNet network. The RGB image data, panoramic segmentation image data, and sparse depth image data are input into the 3U-DenseNet network, and the preliminary depth completion result is output after passing through the multimodal attention fusion module MMTSAFB.

[0033] The depth completion module is used to construct a convolutional spatial propagation network CSPN++, input the preliminary depth completion result into CSPN++, obtain the depth completion result, and complete the depth completion based on attention-guided panoramic perception.

[0034] On the one hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the aforementioned depth completion method based on attention-guided panoramic perception.

[0035] On the one hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement the above-described depth completion method based on attention-guided panoramic perception.

[0036] The above technical solution has at least the following advantages compared with the existing technology:

[0037] The above scheme achieves end-to-end depth completion by designing a 3U-DenseNet network and combining it with a convolutional spatial propagation network (CSPN++) for post-processing. The attention-based panoramic perception-guided depth completion method has high accuracy and good completion effect. By fusing panoramic segmentation images, the depth changes at object boundaries are more subtle. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1This is a flowchart of the depth completion method based on attention-guided panoramic perception provided in an embodiment of the present invention;

[0040] Figure 2 This is a flowchart of the 3U-DenseNet network for the depth completion method based on attention-based panoramic perception provided in this embodiment of the invention.

[0041] Figure 3 This is the SC branch network diagram of the depth completion method based on attention-based panoramic perception guidance provided in this embodiment of the invention;

[0042] Figure 4 This is a BT structure diagram of the depth completion method based on attention-guided panoramic perception provided in this embodiment of the invention;

[0043] Figure 5 This is an MHSA structure diagram of the depth completion method based on attention-guided panoramic perception provided in this embodiment of the invention;

[0044] Figure 6 This is the SP branch network diagram of the depth completion method based on attention-based panoramic perception guidance provided in this embodiment of the invention;

[0045] Figure 7 This is a diagram of the MMTAFB structure of the depth completion method based on attention-based panoramic perception guidance provided in this embodiment of the invention.

[0046] Figure 8 This is a 3S branch network diagram of the depth completion method based on attention-based panoramic perception guidance provided in this embodiment of the invention;

[0047] Figure 9 This is a structural diagram of the depth completion device based on attention-guided panoramic perception provided in an embodiment of the present invention;

[0048] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0050] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms “first,” “second,” and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an,” “a,” or “the,” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “comprising,” “including,” or “including,” and similar terms mean that the element or object preceding the word encompasses the element or object listed following the word and its equivalents, without excluding other elements or objects. The terms “connected,” “linked,” or “connected,” and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0051] This invention provides a depth completion method based on attention-guided panoramic perception, such as... Figure 1 The flowchart shown below illustrates the depth completion method based on attention-based panoramic perception guidance, including:

[0052] S101: Acquire RGB image data captured by a visible light camera and sparse depth image data scanned by a lidar;

[0053] In one feasible implementation, the present invention uses RGB images captured by a visible light camera and sparse depth image data scanned by a lidar sensor for fusion and completion. The image data acquired by the visible light camera lacks depth information but possesses rich texture and color information; while the depth information from the lidar scan is very sparse but less affected by ambient light. Therefore, fusing information from these two sensors to complete the sparse depth image is a feasible and efficient method.

[0054] S102: The RGB image data is processed into panoramic segmentation image data of the image to be processed through a panoramic segmentation network.

[0055] In one feasible implementation, an RGB image is converted into panoramic segmentation semantic image data using a pre-trained Mask2Former model.

[0056] S103: Construct a 3U-DenseNet network, input RGB image data, panoramic segmentation image data, and sparse depth image data into the 3U-DenseNet network, and output preliminary depth completion results through MMTSAFB (Multi-model Transformer-SpatialAttention Fusion Block).

[0057] One feasible implementation method is, for example Figure 2As shown, this invention designs a 3U-DenseNet network and uses CSPN++ (Learning Context and Resource Aware Convolutional Spatial Propagation Networks for Depth Completion, combined with a convolutional spatial propagation network) for post-processing to achieve end-to-end depth completion.

[0058] In one feasible implementation, the 3U-DenseNet network includes three branches: a color-guided SC branch, a panoptic segmentation semantic-guided SP branch, and a depth-guided 3S branch.

[0059] One feasible implementation method is, for example Figure 3 This is the structure diagram of the SC (Sparse-Color) branch network. Figure 4 This is the BT architecture diagram. Both the color-guided SC branch and the panoptic segmentation semantic-guided SP branch employ a densely connected encoder-decoder network architecture.

[0060] The encoder includes a convolutional layer and five ResNet blocks. After the convolutional layer and the residual (ResNet) blocks, there are also six BT (Bottleneck-Transformer) blocks forming another branch.

[0061] The decoder consists of one convolutional layer and five transposed convolutional layers, which upsample the input data and fuse the features of each decoder layer.

[0062] In one feasible implementation, RGB image data and sparse depth image data are input into the color-guided SC branch to obtain color density depth SC-Depth data with scene color cues.

[0063] The predicted dense depth map of the SC branch, along with the panoramic semantic image and sparse depth map, are input into the panoramic segmentation semantic guidance SP branch to obtain panoramic semantic depth data.

[0064] Sparse depth image data, color depth data, and panoramic semantic depth data are input into the 3S branch to obtain density depth map data.

[0065] In this embodiment of the invention, the purpose of the SC branch is to learn important color cues in dense depth completion. It takes a color image concatenated with an aligned sparse depth map as input and outputs a color dense depth map (SC-Depth). Concatenating the sparse depth map aligned with the color image helps in predicting the dense depth map. The SC branch employs a densely skip-connected encoder-decoder network architecture. The encoder consists of one convolutional layer and five ResNet blocks, with an additional branch consisting of six BT (Bottleneck-Transformer) blocks following the convolutional layer and ResNet. The decoder consists of one convolutional layer and five transposed convolutional layers for upsampling and fusing features from each decoder layer. The output of this branch consists of a dense depth map (SC-Depth) and a confidence map (SC-Confidence). (Unlike the previous description) The dense depth map (SC-Depth) output by this branch is a noisy depth estimate, but it provides the foundation for learning the structural information of the scene in other branches.

[0066] like Figure 5 For the MHSA structure diagram, ⊕ and These represent element-wise summation and matrix multiplication, respectively; while 1×1 represents pointwise convolution. In this invention, we used four heads in multi-head attention. In addition to using multiple heads, the highlighted blue boxes indicate positional encoding and value projection. Furthermore, and R... w The height and width of the input feature map are encoded separately. The attention log is qk. T +qr T Here, q, k, and r represent the query, key, and position encoding, respectively. Similarly, we use relative distance for encoding.

[0067] In one feasible implementation, the SC branch takes a sparse depth map and an RGB image as input and generates a color depth (SC-Depth) that includes color cues from the scene. The predicted dense depth map of the SC branch is then compared with the panoramic semantic image and the sparse depth map. Figure 1 The input is passed to the SP (Sparse-Panoptic) branch to estimate the panoramic semantic depth (SP-Depth).

[0068] RGB image data and sparse depth image data are concatenated and input into the color-guided SC branch to output a dense depth map SC-Depth.

[0069] In one feasible implementation, the decoder features of the SC branch are fused into the encoder features of the corresponding SP branch through MMTAFB. The SP branch takes the concatenation of SC-Depth, panoramic semantic image and sparse depth map as input, and then outputs a dense depth map. The features of SC and SP are input into MMTAFB, and then MMTAFB outputs a refined fused depth feature map SP-Depth.

[0070] In one feasible implementation, panoramic semantic cues aid in scene understanding, particularly by providing clearer boundaries at object occlusion points. However, the SC branch alone is insufficient for learning panoramic semantic information. Therefore, this invention proposes an SP branch to learn effective panoramic segmentation semantic cues and supplements the RGB image with depth completion. The encoder and decoder of the SP branch also consist of a convolutional layer and five ResNet blocks, with the decoder also consisting of a convolutional layer and five transposed convolutions, and dense skip connections are also employed. The SP branch takes the concatenation of SC-Depth, the panoramic semantic image, and the sparse depth map as input and outputs a dense depth map. We use the KITTI depth completion dataset for training, but it does not provide panoramic segmentation semantic maps corresponding to RGB images. Therefore, this invention chooses to convert RGB images into panoramic segmentation semantic images (Panoptic Images) using a pre-trained Mask2Former model. Furthermore, the decoder features of the SC branch are fused into the encoder features of the corresponding SP branch via MMTAFB. The features of SC and SP are input into MMTAFB, and then MMTAFB outputs a refined fused depth feature map SP-Depth. The SP branch network is as follows: Figure 6 As shown. The MMTSAFB structure is as follows. Figure 7 As shown. In Figure 7In MMTSAFB, the input is a concatenation of an RGB image, a panoptic segmentation semantic image, and sparse depth information. The algorithm focuses on connected features in both channel and spatial directions to generate refined fused feature maps. The first stage downsamples the input features using mean pooling, max pooling, and median pooling, reducing the feature resolution to the n×n size required for MMTSAFB in each branch. Then, a shared Transformer layer is used to concatenate the features across the channels. Finally, an n×n convolution and a sigmoid function are applied to obtain a Transformer-based three-channel attention layer. The second stage performs spatial fusion, focusing the input features using mean pooling, max pooling, and median pooling. A convolution and a sigmoid function are then applied to obtain spatial attention, which is finally multiplied and fused with the input features to obtain the final fused feature. MMTSAFB effectively fuses information from the RGB image, panoptic segmentation semantic image, and sparse depth image, while reducing the impact of noise on the completion result.

[0071] In one feasible implementation, decoder features from the SC and SP branches are fused into encoder features of the corresponding 3S branch via MMTAFB; in the 3S branch, MMTAFB uses feature maps of the three modes as input and output to refine the fused feature maps.

[0072] In one feasible implementation, an attention fusion module MMTSAFB with different input resolutions is constructed to fuse features between the three branches to obtain preliminary depth completion results. Color depth, panoramic segmentation semantic depth and sparse depth are adaptively fused to generate the output of our designed 3U-DenseNet backbone.

[0073] One feasible implementation method is, for example Figure 8As shown, the 3S branch network primarily learns deep features from the coarse depth maps (SC-Depth and SP-Depth) and sparse depth maps (Sparse-Depth) of the preceding SC and SP branches. These features help generate accurate dense depth maps. It takes the concatenation of SC-Depth, SP-Depth, and Sparse-Depth as input to output a dense depth map (3S-Depth). Similar to the feature fusion strategy in the SC and SP branches, we fuse the decoder features from the SC and SP branches into the encoder features of the corresponding 3S branch using MMTAFB. In the 3S branch, MMTAFB uses the feature maps of the three modes as input and output to refine the fused feature map. The refined fused feature map contains useful information from the SC and SP branches, guiding the 3S branch to learn effective deep feature representations. The network structure is similar to the SP branch; the encoder consists of one convolutional layer and five ResNet blocks, while MMTAFB is used for focusing and fusion. The decoder also consists of one convolutional layer and five transposed layers. The connection between the encoder and decoder also employs dense skip connections to more fully integrate deep features from different layers.

[0074] S104: Construct a convolutional spatial propagation network CSPN++, input the preliminary depth completion results into CSPN++, obtain the depth completion results, and complete the attention-based panoramic perception-guided depth completion.

[0075] In one feasible implementation, the present invention further uses CSPN++ to refine the dense depth map generated by the 3U-DenseNet backbone. CSPN++ can first recover depth values ​​at effective pixels, and secondly, it can make the completed dense depth map transition smoothly between adjacent objects.

[0076] In this embodiment of the invention, a 3U-DenseNet network was designed and combined with a Convolutional Spatial Propagation Network (CSPN++) for post-processing to achieve end-to-end depth completion. The attention-based panoramic perception-guided depth completion method has high accuracy and good completion effect. By fusing panoramic segmentation images, the depth changes at object boundaries are more subtle.

[0077] On the one hand, such as Figure 9 As shown, a depth completion device 300 based on attention-based panoramic perception guidance is provided, which is applied to any of the methods described above; including:

[0078] The data acquisition module 310 is used to acquire RGB image data captured by a visible light camera and sparse depth image data scanned by a lidar.

[0079] The RGB image processing module 320 is used to process RGB image data into panoramic segmentation image data of the image to be processed through a panoramic segmentation network.

[0080] The preliminary depth completion module 330 is used to construct a 3U-DenseNet network. It inputs the RGB image data, panoramic segmentation image data and sparse depth image data into the 3U-DenseNet network, and outputs the preliminary depth completion result through the multimodal attention fusion module MMTSAFB.

[0081] The depth completion module 340 is used to construct a convolutional spatial propagation network CSPN++, input the preliminary depth completion result into CSPN++, obtain the depth completion result, and complete the depth completion based on attention-guided panoramic perception.

[0082] Optionally, RGB images can be converted into panoramic segmentation semantic image data using a pre-trained Mask2Former model.

[0083] Optionally, the 3U-DenseNet network includes three branches: a color-guided SC branch, a panoptic segmentation semantic-guided SP branch, and a depth-guided 3S branch.

[0084] Optionally, both the color-guided SC branch and the panorama segmentation semantic-guided SP branch adopt a densely skip-connected encoder-decoder network architecture.

[0085] The encoder includes a convolutional layer and five ResNet blocks, and six BT blocks are added after the convolutional layer and ResNet blocks to form another branch.

[0086] The decoder consists of one convolutional layer and five transposed convolutional layers, which upsample the input data and fuse the features of each decoder layer.

[0087] Optionally, the preliminary completion module 330 is further used to input the RGB image data and sparse depth image data into the color guide SC branch to obtain color dense depth map SC-Depth data with scene color cues;

[0088] The predicted color dense depth map (SC-Depth) data of the SC branch, as well as the panoramic semantic image and sparse depth image data, are input into the panoramic segmentation semantic guidance (SP) branch to obtain panoramic semantic depth data.

[0089] Sparse depth image data, color dense depth map (SC-Depth) data, and panoramic semantic depth data are input into the depth-guided 3S branch to obtain dense depth map data.

[0090] Optionally, the preliminary completion module 330 is further used to input RGB image data and sparse depth image data into the color-guided SC branch in a concatenated manner, and output color dense depth map SC-Depth data.

[0091] Optionally, the preliminary completion module 330 is further used to fuse the decoder features of the SC branch into the encoder features of the corresponding SP branch through MMTAFB; the SP branch takes the stitching of color dense depth map SC-Depth data, panoramic semantic image and sparse depth map as input, and outputs SP dense depth map, which is panoramic semantic depth data.

[0092] Optionally, the preliminary completion module 330 is further used to fuse the decoder features from the SC and SP branches into the encoder features of the corresponding 3S branch through the multimodal attention fusion module MMTSAFB; wherein, in the 3S branch, MMTSAFB uses the feature maps of the three modes as input and output to refine the fused feature maps.

[0093] Optionally, the preliminary completion module 330 is further used to construct an attention fusion module MMTSAFB with different input resolutions to fuse the features between the three branches and obtain preliminary depth completion results.

[0094] In this embodiment of the invention, a 3U-DenseNet network was designed and combined with a Convolutional Spatial Propagation Network (CSPN++) for post-processing to achieve end-to-end depth completion. The attention-based panoramic perception-guided depth completion method has high accuracy and good completion effect. By fusing panoramic segmentation images, the depth changes at object boundaries are more subtle.

[0095] Figure 10 This is a schematic diagram of the structure of an electronic device 400 provided in an embodiment of the present invention. The electronic device 400 can vary considerably due to differences in configuration or performance. It may include one or more central processing units (CPUs) 401 and one or more memories 402. The memory 402 stores at least one instruction, which is loaded and executed by the processor 401 to implement the steps of the following method for automatic modeling of a robotic warehousing system:

[0096] S1: Acquire RGB image data captured by a visible light camera and sparse depth image data scanned by a lidar;

[0097] S2: The RGB image data is processed into panoramic segmentation image data of the image to be processed through a panoramic segmentation network;

[0098] S3: Construct a 3U-DenseNet network, input the RGB image data, panoramic segmentation image data and sparse depth image data into the 3U-DenseNet network, and output the preliminary depth completion result through the multimodal attention fusion module MMTSAFB;

[0099] S4: Construct a convolutional spatial propagation network CSPN++, input the preliminary depth completion result into CSPN++, obtain the depth completion result, and complete the depth completion based on attention-guided panoramic perception.

[0100] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to complete the method for automatically modeling the robotic warehousing system described above. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0101] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A depth completion method based on attention-based panoramic perception guidance, characterized in that, include: S1: Acquire RGB image data captured by a visible light camera and sparse depth image data scanned by a lidar; S2: The RGB image data is processed into panoramic segmentation image data of the image to be processed through a panoramic segmentation network; S3: Construct a 3U-DenseNet network, input the RGB image data, panoramic segmentation image data and sparse depth image data into the 3U-DenseNet network, and output the preliminary depth completion result through the multimodal attention fusion module MMTSAFB; The RGB image data and sparse depth image data are input into the color-guided SC branch to obtain color dense depth map SC-Depth data with scene color cues. The predicted color dense depth map (SC-Depth) data of the SC branch, as well as the panoramic semantic image and sparse depth image data, are input into the panoramic segmentation semantic guidance (SP) branch to obtain panoramic semantic depth data. Sparse depth image data, color dense depth map (SC-Depth) data, and panoramic semantic depth data are input into the depth-guided 3S branch to obtain dense depth map data. RGB image data and sparse depth image data are input into the color-guided SC branch in a concatenated manner, and color dense depth map SC-Depth data is output. The decoder features of the SC branch are fused into the encoder features of the corresponding SP branch through MMTAFB. The SP branch takes the stitching of color dense depth map SC-Depth data, panoramic semantic image and sparse depth map as input and outputs SP dense depth map, which is panoramic semantic depth data. The decoder features from the SC and SP branches are fused into the encoder features of the corresponding 3S branches through the multimodal attention fusion module MMTSAFB; in the 3S branches, MMTSAFB uses the feature maps of the three modes as input and output to refine the fused feature maps. An attention fusion module MMTSAFB with different input resolutions is constructed to fuse the features between the three branches and obtain preliminary depth completion results; S4: Construct a convolutional spatial propagation network CSPN++, input the preliminary depth completion result into CSPN++, obtain the depth completion result, and complete the depth completion based on attention-guided panoramic perception.

2. The depth completion method based on attention-guided panoramic perception according to claim 1, characterized in that, In step S2, the RGB image data is processed into panoramic segmentation data of the image to be processed using a panoramic segmentation network, including: RGB images are converted into panoramic segmentation semantic image data using a pre-trained Mask2Former model.

3. The depth completion method based on attention-guided panoramic perception according to claim 1, characterized in that, In step S3, the 3U-DenseNet network includes three branches: the color-guided SC branch, the panoptic segmentation semantic-guided SP branch, and the depth-guided 3S branch.

4. The depth completion method based on attention-guided panoramic perception according to claim 3, characterized in that, Both the color-guided SC branch and the panoramic segmentation semantic-guided SP branch adopt a densely skip-connected encoder-decoder network architecture. The encoder includes a convolutional layer and five ResNet blocks, and six BT blocks are added after the convolutional layer and ResNet blocks to form another branch. The decoder consists of one convolutional layer and five transposed convolutional layers, which upsample the input data and fuse the features of each decoder layer.

5. A depth completion device based on attention-guided panoramic perception, characterized in that, The apparatus is used in the method as described in any one of claims 1-4; comprising: The data acquisition module is used to acquire RGB image data captured by a visible light camera and sparse depth image data scanned by a lidar. The RGB image processing module is used to process RGB image data into panoramic segmentation image data of the image to be processed through a panoramic segmentation network. The preliminary depth completion module is used to construct a 3U-DenseNet network. The RGB image data, panoramic segmentation image data, and sparse depth image data are input into the 3U-DenseNet network, and the preliminary depth completion result is output after passing through the multimodal attention fusion module MMTSAFB. The depth completion module is used to construct a convolutional spatial propagation network CSPN++, input the preliminary depth completion result into CSPN++, obtain the depth completion result, and complete the depth completion based on attention-guided panoramic perception.

Images