Depth image completion method and device based on double-branch cross-modal diffusion, equipment and medium

Abstract

The embodiment of the invention provides a depth image completion method and device based on double-branch cross-modal diffusion, equipment and a medium, and relates to the field of image processing. The method comprises the following steps: performing feature extraction on an RGB image through a visual semantic branch of a pre-trained depth image completion model to obtain an RGB feature map, processing the RGB feature map through an encoder in the depth completion branch to obtain a depth feature map, endowing the depth feature map with a first pixel-by-pixel weight through a feature similarity network to obtain a first pixel-by-pixel weight and a second pixel-by-pixel weight; and giving a second pixel-by-pixel weight to the RGB feature map, fusing the depth feature map and the RGB feature map at each pixel position according to the second pixel-by-pixel weight, and decoding the fused feature map to obtain a complete depth map. Through the mode, global context perception is enhanced, and RGB textures and depth sets are dynamically fused, so that requirements of different regions are adapted, plane consistency and edge smoothness are constrained, geometric alignment is improved, and efficient and accurate completion of the depth image is realized.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a method, apparatus, device and medium for depth image completion based on bi-branch cross-modal diffusion. Background Technology

[0002] Depth completion is a key technology for recovering dense and accurate depth maps from sparse or incomplete depth data, providing fundamental geometric information for downstream 3D vision tasks. Among related technologies, traditional depth completion methods mainly fall into three categories: filter-based local propagation methods, which utilize the spatial proximity and depth similarity of depth maps to fill missing values ​​through weighted averaging, but cannot handle large-area missing areas or completion across object boundaries, easily leading to depth blurring; geometric prior-based methods, which estimate the depth of missing regions through geometric projection, but are only applicable to scenes strictly following the Manhattan structure, lacking generalization ability; and iterative methods based on sparse propagation, which propagate sparse depth values ​​iteratively, but rely on hand-designed affinity rules, cannot capture long-distance dependencies, and are prone to noise.

[0003] However, in the process of realizing this application, the inventors found that the prior art has at least the following problems: insufficient geometric consistency, misalignment between the completed area and the known depth structure; limited global perception capability of the backbone network, resulting in discontinuous completion of large-area missing areas; and coarse cross-modal fusion, which cannot adapt to the information needs of different regions.

[0004] Therefore, how to efficiently and accurately complete depth images has become an urgent problem to be solved. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a depth image completion method, apparatus, device, and medium based on bi-branch cross-modal diffusion, aiming to efficiently and accurately complete depth images.

[0006] The first aspect of this application provides a depth image completion method based on bi-branch cross-modal diffusion, the method comprising: The RGB image is feature-extracted by the visual semantic branch of a pre-trained deep image completion model to obtain an RGB feature map. The encoder in the depth completion branch of the pre-trained depth image completion model processes the mask image corresponding to the RGB image, the preliminary completed depth map corresponding to the RGB image, the masked depth map, and the noisy depth map after adding noise to the preliminary completed depth map to obtain a depth feature map. The feature similarity network in the encoder assigns a first pixel-wise weight to the depth feature map and a second pixel-wise weight to the RGB feature map. At each pixel location, the depth feature map and the RGB feature map are fused based on the first pixel-wise weight and the second pixel-wise weight to fuse the RGB texture and depth geometry, resulting in a fused feature map. The fused feature map is processed by the decoder in the depth completion branch to obtain the complete depth map corresponding to the RGB image; the encoder and decoder in the depth completion branch form a U-Net architecture based on diffusion denoising.

[0007] Optionally, the visual semantic branch is a four-layer pyramid Transformer encoder corresponding to four stages; the method involves extracting features from the RGB image using the visual semantic branch of a pre-trained deep image completion model to obtain an RGB feature map, and includes: In the first stage, a multi-head attention mechanism is adopted to process the RGB image based on N first-layer Transformer blocks to obtain a first feature map. The number of channels of the first feature map is C1, and the resolution is 1 / 4 of that of the RGB image. In the second stage, a multi-head attention mechanism is adopted to process the first feature map based on N second-layer Transformer blocks to obtain a second feature map. The second feature map has C2 channels and a resolution of 1 / 8 of the RGB image; N is an integer greater than or equal to 2. In the third stage, a multi-head attention mechanism is adopted to process the second feature map based on M third-layer Transformer blocks to obtain a third feature map. The second feature map has C3 channels and a resolution of 1 / 16 of the RGB image. In the fourth stage, a multi-head attention mechanism is adopted to process the third feature map based on M fourth-layer Transformer blocks to obtain the RGB feature map. The number of channels of the RGB feature map is C4, the resolution is 1 / 32 of the RGB image, and M is an integer greater than N.

[0008] Optionally, features are extracted from the RGB image using the visual semantic branch of a pre-trained deep image completion model to obtain an RGB feature map. The method includes: In the second stage, a first spatial reduction attention layer is deployed before the N second-layer Transformer blocks; the first feature map is processed through the first spatial reduction attention layer, and a multi-head attention mechanism is adopted to process the output of the first spatial reduction attention layer based on the N second-layer Transformer blocks to obtain the second feature map; In the third stage, a second spatial reduction attention layer is deployed before the M third-layer Transformer blocks; the second feature map is processed through the second spatial reduction attention layer, and a multi-head attention mechanism is adopted to process the output of the second spatial reduction attention layer based on the M second-layer Transformer blocks to obtain the third feature map; In the fourth stage, a third spatial reduction attention layer is deployed before the M fourth-layer Transformer blocks; the third feature map is processed through the third spatial reduction attention layer, and a multi-head attention mechanism is adopted to process the output of the third spatial reduction attention layer based on the M fourth-layer Transformer blocks to obtain the RGB feature map.

[0009] Optionally, a first pixel-wise weight is assigned to the deep feature map and a second pixel-wise weight is assigned to the RGB feature map through a feature similarity network in the encoder, the method comprising: The alignment module in the encoder performs alignment processing on the RGB feature map and the depth feature map; The semantic similarity and geometric consistency between the aligned RGB feature map and the aligned depth feature map at each pixel position are determined by the feature similarity network in the encoder. Based on the semantic similarity and geometric consistency of each pixel location, a first pixel-wise weight is assigned to the deep feature map, and a second pixel-wise weight is assigned to the RGB feature map.

[0010] Optionally, the relationship between the first pixel-wise weight and the second pixel-wise weight is determined according to the following steps: For a given pixel location, if the semantic similarity of the pixel location is greater than the geometric consistency, it is determined that the texture information corresponding to the pixel location is greater than the structural information corresponding to the pixel location, and the second weight corresponding to the pixel location is greater than the first weight corresponding to the pixel location. For a given pixel location, if the semantic similarity of that pixel location is less than the geometric consistency, it is determined that the texture information corresponding to that pixel location is less than the structural information corresponding to that pixel location, and the second weight corresponding to that pixel location is less than the first weight corresponding to that pixel location.

[0011] Optionally, the method further includes: The pre-trained depth image completion model is trained using multiple training samples. Each training sample carries a depth label and includes: an RGB image sample, a mask image sample corresponding to the RGB image sample, a preliminary completed depth map sample corresponding to the RGB image sample, a masked depth map sample, and a noisy depth map sample after adding noise to the preliminary completed depth map sample. The loss function values ​​used to obtain the pre-trained depth image completion model include at least the edge smoothing loss function value, which is the edge gradient calculated using the Sobel gradient operator for the completed depth map output by the depth image completion model to be trained during the training process.

[0012] Optionally, the method further includes obtaining the loss function value used by the pre-trained deep image completion model, and then further comprising: The plane fitting loss function value is obtained according to the following steps: Planar regions are obtained by performing planar recognition on the completed depth map output by the depth image completion model to be trained during the training process. The predicted depth of the planar region is determined based on the completed depth map output by the depth image completion model to be trained during the training process. The plane fitting loss function value is determined based on the predicted depth of the planar region and the depth labels carried by the training samples of the depth image completion model to be trained.

[0013] A second aspect of this application provides a depth image completion device based on bi-branch cross-modal diffusion, the device comprising: The RGB feature map determination module is used to extract features from RGB images through the visual semantic branch of a pre-trained deep image completion model to obtain RGB feature maps. The depth feature map determination module is used to process the mask image corresponding to the RGB image, the preliminary completed depth map corresponding to the RGB image, the masked depth map, and the noisy depth map after adding noise to the preliminary completed depth map through the encoder in the depth completion branch of the pre-trained depth image completion model to obtain the depth feature map. The weighting module is used to assign a first pixel-wise weight to the depth feature map and a second pixel-wise weight to the RGB feature map through the feature similarity network in the encoder. The feature map fusion module is used to fuse the depth feature map and the RGB feature map at each pixel location based on the first pixel-wise weight and the second pixel-wise weight, so as to fuse the RGB texture and depth geometry to obtain the fused feature map; The decoding module is used to process the fused feature map through the decoder in the depth completion branch to obtain the complete depth map corresponding to the RGB image; the encoder and decoder in the depth completion branch form a U-Net architecture based on diffusion denoising.

[0014] Optionally, the RGB feature map determination module includes: The first feature map determination submodule is used in the first stage to process the RGB image based on N first-layer Transformer blocks using a multi-head attention mechanism to obtain a first feature map. The first feature map has C1 channels and a resolution of 1 / 4 of the RGB image. The second feature map determination submodule is used in the second stage to process the first feature map using a multi-head attention mechanism based on N second-layer Transformer blocks to obtain a second feature map. The second feature map has C2 channels and a resolution of 1 / 8 of the RGB image; N is an integer greater than or equal to 2. The third feature map determination submodule is used in the third stage to process the second feature map based on M third-layer Transformer blocks using a multi-head attention mechanism to obtain the third feature map. The second feature map has C3 channels and a resolution of 1 / 16 of the RGB image. The RGB feature map determination submodule is used in the fourth stage to process the third feature map using a multi-head attention mechanism based on M fourth-layer Transformer blocks to obtain the RGB feature map. The number of channels of the RGB feature map is C4, the resolution is 1 / 32 of the RGB image, and M is an integer greater than N.

[0015] Optionally, the RGB feature map determination module further includes: The first spatial reduction attention submodule is used in the second stage to deploy a first spatial reduction attention layer before N second-layer Transformer blocks; the first feature map is processed through the first spatial reduction attention layer, and a multi-head attention mechanism is adopted to process the output of the first spatial reduction attention layer based on N second-layer Transformer blocks to obtain the second feature map; The second spatial reduction attention submodule is used to deploy a second spatial reduction attention layer before the M third-layer Transformer blocks in the third stage; the second feature map is processed through the second spatial reduction attention layer, and a multi-head attention mechanism is adopted to process the output of the second spatial reduction attention layer based on the M second-layer Transformer blocks to obtain the third feature map; The third spatial reduction attention submodule is used to deploy a third spatial reduction attention layer before the M fourth-layer Transformer blocks in the fourth stage; the third feature map is processed through the third spatial reduction attention layer, and a multi-head attention mechanism is adopted to process the output of the third spatial reduction attention layer based on the M fourth-layer Transformer blocks to obtain the RGB feature map.

[0016] Optionally, the weight assignment module includes: The alignment processing submodule is used to align the RGB feature map and the depth feature map using the alignment module in the encoder; The feature similarity judgment submodule is used to determine the semantic similarity and geometric consistency between the aligned RGB feature map and the aligned depth feature map at each pixel position through the feature similarity network in the encoder. The weighting submodule is used to assign a first pixel-wise weight to the deep feature map and a second pixel-wise weight to the RGB feature map based on the semantic similarity and geometric consistency of each pixel location.

[0017] Optionally, the weights are assigned to the sub-modules, including: The first weight determination unit is used to determine, for a pixel position, that if the semantic similarity of the pixel position is greater than the geometric consistency, the texture information corresponding to the pixel position is greater than the structural information corresponding to the pixel position, and the second weight corresponding to the pixel position is greater than the first weight corresponding to the pixel position. The second weight determination unit is used to determine, for a pixel position, that if the semantic similarity of the pixel position is less than the geometric consistency, the texture information corresponding to the pixel position is less than the structural information corresponding to the pixel position, and the second weight corresponding to the pixel position is less than the first weight corresponding to the pixel position.

[0018] Optionally, the device further includes: The image completion model training module is used to train the depth image completion model to be trained using multiple training samples to obtain the pre-trained depth image completion model. Each training sample carries a depth label and includes: an RGB image sample, a mask image sample corresponding to the RGB image sample, a preliminary completed depth map sample corresponding to the RGB image sample, a masked depth map sample, and a noisy depth map sample after adding noise to the preliminary completed depth map sample. The first loss function module is used to obtain the loss function value used by the pre-trained depth image completion model, including at least: edge smoothing loss function value, which is the edge gradient calculated using the Sobel gradient operator for the completed depth map output by the depth image completion model to be trained during the training process.

[0019] Optionally, the device further includes: a second loss function module for plane fitting of loss function values; The second loss function module includes: The planar recognition submodule is used to perform planar recognition on the completed depth map output by the depth image completion model to be trained during the training process, so as to obtain planar regions. The predicted depth determination submodule is used to determine the predicted depth of the planar region based on the completed depth map output by the depth image completion model to be trained during the training process. The plane fitting loss function value determination submodule is used to determine the plane fitting loss function value based on the predicted depth of the plane region and the depth labels carried by the training samples of the depth image completion model to be trained.

[0020] A third aspect of this application provides an electronic device, the electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the depth image completion method based on bi-branch cross-modal diffusion as described in the first aspect of this application.

[0021] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the depth image completion method based on bi-branch cross-modal diffusion according to the first aspect of this application.

[0022] The fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the depth image completion method based on bi-branch cross-modal diffusion of the first aspect of this application.

[0023] The depth image completion method based on bi-branch cross-modal diffusion in this application first extracts features from an RGB image using the visual semantic branch of a pre-trained depth image completion model to obtain an RGB feature map. Then, the encoder in the depth completion branch processes the mask image corresponding to the RGB image, the initially completed depth map corresponding to the RGB image, the masked depth map, and the noisy depth map with noise added to the initially completed depth map to obtain a depth feature map. Next, the feature similarity network in the encoder assigns a first pixel-wise weight to the depth feature map and a second pixel-wise weight to the RGB feature map. At each pixel position, based on the first and second pixel-wise weights, the depth feature map and the RGB feature map are fused to fuse RGB texture and depth geometry to obtain a fused feature map. Finally, the decoder in the depth completion branch processes the fused feature map to obtain the complete depth map corresponding to the RGB image. The encoder and decoder in the depth completion branch form a U-Net architecture based on diffusion denoising.

[0024] Unlike related technologies that perform depth completion, this application employs a dual-branch approach, simultaneously processing RGB and depth features. By combining the processing results of each feature map to determine the final depth map, it enhances global context awareness and addresses the issue of structural inconsistencies in the completion result caused by large areas of depth information. Furthermore, by determining different weights for RGB and depth feature maps, it dynamically fuses RGB textures and depth sets, thus adapting to the needs of different regions. Depth completion is achieved during iterative denoising. Additionally, it proposes incorporating geometric prior loss to constrain plane consistency and edge smoothness, improving geometric alignment and achieving efficient and accurate depth image completion.

[0025] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0026] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart of a depth image completion method based on bi-branch cross-modal diffusion proposed in an embodiment of this application; Figure 2 This is a schematic diagram of the depth image completion process proposed in an embodiment of this application; Figure 3This is a structural block diagram of a depth image completion device based on dual-branch cross-modal diffusion proposed in an embodiment of this application; Figure 4 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0027] The embodiments of this application will now be described in detail. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0028] In related technologies, traditional depth completion methods fall into several categories, including filtering methods, geometric prior methods, and morphological methods. However, each has its limitations. For example, the local window weighted averaging in filtering methods relies too heavily on local information, leading to long-distance semantic breaks and making them unsuitable for handling large-area depth gaps. Alternatively, geometric prior methods, which use geometric projection to complete missing parts, are ineffective in unstructured scenes, and fixed geometric constraints cannot adapt to dynamic changes. Morphological methods gradually fill holes in the depth map through operations such as dilation and erosion, but achieving real-time completion solely through pure depth manipulation limits their processing capabilities when dealing with complex geometric structures.

[0029] Building upon traditional depth completion methods, researchers have proposed a deep learning approach that utilizes neural networks to automatically extract the correlation between depth and visual features. This avoids the limitations of manually designed rules in traditional methods, achieving more robust completion. However, deep learning methods, whether focusing on completing globally sparse depth data into dense depth data or using monocular depth estimators to infer the depth of a single image, all have limitations. They may lack generalization ability, limiting applicability to downstream tasks, or produce severe geometric inconsistencies in situations with complex geometric structures such as edges.

[0030] Therefore, in order to at least partially solve one or more of the above problems and other potential problems, this application proposes a depth image completion method based on bi-branch cross-modal diffusion. This method can not only enhance global context awareness and solve the problem of discontinuous large-area missing structures, but also adapt to different regional requirements, complete depth completion in iterative denoising, and improve geometric alignment by adding geometric prior loss to constrain plane consistency and edge smoothness.

[0031] Please refer to the details. Figure 1 , Figure 1 This is a flowchart of a depth image completion method based on bi-branch cross-modal diffusion proposed in an embodiment of this application. Figure 1 As shown, the method may include steps S101 to S105: Step S101: Extract features from the RGB image using the visual semantic branch of the pre-trained deep image completion model to obtain the RGB feature map; Step S102: The encoder in the depth completion branch of the pre-trained depth image completion model processes the mask image corresponding to the RGB image, the preliminary completed depth map corresponding to the RGB image, the masked depth map, and the noisy depth map after adding noise to the preliminary completed depth map to obtain a depth feature map. Step S103: Assign a first pixel-wise weight to the depth feature map and a second pixel-wise weight to the RGB feature map through the feature similarity network in the encoder; Step S104: At each pixel location, based on the first pixel-wise weight and the second pixel-wise weight, the depth feature map and the RGB feature map are fused to fuse the RGB texture and depth geometry to obtain the fused feature map; Step S105: The fused feature map is processed by the decoder in the depth completion branch to obtain the complete depth map corresponding to the RGB image; the encoder and decoder in the depth completion branch form a U-Net architecture based on diffusion denoising.

[0032] In this application, during the depth completion process of the target image, the entire completion process is divided into two branches with different modalities by a pre-trained depth image completion model. First, the visual semantic branch is used to extract features from the RGB image to obtain the RGB feature map of the target image in the visual semantic branch, which is used to extract the global semantics and texture details of the scene. Then, the depth information of the target image is processed in the depth completion branch of the depth image completion model, focusing on the geometric restoration of the depth-deficient region. The depth feature map corresponding to the RGB image is obtained by encoding and processing the mask image corresponding to the RGB image, the preliminary completed depth map corresponding to the RGB image, the masked depth map, and the noisy depth map after adding noise to the preliminary completed depth map.

[0033] The target image is processed separately by a visual semantic branch and a depth completion branch to obtain feature maps for each branch. Using the similarity feature network proposed in this application, weight values ​​are assigned pixel-wise to the feature maps of both branches, namely, a first pixel-wise weight and a second pixel-wise weight. This allows for dynamic fusion of the RGB feature map and the depth feature map based on the weight values ​​at each pixel position, thereby combining RGB texture and depth geometry to determine the fused feature map. Finally, the fused feature map is decoded, and diffusion denoising is performed during the encoding and decoding processes of the entire depth branch, ultimately yielding the completed depth map.

[0034] In this way, feature information is extracted from the visual semantic branch and the depth completion branch respectively. Then, based on the weight relationship between the two, the influence of the visual semantic branch is enhanced in texture-rich areas, and depth geometric information is preferentially retained in structured areas to improve the influence of the depth completion branch. This enables dynamic cross-modal feature fusion, thereby achieving accurate completion of depth-deficient areas and generating a complete depth map that is highly aligned with the known depth in terms of geometric structure and size.

[0035] Step S101: Extract features from the RGB image using the visual semantic branch of the pre-trained deep image completion model to obtain the RGB feature map.

[0036] In this embodiment, when the target image to be processed is obtained, the RGB image needs to be determined first, and then converted into a form that can be processed by the visual semantic branch of the pre-trained deep image completion model. Specifically, in an optional embodiment, the RGB image can be converted into a 4-channel feature map by a 4-channel feature encoder, for example by using 3×3 convolution combined with BatchNorm (batch normalization) and ReLU activation function, preserving the texture details and global semantics, and then encoded by the encoder to be converted into the RGB feature map required for subsequent fusion.

[0037] Step S102: The encoder in the depth completion branch of the pre-trained depth image completion model processes the mask image corresponding to the RGB image, the preliminary completed depth map corresponding to the RGB image, the masked depth map, and the noisy depth map after adding noise to the preliminary completed depth map to obtain a depth feature map.

[0038] In this embodiment, when the target image to be processed is acquired, the depth completion branch of the depth image completion model, which can be pre-trained and runs concurrently with the visual semantic branch, performs geometric restoration of the depth-deficient region. Specifically, based on the target image, the incomplete depth map, the corresponding mask image, and the noisy depth map are determined and processed to obtain the masked depth map, which is then used as input and processed by the depth completion branch. In an optional embodiment, the depth completion branch can be based on the U-Net architecture and accurately restore the depth geometric features and determine the depth feature map through an "encoder-decoder + skip connection" approach. By inputting the masked depth potential data and the encoded mask, fine-grained shape and boundary information that is crucial for accurate depth restoration can be preserved.

[0039] Step S103: Assign a first pixel-wise weight to the depth feature map and a second pixel-wise weight to the RGB feature map through the feature similarity network in the encoder.

[0040] In this embodiment, after the RGB feature map and depth feature map are determined by the two branches of the pre-trained depth image completion model, they can be fused. Before fusion, a first pixel-wise weight can be assigned to the depth feature map and a second pixel-wise weight to the RGB feature map based on the semantic similarity and geometric consistency at each pixel position in the RGB and depth feature maps. This avoids the adaptation problem of splicing with "fixed weights" in related technologies. By comparing the semantic similarity and geometric consistency of the two branches, higher weights can be assigned to the RGB feature map in texture-rich regions and higher weights to the depth feature map in structured regions, thus achieving dynamic adaptation.

[0041] Step S104: At each pixel location, based on the first pixel-wise weight and the second pixel-wise weight, the depth feature map and the RGB feature map are fused to fuse the RGB texture and depth geometry to obtain the fused feature map.

[0042] In this embodiment of the application, after determining the first pixel weight and the second pixel weight of each pixel position, the relative position of each pixel in the depth feature map and the RGB feature map can be weighted and fused according to the first pixel weight and the second pixel weight, thereby realizing the dynamic fusion of the depth feature map and the RGB feature map, enhancing the influence of RGB texture features in texture-rich areas, while preserving depth geometric information in structured areas, and determining the fused feature map.

[0043] Step S105: The fused feature map is processed by the decoder in the depth completion branch to obtain the complete depth map corresponding to the RGB image; the encoder and decoder in the depth completion branch form a U-Net architecture based on diffusion denoising.

[0044] In this embodiment, after determining the fused feature map, it can be processed by the decoder in the depth completion branch. Specifically, the decoder can restore the resolution through transposed convolution upsampling, and combine skip connections to stitch together features at the same level as the decoder to supplement the shallow details lost during downsampling (such as the depth of small objects in the image and the transition of edges), thereby obtaining the complete depth map corresponding to the RGB image. Furthermore, in the depth completion branch, a diffusion denoising U-Net architecture is formed based on the encoder and decoder. Taking the noisy depth map as the starting input, it iterates during the encoding and decoding process to gradually remove noise and restore clear depth information. Moreover, since this application integrates the visual semantic branch and the depth completion branch, during the iteration process, the depth image completion model can adjust the denoising operation according to the RGB features and depth features of the target image, thereby determining the final complete depth map with noise removed.

[0045] In conjunction with the above embodiments, in one implementation, this application also provides a depth image completion method based on dual-branch cross-modal diffusion, wherein the visual semantic branch is a four-layer pyramid Transformer encoder corresponding to four stages. The RGB image is feature-extracted by using the visual semantic branch of a pre-trained deep image completion model, resulting in an RGB feature map, which includes the following: First, in the first stage, a multi-head attention mechanism is adopted to process the RGB image based on N first-layer Transformer blocks to obtain a first feature map. The number of channels of the first feature map is C1, and the resolution is 1 / 4 of that of the RGB image.

[0046] In this embodiment, to maximize the preservation of texture details and global semantics of the target image in the visual semantic branch, a four-layer pyramid Transformer encoder is proposed to extract features from the RGB image in the visual semantic branch. This allows for accurate capture of the texture, edges, and scene layout of the RGB image. Specifically, the four-layer pyramid Transformer encoder is divided into four stages in the visual semantic branch. First, in the first stage, the RGB image is segmented into multiple smaller patch blocks through patch embedding, and spatial location information is encoded by overlaying position embedding. Next, multi-head attention is used to capture multi-scale details (such as the edges of furniture or the layout on the wall in the image) in parallel. More attention is allocated to parts with more complex textures and higher importance, while less attention is allocated to parts with relatively simple texture information. Finally, the processing result of this layer is used as the first feature map. The non-linear interaction of features is enhanced through a feed-forward neural network (FFN), and layer normalization and element-wise addition are used. The method of Add) is used to achieve residual connections, stabilize training and preserve feature levels, so that the first feature map obtained in the first stage is used as input to the next stage for processing.

[0047] In this process, the first feature map obtained in the first stage has C1 channels and a resolution of 1 / 4 that of the RGB image. In other words, in the four-layer pyramid Transformer encoder proposed in this application, the earlier processing stages focus more on the shallower and more comprehensive semantic information in the RGB image. In the subsequent processing stages, the processing layers are increased layer by layer to gradually increase the attention to texture details, thereby ultimately extracting global scene semantics and local texture features. This effectively captures the global layout of the scene and the long-distance deep correlation, solves the problem of limited receptive field and difficulty in handling large-area continuous missing areas in traditional networks, and solves the problem of fragmented and discontinuous structure of the completion area caused by this processing in related technologies. This improves the overall coherence of the completion results in scenes with large-area missing areas.

[0048] Then, in the second stage, a multi-head attention mechanism is used to process the first feature map based on N second-layer Transformer blocks to obtain a second feature map. The second feature map has C2 channels and a resolution of 1 / 8 of the RGB image; N is an integer greater than or equal to 2. In the third stage, a multi-head attention mechanism is adopted to process the second feature map based on M third-layer Transformer blocks to obtain a third feature map. The second feature map has C3 channels and a resolution of 1 / 16 of the RGB image. In the fourth stage, a multi-head attention mechanism is adopted to process the third feature map based on M fourth-layer Transformer blocks to obtain the RGB feature map. The number of channels of the RGB feature map is C4, the resolution is 1 / 32 of the RGB image, and M is an integer greater than N.

[0049] In this embodiment, after completing the first stage of RGB image processing, subsequent processing stages are required. The processing methods of each stage are similar to those of the first stage, all using a multi-head attention mechanism based on Transformer blocks to process the feature map. However, the difference lies in the fact that the resolution of the feature map is gradually reduced by adjusting the size of the patch blocks segmented in each layer, from 1 / 4 of the RGB image in the first stage to 1 / 32 of the RGB image in the fourth stage. Furthermore, by setting different numbers of Transformer blocks, a feature extraction structure in the form of a "pyramid" is formed, thereby achieving both the preservation of detailed information in the RGB image and the capture of global contextual information.

[0050] In an optional embodiment, the structure of the four-layer pyramid Transformer encoder corresponding to the four stages proposed in this application can be as shown in Table 1. Each stage includes a patch embedding and a Transformer encoder, but the number of Transformer encoders varies slightly depending on the stage progress. There are two in stages 1 and 2, and three in stages 3 and 4. In the first stage, the input RGB image is segmented into fine-grained patches of 4×4 pixels, which are transformed into embedding vectors through linear projection and combined with position embeddings and passed to the Transformer encoder to output a high-resolution feature map. In subsequent stages, the feature map of the previous stage is processed in a similar manner to obtain feature maps with different numbers of channels and resolutions as outputs. Finally, after processing in the fourth stage, the RGB feature map is output.

[0051] Table 1. Structure of the Four-Layer Pyramid Transformer Encoder

[0052] In conjunction with the above embodiments, in one implementation, this application also provides a depth image completion method based on bi-branch cross-modal diffusion. This method extracts features from an RGB image using the visual semantic branch of a pre-trained depth image completion model to obtain an RGB feature map, specifically including the following: First, in the second stage, a first spatial reduction attention layer is deployed before the N second-layer Transformer blocks; the first feature map is processed through the first spatial reduction attention layer, and a multi-head attention mechanism is adopted to process the output of the first spatial reduction attention layer based on the N second-layer Transformer blocks to obtain the second feature map; Then, in the third stage, a second spatial reduction attention layer is deployed before the M third-layer Transformer blocks; the second feature map is processed through the second spatial reduction attention layer, and a multi-head attention mechanism is adopted to process the output of the second spatial reduction attention layer based on the M second-layer Transformer blocks to obtain the third feature map; Finally, in the fourth stage, a third spatial reduction attention layer is deployed before the M fourth-layer Transformer blocks; the third feature map is processed through the third spatial reduction attention layer, and a multi-head attention mechanism is adopted to process the output of the third spatial reduction attention layer based on the M fourth-layer Transformer blocks to obtain the RGB feature map.

[0053] In this embodiment, during feature extraction from RGB images in the visual semantic branch, due to the varying resolutions of the processed images, even with the proposed four-layer pyramid Transformer encoder structure, there are instances where excessively high RGB image resolution leads to severe resource consumption. Continuing image processing in such cases significantly increases computational and memory costs, reducing efficiency. Therefore, to reduce computational and memory costs when processing high-resolution feature maps, this application proposes a Spatial Reduction Attention (SRA) layer. Specifically, this layer is placed before the Transformer block, replacing the traditional multi-head attention layer in related technologies. Before attention calculation, SRA reduces the spatial dimensions of K and V, decreasing sequence length through reshaping and linear projection operations, thus significantly reducing resource consumption while retaining sufficient feature information to maintain effective feature learning capabilities. The Spatial Reduction Attention layer enables the deep image completion model to process larger RGB images, improving processing efficiency and ensuring efficiency even when facing numerous intensive image processing tasks simultaneously, meeting user needs.

[0054] In conjunction with the above embodiments, in one implementation, this application also provides a depth image completion method based on bi-branch cross-modal diffusion. This method assigns a first pixel-wise weight to the depth feature map and a second pixel-wise weight to the RGB feature map through a feature similarity network in the encoder. Specifically, it includes the following: First, the RGB feature map and the depth feature map are aligned using the alignment module in the encoder.

[0055] In this embodiment of the application, after determining the RGB feature map obtained from the visual semantic branch and the depth feature map obtained from the depth completion branch, it is necessary to perform cross-modal fusion between the two. Specifically, RGB semantic and depth geometric features can be dynamically fused by weighted adaptive instance normalization (W-AdaIN). Before fusion, the RGB feature map and the depth feature map need to be aligned. The mean and variance of the RGB feature map and the depth feature map can be aligned by AdaIN, thereby avoiding scale bias during cross-modal fusion.

[0056] Then, the semantic similarity and geometric consistency between the aligned RGB feature map and the aligned depth feature map at each pixel position are determined by the feature similarity network in the encoder.

[0057] In this embodiment, after aligning the RGB feature map and the depth feature map, the weights of the RGB feature map and the depth feature map can be learned pixel by pixel through the feature similarity network in the encoder of the depth completion branch. Specifically, the feature similarity network can perform feature interaction modeling through a lightweight convolutional network, and successively downsample through 3×3 convolution, ReLU activation and 2×2 max pooling to compress the spatial dimension and multiply the number of channels, extract multi-scale depth geometric information, and calculate the semantic similarity and geometric consistency of the aligned RGB feature map and the aligned depth feature map at each pixel position.

[0058] Finally, based on the semantic similarity and geometric consistency of each pixel location, a first per-pixel weight is assigned to the deep feature map, and a second per-pixel weight is assigned to the RGB feature map.

[0059] In this embodiment, after determining the semantic similarity and geometric consistency of each pixel location, a first pixel-wise weight is assigned to the depth feature map and a second pixel-wise weight is assigned to the RGB feature map based on the semantic similarity and geometric consistency of each pixel location. Through the adaptive instance normalization dynamic fusion strategy of this application, a pixel-wise weight adjustment mechanism based on feature similarity is constructed, thereby enhancing the influence of RGB texture features in texture-rich regions and preferentially preserving depth geometric information in structured regions, avoiding scale bias in texture-rich regions and geometric distortion in structured regions.

[0060] In conjunction with the above embodiments, in one implementation, this application also provides a depth image completion method based on bi-branch cross-modal diffusion, wherein the magnitude relationship between the first pixel-wise weight and the second pixel-wise weight is determined according to the following steps: For a given pixel location, if the semantic similarity of the pixel location is greater than the geometric consistency, it is determined that the texture information corresponding to the pixel location is greater than the structural information corresponding to the pixel location, and the second weight corresponding to the pixel location is greater than the first weight corresponding to the pixel location. For a given pixel location, if the semantic similarity of that pixel location is less than the geometric consistency, it is determined that the texture information corresponding to that pixel location is less than the structural information corresponding to that pixel location, and the second weight corresponding to that pixel location is less than the first weight corresponding to that pixel location.

[0061] In this embodiment, during the process of assigning a first pixel-wise weight to the depth feature map and a second pixel-wise weight to the RGB feature map based on the semantic similarity and geometric consistency of each pixel location, as described above, the first and second pixel-wise weights assigned to each pixel location are different. Specifically, they are determined based on the texture and structural information corresponding to each pixel location. For a given pixel location, if it contains rich texture information, it indicates that the texture at that pixel location is relatively rich in the RGB image. For example, the patterns on sofas and clothes often contain rich texture information. Pixel locations in structured regions often contain richer structural information. For example, the boundary between a flat wall and the ground has obvious structural changes, thus containing more structural information. In related technologies, the fusion strategy for the processing results of different branches is relatively simple and crude. In regions with rich texture, the RGB feature map is assigned an absolutely high weight value, thereby ignoring depth geometric information. When both rich texture and structural information are present, depth geometric information is lost, resulting in a deviation between the completed depth and the actual depth.

[0062] In this application, the semantic similarity and geometric consistency of each pixel location are compared to determine the texture and structural information at each pixel location. If the texture information at a pixel location is greater than the structural information at that pixel location, a second weight is assigned to that pixel location that is greater than a first weight; conversely, if the texture information at a pixel location is less than the structural information at that pixel location, a second weight is assigned to that pixel location that is less than a first weight. This approach avoids geometric distortion caused by over-reliance on texture, thereby achieving dynamic adaptation of the RGB feature map and the depth feature map, ultimately outputting a fused feature map.

[0063] In conjunction with the above embodiments, in one implementation, this application also provides a depth image completion method based on bi-branch cross-modal diffusion. The pre-trained depth image completion model is obtained by training a depth image completion model to be trained using multiple training samples. Each training sample carries a depth label and includes: an RGB image sample, a mask image sample corresponding to the RGB image sample, a preliminary completed depth map sample corresponding to the RGB image sample, a masked depth map sample, and a noisy depth map sample after adding noise to the preliminary completed depth map sample.

[0064] In this embodiment of the application, before performing depth completion of the target image using a pre-trained depth image completion model, it is necessary to train the depth image completion model. Specifically, the depth image completion model is trained multiple times using training samples carrying depth labels. The depth labels determine the true depth information of each pixel position in the training sample image, serving as a benchmark for the training results. Each training sample also includes: an RGB image sample, a mask image sample corresponding to the RGB image sample, a preliminary completed depth map sample corresponding to the RGB image sample, a masked depth map sample, and a noisy depth map sample with noise added to the preliminary completed depth map sample, which are used as model inputs when the model processes real depth image completion.

[0065] Then, the loss function values ​​used by the pre-trained depth image completion model include at least the edge smoothing loss function value, which is the edge gradient calculated using the Sobel gradient operator for the completed depth map output by the depth image completion model to be trained during the training process.

[0066] In this embodiment of the application, during the training process of the depth image completion model, after obtaining the final completed depth map from the training samples, it is necessary to compare it with the depth label and determine the training loss of the model based on the loss function value, which serves as the basis for the next model training. Specifically, the loss function value includes at least the edge smoothing loss function value. The Sobel gradient operator is used to calculate the edge gradient of the completed depth map output by the depth image completion model to be trained during the training process. The L1 norm can be used to suppress "step-like" misalignment to ensure the continuity of the depth gradient.

[0067] In conjunction with the above embodiments, in one implementation, this application also provides a depth image completion method based on bi-branch cross-modal diffusion. The loss function value used to obtain the pre-trained depth image completion model further includes: a plane fitting loss function value, which is obtained according to the following steps: First, planar recognition is performed on the completed depth map output by the depth image completion model to be trained during the training process to obtain planar regions; Then, based on the completed depth map output by the depth image completion model to be trained during the training process, the predicted depth of the planar region is determined; Finally, based on the predicted depth of the planar region and the depth labels carried by the training samples of the depth image completion model to be trained, the value of the planar fitting loss function is determined.

[0068] In this embodiment, in the process of determining the training loss of the model through the loss function value as described above, the loss function value can also be calculated through plane fitting. Specifically, plane recognition is performed on the completed depth map output by the depth image completion model to be trained during the training process to obtain planar regions. Specifically, the RANSAC plane fitting algorithm can be used to identify planar regions, such as the ground, walls, and ceiling in the image. Then, based on the completed depth map output by the depth image completion model to be trained during the training process, the predicted depth of the planar region can be determined. The predicted depth of the fitted plane is used as the input for loss calculation. Finally, by calculating the error between the fitted plane and the actual depth of the plane represented by the depth label, the geometric relationship of the plane is constrained in the next round of training, such as the horizontal parallel relationship between the ceiling and the ground, and the vertical relationship between the wall and the ground, thereby compensating for the plane fitting loss.

[0069] By using the edge smoothing loss function and the plane fitting loss function mentioned above, the training results of the depth image completion model can be fed back during the training process. By weighted summing of the two types of losses, the completion results can be forced to conform to physical laws, thereby solving the problems of misalignment between the completed area and the known depth structure and insufficient geometric consistency. This makes the completion results more in line with physical laws in terms of the perpendicularity of the wall and the ground, and the continuity of furniture edges.

[0070] In an optional embodiment, the depth image completion scheme based on bi-branch cross-modal diffusion proposed in this application is as follows: Figure 2 As shown, Figure 2 This is a schematic diagram of the depth image completion process proposed in an embodiment of this application. The pre-trained depth image completion model is divided into a visual semantic branch and a depth completion branch. In the visual semantic branch, after encoding the RGB image into a 4-channel feature map, it passes through a multi-scale pyramid Transformer and then undergoes operations such as patch mosaicking, spatial attention reduction, and multi-head attention processing. The normalized processing result is then input into the next stage via a feedforward network for further processing to obtain the RGB feature map. In the depth completion branch, the depth feature map is obtained by processing the mask image corresponding to the RGB image, the initially completed depth map corresponding to the RGB image, the masked depth map, and the noisy depth map after adding noise to the initially completed depth map. Then, the depth feature map and the RGB feature map are assigned a first pixel-wise weight and a second pixel-wise weight, respectively, and the depth feature map and the RGB feature map are fused according to the weight values ​​to obtain the fused feature map. The fused feature map is then decoded through operations such as transpose convolution to obtain the final complete depth map of the RGB image. During model training, the completion results can be fed back by calculating the edge smoothing loss function value and the plane fitting loss function value, thereby improving the plane consistency, edge smoothness and geometric alignment of the incomplete results.

[0071] Based on the same design concept, one embodiment of this application provides a depth image completion device based on dual-branch cross-modal diffusion. (Reference) Figure 3 , Figure 3 This is a structural block diagram of a depth image completion device based on bi-branch cross-modal diffusion provided in an embodiment of this application. Figure 3 As shown, the device includes: The RGB feature map determination module is used to extract features from RGB images through the visual semantic branch of a pre-trained deep image completion model to obtain RGB feature maps. The depth feature map determination module is used to process the mask image corresponding to the RGB image, the preliminary completed depth map corresponding to the RGB image, the masked depth map, and the noisy depth map after adding noise to the preliminary completed depth map through the encoder in the depth completion branch of the pre-trained depth image completion model to obtain the depth feature map. The weighting module is used to assign a first pixel-wise weight to the depth feature map and a second pixel-wise weight to the RGB feature map through the feature similarity network in the encoder. The feature map fusion module is used to fuse the depth feature map and the RGB feature map at each pixel location based on the first pixel-wise weight and the second pixel-wise weight, so as to fuse the RGB texture and depth geometry to obtain the fused feature map; The decoding module is used to process the fused feature map through the decoder in the depth completion branch to obtain the complete depth map corresponding to the RGB image; the encoder and decoder in the depth completion branch form a U-Net architecture based on diffusion denoising.

[0072] Optionally, the RGB feature map determination module includes: The first feature map determination submodule is used in the first stage to process the RGB image based on N first-layer Transformer blocks using a multi-head attention mechanism to obtain a first feature map. The first feature map has C1 channels and a resolution of 1 / 4 of the RGB image. The second feature map determination submodule is used in the second stage to process the first feature map using a multi-head attention mechanism based on N second-layer Transformer blocks to obtain a second feature map. The second feature map has C2 channels and a resolution of 1 / 8 of the RGB image; N is an integer greater than or equal to 2. The third feature map determination submodule is used in the third stage to process the second feature map based on M third-layer Transformer blocks using a multi-head attention mechanism to obtain the third feature map. The second feature map has C3 channels and a resolution of 1 / 16 of the RGB image. The RGB feature map determination submodule is used in the fourth stage to process the third feature map using a multi-head attention mechanism based on M fourth-layer Transformer blocks to obtain the RGB feature map. The number of channels of the RGB feature map is C4, the resolution is 1 / 32 of the RGB image, and M is an integer greater than N.

[0073] Optionally, the RGB feature map determination module further includes: The first spatial reduction attention submodule is used in the second stage to deploy a first spatial reduction attention layer before N second-layer Transformer blocks; the first feature map is processed through the first spatial reduction attention layer, and a multi-head attention mechanism is adopted to process the output of the first spatial reduction attention layer based on N second-layer Transformer blocks to obtain the second feature map; The second spatial reduction attention submodule is used to deploy a second spatial reduction attention layer before the M third-layer Transformer blocks in the third stage; the second feature map is processed through the second spatial reduction attention layer, and a multi-head attention mechanism is adopted to process the output of the second spatial reduction attention layer based on the M second-layer Transformer blocks to obtain the third feature map; The third spatial reduction attention submodule is used to deploy a third spatial reduction attention layer before the M fourth-layer Transformer blocks in the fourth stage; the third feature map is processed through the third spatial reduction attention layer, and a multi-head attention mechanism is adopted to process the output of the third spatial reduction attention layer based on the M fourth-layer Transformer blocks to obtain the RGB feature map.

[0074] Optionally, the weight assignment module includes: The alignment processing submodule is used to align the RGB feature map and the depth feature map using the alignment module in the encoder; The feature similarity judgment submodule is used to determine the semantic similarity and geometric consistency between the aligned RGB feature map and the aligned depth feature map at each pixel position through the feature similarity network in the encoder. The weighting submodule is used to assign a first pixel-wise weight to the deep feature map and a second pixel-wise weight to the RGB feature map based on the semantic similarity and geometric consistency of each pixel location.

[0075] Optionally, the weights are assigned to the sub-modules, including: The first weight determination unit is used to determine, for a pixel position, that if the semantic similarity of the pixel position is greater than the geometric consistency, the texture information corresponding to the pixel position is greater than the structural information corresponding to the pixel position, and the second weight corresponding to the pixel position is greater than the first weight corresponding to the pixel position. The second weight determination unit is used to determine, for a pixel position, that if the semantic similarity of the pixel position is less than the geometric consistency, the texture information corresponding to the pixel position is less than the structural information corresponding to the pixel position, and the second weight corresponding to the pixel position is less than the first weight corresponding to the pixel position.

[0076] Optionally, the device further includes: The image completion model training module is used to train the depth image completion model to be trained using multiple training samples to obtain the pre-trained depth image completion model. Each training sample carries a depth label and includes: an RGB image sample, a mask image sample corresponding to the RGB image sample, a preliminary completed depth map sample corresponding to the RGB image sample, a masked depth map sample, and a noisy depth map sample after adding noise to the preliminary completed depth map sample. The first loss function module is used to obtain the loss function value used by the pre-trained depth image completion model, including at least: edge smoothing loss function value, which is the edge gradient calculated using the Sobel gradient operator for the completed depth map output by the depth image completion model to be trained during the training process.

[0077] Optionally, the device further includes: a second loss function module for plane fitting of loss function values; The second loss function module includes: The planar recognition submodule is used to perform planar recognition on the completed depth map output by the depth image completion model to be trained during the training process, so as to obtain planar regions. The predicted depth determination submodule is used to determine the predicted depth of the planar region based on the completed depth map output by the depth image completion model to be trained during the training process. The plane fitting loss function value determination submodule is used to determine the plane fitting loss function value based on the predicted depth of the plane region and the depth labels carried by the training samples of the depth image completion model to be trained.

[0078] Based on the same design concept, another embodiment of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the depth image completion method based on bi-branch cross-modal diffusion as described in any of the above embodiments of this application.

[0079] Based on the same design concept, another embodiment of this application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps in the depth image completion method based on bi-branch cross-modal diffusion as described in any of the above embodiments of this application.

[0080] Based on the same design concept, another embodiment of this application provides an electronic device, such as... Figure 4 As shown. Figure 4 This is a schematic diagram of an electronic device according to an embodiment of this application. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the program implements the steps of the depth image completion method based on bi-branch cross-modal diffusion described in any of the above embodiments of this application.

[0081] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0082] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0083] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0084] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0085] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0086] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0087] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0088] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0089] The above provides a detailed description of the depth image completion method, apparatus, device, and medium based on bi-branch cross-modal diffusion provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A deep image inpainting method based on two-branch cross-modal diffusion, characterized in that, The application relates to a method for generating a complete depth map corresponding to an RGB image based on a pre-trained depth image completion model. The method comprises the following steps: extracting features of the RGB image through a visual semantic branch of the pre-trained depth image completion model to obtain an RGB feature map; processing a mask image corresponding to the RGB image, a preliminarily completed depth map corresponding to the RGB image, a masked depth map, and a noisy depth map obtained by adding noise to the preliminarily completed depth map through an encoder in a depth completion branch of the pre-trained depth image completion model to obtain a depth feature map; assigning a first pixel-wise weight to the depth feature map and a second pixel-wise weight to the RGB feature map through a feature similarity network in the encoder; fusing the depth feature map and the RGB feature map at each pixel position based on the first pixel-wise weight and the second pixel-wise weight to fuse RGB texture and depth geometry and obtain a fused feature map; processing the fused feature map through a decoder in the depth completion branch to obtain a complete depth map corresponding to the RGB image; 2. The deep image inpainting method based on two-branch cross-modal diffusion according to claim 1, characterized in that, the encoder and the decoder in the depth completion branch form a U-Net architecture based on diffusion denoising. The visual semantic branch is a four-layer pyramid Transformer encoder corresponding to four stages. The method comprises the following steps: in the first stage, processing the RGB image based on N first-layer Transformer blocks through a multi-head attention mechanism to obtain a first feature map, wherein the number of channels of the first feature map is C1, and the resolution of the first feature map is 1 / 4 of the RGB image; in the second stage, processing the first feature map based on N second-layer Transformer blocks through a multi-head attention mechanism to obtain a second feature map, wherein the number of channels of the second feature map is C2, and the resolution of the second feature map is 1 / 8 of the RGB image; N is an integer greater than or equal to 2; in the third stage, processing the second feature map based on M third-layer Transformer blocks through a multi-head attention mechanism to obtain a third feature map, wherein the number of channels of the second feature map is C3, and the resolution of the second feature map is 1 / 16 of the RGB image; 3. The deep image inpainting method based on two-branch cross-modal diffusion according to claim 2, characterized in that, in the fourth stage, processing the third feature map based on M fourth-layer Transformer blocks through a multi-head attention mechanism to obtain the RGB feature map, wherein the number of channels of the RGB feature map is C4, and the resolution of the RGB feature map is 1 / 32 of the RGB image; M is an integer greater than N. The method comprises the following steps: in the second stage, a first spatial reduction attention layer is arranged before the N second-layer Transformer blocks; the first spatial reduction attention layer is used to process the first feature map, and the output of the first spatial reduction attention layer is processed based on the N second-layer Transformer blocks through a multi-head attention mechanism to obtain a second feature map. In the third stage, a second spatial reduction attention layer is arranged before M third layer Transformer blocks; the second feature map is processed through the second spatial reduction attention layer, and a multi-head attention mechanism is adopted to process the output of the second spatial reduction attention layer based on M second layer Transformer blocks to obtain a third feature map; In the fourth stage, a third spatial reduction attention layer is arranged before M fourth layer Transformer blocks; the third feature map is processed through the third spatial reduction attention layer, and a multi-head attention mechanism is adopted to process the output of the third spatial reduction attention layer based on M fourth layer Transformer blocks to obtain the RGB feature map.

4. The deep image inpainting method based on two-branch cross-modal diffusion according to claim 1, characterized in that, The feature similarity network in the encoder is used to assign a first pixel-by-pixel weight to the depth feature map and a second pixel-by-pixel weight to the RGB feature map, including: The alignment module in the encoder is used to perform alignment processing on the RGB feature map and the depth feature map; The feature similarity network in the encoder is used to determine the semantic similarity and geometric consistency of the RGB feature map and the depth feature map at each pixel position after alignment processing; Based on the semantic similarity and geometric consistency at each pixel position, a first pixel-by-pixel weight is assigned to the depth feature map, and a second pixel-by-pixel weight is assigned to the RGB feature map.

5. The deep image completion method based on two-branch cross-modal diffusion according to claim 4, characterized in that, The size relationship between the first pixel-by-pixel weight and the second pixel-by-pixel weight is determined according to the following steps: For a pixel position, if the semantic similarity at the pixel position is greater than the geometric consistency, it is determined that the texture information corresponding to the pixel position is greater than the structure information corresponding to the pixel position, and the second weight corresponding to the pixel position is greater than the first weight corresponding to the pixel position; For a pixel position, if the semantic similarity at the pixel position is less than the geometric consistency, it is determined that the texture information corresponding to the pixel position is less than the structure information corresponding to the pixel position, and the second weight corresponding to the pixel position is less than the first weight corresponding to the pixel position.

6. The deep image completion method based on two-branch cross-modal diffusion according to claim 1, characterized in that, The pre-trained depth image completion model is obtained by training a depth image completion model to be trained using a plurality of training samples, each training sample carrying a depth label, each training sample including: an RGB image sample, a mask image sample corresponding to the RGB image sample, a preliminary completed depth image sample corresponding to the RGB image sample, a masked depth image sample, and a noisy depth image sample obtained by adding noise to the preliminary completed depth image sample; The loss function value used by the pre-trained depth image completion model includes at least an edge smoothing loss function value, which is calculated using a Sobel gradient operator for the completed depth image output by the depth image completion model to be trained during training.

7. The deep image completion method based on two-branch cross-modal diffusion according to claim 6, characterized in that, The loss function value used by the pre-trained depth image completion model is further obtained by a plane fitting loss function value, which is obtained by the following steps: plane recognition is performed on the completed depth map output by the depth image completion model to be trained in the training process, to obtain a plane region; a predicted depth of the plane region is determined based on the completed depth map output by the depth image completion model to be trained in the training process; the plane fitting loss function value is determined based on the predicted depth of the plane region and the depth label carried by the training sample input into the depth image completion model to be trained.

8. A device for deep image inpainting based on two-branch cross-modal diffusion, characterized in that, The device comprises: an RGB feature map determination module configured to perform feature extraction on an RGB image by a visual semantic branch of a pre-trained depth image completion model, to obtain an RGB feature map; a depth feature map determination module configured to process a mask image corresponding to the RGB image, a preliminarily completed depth map corresponding to the RGB image, a masked depth map, and a noisy depth map obtained by adding noise to the preliminarily completed depth map, by an encoder in a depth completion branch of the pre-trained depth image completion model, to obtain a depth feature map; a weight assignment module configured to assign a first pixel-wise weight to the depth feature map and a second pixel-wise weight to the RGB feature map by a feature similarity network in the encoder; a feature map fusion module configured to fuse the depth feature map and the RGB feature map based on the first pixel-wise weight and the second pixel-wise weight at each pixel position, to fuse RGB texture and depth geometry, to obtain a fused feature map; a decoding module configured to process the fused feature map by a decoder in the depth completion branch, to obtain a complete depth map corresponding to the RGB image; the encoder and the decoder in the depth completion branch form a U-Net architecture based on diffusion denoising.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The computer program is executed by the processor to implement the depth image completion method based on double-branch cross-modal diffusion according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the depth image completion method based on double-branch cross-modal diffusion according to any one of claims 1 to 7.

Images