A Deep Completion Method for the Intestinal Environment Based on Multi-Scale Confidence and Self-Attention Mechanisms

Abstract

A depth completion method for the intestinal environment based on multi-scale confidence and self-attention mechanisms belongs to the fields of computer vision and artificial intelligence. This method constructs a multi-scale confidence neural network to mine potential geometric features in sparse depth data. By spreading effective depth pixel values ​​to neighboring pixels, holes in the sparse depth map are filled, resulting in a complete and dense depth feature map with a balanced pixel distribution. This facilitates cross-modal fusion with the corresponding RGB image. Next, the depth feature map and confidence map are input into the depth completion network, and cross-modal features are fused under the guidance of the corresponding RGB image for depth completion. Finally, the fused multi-modal features are further enhanced by a structure-aware module based on a self-attention mechanism, improving the geometric and texture features in the image and depth data and increasing the accuracy of depth completion.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and artificial intelligence technology, specifically relating to a method for deep completion of the intestinal environment based on multi-scale confidence and self-attention mechanisms. Background Technology

[0002] Typically, doctors can only perform limited unidirectional intestinal endoscopy. During these examinations, even state-of-the-art surgical robots equipped with depth sensors can only provide sparse and incomplete depth information. However, dense, accurate, and real-time intestinal depth estimation is crucial for doctors to determine the three-dimensional location and shape of intestinal tissue. Otherwise, it would severely impact human-machine interaction between the doctor and the surgical robot, such as the doctor's subsequent movement and manipulation of the probe.

[0003] With the emergence of large datasets and the continuous development of deep learning algorithms, deep learning-based depth estimation methods have flourished. This method mainly involves training the network by inputting data into a designed deep neural network model and using the trained model to predict dense depth maps.

[0004] Depth estimation models can be broadly categorized into two types. The first type is depth regression models, which directly extract features and perform depth regression on the input RGB image to obtain dense depth predictions. Depth regression methods have recently been applied to in vivo environments; however, due to the strong illumination variations and weak texture features present in these environments, the generated depth maps are largely data-driven, and their accuracy and generalization remain limited. Scale ambiguity also exists in the generated depth maps due to the lack of utilization of environmental geometry. The second type of depth estimation method is depth completion methods, which reconstruct depth maps by combining brightness images with sparse depth information from SFM (Structure From Motion) / SLAM (Simultaneous Localization and Mapping) algorithms or depth sensors. However, most methods simply concatenate the image and depth map without exploring the role of cross-modal feature fusion in promoting depth estimation. Furthermore, most current methods are designed for indoor and outdoor human activity scenarios; there are no depth completion methods designed for the intestinal or other body cavity environments. Summary of the Invention

[0005] To address the limitations of existing methods, such as scale ambiguity, simplistic and crude cross-modal fusion approaches, and significant differences between the intestinal environment and human activity scenarios, the present invention aims to provide a deep learning-based method for dense depth estimation of the intestinal environment, which utilizes endoscopic images and sparse depth measurement data to complete the depth of the intestinal environment.

[0006] This invention provides the following technical solution:

[0007] A method for deep completion of the gut environment based on multi-scale confidence and self-attention mechanisms includes the following steps:

[0008] S1. Construct a multi-scale confidence neural network to mine potential geometric features in sparse depth data. By spreading effective depth pixel values ​​to adjacent pixels, the holes in the sparse depth map are filled, and a complete and dense depth feature map and confidence map with balanced pixel distribution are obtained, which is beneficial for cross-modal fusion with the corresponding RGB image.

[0009] S2. Construct a deep completion network. Input the depth feature map and confidence map into the deep completion network, and perform deep completion by fusing cross-modal features under the guidance of the corresponding RGB image.

[0010] S3. Construct a structure-aware module based on a self-attention mechanism in the deep completion network. The multimodal features fused are adopted by the structure-aware module to further enhance the geometric and texture features in the image and depth data and improve the accuracy of deep completion.

[0011] Furthermore, the multi-scale confidence neural network constructed in step S1 adopts an encoder-decoder structure, replacing the standard convolutional layer in the traditional encoder-decoder framework with a normalized convolutional layer; based on confidence signal theory, normalized convolution can use effective pixels to represent its neighboring pixels, thereby achieving pixel diffusion and densification.

[0012] Furthermore, the workflow of the normalized convolutional layer is as follows:

[0013] 1) The normalized convolutional layer accepts two inputs simultaneously: a sparse depth map and its corresponding confidence map. The process of generating the estimated depth value is shown in Equation (1):

[0014]

[0015] Where l represents the number of normalized convolutional layers, and l-1 represents the previous convolutional layer. This indicates the correlation between the confidence image at the corresponding scale and the original image; the function Γ(·)=log(1+exp(·)) will... The upper domain of Z is transformed to a non-negative value while preserving the surface trend; l-1 Represents the depth map of layer l-1, C l-1 This represents the confidence plot for layer l-1, b l It is a bias term. This is a constant added to prevent the denominator from being zero;

[0016] 2) In order to propagate the confidence mask corresponding to the sparse depth map between layers, the denominator term calculated in formula (1) is normalized to the sum of the filter elements. The specific processing is shown in formula (2):

[0017]

[0018] Among them, C l-1 Representing the confidence value of layer l-1, the function Γ(·) = log(1 + exp(·)) converts the upper domain of the two-dimensional surface into non-negative values ​​while preserving the surface's trend. It is a constant;

[0019] 3) Based on the Γ function, the backpropagation of the multi-scale confidence network is expressed by the following formula:

[0020]

[0021] Here, E represents the loss between the deep prediction from the network and the true value.

[0022] Furthermore, the specific process of step S2 is as follows:

[0023] A U-Net-based deep completion network is constructed. The depth features and confidence mask output from the proposed multi-scale confidence prediction network are fused with the corresponding RGB image and input into the deep completion network. The image-guided deep completion neural network estimates the dense depth of the sparse depth map and predicts the dense depth map. Guided by the RGB image features, the deep completion network makes the influence of sparse depth more accurate and efficient.

[0024] Furthermore, the specific process of step S3 is as follows:

[0025] A structure-aware module based on a self-attention mechanism is introduced between the encoder and decoder of a deep completion network to simulate the interdependence between responses in different regions and to fuse deep texture and geometric features. This includes the following steps:

[0026] 1) The feature maps F∈R obtained from the U-Net-based encoder C×H×W Remodeling into R C×N Where C represents the number of channels, H represents the height of the image, W represents the width of the image, and N = H × W represents the size of the feature map.

[0027] 2) Perform matrix multiplication on F and its transpose, and calculate the feature similarity S∈R. C×C The similarity between different channel maps reflects the spatial relationship of regional response.

[0028] 3) By performing pixel-by-pixel subtraction, the similarity S is transformed into a feature discriminant D∈R.C×C This integrates more responses from different regions; through a softmax layer, an attention mapping A∈R is obtained. C×C It can simulate the relationship between two channel mappings;

[0029] 4) Perform matrix multiplication on the transposes of A and F, and the result is reshaped into R. C×H×W ;

[0030] 5) Perform element-wise summation on F and the result R from step 4) to obtain the output E; the final feature of each channel is the weighted sum of the features of all channels and the original features.

[0031] By employing the above-described technology, the beneficial effects of the present invention compared to the prior art are as follows:

[0032] In this invention, the original sparse depth map is used to generate a dense depth feature map and a corresponding confidence map through a multi-scale confidence prediction module. Then, under the guidance of the corresponding RGB image, a structure-aware depth completion network is used to enhance texture and structural features, thereby generating a dense depth map with rich details. This method is based on a multi-scale confidence network, which can effectively mine the features of sparse depth and finally generate a feature map with dense confidence. Furthermore, the structure-aware module based on the self-attention mechanism further enhances texture and geometric features. Attached Figure Description

[0033] Figure 1 This is a network framework diagram of the method of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0035] Conversely, this invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of the invention as defined in the claims. Furthermore, to provide a better understanding of the invention, certain specific details are described in detail below. However, those skilled in the art will fully understand the invention even without these detailed descriptions.

[0036] Please see Figure 1 A method for deep completion of the gut environment based on multi-scale confidence and self-attention mechanisms is described below:

[0037] First, a multi-scale confidence neural network is designed to mine potential geometric features in sparse depth data. By spreading effective depth pixel values ​​to neighboring pixels, holes in the sparse depth map are filled, resulting in a complete and dense depth feature map with a balanced pixel distribution. This facilitates cross-modal fusion with the corresponding RGB image. Next, the depth feature map and confidence map are input into a depth completion network. Guided by the corresponding RGB image, cross-modal features are fused for depth completion. Finally, the fused multimodal features are further enhanced by a structure-aware module based on a self-attention mechanism, improving the geometric and texture features in the image and depth data, and increasing the accuracy of depth completion. The specific steps are as follows:

[0038] (1) Construct a multi-scale confidence neural network

[0039] The multi-scale confidence network employs an encoder-decoder structure, replacing the standard convolutional layers in the traditional encoder-decoder framework with normalized convolutional layers. Based on confidence signal theory, normalized convolution can utilize effective pixels to represent their neighboring pixels, thereby achieving pixel diffusion and densification. The workflow of the normalized convolutional layer is as follows:

[0040] The normalized convolutional layer accepts two inputs simultaneously: a sparse depth map Z and its corresponding confidence map C. First, the depth value of this layer is calculated:

[0041]

[0042] Where l represents the number of normalized convolutional layers, and l-1 represents the previous convolutional layer. This represents the correlation between the confidence image at the corresponding scale and the original image. Γ(·)=log(1+exp(·)) The upper domain is transformed to non-negative values ​​while preserving the surface trend, Z l-1 C represents the depth value of layer l-1. l-1 b represents the confidence value of layer l-1. l It is a bias term, and This is a constant added to prevent the denominator from being zero. C is a confidence plot.

[0043] First, calculate the depth value Z of layer l-1. l-1 and confidence level C l-1 The corresponding element-wise product, multiplied by the Γ function compression. The results are then summed to obtain an intermediate depth map, and C is also processed. l-1 And the compressed Γ function Multiply the results to get the denominator, and finally add the bias value to get the updated depth value.

[0044] Next, calculate the updated confidence level:

[0045] To propagate the confidence mask corresponding to the sparse depth map between layers, the calculated denominator is normalized to the sum of the filter elements, as shown in the following formula:

[0046]

[0047] Among them, C l-1 Let Γ(·) represent the confidence value of layer l-1. Γ(·) = log(1 + exp(·)) converts the upper domain of the two-dimensional surface to a non-negative value while preserving the surface's trend. It is a constant. Multiply the confidence level of layer l-1 by the compressed result of the Γ function, and add... A constant, then divided by the compressed Γ function The updated confidence level is obtained.

[0048] Based on the Γ function, the backpropagation of the multi-scale confidence network is modified to formula (3):

[0049]

[0050] Where E represents the loss between the depth prediction from the network and the ground truth, and Z represents the corresponding depth image.

[0051] Construct a deep completion network based on a structure-aware module;

[0052] A U-Net-based deep completion network is constructed, which uses the depth feature map from the multi-scale confidence prediction module and the input of the relevant RGB image to predict dense depth maps. Furthermore, in this embodiment, a structure-aware module based on a self-attention mechanism is built between the encoder and decoder of the deep completion network. This module can perform deep fusion of texture and geometric features on the dense depth map, allowing highly feasible sparse depth values ​​to better influence the depth completion result. The specific details are as follows:

[0053] 2-1) Image-guided deep completion neural network

[0054] In this embodiment, the depth feature map and confidence mask from the multi-scale confidence prediction module are fused with the corresponding RGB image and input into the U-Net-based depth completion network. The depth completion neural network guided by the RGB image can estimate the dense depth of the sparse depth map and predict the dense depth map. Under the guidance of RGB image features, the depth completion network makes the influence of sparse depth more accurate and efficient.

[0055] 2-2) Structure perception module based on self-attention mechanism

[0056] The traditional U-Net network is a typical non-discriminatory network that does not enhance any feature information. In depth estimation, each extracted feature map can be viewed as a response of a specific region, and the responses of different regions are interrelated. It is assumed that each channel map captures more diverse region responses from all other channel maps. In this case, it will obtain more relative depth information from more distant regions and significantly enhance the perception of scene structure. Therefore, we introduce a structure-aware module based on a self-attention mechanism between the encoder and decoder of the deep completion network. This module can simulate the interdependencies between responses of different regions, fusing deep texture and geometric features.

[0057] The specific process is as follows: First, the feature map F∈R obtained by the U-net-based encoder is... C×H×W Remodeling into R C×N Where N = H × W represents the size of the feature map. Then, matrix multiplication is performed on F and its transpose to calculate the feature similarity S ∈ R. C ×C The similarity between different channel images reflects the spatial relationship of regional responses. That is, two highly similar feature images have a strong response in the same region. Then, we transform the similarity S into a feature discriminant D∈R through pixel-by-pixel subtraction. C×C This integrates responses from different regions. Through a single softmax layer, an attention map A∈R is obtained. C×C It can simulate the relationship between two channel mappings. Next, matrix multiplication is performed on the transposes of A and F, and the result is reshaped into R. C×H×W Finally, F and the result are summed element-wise to obtain the output E. The final feature of each channel is the weighted sum of the features of all channels and the original features. The fused multimodal features, through a structure-aware module based on a self-attention mechanism, further enhance the geometric and texture features in the image and depth data, improving the accuracy of depth completion.

[0058] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for deep completion of the intestinal environment based on multi-scale confidence and self-attention mechanisms, characterized in that, Includes the following steps: S1. Construct a multi-scale confidence neural network to mine potential geometric features in sparse depth data. By spreading effective depth pixel values ​​to adjacent pixels, the holes in the sparse depth map are filled, and a complete and dense depth feature map and confidence map with balanced pixel distribution are obtained, which is beneficial for cross-modal fusion with the corresponding RGB image. S2. Construct a deep completion network. Input the depth feature map and confidence map into the deep completion network, and perform deep completion by fusing cross-modal features under the guidance of the corresponding RGB image. S3. Construct a structure awareness module based on a self-attention mechanism in the deep completion network. The fused multimodal features are further enhanced by the structure awareness module to improve the geometric and texture features in the image and depth data, thereby improving the accuracy of deep completion. The multi-scale confidence neural network constructed in step S1 adopts an encoder-decoder structure, replacing the standard convolutional layer in the traditional encoder-decoder framework with a normalized convolutional layer. Based on confidence signal theory, normalized convolution can use effective pixels to represent its neighboring pixels, thereby achieving pixel diffusion and densification. The workflow of the normalized convolutional layer is as follows: 1) The normalized convolutional layer accepts two inputs simultaneously: a sparse depth map and its corresponding confidence map. The process of generating the estimated depth value is shown in Equation (1): Where l represents the number of normalized convolutional layers, and l-1 represents the previous convolutional layer. This indicates the correlation between the confidence image at the corresponding scale and the original image; the function Γ(·)=log(1+exp(·)) will... The upper domain of Z is transformed to non-negative values ​​while preserving the surface trend; l-1 Represents the depth map of layer l-1, C l-1 This represents the confidence plot for layer l-1, b l It is a bias term. This is a constant added to prevent the denominator from being zero; 2) In order to propagate the confidence mask corresponding to the sparse depth map between layers, the denominator term calculated in formula (1) is normalized to the sum of the filter elements. The specific processing is shown in formula (2): Among them, C l-1 Representing the confidence value of layer l-1, the function Γ(·) = log(1 + exp(·)) converts the upper domain of the two-dimensional surface into non-negative values ​​while preserving the surface's trend. It is a constant; 3) Based on the Γ function, the backpropagation of the multi-scale confidence network is expressed by the following formula: Here, E represents the loss between the deep prediction from the network and the true value.

2. The intestinal environment depth completion method based on multi-scale confidence and self-attention mechanism according to claim 1, characterized in that... The specific process of step S2 is as follows: A U-Net-based deep completion network is constructed. The depth features and confidence mask output from the proposed multi-scale confidence prediction network are fused with the corresponding RGB image and input into the deep completion network. The image-guided deep completion neural network estimates the dense depth of the sparse depth map and predicts the dense depth map. Guided by the RGB image features, the deep completion network makes the influence of sparse depth more accurate and efficient.

3. The intestinal environment depth completion method based on multi-scale confidence and self-attention mechanism according to claim 2, characterized in that... The specific process of step S3 is as follows: A structure-aware module based on a self-attention mechanism is introduced between the encoder and decoder of a deep completion network to simulate the interdependence between responses in different regions and to fuse deep texture and geometric features. This includes the following steps: 1) The feature maps F∈R obtained from the U-Net-based encoder C×H×W Remodeling into R C×N Where C represents the number of channels, H represents the height of the image, W represents the width of the image, and N = H × W represents the size of the feature map. 2) Perform matrix multiplication on F and its transpose, and calculate the feature similarity S∈R. C×C The similarity between different channel maps reflects the spatial relationship of regional response. 3) By performing pixel-by-pixel subtraction, the similarity S is transformed into a feature discriminant D∈R. C×C This integrates more responses from different regions; through a softmax layer, an attention mapping A∈R is obtained. C×C It can simulate the relationship between two channel mappings; 4) Perform matrix multiplication on the transposes of A and F, and the result is reshaped into R. C×H×W ; 5) Perform element-wise summation on F and the result R from step 4) to obtain the output E; the final feature of each channel is the weighted sum of the features of all channels and the original features.

Images