Human intestinal tract dense depth estimation method

Abstract

The invention provides a human intestinal tract dense depth estimation method, and belongs to the field of computer vision and artificial intelligence, and the method comprises the steps: S1, obtaining a real data set and a virtual data set; s2, constructing a domain adaptive network # imgabs0 # comprising a convolution-multi-head self-attention module, a multi-scale pyramid module and a feature decoding module, the weight of the domain adaptive network being # imgabs1 #, the weight of the multi-scale pyramid module being # imgabs2 #, and the weight of the multi-scale pyramid module being # imgabs2 #; a total loss function of the domain adaptive network depends on a self-supervision loss function # imgabs2 # based on a real data set, a supervision loss function # imgabs3 # based on a virtual data set and a domain adaptive loss function # imgabs4 # of a domain classifier based on gradient inversion; and S3, training the domain adaptive network by forward transfer-reverse transfer to minimize a total loss function of the domain adaptive network so as to obtain an optimal weight # imgabs5 #. The method can effectively improve the estimation precision.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision and artificial intelligence, and particularly relates to a method for estimating the dense depth of the human intestine. Background Art

[0002] During the inspection and treatment of intestinal diseases, it is usually necessary to combine the depth information of intestinal images for analysis. However, the human intestinal environment has a narrow space, insufficient texture, and high geometric feature similarity, which brings challenges to the depth estimation task. Existing learning-based human intestinal depth estimation methods mainly include supervised learning methods and self-supervised methods. Supervised learning methods have significant overall depth prediction effects. These methods rely on accurate dense depth labels as supervisory signals to train the model, but due to the limitations of the use of in vivo sensors, it is difficult to obtain accurate dense depth labels for the intestinal environment in vivo. Self-supervised methods use continuous multi-frame colonoscopy images for training. However, these methods have difficulty in obtaining accurate encoding features in intestinal environments with scarce textures and high feature similarity, resulting in insufficient accuracy in intestinal environment depth estimation. In addition, self-supervised methods rely on photometric consistency between adjacent frames for training and cannot consider the continuity of the depth of corresponding points from a global perspective. Therefore, there is a problem of inconsistent depth value scales of the same point between different frames, which is the scale ambiguity problem. Summary of the Invention

[0003] The purpose of this invention is to propose a method for estimating the dense depth of the human intestine, which can effectively improve the estimation accuracy.

[0004] The present invention is achieved through the following technical solutions: A method for estimating the dense depth of the human intestine comprises the following steps: Step S1: obtaining a real dataset and a virtual dataset, wherein the real dataset includes a sequence of real image frames collected during a real colonoscopy examination, and the virtual dataset includes a sequence of virtual image frames collected during a simulated colonoscopy examination in a virtual reality colonoscopy platform, wherein each virtual image frame has a corresponding true depth value; Step S2: Construct a domain adaptation network including a convolution-multi-head self-attention module, a multi-scale pyramid module, and a feature decoding module , the weight of the domain adaptation network is , 、 、 They are the weights of the convolution-multi-head self-attention module, the multi-scale pyramid module, and the feature decoding module. The total loss function of the domain adaptation network depends on the self-supervised loss function based on the real dataset. , supervised loss function based on virtual dataset and domain adaptation loss function for gradient reversal-based domain classifier , self-supervised loss function and self-supervision weights Related, domain adaptation loss function and domain classifier weights related; Step S3: Train the domain adaptation network by forward propagation-backward propagation to minimize the total loss function of the domain adaptation network to obtain the optimal weights. , introduce the equalization factor in training To balance the supervision loss and self-supervision loss.

[0005] Furthermore, in step S1, the real image frame sequence is represented as ,in express t The real image frame acquired at every moment, N r represents the number of real image frames, the virtual reality colonoscopy platform is the VR-Caps platform, and the virtual image frame sequence is represented by , VR-Caps platform provides depth information obtained by converting the 3D model based on computed tomography. This depth information is the true depth value corresponding to the virtual image frame. The total true depth value is expressed as ,in, express t The real image frame acquired at every moment, N v Indicates the number of virtual image frames, for The corresponding true depth value.

[0006] Furthermore, in step S2, the self-supervisory loss function is expressed as ,in, is the weight coefficient, Represents the depth map estimated by the smoothing domain adaptation network while retaining some edge details, represents the actual luminosity loss, for Two frames at adjacent moments, express The mean of , , represents an Iverson bracket applied pixel by pixel, , for and The pixel photometric error between Indicates that based on continuous frames 、 and the synthetic frame generated by the depth map estimated by the adaptive network, Represents pixel-by-pixel multiplication.

[0007] Furthermore, in step S2, the supervision loss function is expressed as ,in, represents the image mask, which depends on and , is the maximum depth in the real depth map, calculate When, abandon Pixels .

[0008] Furthermore, in step S2, the domain adaptation loss function is expressed as , among which, if ,but Output 1, if ,but Output 0.

[0009] Furthermore, in step S3, one training session specifically includes the following steps: Step S31, supervised training process: input get , the loss error Perform back propagation to obtain the updated Parameters ; Step S32, self-supervised training process: input get , the loss error Perform back propagation to obtain the updated Parameters ; Step S33: perform the following calculations based on the results of step S31 and step S32: 、 、 , for the final back-propagation process.

[0010] Furthermore, in step S2, a four-layer encoder-decoder structure is adopted, and the encoder has a linear combination of a convolution-multi-head self-attention module and a multi-scale pyramid module. The convolution-multi-head self-attention module includes three parallel multi-head attention modules and a convolution block. The convolution block extracts the local relationship between adjacent features in the feature. The three multi-head attention modules use the self-attention mechanism to embed feature codes of different sizes in parallel through the multi-scale pyramid module. The decoder uses the multi-scale features generated by the encoder and adopts cross-layer and cross-scale connections to gradually improve the spatial resolution.

[0011] Furthermore, before the supervised training process, the global scaling factor is calculated , the global scale factor Make the final depth estimation result With absolute scale, is the total depth truth value The median value of Represents the median depth value predicted from the virtual dataset.

[0012] The present invention has the following beneficial effects: The present invention first obtains real data sets and virtual data sets, and then constructs a domain adaptation network. The weight of the domain adaptation network is determined by the weight of the convolution-multi-head self-attention module, the weight of the multi-scale pyramid module and the weight of the feature decoding module. The total loss function of the domain adaptation network depends on the self-supervised loss function based on the real data set. , supervised loss function based on virtual dataset and domain adaptation loss function for gradient reversal-based domain classifier , the encoder is jointly trained using a domain classifier, so that the domain adaptation network can transfer the experience learned in the virtual intestinal domain where depth data is easy to obtain to the real intestinal domain. In terms of feature encoding, a multi-head self-attention mechanism is adopted, and a "convolution-multi-head self-attention block" is designed to replace the traditional convolution block, enhancing the network's performance in accurately encoding features in an intestinal environment with insufficient texture and high feature similarity, thereby effectively improving the estimation accuracy.

[0013] The present invention calculates a global scale factor before supervised training and uses the global scale factor to predict a depth map with absolute scale based on a real intestinal image during testing, which can effectively solve the problem of scale ambiguity in depth estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The present invention will be further described in detail below with reference to the accompanying drawings.

[0015] Figure 1 Flowchart of the present invention. DETAILED DESCRIPTION

[0016] like Figure 1 As shown, a method for estimating the dense depth of the human intestine includes the following steps: Step S1: obtaining a real dataset and a virtual dataset, wherein the real dataset includes a sequence of real image frames collected during a real colonoscopy examination, and the virtual dataset includes a sequence of virtual image frames collected during a simulated colonoscopy examination in a virtual reality colonoscopy platform, wherein each virtual image frame has a corresponding true depth value; Specifically, the real image frame sequence is represented as ,in expresst The real image frame acquired at every moment, N r Represents the number of real image frames. Due to the limitations of the sensor, these real image frames do not have corresponding real depth information. The virtual reality colonoscopy platform is the VR-Caps platform, and the virtual image frame sequence is represented by The VR-Caps platform provides depth information obtained by converting the 3D model based on computed tomography (CT). This depth information is the true depth value corresponding to the virtual image frame. The total true depth value is expressed as ,in, express t The real image frame acquired at every moment, N v Indicates the number of virtual image frames, for The corresponding true depth value.

[0017] Step S2: Construct a domain adaptation network including a convolution-multi-head self-attention module, a multi-scale pyramid module, and a feature decoding module , the weight of the domain adaptation network is , 、 、 They are the weights of the convolution-multi-head self-attention module, the multi-scale pyramid module, and the feature decoding module. The total loss function of the domain adaptation network depends on the self-supervised loss function based on the real dataset. , supervised loss function based on virtual dataset and domain adaptation loss function for gradient reversal-based domain classifier , self-supervised loss function and self-supervision weights Related, domain adaptation loss function and domain classifier weights related; Among them, the convolution-multi-head self-attention module is used to encode the features in the input image, and the multi-scale pyramid module is used to parallelize images of different scales, and to deeply associate the back propagation of the encoder with the decoder. During training, real colonoscopy images are used. and virtual colonoscopy images , during testing, only real colonoscopy images are used .

[0018] Specifically, the self-supervised loss function is expressed as ,in, is the weight coefficient, Represents an edge-aware exponentially weighted smoothing loss term, which is used to smooth the depth map estimated by the domain adaptation network while retaining some The edge details, where , and Respectively t Anti-depth prediction of the moment (Inverse of the depth map) x Direction and y The gradient of the direction, and Refers to the input image exist x Direction and y The gradient in the direction is subjected to exponential weighting operation to obtain and This operation can reduce the impact of large gradient problems when the network learns the edge of the image, so that the final depth map retains the image. It can also achieve smooth depth prediction while retaining edge structure information.

[0019] represents the actual photometric loss, which involves the depth estimation weight and self-supervision weights Joint training, during the training process, two consecutive frames As input, it outputs the pose information between two frames, including rotation and translation. , then, using Generate synthetic frames with the depth map estimated by the adaptive network , in luminosity loss In the optimization, the goal is to minimize the synthetic frame and The difference between. The formula is expressed as in, for Two frames at adjacent moments, express The mean of Represents the photometric error of a single pixel, in order to obtain , first calculate the photometric error measurement of the two pixels and ,in, , for and The pixel photometric error between the two is then used to calculate the binary automatic mask pixel by pixel. , the calculation formula is , represents the Iverson brackets applied pixel by pixel, and finally, The calculation formula is , Represents pixel-by-pixel multiplication. Automatic masking It is used to eliminate the influence of pixels whose photometric values ​​remain unchanged between adjacent frames during training. These pixels usually violate the photometric loss principle, such as static frames, objects moving with the colonoscope, or low-texture areas. The domain adaptation network does not need to be tested. participate.

[0020] The supervised monocular depth estimation relies on the real depth value, but it is difficult to obtain the real depth information in the real intestinal environment. The present invention uses the colonoscopy image obtained in the virtual intestinal environment to obtain the real depth information. and the corresponding depth information For supervised training, we add on As the network structure of the virtual domain. The supervision loss function is expressed as ,in, represents the image mask, which depends on and , is the maximum depth in the real depth map, but Different, can be calculated offline , that is, before starting training, calculate , calculate the value of When, abandon Pixels .

[0021] The goal of domain adaptation is to train the encoder (convolution-multi-head self-attention module) , making it impossible to distinguish whether it is generated from a real colonoscopy input frame (real domain) or a virtual colonoscopy input frame (virtual domain). To ensure accuracy and domain invariance is an adversarial goal. Therefore, the present invention uses the idea of ​​gradient inversion layer, The domain invariance of is given by the weights is measured by a binary target / source classifier of log Loss is used to train the domain classifier, that is, the domain adaptation loss function is expressed as , among which, if ,but Output 1, if ,but Output 0. The gradient inversion layer has no parameters. and This means that during the forward pass of training, it acts as the identity function, while during the backpropagation, it reverses the gradient vector passing through it. , which is not required during testing.

[0022] Step S3: Train the domain adaptation network by forward propagation-backward propagation to minimize the total loss function of the domain adaptation network to obtain the optimal weights. , introduce the equalization factor in training To balance the supervision loss and self-supervision loss.

[0023] Specifically, the process of training a domain adaptation network is essentially to find the optimal weight value by minimizing the loss function ,in, is the loss function, represents the use of virtual intestinal image frames and their real depth map labels, Relying on three different losses, namely the self-supervised loss function based on real intestinal data , supervised loss function based on virtual intestinal data and domain adaptation loss function based on domain classifier , self-supervised loss function Training network weights based on self-supervision , so only real colonoscopy data is used, and the goal of self-supervised learning is to In the neural network, minimize the adjacent real frames Photometric loss between, self-supervised loss function Recorded as .Supervisory loss function Using virtual colonoscopy data and its corresponding real depth value Train network weights in a supervised manner , supervised loss function Recorded as .

[0024] During the training process of the present invention is the ratio between the supervised loss and the self-supervised loss, and introduces a scaling factor , control optimization To obtain a trade-off between accurate depth estimation and domain invariance.

[0025] A training session includes the following steps: Step S31, supervised training process: input get , the loss error Perform back propagation to obtain the updated Parameters , represents the loss error that must be calculated using dummy data; Step S32, self-supervised training process: input get , the loss error Perform back propagation to obtain the updated Parameters , Represents the loss error that must be calculated using real data; Step S33: perform the following calculations based on the results of step S31 and step S32: 、 、 , for the final back-propagation process; From then on, a complete training session was completed.

[0026] In the intestinal environment, where texture is scarce and feature similarity is high, the encoder structure is crucial for effective feature extraction. Long-range correlation encoding (LRC) offers superior feature resolution and extraction capabilities compared to convolution in this environment. The multi-head self-attention mechanism is a high-performance LRC encoding module.

[0027] The present invention adopts a four-layer encoder-decoder structure. The encoder has a linear combination of a convolution-multi-head self-attention module and a multi-scale pyramid module. The convolution-multi-head self-attention module includes three parallel multi-head attention modules and a convolution block. The convolution block extracts the local relationship between adjacent features in the feature. The three multi-head attention modules use the self-attention mechanism to model the information interaction of the entire input space in the feature. More specifically, the three multi-head attention modules embed the feature encoding sequence through a multi-scale pyramid embedding layer and project it into the query (Q), key (K) and value ( ,in N is the number of pixels in a single image, C The self-attention mechanism is implemented in an efficient factorization way: ,in, C is the embedding dimension. Finally, the feature fusion module is used to collect and further enhance the i Interaction between local and global features extracted by the "Convolution-Multi-Head Self-Attention Block" stage: , ,in, CNN 、 G Feature extraction results of convolutional layer and multi-head self-attention layer respectively, is the aggregation feature, It is a 1×1 convolution layer used to fuse the features of each channel and prepare for the next stage. i +1 for generating the final feature Combining convolution with multi-head self-attention significantly improves the network's ability to extract features, helping the network perceive tissue structure and relative position in an in vivo environment with insufficient texture and high feature similarity, including encoding the features of lesion areas in the intestine.

[0028] The decoder leverages the multi-scale features generated by the encoder, employing cross-layer and cross-scale connections to gradually increase spatial resolution. Given the contextual differences between features at different scales, this paper utilizes spatial and channel-wise attention mechanisms to enhance cross-scale feature fusion. A four-layer decoder, consisting of two convolutional layers and a sigmoid activation layer, predicts depth based on the corresponding aggregated features, outputting depth maps at full resolution, 1.2 sigmoid resolution, 1.4 sigmoid resolution, and 1.8 sigmoid resolution, respectively.

[0029] The present invention obtains absolute depth value from virtual data to supervise the training network , obtain relative depth value training from real data without depth labels The unsupervised domain adaptation method based on domain classification, the encoder in this invention is trained to have domain-invariant properties. However, using only the encoder is not enough to make Generate accurate absolute depth values ​​during testing. The present invention calculates the global scale factor before supervised training ,in, , is the total true depth value in the virtual domain The median value of Represents the median of the depth values ​​predicted from the virtual dataset. Global scale factor Make the final depth estimation result This preprocessing step only needs to be performed once before training, and there is no need to calculate new value.

[0030] The above description is merely a preferred embodiment of the present invention and therefore cannot be used to limit the scope of the present invention. In other words, equivalent changes and modifications made according to the scope of the patent application and the contents of the specification should still fall within the scope of the patent of the present invention.

Claims

1. A method for estimating the dense depth of the human intestine, characterized by: The steps include: Step S1, obtaining a real data set and a virtual data set, wherein the real data set includes a real image frame sequence collected during a real colonoscopy, and the virtual data set includes a virtual image frame sequence collected during a simulated colonoscopy in a virtual reality colonoscopy platform, and each virtual image frame has a corresponding real depth value; Step S2: construct a domain adaptation network including a convolution-multi-head self-attention module, a multi-scale pyramid module, and a feature decoding module , the weight of the domain adaptation network is , , , They are the weights of the convolution-multi-head self-attention module, the multi-scale pyramid module, and the feature decoding module. The total loss function of the domain adaptation network depends on the self-supervised loss function based on the real dataset. , supervised loss function based on virtual dataset and domain adaptation loss function for gradient reversal based domain classifier , self-supervised loss function With self-supervision weight Related, domain adaptation loss function and domain classifier weights related; Step S3: Train the domain adaptation network by forward propagation-backward propagation to minimize the total loss function of the domain adaptation network to obtain the best weights. , introduce equalization factor in training To balance the supervision loss and self-supervision loss.

2. A method for estimating dense depth of human intestine according to claim 1, characterized in that: In step S1, the real image frame sequence is represented as ,in express t The real image frame acquired at every moment, N r represents the number of real image frames, the virtual reality colonoscopy platform is the VR-Caps platform, and the virtual image frame sequence is represented by , VR-Caps platform provides depth information obtained by converting the 3D model based on computed tomography. The depth information is the real depth value corresponding to the virtual image frame. The total real depth value is expressed as ,in, express t The virtual image frame acquired at each moment, N v Indicates the number of virtual image frames, for The corresponding depth truth value.

3. A method for estimating the dense depth of human intestine according to claim 2, characterized in that: In step S2, the self-supervised loss function is expressed as ,in, is the weight coefficient, represents an edge-aware exponentially weighted smoothing loss, which is used to smooth the depth map estimated by the domain adaptation network while retaining some The edge details, represents the actual luminosity loss, for Two frames at adjacent times, express The mean of , , represents an Iverson bracket applied pixel by pixel, , for and The pixel brightness error between Indicates that based on the continuous frame , and the synthetic frame generated by the depth map estimated by the adaptive network, Represents pixel-by-pixel multiplication.

4. A method for estimating the dense depth of the human intestine according to claim 1, 2 or 3, characterized in that: In step S2, the supervision loss function is expressed as ,in, represents the image mask, which depends on and , is the maximum depth in the real depth map, and is calculated When, abandon Pixels .

5. A method for estimating dense depth of human intestine according to claim 1, 2 or 3, characterized in that: In step S2, the domain adaptation loss function is expressed as , among which, if ,but Output 1, if ,but Output 0.

6. A method for estimating dense depth of human intestine according to claim 2 or 3, characterized in that: In step S3, one training specifically includes the following steps: Step S31, supervised training process: input get , the loss error Perform back propagation to obtain the updated Parameters ; Step S32, self-supervised training process: input get , the loss error Perform back propagation to obtain the updated Parameters ; Step S33: perform the following calculations according to the results of step S31 and step S32: , , , for the final back-propagation process.

7. A method for estimating the dense depth of the human intestine according to claim 1, 2 or 3, characterized in that: In the step S2, a four-layer encoder-decoder structure is adopted, the encoder has a linear combination of a convolution-multi-head self-attention module and a multi-scale pyramid module, the convolution-multi-head self-attention module includes three parallel multi-head attention modules and a convolution block, the convolution block extracts the local relationship between adjacent features in the feature, the three multi-head attention modules use the self-attention mechanism, and embed feature codes of different sizes in parallel through the multi-scale pyramid module. The decoder uses the multi-scale features generated by the encoder and adopts cross-layer and cross-scale connections to gradually improve the spatial resolution.

8. The method for estimating the dense depth of the human intestine according to claim 6, characterized in that: Before the supervised training process, the global scaling factor is calculated , the global scale factor The final depth estimation result With absolute scale, is the total true depth value The median value of Represents the median of the depth values ​​predicted from the virtual dataset.

Images