A dense depth estimation method for human intestine

Abstract

The present invention provides a method for estimating the dense depth of the human intestine, which belongs to the field of computer vision and artificial intelligence, and includes: step S1, obtaining a real data set and a virtual data set; step S2, constructing a domain adaptation network including a convolution-multi-head self-attention module, a multi-scale pyramid module and a feature decoding module, the weight of the domain adaptation network is, and the total loss function of the domain adaptation network depends on the self-supervised loss function based on the real data set, the supervised loss function based on the virtual data set and the domain adaptation loss function of the domain classifier based on gradient reversal; step S3, training the domain adaptation network by forward transfer-backward transfer to minimize the total loss function of the domain adaptation network to obtain the best weight. The present invention can effectively improve the estimation accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and artificial intelligence, and in particular relates to a method for estimating the dense depth of human intestine. Background Art

[0002] In the process of inspection and treatment of intestinal diseases, it is usually necessary to combine the depth information of intestinal images for analysis. However, the human intestinal environment has a narrow space, insufficient texture, and high geometric feature similarity, which brings challenges to the depth estimation task. Existing learning-based human intestinal depth estimation methods mainly include supervised learning methods and self-supervised methods. Supervised learning methods have significant overall depth prediction effects. Such methods rely on accurate dense depth labels as supervisory signals to train models, but due to the limitations of the use of in vivo sensors, it is difficult to obtain accurate dense depth labels for the in vivo intestinal environment. Self-supervised methods use continuous multi-frame colonoscopy images for training. However, such methods are difficult to obtain accurate encoding features in intestinal environments with scarce textures and high feature similarity, resulting in insufficient accuracy in intestinal environment depth estimation. In addition, self-supervised methods rely on photometric consistency between adjacent frames for training, and cannot consider the continuity of the depth of corresponding points from a global perspective. Therefore, there is a problem of inconsistent depth value scales of the same point between different frames, that is, the scale ambiguity problem. Summary of the invention

[0003] The purpose of the present invention is to propose a method for estimating the dense depth of the human intestine, which can effectively improve the estimation accuracy.

[0004] The present invention is achieved through the following technical solutions:

[0005] A method for estimating the dense depth of human intestine comprises the following steps:

[0006] Step S1, obtaining a real data set and a virtual data set, wherein the real data set includes a real image frame sequence collected during a real colonoscopy, and the virtual data set includes a virtual image frame sequence collected during a simulated colonoscopy in a virtual reality colonoscopy platform, and each virtual image frame has a corresponding real depth value;

[0007] Step S2: construct a domain adaptation network including a convolution-multi-head self-attention module, a multi-scale pyramid module, and a feature decoding module , the weight of the domain adaptation network is , , , They are the weights of the convolution-multi-head self-attention module, the multi-scale pyramid module, and the feature decoding module. The total loss function of the domain adaptation network depends on the self-supervised loss function based on the real dataset. , supervised loss function based on virtual dataset and domain adaptation loss function for gradient reversal based domain classifier , self-supervised loss function With self-supervision weight Related, domain adaptation loss function and domain classifier weights related;

[0008] Step S3: Train the domain adaptation network by forward propagation-backward propagation to minimize the total loss function of the domain adaptation network to obtain the best weights. , introduce equalization factor in training To balance the supervision loss and self-supervision loss.

[0009] Furthermore, in step S1, the real image frame sequence is represented as ,in express t The real image frames acquired at every moment, N r represents the number of real image frames, the virtual reality colonoscopy platform is the VR-Caps platform, and the virtual image frame sequence is represented by The VR-Caps platform provides depth information obtained by converting the 3D model based on computed tomography. The depth information is the true depth value corresponding to the virtual image frame. The total true depth value is expressed as ,in, express t The real image frame acquired at every moment, N v Indicates the number of virtual image frames, for The corresponding depth truth value.

[0010] Furthermore, in step S2, the self-supervised loss function is expressed as ,in, is the weight coefficient, Represents, which is used to smooth the depth map estimated by the domain adaptation network while retaining part The edge details, represents the actual luminosity loss, for Two frames at adjacent times, express The mean of , , represents an Iverson bracket applied pixel by pixel, , for and The pixel brightness error between Indicates that based on the continuous frame , and the synthetic frame generated by the depth map estimated by the adaptive network, Represents pixel-by-pixel multiplication.

[0011] Furthermore, in step S2, the supervision loss function is expressed as ,in, represents the image mask, which depends on and , is the maximum depth in the real depth map, and is calculated When, abandon Pixels .

[0012] Furthermore, in step S2, the domain adaptation loss function is expressed as , among which, if ,but Output 1, if ,but Output 0.

[0013] Furthermore, in step S3, one training specifically includes the following steps:

[0014] Step S31, supervised training process: input get , the loss error Perform back propagation to obtain the updated Parameters ;

[0015] Step S32, self-supervised training process: input get , the loss error Perform back propagation to obtain the updated Parameters ;

[0016] Step S33: perform the following calculations according to the results of step S31 and step S32: , , , for the final back-propagation process.

[0017] Furthermore, in step S2, a four-layer encoder-decoder structure is adopted, and the encoder has a linear combination of a convolution-multi-head self-attention module and a multi-scale pyramid module. The convolution-multi-head self-attention module includes three parallel multi-head attention modules and a convolution block. The convolution block extracts the local relationship between adjacent features in the feature. The three multi-head attention modules utilize the self-attention mechanism to embed feature codes of different sizes in parallel through the multi-scale pyramid module. The decoder utilizes the multi-scale features generated by the encoder and adopts cross-layer and cross-scale connections to gradually improve the spatial resolution.

[0018] Furthermore, before the supervised training process, the global scaling factor is calculated , the global scale factor The final depth estimation result With absolute scale, is the total true depth value The median value of Represents the median of the depth values ​​predicted from the virtual dataset.

[0019] The present invention has the following beneficial effects:

[0020] The present invention first obtains a real data set and a virtual data set, and then constructs a domain adaptation network. The weight of the domain adaptation network is determined by the weight of the convolution-multi-head self-attention module, the weight of the multi-scale pyramid module and the weight of the feature decoding module. The total loss function of the domain adaptation network depends on the self-supervised loss function based on the real data set. , supervised loss function based on virtual dataset and domain adaptation loss function for gradient reversal based domain classifier , the encoder is jointly trained with the domain classifier, so that the domain adaptation network can transfer the experience learned in the virtual intestinal domain where deep data is easy to obtain to the real intestinal domain. In terms of feature encoding, a multi-head self-attention mechanism is adopted, and a "convolution-multi-head self-attention block" is designed to replace the traditional convolution block, enhancing the network's performance in accurately encoding features in an intestinal environment with insufficient texture and high feature similarity, thereby effectively improving the estimation accuracy.

[0021] The present invention calculates a global scale factor before supervised training, and uses the global scale factor to predict a depth map with an absolute scale according to a real intestinal image during testing, which can effectively solve the problem of scale ambiguity in depth estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The present invention is further described in detail below with reference to the accompanying drawings.

[0023] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0024] like Figure 1 As shown, a method for estimating the dense depth of the human intestine includes the following steps:

[0025] Step S1, obtaining a real data set and a virtual data set, wherein the real data set includes a real image frame sequence collected during a real colonoscopy, and the virtual data set includes a virtual image frame sequence collected during a simulated colonoscopy in a virtual reality colonoscopy platform, and each virtual image frame has a corresponding real depth value;

[0026] Specifically, the real image frame sequence is represented as ,in express t The real image frame acquired at every moment, N r Represents the number of real image frames. Due to the limitation of the sensor, these real image frames do not have corresponding real depth information. The virtual reality colonoscopy platform is the VR-Caps platform, and the virtual image frame sequence is represented by The VR-Caps platform provides depth information obtained by converting the 3D model based on computed tomography (CT). The depth information is the true depth value corresponding to the virtual image frame. The total true depth value is expressed as ,in, express t The real image frame acquired at every moment, N v Indicates the number of virtual image frames, for The corresponding depth truth value.

[0027] Step S2: construct a domain adaptation network including a convolution-multi-head self-attention module, a multi-scale pyramid module, and a feature decoding module , the weight of the domain adaptation network is , , , They are the weights of the convolution-multi-head self-attention module, the multi-scale pyramid module, and the feature decoding module. The total loss function of the domain adaptation network depends on the self-supervised loss function based on the real dataset. , supervised loss function based on virtual dataset and domain adaptation loss function for gradient reversal based domain classifier , self-supervised loss function With self-supervision weight Related, domain adaptation loss function and domain classifier weights related;

[0028] Among them, the function of the convolution-multi-head self-attention module is to encode the features in the input image, and the function of the multi-scale pyramid module is to parallelize images of different scales, and to deeply associate the back propagation of the encoder with the decoder. During training, real colonoscopy images are used. and virtual colonoscopy images , during testing, only real colonoscopy images are used .

[0029] Specifically, the self-supervised loss function is expressed as ,in, is the weight coefficient, represents an edge-aware exponentially weighted smoothing loss term, which is used to smooth the depth map estimated by the domain adaptation network while retaining some The edge details, where , and Respectively t Anti-depth prediction of the moment (Inverse of the depth map) x Direction and y The gradient of the direction, and The input image exist x Direction and y The gradient in the direction is exponentially weighted to obtain and This operation can reduce the impact of large gradient problems when the network learns the edge of the image, so that the final depth map retains the image It can also achieve smooth depth prediction while retaining edge structure information.

[0030] represents the actual photometric loss, which involves the depth estimation weight and self-supervision weights During the joint training, two consecutive frames are As input, it outputs the pose information between two frames, including rotation and translation , then, using Generate synthetic frames with the depth map estimated by the adaptive network , in luminosity loss In the optimization, the goal is to minimize the synthetic frame and The difference between. The formula is expressed as in, for Two frames at adjacent times, express The mean of Represents the photometric error of a single pixel. In order to obtain , first calculate the photometric error measurement of the two pixels and ,in, , for and The pixel photometric error between the two is then used to calculate the binary automatic mask pixel by pixel. , the calculation formula is , represents the Iverson bracket applied pixel by pixel, and finally, The calculation formula is , Represents pixel-by-pixel multiplication. Automatic masking It is used to eliminate the influence of pixels whose photometric values ​​remain unchanged between adjacent frames during training. These pixels usually violate the photometric loss principle, such as static frames, objects moving with the colonoscope, or low-texture areas. The domain adaptation network does not need to be used during testing. participate.

[0031] The supervised monocular depth estimation relies on the real depth value, but it is difficult to obtain the real depth information in the real intestinal environment. The present invention uses the colonoscopy image obtained in the virtual intestinal environment to obtain the real depth information. And the corresponding depth information For supervised training, we add on As the network structure of the virtual domain. The supervision loss function is expressed as ,in, represents the image mask, which depends on and , is the maximum depth in the real depth map, but Different, can be calculated offline , that is, before starting training, calculate The value of When, abandon Pixels .

[0032] The goal of domain adaptation is to train the encoder (convolution-multi-head self-attention module) , making it impossible to distinguish whether it is generated from a real colonoscopy input frame (real domain) or a virtual colonoscopy input frame (virtual domain). To ensure accuracy and domain invariance is an adversarial goal. Therefore, the present invention uses the idea of ​​gradient inversion layer, The domain invariance of is given by the weights is measured by a binary target / source domain classifier of log Loss is used to train the domain classifier, that is, the domain adaptation loss function is expressed as , among which, if ,but Output 1, if ,but Output 0. The gradient inversion layer has no parameters. and This means that during the forward pass of training, it acts as the identity function, while during the back-propagation, it inverts the gradient vector passing through it. Gradient inversion layers and , which is not required during testing.

[0033] Step S3: Train the domain adaptation network by forward propagation-backward propagation to minimize the total loss function of the domain adaptation network to obtain the best weights. , introduce equalization factor in training To balance the supervision loss and self-supervision loss.

[0034] Specifically, the process of training a domain adaptation network is essentially to find the optimal weight value by minimizing the loss function ,in, is the loss function, represents the use of virtual intestinal image frames and their true depth map labels, Relies on three different losses, namely a self-supervised loss function based on real intestinal data , supervised loss function based on virtual intestinal data and domain adaptation loss function based on domain classifier , self-supervised loss function Training network weights based on self-supervision , so only real colonoscopy data is used, and the goal of self-supervised learning is to In the neural network, minimize the adjacent real frames Photometric loss between, self-supervised loss function Recorded as .Supervisory loss function Using virtual colonoscopy data And its corresponding real depth value Train the network weights in a supervised manner , supervised loss function Recorded as .

[0035] During the training process of the present invention is the ratio between the supervised loss and the self-supervised loss, and introduces a scaling factor , control optimization To achieve a trade-off between accurate depth estimation and domain invariance.

[0036] A training session includes the following steps:

[0037] Step S31, supervised training process: input get , the loss error Perform back propagation to obtain the updated Parameters , represents the loss error that must be calculated using dummy data;

[0038] Step S32, self-supervised training process: input get , the loss error Perform back propagation to obtain the updated Parameters , Represents the loss error that must be calculated using real data;

[0039] Step S33: perform the following calculations according to the results of step S31 and step S32: , , , for the final back-propagation process;

[0040] From then on, a complete training was completed.

[0041] In the intestinal environment with insufficient texture and high feature similarity, the structure of the encoder is crucial for effective feature extraction. Long-distance correlation coding has stronger feature resolution and extraction capabilities than convolution in the intestinal environment with insufficient texture and high feature similarity. The multi-head self-attention mechanism is a high-performance long-distance correlation coding module.

[0042] The present invention adopts a four-layer encoder-decoder structure. The encoder has a linear combination of a convolution-multi-head self-attention module and a multi-scale pyramid module. The convolution-multi-head self-attention module includes three parallel multi-head attention modules and a convolution block. The convolution block extracts the local relationship between adjacent features in the feature. The three multi-head attention modules use the self-attention mechanism to model the information interaction of the entire input space in the feature. More specifically, the three multi-head attention modules embed the feature encoding sequence through a multi-scale pyramid embedding layer and project it to the query (Q), key (K) and value ( ,in N is the number of pixels in a single image, C The self-attention mechanism is implemented in an efficient factorization way: ,in, C is the embedding dimension. Finally, the feature fusion module is used to collect and further enhance the i Interaction between local and global features extracted by the stage “Convolution-Multi-Head Self-Attention Block”: , ,in, CNN , G Feature extraction results of convolutional layer and multi-head self-attention layer respectively. is the aggregation feature, It is a 1×1 convolution layer, which is used to fuse the features of each channel and prepare for the next stage. i +1 for generating the final feature Combining convolution with multi-head self-attention significantly improves the network's ability to extract features, helping the network to perceive tissue structure and relative position in an in vivo environment with insufficient texture and high feature similarity, including encoding the features of lesion areas in the intestine.

[0043] The decoder uses the multi-scale features generated by the encoder and adopts cross-layer and cross-scale connections to gradually improve the spatial resolution. There are contextual differences between features of different scales. The present invention uses spatial and channel attention mechanisms to enhance cross-scale feature fusion. The four-layer decoder consisting of two convolutional layers and a sigmoid activation layer performs depth prediction based on the corresponding aggregated features, and outputs depth maps of full resolution, 1.2 resolution, 1.4 resolution, and 1.8 resolution, respectively.

[0044] The present invention obtains absolute depth value from virtual data to supervise the training network , obtain relative depth value training from real data without depth labels . Unsupervised domain adaptation method based on domain classification, the encoder in the present invention is trained to have domain-invariant properties. However, using only the encoder is not enough to make Generate accurate absolute depth values ​​during testing. The present invention calculates the global scale factor before supervised training ,in, , is the total true depth value in the virtual domain The median value of Represents the median of the depth values ​​predicted from the virtual dataset. Global scale factor The final depth estimation result This preprocessing step only needs to be performed once before training, and there is no need to calculate new value.

[0045] The above description is only a preferred embodiment of the present invention, and therefore cannot be used to limit the scope of implementation of the present invention. That is, equivalent changes and modifications made according to the scope of the patent application of the present invention and the contents of the specification should still fall within the scope covered by the patent of the present invention.

Claims

1. A method for estimating the dense depth of the human intestine, characterized by: The steps include: Step S1, obtaining a real data set and a virtual data set, wherein the real data set includes a real image frame sequence collected during a real colonoscopy, and the virtual data set includes a virtual image frame sequence collected during a simulated colonoscopy in a virtual reality colonoscopy platform, and each virtual image frame has a corresponding real depth value; Step S2: construct a domain adaptation network including a convolution-multi-head self-attention module, a multi-scale pyramid module, and a feature decoding module , the weight of the domain adaptation network is , , , They are the weights of the convolution-multi-head self-attention module, the multi-scale pyramid module, and the feature decoding module. The total loss function of the domain adaptation network depends on the self-supervised loss function based on the real dataset. , supervised loss function based on virtual dataset and domain adaptation loss function for gradient reversal based domain classifier , self-supervised loss function With self-supervision weight Related, domain adaptation loss function and domain classifier weights related; Step S3: Train the domain adaptation network by forward propagation-backward propagation to minimize the total loss function of the domain adaptation network to obtain the best weights. , introduce equalization factor in training To balance the supervision loss and self-supervision loss; In step S2, the self-supervised loss function is expressed as ,in, is the weight coefficient, represents an edge-aware exponentially weighted smoothing loss, which is used to smooth the depth map estimated by the domain adaptation network while retaining some The edge details, represents the actual luminosity loss, for Two frames at adjacent times, express The mean of , , represents an Iverson bracket applied pixel by pixel, , for and The pixel brightness error between Indicates that based on the continuous frame , and the synthetic frame generated by the depth map estimated by the adaptive network, represents pixel-by-pixel multiplication; In step S2, the supervision loss function is expressed as ,in, represents the image mask, which depends on and , is the maximum depth in the real depth map, and is calculated When, abandon Pixels , Indicates the norm; In step S2, the domain adaptation loss function is expressed as , among which, if ,but Output 1, if ,but Output 0; In step S3, one training specifically includes the following steps: Step S31, supervised training process: input get , the loss error Perform back propagation to obtain the updated Parameters ; Step S32, self-supervised training process: input get , the loss error Perform back propagation to obtain the updated Parameters ; Step S33: perform the following calculations according to the results of step S31 and step S32: , , , for the final back-propagation process; in, L r represents a real image frame sequence, L v represents a sequence of virtual image frames, express t The real image frame acquired at every moment, express t The virtual image frame acquired at each moment, for The corresponding depth truth value, Z v Represents the total depth truth value of the virtual image frame sequence.

2. A method for estimating dense depth of human intestine according to claim 1, characterized in that: In step S1, the real image frame sequence is represented as ,in express t The real image frame acquired at every moment, N r represents the number of real image frames, the virtual reality colonoscopy platform is the VR-Caps platform, and the virtual image frame sequence is represented by , VR-Caps platform provides depth information obtained by converting the 3D model based on computed tomography. The depth information is the real depth value corresponding to the virtual image frame. The total real depth value is expressed as ,in, express t The virtual image frame acquired at each moment, N v Indicates the number of virtual image frames, for The corresponding depth truth value.

3. A method for estimating the dense depth of human intestine according to claim 1 or 2, characterized in that: In the step S2, a four-layer encoder-decoder structure is adopted, the encoder has a linear combination of a convolution-multi-head self-attention module and a multi-scale pyramid module, the convolution-multi-head self-attention module includes three parallel multi-head attention modules and a convolution block, the convolution block extracts the local relationship between adjacent features in the feature, the three multi-head attention modules use the self-attention mechanism, and embed feature codes of different sizes in parallel through the multi-scale pyramid module. The decoder uses the multi-scale features generated by the encoder and adopts cross-layer and cross-scale connections to gradually improve the spatial resolution.

4. A method for estimating the dense depth of human intestine according to claim 3, characterized in that: Before the supervised training process, the global scaling factor is calculated , the global scale factor The final depth estimation result With absolute scale, is the total true depth value The median value of Represents the median of the depth values ​​predicted from the virtual dataset.

Images