Self-supervised monocular depth estimation method based on CNN and visual attention architecture

Abstract

The embodiment of the invention discloses a CNN and visual attention architecture-based self-supervised monocular depth estimation method, which comprises the steps of training a self-supervised monocular depth estimation network by adopting public and clinically collected data sets, and inputting a target image into the depth estimation network to obtain a predicted depth map; inputting the target image and an adjacent frame image of the target image into a posture estimation network to obtain a relative posture matrix of the camera; according to the obtained depth map and the relative pose matrix, a re-projection formula and bilinear sampling are used to obtain a synthetic target image; the network is optimized using a luminosity error loss function and a smoothness error loss function between the minimized target image and the reconstructed target image. Different level features are fused, the feature expression of the weak texture area is enhanced, and the monocular depth estimation precision of weak texture scenes such as ear canals and intestines and stomach is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a self-supervised monocular depth estimation method based on CNN and visual attention architecture. Background Art

[0002] With the development of medical technology, minimally invasive surgery has become feasible and gradually popularized. Endoscopic minimally invasive surgery is widely used in clinical surgeries, such as gastrointestinal, otolaryngology, and urology surgeries. During the operation, a surgical navigation system is needed to provide doctors with three-dimensional information to help them locate surgical instruments, avoid key structures of the human body, and ensure the safety of the operation. Traditional depth estimation methods based on sensors and geometric principles require additional sensors or calibrated stereo images, but due to the limitations of minimally invasive surgical scenarios such as the ear canal, these conditions are difficult to meet, and traditional methods are difficult to apply.

[0003] With the development of artificial intelligence and the deepening of computer vision technology research, depth estimation methods based on deep learning have flourished. They do not require sensors and binocular cameras, but directly use visual clues in two-dimensional images to perform monocular depth estimation. There are two types of depth estimation methods based on deep learning: supervised and self-supervised. Supervised methods require the provision of real depth labels, but real depth labels for narrow scenes such as the human ear canal are difficult to obtain, making it difficult to implement supervised depth estimation methods. Summary of the invention

[0004] The purpose of the present invention is to solve the problems raised in the above-mentioned background technology. The present invention provides a self-supervised monocular depth estimation method based on CNN and visual attention architecture, which integrates features at different levels, enhances the feature expression of weak texture areas, and improves the monocular depth estimation accuracy for weak texture scenes such as the ear canal and gastrointestinal tract.

[0005] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0006] A self-supervised monocular depth estimation method based on CNN and visual attention architecture, including:

[0007] S1. Use public and clinically collected datasets to train a self-supervised monocular depth estimation network, and input the target image into the depth estimation network to obtain a predicted depth map;

[0008] S2, inputting the target image and the adjacent frame images of the target image into the posture estimation network to obtain the relative posture matrix of the camera;

[0009] S3, according to the obtained depth map and relative pose matrix, a synthetic target image is obtained by using a reprojection formula and bilinear sampling;

[0010] S4. The network is optimized by minimizing the photometric error loss function and the smoothness error loss function between the target image and the reconstructed target image.

[0011] Beneficial effects:

[0012] The present invention does not require a data set with real depth labels. It only needs to input a monocular video sequence and camera internal parameters to perform network training. It provides a good depth estimation method for minimally invasive surgery scenarios such as the ear canal, nasal cavity, and gastroenteroscopy, where it is difficult to collect real depth labels and only a monocular endoscope can be used.

[0013] For scenes with weak texture features, the present invention adopts a pyramid-shaped encoder and decoder structure, uses CNN and visual attention to extract feature information in images at multiple scales, and effectively fuses multi-scale features to help enhance the feature expression of weak texture areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 A flowchart of an embodiment of the present invention;

[0016] Figure 2 A network model diagram of an embodiment of the present invention;

[0017] Figure 3 1 is a structural diagram of a convolution module and an attention module according to an embodiment of the present invention;

[0018] Figure 4 Schematic diagram of a self-supervised monocular depth estimation method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0020] As shown in the figure, the embodiment of the present invention discloses a self-supervised monocular depth estimation method based on CNN and visual attention architecture, which is characterized by comprising:

[0021] S1. Use public and clinically collected datasets to train a self-supervised monocular depth estimation network, and input the target image into the depth estimation network to obtain a predicted depth map;

[0022] S2, inputting the target image and the adjacent frame images of the target image into the posture estimation network to obtain the relative posture matrix of the camera;

[0023] S3, according to the obtained depth map and relative pose matrix, a synthetic target image is obtained by using a reprojection formula and bilinear sampling;

[0024] S4. The network is optimized by minimizing the photometric error loss function and the smoothness error loss function between the target image and the reconstructed target image.

[0025] In one embodiment of the present invention, in step S1, a target image of size H×W is input into a depth estimation network of an encoder-decoder structure. In order to better extract features, the target image will reduce its resolution through an average pooling operation and input it into the corresponding scale layer. The encoder extracts features from the input image through a convolution module and an attention module. After the features of the current resolution are extracted, downsampling is performed, and feature extraction is performed again on the target image at different resolutions. This ensures that the encoder can extract more diverse features from more scales. The features extracted at different resolutions will be transmitted to the next scale, combined with the features obtained after pooling the target image, and then input.

[0026] In one embodiment of the present invention, in step S1, the features obtained by the encoder are input into the decoder, and the decoder fuses and upsamples these features and outputs the predicted depth map D t , a skip connection is added between the encoder and decoder.

[0027] In one embodiment of the present invention, in step S2, the target image I t and adjacent frame images I s Input to the pose estimation network to obtain the relative pose matrix T containing the rotation matrix and translation matrix t→s .

[0028] In one embodiment of the present invention, in step S3, the final task of the network is designed as an image reconstruction task, that is, according to the predicted depth map D t And the relative pose matrix T obtained by the pose estimation network t→s , reconstruct the target image I using reprojection and bilinear sampling t , as attached Figure 4 As shown in the figure, the essence of self-supervised monocular depth estimation is view reconstruction, using the predicted target image depth map D t , relative pose matrix T t→s Projection to get the target view I t Reconstructed images from the perspective of the camera poses in adjacent frames Finally, the target image I t and reconstruct the image As a supervisory signal, the network is optimized and the reconstruction task is used to realize self-supervisory training of the network, which solves the problem that it is difficult to obtain real labels in minimally invasive surgery scenes. In the process of reconstructing the target image, the reprojection formula is used to transform the pixel coordinates, and bilinear sampling is used to obtain the pixel coordinates from the adjacent images I. s Get pixel values ​​to reconstruct the target image I t , the formula is as follows:

[0029] p s ~KT t→s D t (p t )K -1 p t

[0030]

[0031] Among them, p t is the homogeneous coordinate of the pixel in the target image, K is the intrinsic parameter of the monocular camera, and D t is the predicted depth map of the target image, T t→s is the relative pose between the target image and the adjacent images, p s is the target image pixel p t The projection coordinates in the pose of the adjacent image, because the projection coordinates p s It is a continuous value, and the projection coordinates may not have corresponding pixel values, so bilinear sampling is used to obtain the reconstructed image from the corresponding coordinates of the adjacent images. The pixel value of .

[0032] In one embodiment of the present invention, in step S4, the network is optimized using a loss function that mixes photometric error loss and smoothness error.

[0033] In one embodiment of the present invention, the encoder backbone includes a Conv Block and three Attention Blocks. When the target image I of size H×W t After being input into the depth estimation network, it first passes through the Conv Block layer convolution to obtain a size of The feature map is input to the next layer and is obtained by downsampling the target image. The feature maps are merged and input into the Attention Block, and the output of the Attention Block is also operated in the same way;

[0034] After multiple downsampling and feature extraction, the feature map is input into the decoder, and then upsampled by UpsamplingBlock to gradually restore it to its original size. During the upsampling process, in order to avoid the loss of detail features, a jump connection is added between the encoder and the decoder. Finally, the upsampled feature map is input into the Depth Regressor, and the predicted depth map D is obtained through two layers of convolution and Sigmoid layer. t The two most important modules in the deep convolutional network are ConvBlock and Attention Block. Figure 3 . Among them, Conv Block is mainly convolution. In addition to the ordinary convolution layer, a deep separable convolution (DW Conv) is also added, which can reduce parameters and calculations while maintaining the efficient feature extraction ability of convolution. Attention Block is the main innovation of the present invention. It combines visual attention and convolution to better extract local and global features of the image. The ViT (Vision Transformer) in the module can use the ViT model pre-trained in a large-scale dataset. After fine-tuning, it can achieve good global feature extraction effects. In this way, training the model with less endoscopic data can also achieve good results.

[0035] In one embodiment of the present invention, as shown in the attached Figure 2 As shown, the pose estimation network of the model adopts the ResNet18 architecture, and the input target image I t and adjacent frame images I s , output the relative pose T with 6 degrees of freedom t→s ResNet18 is efficient and lightweight, and can accurately classify images of different categories. As a feature extractor, ResNet18 can provide rich image features for the target detection algorithm and improve the accuracy of target detection.

[0036] In one embodiment of the present invention, in step S4, the network model is optimized according to the minimum photometric error loss function and the smoothness error loss function, and the formula of the photometric error loss function is as follows:

[0037]

[0038] The photometric error loss function is used to measure the reconstruction effect, where α is the weight, which is a hyperparameter and is set to 0.85, and SSIM is the structural similarity;

[0039] The formula of the smoothness error loss function is as follows:

[0040]

[0041] The smoothness error loss function is used to reduce blur and preserve depth details, where is the first-order derivative along the spatial direction;

[0042] The total loss function is expressed as follows:

[0043] L=λL p +βL smooth

[0044] The total function is a weighted sum of the photometric error loss function and the smoothness error loss function. Among them, λ and β are weight parameters, L p is the photometric error loss function, L smooth It is the smoothness error loss function. The total function can better supervise network training and optimize the network by combining the two loss functions.

[0045] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A self-supervised monocular depth estimation method based on CNN and visual attention architecture, characterized in that: include: S1. Use public and clinically collected datasets to train a self-supervised monocular depth estimation network, and input the target image into the depth estimation network to obtain a predicted depth map; S2, inputting the target image and the adjacent frame images of the target image into the posture estimation network to obtain the relative posture matrix of the camera; S3, according to the obtained depth map and relative pose matrix, a synthetic target image is obtained by using a reprojection formula and bilinear sampling; S4. The network is optimized by minimizing the photometric error loss function and the smoothness error loss function between the target image and the reconstructed target image.

2. The method of self-supervised monocular depth estimation based on CNN and visual attention architecture according to claim 1, characterized in that: In step S1, a target image of size H×W is input into a depth estimation network of an encoder-decoder structure. The target image is reduced in resolution through an average pooling operation and input into the corresponding scale layer. The encoder extracts features from the input image through a convolution module and an attention module. After the features of the current resolution are extracted, downsampling is performed, and features of the target image are extracted again at different resolutions. The features extracted at different resolutions are transmitted to the next scale, combined with the features obtained after pooling of the target image, and then input.

3. The self-supervised monocular depth estimation method based on CNN and visual attention architecture according to claim 2, characterized in that: In step S1, the features obtained by the encoder are input into the decoder, which fuses and upsamples these features and outputs the predicted depth map D t , a skip connection is added between the encoder and decoder.

4. The method of self-supervised monocular depth estimation based on CNN and visual attention architecture according to claim 1, characterized in that: In step S2, the target image I t and adjacent frame images I s Input to the pose estimation network to obtain the relative pose matrix T containing the rotation matrix and translation matrix t→s .

5. The method of self-supervised monocular depth estimation based on CNN and visual attention architecture according to claim 1, characterized in that: In step S3, the final task of the network is designed as an image reconstruction task, that is, according to the predicted depth map D t And the relative pose matrix T obtained by the pose estimation network t→s , reconstruct the target image I using reprojection and bilinear sampling t , using the predicted target image depth map D t , relative pose matrix T t→s Projection to get the target view I t Reconstructed images from the perspective of the camera poses in adjacent frames Finally, the target image I t and reconstruct the image As a supervisory signal to supervise the optimization of the network; in the process of reconstructing the target image, the reprojection formula is used to transform the pixel coordinates, and bilinear sampling is used to extract the pixel coordinates from the adjacent image I s Get pixel values ​​to reconstruct the target image I t , the formula is as follows: p s ~KT t→s D t (p t )K -1 p t Among them, p t is the homogeneous coordinate of the pixel in the target image, K is the intrinsic parameter of the monocular camera, and D t is the predicted depth map of the target image, T t→s is the relative pose between the target image and the adjacent images, p s is the target image pixel p t The projection coordinates under the posture of the adjacent image, the projection coordinates p s It is a continuous value, and the reconstructed image is obtained from the corresponding coordinates of adjacent images using bilinear sampling. The pixel value of .

6. The method of self-supervised monocular depth estimation based on CNN and visual attention architecture according to claim 1, characterized in that: In step S4, the network is optimized with a loss function that mixes photometric error loss and smoothness error.

7. The method of self-supervised monocular depth estimation based on CNN and visual attention architecture according to claim 3, characterized in that: The encoder backbone consists of a Conv Block and three Attention Blocks. When the target image I of size H×W t After being input into the depth estimation network, it first passes through the Conv Block layer convolution to obtain a size of The feature map is input to the next layer and is obtained by downsampling the target image. The feature maps are merged and input into the Attention Block, and the output of the Attention Block is also operated in the same way; After multiple downsampling and feature extraction, the feature map is input into the decoder, and then gradually restored to its original size through upsampling by Upsampling Block. During the upsampling process, a jump connection is added between the encoder and the decoder. Finally, the upsampled feature map is input into the Depth Regressor, and the predicted depth map D is obtained through two layers of convolution and Sigmoid layer. t .

8. The method of self-supervised monocular depth estimation based on CNN and visual attention architecture according to claim 4, characterized in that: The pose estimation network of the model adopts the ResNet18 architecture and inputs the target image I t and adjacent frame images I s , output the relative pose T with 6 degrees of freedom t→s .

9. The method of self-supervised monocular depth estimation based on CNN and visual attention architecture according to claim 1, characterized in that: In step S4, the network model is optimized according to the minimum photometric error loss function and the smoothness error loss function. The formula of the photometric error loss function is as follows: The photometric error loss function is used to measure the reconstruction effect, where α is the weight, which is a hyperparameter and is set to 0.85, and SSIM is the structural similarity; The formula of the smoothness error loss function is as follows: The smoothness error loss function is used to reduce blur and preserve depth details, where is the first-order derivative along the spatial direction; The overall function is expressed as follows: L=λL p +βL smooth Among them, λ and β are weight parameters, L p is the photometric error loss function, L smooth is the smoothness error loss function.

Images