Method and device for unsupervised depth and motion estimation of monocular endoscope

Abstract

A method and device for unsupervised depth and motion estimation of a monocular endoscope, the method comprising: (1) obtaining a dense descriptor map: performing feature matching on an input source-target image pair to obtain sparse matching point pairs; obtaining a dense descriptor map corresponding to the input image through a feature learning network; establishing a relative response loss using the sparse matching point pairs and the descriptor map; (2) performing depth and pose estimation: performing dense feature matching on the dense descriptor map obtained by feature extraction to obtain N pairs of dense matching point pairs; a depth estimation network and a pose estimation network respectively obtain a depth map and a relative pose of the input image; in order to provide supervision signals to the depth and pose estimation networks, the descriptor map and the dense matching point pairs are respectively used to construct a feature consistency loss and an epipolar geometric consistency loss.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a method for unsupervised depth and motion estimation of a monocular endoscope, and a device for unsupervised depth and motion estimation of a monocular endoscope. Background Art

[0002] In minimally invasive surgery (MIS), such as urology, abdominal surgery, and thoracic surgery, surgeons usually need to use medical endoscopes to observe anatomical structures in the body. However, the incision in MIS is usually very small, and it is still difficult for surgeons to perform precise surgical operations only through the narrow range of visual observation provided by the endoscope. Fortunately, endoscope-based surgical navigation systems can provide surgeons with real-time anatomical structure and position information. However, in such systems, since endoscopic imaging usually loses three-dimensional structural information, in order to provide doctors with more effective surgical guidance, it is usually necessary to recover dense 3D information and the relative position of the endoscope from intraoperative endoscopic images. Therefore, it is necessary to accurately estimate the image depth and camera pose of the endoscopic image so that the lost 3D information can be restored and the position information of the endoscope can be obtained.

[0003] In order to obtain accurate depth and pose information while observing with an endoscope, the following methods can usually be used: stereo reconstruction using binocular vision principles; direct depth measurement using structured light. Compared with the above two methods, the method of directly estimating depth through a monocular endoscope has the advantages of small hardware size, simple structure, low cost, and easy integration into clinical surgical procedures. However, endoscopic images usually have the characteristics of mirror reflection, non-constant luminosity, weak texture, image disturbance (smoke, blood, etc.), and tissue deformation. Commonly used depth measurement methods based on endoscopic image features, such as motion structure reconstruction (SFM) and simultaneous localization and mapping (SLAM), usually have difficulty in obtaining stable and reliable image features, and thus fail when calculating image depth and camera pose through feature matching and triangulation.

[0004] In recent years, deep learning has been widely used in various fields of medical image processing. Endoscopic image depth estimation based on deep learning can perform dense depth estimation through an end-to-end network model. However, due to the inherent characteristics of endoscopic images such as non-constant luminosity and similar texture, unsupervised monocular depth estimation methods based on view synthesis, which have performed well in the field of computer vision, usually find it difficult to obtain good depth estimation. At the same time, Bian et al. pointed out in the unsupervised depth estimation of indoor videos that compared with the movement of cars in the field of computer vision, indoor videos usually have more camera rotations and less camera translations. In endoscopic videos, the movement of the camera also has the same characteristics as indoor videos. Bian et al. proved that for the current unsupervised depth estimation framework based on view synthesis, camera rotation is more like noise, while camera translation will help depth estimation. The size of the camera translation between two frames will affect the accuracy of depth estimation, and a larger translation usually helps to obtain a more accurate depth estimation. However, a larger translation means that the luminosity difference between the two frames is also larger, which will violate the assumption of constant luminosity in the view synthesis method. Summary of the invention

[0005] In order to overcome the defects of the prior art, the technical problem to be solved by the present invention is to provide a method for unsupervised depth and motion estimation of a monocular endoscope, which, on the one hand, solves the problem of failure of depth estimation caused by non-constant luminosity and texture similarity of endoscopic images, and on the other hand, allows the use of two frames of images corresponding to a larger camera translation amount for depth and pose estimation, thereby improving the prediction accuracy.

[0006] The technical solution of the present invention is: the method for unsupervised depth and motion estimation of a monocular endoscope comprises the following steps:

[0007] (1) Obtaining a dense descriptor map: Perform feature matching on the input source-target image pair to obtain sparse matching point pairs; obtain the dense descriptor map corresponding to the input image through the feature learning network; and establish a relative response loss using the sparse matching point pairs and the dense descriptor map;

[0008] (2) Depth and pose estimation: Perform the following operations on the dense descriptor graph obtained by feature extraction:

[0009] Dense feature matching, obtain N pairs of dense matching points The depth estimation network and the pose estimation network obtain the depth map and relative pose of the input image respectively; in order to provide supervision signals for the depth and pose estimation networks, the descriptor map and the dense matching point pairs are They are used to construct feature consistency loss and epipolar geometry consistency loss respectively.

[0010] The present invention builds a feature learning network Feat Net to learn dense feature descriptors, and avoids the chessboard artifacts generated by the Padding technology used in the convolutional layer; the descriptors obtained through learning have a certain degree of stability between two frames of images, which solves the problem of failure of depth estimation caused by inconsistent luminosity and similar texture of endoscopic images, and on the other hand, it allows the use of two frames of images corresponding to a larger camera translation to perform depth and pose estimation. The extracted dense feature descriptors can provide a supervisory signal for depth estimation, and by introducing geometric constraints between video frames, feature matching and depth and pose estimation are jointly learned to improve the prediction accuracy of the proposed method.

[0011] Also provided is an apparatus for unsupervised depth and motion estimation of a monocular endoscope, comprising:

[0012] A dense descriptor acquisition module is configured to perform feature matching on the input source-target image pair to obtain sparse matching point pairs; a dense descriptor map corresponding to the input image is obtained through a feature learning network; a relative response loss is established using the sparse matching point pairs and the dense descriptor map; a depth and pose estimation module is configured to perform dense feature matching on the dense descriptor map obtained by feature extraction to obtain N pairs of dense matching point pairs The depth estimation network and the pose estimation network obtain the depth map and relative pose of the input image respectively; in order to provide supervision signals for the depth and pose estimation networks, the descriptor map and the dense matching point pairs are They are used to construct feature consistency loss and epipolar geometry consistency loss respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a flow chart of a method for unsupervised depth and motion estimation of a monocular endoscope according to the present invention. DETAILED DESCRIPTION

[0014] like Figure 1 As shown, the method for unsupervised depth and motion estimation of a monocular endoscope comprises the following steps:

[0015] (1) Obtaining a dense descriptor map: Perform feature matching on the input source-target image pair to obtain sparse matching point pairs; obtain the dense descriptor map corresponding to the input image through the feature learning network; and establish a relative response loss using the sparse matching point pairs and the dense descriptor map;

[0016] (2) Depth and pose estimation: Perform dense feature matching on the dense descriptor graph obtained by feature extraction to obtain N pairs of dense matching point pairs. The depth estimation network and the pose estimation network obtain the depth map and relative pose of the input image respectively; in order to provide supervision signals for the depth and pose estimation networks, the descriptor map and the dense matching point pairs are They are used to construct feature consistency loss and epipolar geometry consistency loss respectively.

[0017] The present invention builds a feature learning network Feat Net to learn dense feature descriptors, and avoids the chessboard artifacts generated by the Padding technology used in the convolutional layer; the descriptors obtained through learning have a certain degree of stability between two frames of images, which solves the problem of failure of depth estimation caused by inconsistent luminosity and similar texture of endoscopic images, and on the other hand, it allows the use of two frames of images corresponding to a larger camera translation to perform depth and pose estimation. The extracted dense feature descriptors can provide a supervisory signal for depth estimation, and by introducing geometric constraints between video frames, feature matching and depth and pose estimation are jointly learned to improve the prediction accuracy of the proposed method.

[0018] Preferably, in the step (1), an end-to-end feature learning network is used to obtain the corresponding descriptor map from the source-target input endoscopic image; the feature learning network Feat Net is composed of two weight-sharing branches, and each branch of the network is designed using a residual block, wherein the encoder is composed of a convolutional layer and 9 ResNet-v2 blocks, and the input image 3×H×W is downsampled twice through an average pooling layer to finally extract a feature map C×H / 4×W / 4, wherein H and W are the height and width of the input image, and C is the number of channels for extracting the feature map; the decoder network is composed of three convolutional layers, and after two bilinear upsamplings, a dense feature descriptor map of L×H×W is finally obtained, wherein L is the dimension of the dense feature descriptor, and L=32.

[0019] Preferably, in the step (1), after each convolution operation, a bilinear interpolation method is used to keep the image size unchanged before and after the convolution; at the same time, in the decoder, a convolution layer and a bilinear upsampling layer are used to restore the image size; and the descriptor is L2 normalized along the channel.

[0020] Preferably, in step (1), an endoscopic image feature matching method is used to obtain sparse matching point pairs between adjacent frames. These sparse matching point pairs are used to construct the relative response loss supervision network training. For a pair of input source images I s and the target image I t , the feature descriptor learning network will generate its corresponding dense feature descriptor graph F s and F t , the input image and dense descriptor map sizes are 3×H×W and L×H×W respectively, and the relative response loss is mathematically defined as:

[0021]

[0022] Where R t is the response graph, indicating F s (x s ) and F t The similarity of each position on the y-axis is calculated, and the square of the Euclidean distance between the two descriptors is calculated as the similarity evaluation index. Since each descriptor has been normalized to a unit vector, the calculation of the square of the Euclidean distance between the descriptors is simplified to dist 2 =(2-2cos(f 1 ,f 2 )), where f 1 ∈F s ,f 2 ∈F t In order to avoid the distance range being too large, Normalize it to [0,1], then perform a softmax operation on the response map. At the same time, the scaling factor σ expands R t The value range of makes the network pay more attention to the areas with greater similarity during the learning process, reduce other large responses except the responses of the true position, and ignore the descriptors with large differences to a certain extent.

[0023] Preferably, in step (1), the response graph R t The maximum value position is taken as the key point x detected by the network det , for k pairs of sparse matching points

[0024]

[0025]

[0026] L RR =λ i L rr (4)

[0027] in, Yes By taking the average value and implicitly weighting the distance between the key points detected by the network and the true position into the relative response loss, the weight of the incorrectly matched point pairs will be very small, and the network will optimize most of the correctly matched point pairs, thereby effectively avoiding the influence of the incorrectly matched point pairs and accelerating the network convergence.

[0028] Preferably, in step (2), for the input source-target image pair (I s ,I t ), and predict the depth map (D s ,D t), where each branch consists of an encoder-decoder, the encoder uses ResNet34, and the activation function of the prediction layer is expressed as:

[0029]

[0030] Where x is the output value of the sigmoid activation function of the prediction layer, a = (1 / d min -1 / d max ),b=1 / d max , d min and d max denote the minimum and maximum depths of the endoscopic image, respectively, and y is the predicted depth of the current frame;

[0031] The relative pose estimation network PoseNet without mask prediction branch is used to estimate the input image pair (I s ,I t )’s 6DoF relative pose PoseNet consists of seven convolutional layers, and the output value of the network is multiplied by the proportional coefficient ρ to limit the output value range.

[0032] Preferably, in step (2), for the target view I t Any pixel coordinate ρ t , assuming it is visible in the source view, then the corresponding pixel coordinates in the source view are s Determined by formula (6)

[0033]

[0034] Among them, ~ indicates that the homogeneous coordinates are equal, K s and K t is the intrinsic parameter matrix of the source view and the target view corresponding to the camera, D t (p t ) is the coordinate point p in the target view t The depth of the place.

[0035] Preferably, in step (2), the coordinate correspondence between the target view and the source view is obtained by formula (6), and the source descriptor image F obtained by the feature descriptor learning network is s and the target descriptor graph F t , using the above coordinate correspondence and bilinear sampling method, from the source descriptor graph F s Generate synthetic object descriptor graph Then, the original object descriptor graph F is minimized by formula (7): t and synthetic object descriptor graph The error between

[0036]

[0037] Where M is a binary mask, indicating whether the coordinate point in the target view falls into the valid area in the source view after the warp transformation, which can be determined by the warp transformation;

[0038] Then, the depth map is smoothed using an edge-aware smoothness function.

[0039]

[0040] in and Find the derivatives in the x and y directions respectively, is the average normalized inverse depth used to avoid depth shrinkage;

[0041] In the above distortion transformation process, the target depth map is transformed to synthesize the source depth map Coordinate correspondence obtained using distortion transformation For the original source depth map D s Sampling is performed to obtain the sampling source depth map D′ s , calculate the synthetic source depth map and the sampled source depth map D′ s The consistency between

[0042]

[0043] M diff =1-D diff (10)

[0044]

[0045] For the feature matching point set between the source and target input images Should satisfy the epipolar geometry constraint p′ T Fp=0,where F=K -T [t] × R -1 is the fundamental matrix, [·] × Represents the cross multiplication operation, therefore, the relative pose T estimated by the network t→s =[R t→s |t t→s ]∈SE(3), combined with the feature matching point set The antipodal loss function is established as follows:

[0046]

[0047] Where dist(·,·) means the distance from a point to a line, and Fp is the epipolar line;

[0048] For the feature matching point set of the target view and the source view Establish the error function between S and the reprojection coordinate point, where the reprojection coordinate is obtained by distortion transformation, and the reprojection loss is as follows

[0049]

[0050] Preferably, in step (2), three views are used each time, and the loss is calculated between each two of them. First, let I 2 Target view I t , let I 1 and I 3 Source view I s , calculate the above loss, and then let I 1 Target view I t , let I 3 Source view I s , the final loss function L is calculated as follows:

[0051] L = k 1 L RR +k 2 L F +k 3 L s +k 4 L c +k 5 L epi +k 6 L reproject (14)

[0052] The weights k of different losses 1 =1,k 2 =1,k 3 =0.1,k 4 =2,k 5 =0.001,k 6 =0.001.

[0053] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes the steps of the above-mentioned embodiment method, and the storage medium can be: ROM / RAM, disk, CD, memory card, etc. Therefore, corresponding to the method of the present invention, the present invention also includes a device for unsupervised depth and motion estimation of a monocular endoscope, which is usually represented in the form of functional modules corresponding to the steps of the method. The device includes:

[0054] A dense descriptor acquisition module is configured to perform feature matching on the input source-target image pair to obtain sparse matching point pairs; a dense descriptor map corresponding to the input image is obtained through a feature learning network; a relative response loss is established using the sparse matching point pairs and the dense descriptor map; a depth and pose estimation module is configured to perform dense feature matching on the dense descriptor map obtained by feature extraction to obtain N pairs of dense matching point pairs The depth estimation network and the pose estimation network obtain the depth map and relative pose of the input image respectively; in order to provide supervision signals for the depth and pose estimation networks, the descriptor map and the dense matching point pairs are They are used to construct feature consistency loss and epipolar geometry consistency loss respectively.

[0055] The present invention is described in more detail below.

[0056] For the dense descriptor acquisition module, an end-to-end feature learning network is used to obtain the corresponding descriptor map from the source-target input endoscopic image. The feature learning network Feat Net consists of two weight-sharing branches, and each branch of the network is designed using a residual block. The encoder consists of a convolutional layer and 9 ResNet-v2 blocks. It downsamples the input image 3×H×W twice through an average pooling layer, and finally extracts a feature map C×H / 4×W / 4. Where H and W are the height and width of the input image, and C is the number of channels for extracting the feature map. The decoder network consists of three convolutional layers, and after two bilinear upsamplings, a dense feature descriptor map of L×H×W is finally obtained. Where L is the dimension of the dense feature descriptor, and L=32 is taken in this article.

[0057] It is worth noting that the padding technique used by the convolution layer to fill the edge of the image with zeros will cause checkerboard artifacts in the edge area of ​​the dense feature descriptor map. In order to avoid checkerboard artifacts in the dense feature descriptor map, the convolution layer does not use zeros to fill the edge of the image. Instead, after each convolution operation, a bilinear interpolation method is used to keep the image size unchanged before and after the convolution. At the same time, in the decoder, instead of using deconvolution layers, convolution layers and bilinear upsampling layers are used to restore the image size. Finally, in order to avoid the influence of image brightness differences and improve the versatility of the descriptor, the descriptor is L2 normalized along the channel.

[0058] Directly use the endoscope image feature matching method to obtain sparse matching point pairs between adjacent frames Furthermore, these sparse matching point pairs are used to construct the relative response loss to supervise the network training. s and the target image I t , the feature descriptor learning network will generate its corresponding dense feature descriptor graph F s and F tThe input image and dense descriptor map sizes are 3×H×W and L×H×W respectively. The relative response loss is mathematically defined as:

[0059]

[0060] Where R t is the response graph, indicating F s (x s ) and F t In this paper, the square of the Euclidean distance between two descriptors is calculated as the similarity evaluation index. Since each descriptor has been normalized to a unit vector, the calculation of the square of the Euclidean distance between descriptors can be simplified to dist 2 =(2-2cos(f 1 ,f 2 )), where f 1 ∈F s ,f 2 ∈F t In fact, in order to avoid the distance range being too large, the It is normalized to [0,1]. Then the response map is subjected to a softmax operation, and the scaling factor σ expands R t The value range of makes the network pay more attention to the areas with greater similarity during the learning process, reduce other large responses except the responses of the true position, and ignore the descriptors with large differences to a certain extent.

[0061] Due to the particularity of endoscopic images, the sparse matching point pairs obtained by traditional feature matching methods may be used as the true correspondence, which will affect the network's accurate positioning of key points. Especially when the brightness difference between the two frames is large, there may be wrong matching point pairs. In order to avoid the influence of wrong matching point pairs, the relative response loss is weighted. Take the response map R t The maximum value position is taken as the key point x detected by the network det For k pairs of sparse matching points

[0062]

[0063]

[0064] L RR =λ i L rr (4)

[0065] in, Yes Take the average value. The distance between the key points detected by the network and the true position is implicitly weighted into the relative response loss. The weight of the wrong matching point pairs will be very small, and the network will optimize most of the correct matching point pairs, thereby effectively avoiding the influence of the wrong matching point pairs and accelerating the network convergence.

[0066] For an input source-target image pair (I s ,I t ), and predict the depth map (D s ,D t ). Each branch consists of an encoder-decoder, and the encoder uses ResNet34. In order to limit the range of predicted depth values, the activation function of the prediction layer is expressed as:

[0067]

[0068] Where x is the output value of the sigmoid activation function of the prediction layer, a = (1 / d min -1 / d max ),b=1 / d max .d min and d max denote the minimum and maximum depths of the endoscopic image, respectively, and y is the predicted depth of the current frame.

[0069] At the same time, the relative pose estimation network PoseNet without mask prediction branch is used to estimate the input image pair (I s ,I t )’s 6DoF relative pose T t→s =[R t→s |t t→s ]∈SE(3). PoseNet consists of seven convolutional layers, and the output value of the network is multiplied by the proportional coefficient ρ to limit the output value range.

[0070] For the target view I t Any pixel coordinate p t , assuming it is visible in the source view, then the corresponding pixel coordinate p in the source view s It can be determined by the following equation, which determines the warp transformation

[0071]

[0072] Among them, ~ indicates that the homogeneous coordinates are equal, K s and K t is the intrinsic parameter matrix of the source view and the target view corresponding to the camera, D t (p t ) is the coordinate point p in the target view t The depth of the place.

[0073] Through the above distortion transformation equation, the coordinate correspondence between the target view and the source view can be obtained. For the source descriptor graph F obtained by the feature descriptor learning network s and the target descriptor graph F t , using the above coordinate correspondence and bilinear sampling method, from the source descriptor graph F s Generate synthetic object descriptor graph Then, the original object descriptor graph F is minimized by the following equation t and synthetic object descriptor graph The error between

[0074]

[0075] Where M is a binary mask, indicating whether the coordinate point in the target view falls into the valid area in the source view after the warp transformation, which can be determined by the warp transformation.

[0076] Then, an edge-aware smoothness function is used to impose a smoothness constraint on the depth map.

[0077]

[0078] in and Find the derivatives in the x and y directions respectively, is the average normalized inverse depth used to avoid depth shrinkage.

[0079] Since the depth estimation network can only estimate the depth of the current frame each time it makes a depth prediction, in order to make the estimated depth of each input image frame consistent. In the above distortion transformation process, we transform the target depth map to synthesize the source depth map Then, the coordinate correspondence obtained by the distortion transformation For the original source depth map D s Sampling is performed to obtain the sampling source depth map D′ s Finally, the synthetic source depth map is calculated and the sampled source depth map D′ s The consistency between them.

[0080]

[0081] M diff =1-D diff (10)

[0082]

[0083] In addition, due to the complexity of unsupervised depth estimation, in order to provide more stable and effective supervision signals for depth and pose estimation, epipolar geometric consistency constraints are imposed on the source and target input images, and epipolar loss and reprojection loss are used to supervise the pose and depth estimation of the network.

[0084] For the feature matching point set between the source and target input images Should satisfy the epipolar geometry constraint p′ T Fp=0. Where F=K -T [t] × R -1 is the fundamental matrix, [·] × represents the cross product operation. Therefore, the relative pose T estimated by the network can be t→s =[R t→s |t t→s ]∈SE(3), combined with the feature matching point set The antipodal loss function is established as follows:

[0085]

[0086] Where dist(·,·) represents the distance from a point to a line, and Fp is the epipolar line.

[0087] For the feature matching point set of the target view and the source view Establish the error function between S and the reprojection coordinate point. The reprojection coordinate can be obtained by distortion transformation. The reprojection loss is as follows

[0088]

[0089] In fact, three views are used each time, and the loss is calculated between each of them. First, let I 2 Target view I t , let I 1 and I 3 Source view I s , and calculate the above loss. Then let I 1 Target view I t , let I 3 Source view I s Through the loss calculation between multiple views, the stability of depth prediction and the propagation of depth consistency between video sequences are fully guaranteed.

[0090] In summary, the final loss function L is calculated as follows:

[0091] L = k 1 L RR +k 2 L F +k 3 Ls +k 4 L c +k 5 L epi +k 6 L reproject (14)

[0092] The weights k of different losses 1 =1,k 2 =1,k 3 =0.1,k 4 =2,k 5 =0.001,k 6 =0.001.

[0093] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the protection scope of the technical solution of the present invention.

Claims

1. A method for unsupervised depth and motion estimation of a monocular endoscope, characterized by: The method comprises the following steps: (1) Obtaining a dense descriptor map: Perform feature matching on the input source-target image pair to obtain sparse matching point pairs; obtain the dense descriptor map corresponding to the input image through the feature learning network; and establish a relative response loss using the sparse matching point pairs and the dense descriptor map; (2) Depth and pose estimation: Perform dense feature matching on the dense descriptor graph obtained by feature extraction to obtain N pairs of dense matching point pairs. The depth estimation network and the pose estimation network obtain the depth map and relative pose of the input image respectively; in order to provide supervision signals for the depth and pose estimation networks, the descriptor map and the dense matching point pairs are They are used to construct feature consistency loss and epipolar geometry consistency loss respectively; In the step (1), an end-to-end feature learning network is used to obtain the corresponding descriptor map from the source-target input endoscopic image; the feature learning network Feat Net is composed of two weight-sharing branches, and each branch of the network is designed using a residual block, wherein the encoder is composed of a convolutional layer and 9 ResNet-v2 blocks, and the input image 3×H×W is downsampled twice through an average pooling layer, and finally a feature map C×H / 4×W / 4 is extracted, wherein H and W are the height and width of the input image, and C is the number of channels of the extracted feature map; the decoder network is composed of three convolutional layers, and after two bilinear upsamplings, a dense feature descriptor map of L×H×W is finally obtained, wherein L is the dimension of the dense feature descriptor, and L=32; In the step (1), an endoscopic image feature matching method is used to obtain sparse matching point pairs between adjacent frames. These sparse matching point pairs are used to construct the relative response loss supervision network training. For a pair of input source images I s and the target image It, the feature descriptor learning network will generate its corresponding dense feature descriptor graph F s and F t , the input image and dense descriptor map sizes are 3×H×W and L×H×W respectively, and the relative response loss is mathematically defined as: Where R t is the response graph, indicating F s (x s ) and F t The similarity of each position on the y-axis is calculated, and the square of the Euclidean distance between the two descriptors is calculated as the similarity evaluation index. Since each descriptor has been normalized to a unit vector, the calculation of the square of the Euclidean distance between the descriptors is simplified to dist 2 =(2-2cos(f1,f2)), where f1∈F s , f2∈F t In order to avoid the distance range being too large, Normalize it to [0, 1], then perform a softmax operation on the response map. At the same time, the scaling factor σ expands R t The value range of makes the network pay more attention to the areas with greater similarity during the learning process, reduce other large responses except the response of the true position, and ignore the descriptors with large differences to a certain extent; In the step (2), for the input source-target image pair (I s , I t ), and predict the depth map (D s , D t ), where each branch consists of an encoder-decoder, the encoder uses ResNet34, and the activation function of the prediction layer is expressed as: Where z is the output value of the sigmoid activation function of the prediction layer, a=(1 / d min -1 / d max ), b=1 / d max , d min and d max denote the minimum and maximum depths of the endoscopic image, respectively, and y is the predicted depth of the current frame; The relative pose estimation network PoseNet without mask prediction branch is used to estimate the input image pair (I s , I t )’s 6DoF relative pose T t→s =[R t→s |t t→s ]∈SE(3), PoseNet consists of seven convolutional layers, and the output value of the network is multiplied by the proportional coefficient ρ to limit the output value range; In step (2), for the target view I t Any pixel coordinate p t , assuming it is visible in the source view, then the corresponding pixel coordinate p in the source view s Determined by formula (6) Among them, ~ indicates that the homogeneous coordinates are equal, K s and K t is the intrinsic parameter matrix of the source view and the target view corresponding to the camera, D t (p t ) is the coordinate point p in the target view t Depth of In step (2), the coordinate correspondence between the target view and the source view is obtained by formula (6), and the source descriptor map F obtained by the feature descriptor learning network is s and the target descriptor graph F t , using the above coordinate correspondence and bilinear sampling method, from the source descriptor graph F s Generate synthetic object descriptor graph Then, the original object descriptor graph F is minimized by formula (7): t and synthetic object descriptor graph The error between Where M is a binary mask, indicating whether the coordinate point in the target view falls into the valid area in the source view after the warp transformation, which can be determined by the warp transformation; Then, the depth map is smoothed using an edge-aware smoothness function. in and Find the derivatives in the x and y directions respectively, is the average normalized inverse depth used to avoid depth shrinkage; In the above distortion transformation process, the target depth map is transformed to synthesize the source depth map Coordinate correspondence obtained using distortion transformation For the original source depth map D s Sampling is performed to obtain the sampling source depth map D′ s , calculate the synthetic source depth map and the sampled source depth map D′ s The consistency between M diff =1-D diff (10) For the feature matching point set between the source and target input images Should satisfy the epipolar geometry constraint p′ T Fp=0,where F=K -T [t] × R -1 is the fundamental matrix, [·] × Represents the cross multiplication operation, therefore, the relative pose T estimated by the network t→s =[R t→s |t t→s ]∈SE(3), combined with the feature matching point set The epipolar loss function is established as follows: Where dist(·,·) represents the distance from a point to a line, and Fp is the epipolar line; For the feature matching point set of the target view and the source view Establish the error function between S and the reprojection coordinate point, where the reprojection coordinate is obtained by distortion transformation, and the reprojection loss is as follows 2. The method for unsupervised depth and motion estimation of a monocular endoscope according to claim 1, characterized in that: In the step (1), after each convolution operation, a bilinear interpolation method is used to keep the image size unchanged before and after the convolution; at the same time, in the decoder, a convolution layer and a bilinear upsampling layer are used to restore the image size; and the descriptor is L2 normalized along the channel.

3. The method for unsupervised depth and motion estimation of a monocular endoscope according to claim 2, characterized in that: In step (1), take the response graph R t The maximum value position is taken as the key point x detected by the network det , for k pairs of sparse matching points L RR =λ i L rr (4) in, Yes By taking the average value and implicitly weighting the distance between the key points detected by the network and the true position into the relative response loss, the weight of the incorrectly matched point pairs will be very small, and the network will optimize most of the correctly matched point pairs, thereby effectively avoiding the influence of the incorrectly matched point pairs and accelerating the network convergence.

4. The method for unsupervised depth and motion estimation of a monocular endoscope according to claim 3, characterized in that: In step (2), three views are used each time, and the loss is calculated between each two of them. First, let I2 be the target view I t , let I1 and I3 be the source views I s , calculate the above loss, and then let I1 be the target view I t , let I3 be the source view I s , the final loss function is calculated as follows: L=k1L RR +k2L F +k3L s +k4L c +k5L epi +k6L reproject (14) The weights of different losses are k1=1, k2=1, k3=0.1, k4=2, k5=0.001, k6=0.

001.

5. A device for unsupervised depth and motion estimation of a monocular endoscope, characterized in that: The method according to claim 1 is implemented, and includes: A dense descriptor acquisition module is configured to perform feature matching on the input source-target image pair to obtain sparse matching point pairs; a dense descriptor map corresponding to the input image is obtained through a feature learning network; a relative response loss is established using the sparse matching point pairs and the dense descriptor map; a depth and pose estimation module is configured to perform dense feature matching on the dense descriptor map obtained by feature extraction to obtain N pairs of dense matching point pairs The depth estimation network and the pose estimation network obtain the depth map and relative pose of the input image respectively; in order to provide supervision signals for the depth and pose estimation networks, the descriptor map and the dense matching point pairs are They are used to construct feature consistency loss and epipolar geometry consistency loss respectively.

Images