Unsupervised monocular endoscope depth estimation method based on large model

Abstract

The invention discloses an unsupervised monocular endoscope depth estimation method based on a large model. A network structure comprises three network modules, namely a distortion transformation module, an intrinsic decomposition module and a synthesis reconstruction module, wherein the distortion transformation module comprises a depth estimation network and a posture estimation network. Wherein the depth estimation network is combined with a large model fine tuning technology, and four loss functions of decomposition synthesis loss, reflection loss, mapping synthesis loss and edge perception depth smooth loss are adopted for training. According to the fine tuning strategy, the adaptability of the model to different scales is enhanced by utilizing a low-rank adaptation based on a random vector, and the capability of capturing local features of the model is improved by utilizing a residual block based on depth separable convolution; the loss function utilizes decomposition synthesis loss to constrain that a synthesized image is the same as an original image, utilizes reflection loss to enhance the consistency of reflectivity, utilizes mapping synthesis loss to improve the similarity of a reconstructed image and an original target image, and utilizes edge perception depth smoothness loss to enhance the smoothness of a non-edge region of the image. According to the scheme, low-rank adaptation based on random vectors and a large model fine tuning strategy based on the residual block of the depth separable convolution are combined, the depth estimation performance is greatly improved, and the safety, precision and efficiency of endoscope minimally invasive surgery are improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and medicine, and more specifically, to an unsupervised monocular endoscope depth estimation method based on a large model. Background Art

[0002] Endoscopic minimally invasive surgery is a cutting-edge field in modern surgery. By inserting an endoscope into a small incision or natural cavity, the doctor uses the real-time image provided by the endoscope to perform precise surgical operations. It has the advantages of less trauma, high surgical precision, and a wide range of applications, and has become an important part of today's surgical operations. In recent years, endoscopic minimally invasive surgery technology has continued to advance and has been combined with robot-assisted surgical systems to effectively improve the accuracy and safety of surgery. Depth estimation is an important part of the augmented reality navigation system in minimally invasive endoscopic surgery, in which accurate 3D depth information is crucial to improving surgical accuracy and safety. When dealing with complex anatomical areas or performing surgery near important structures, the lack of spatial information may lead to misjudgment during surgical operations, affecting surgical accuracy and even causing serious complications.

[0003] Given the difficulty in obtaining ground truth from endoscopic images and the fact that monocular endoscopes are more flexible and practical than stereo endoscopes, unsupervised monocular depth estimation (UMDE) algorithms have attracted more extensive research interests. These algorithms usually transform the depth estimation task into an image synthesis problem between adjacent viewpoints and use the image synthesis error to guide network training. Despite their potential, traditional depth estimation algorithms face great challenges when applied to endoscopic images due to factors such as illumination variations, tissue deformation, and sparse textures.

[0004] In the past year, the emergence of new technologies such as the Foundation Model (FM) has brought new opportunities. The Foundation Model has shown strong representation and generalization capabilities by performing previews on massive training data, providing a more effective solution to the problem of monocular endoscope depth estimation. This paper proposes a monocular unsupervised depth estimation method combined with image intrinsic decomposition, and combines the foundation model to design a fine-tuning strategy for specific endoscope scenes, which improves performance without increasing parameters, effectively reduces the impact of drilling changes in endoscope images on depth estimation accuracy, and further improves the overall performance, safety and success rate of endoscope operations. Summary of the invention

[0005] In order to solve the limitations and shortcomings of the existing technology, an unsupervised monocular endoscope depth estimation method based on a large model is proposed, which includes the following steps:

[0006] Step 1: Build a distortion transformation module including a depth estimation network and a posture estimation network to estimate the depth information and camera posture information of the patient's endoscopic image respectively. Build the posture estimation network Φ T , the source image I s and the target image I t Input pose estimation network Φ T , get the estimated camera relative posture T t→s .

[0007] Step 2: Build a depth estimation network Φ D , the existing depth estimation model (Depth Anything Model) is fine-tuned using a model fine-tuning method based on low-rank decomposition. The fine-tuning strategy consists of two parts: low-rank adaptation based on random vectors (RV-LoRA) and residual block based on depthwise separable convolution (Res-DSC). t Input depth estimation network Φ D , get the estimated depth map D t .

[0008] Step 3: Build the intrinsic decomposition module and the synthetic reconstruction module. s and the target image I t Input eigendecomposition network Φ I , decompose to obtain diffuse reflection S, reflectivity A and specular reflection K, and reconstruct and A s , S s , K s Through the rigid flow extracted from the distortion transformation module, we can obtain A s→t , S s→t , K s→t . Build a lighting adjustment networkΦ L , adjust the diffuse and specular reflection maps and reconstruct

[0009] Step 4: Using the source image I s , target image I t and reconstruct the image Calculate the decomposition and synthesis loss L ds , using the reflectivity map A t and the reconstructed A s→t Calculate the reflection loss L a , using the target image I t and reconstruct the synthetic image Calculate the mapping synthesis loss L ms , using the depth map D t and the target image I t Compute edge-aware depth smoothing loss Les The network training is weighted based on the above four loss functions and iterated until the model converges.

[0010] Step 5: Input the patient's endoscopic image into the network model to output the corresponding depth estimation map.

[0011] Furthermore, the random vector based low rank adaptation (RV-LoRA) fine-tuning strategy in step 2 is based on Low-Rank Adaptation (LoRA), and scales the low rank matrices B and A by adding randomly initialized vectors b and a to enhance the model's ability to adapt to different scales. Kaiming uniform initialization is used for a, b and A, and the low rank matrix B is initialized to 0 to ensure that the weight matrix is ​​not affected during the first forward propagation. This method can be expressed as:

[0012] h=W0x+ΔWx=W0x+Λ b BΛ a Ax (1) in, Rank r<<min(m,n). In this method, the low-rank matrices B and A are trainable, while the scaling vectors b and a are frozen, random, and shared between layers, formally represented as a diagonal matrix Λ b and Λ a , randomly scale the rows of the low-rank matrices B and A.

[0013] Furthermore, the fine-tuning strategy of the residual block (Res-DSC) based on depthwise separable convolution in step 2 is to integrate the convolution module into the Transformer module of the Depth Anything Model. This module consists of three convolution layers. The first layer is a 1×1 convolution to reduce the number of channels; the second layer is a 3×3 depthwise separable convolution; and the last layer is a 1×1 convolution to restore the number of channels. Considering the number of model parameters and the accuracy of depth estimation, Res-DSC is only added after the 3rd, 6th, 9th and 12th Transformer blocks.

[0014] Furthermore, in step 3, the intrinsic decomposition network Φ I , input source image I s and the target image I t Decomposition and Reconstruction and Including: using a jump connection U-shaped structure to decompose diffuse reflection S, reflectivity A and specular reflection K. Then according to the intrinsic decomposition theory, the formula I(u,λ)=L(λ)(S(u)A(u,λ)+K(u)) is calculated, where u is the pixel position, λ is the wavelength, and L is the illumination map, and finally the reconstructed synthetic image is obtained. and

[0015] Furthermore, in step 3, the network Φ is adjusted by illumination L , adjust the diffuse and specular reflection maps and reconstruct Includes: Input original target image I t and the synthesized target image I s→t , a four-layer convolutional network is used, and the last layer uses the activation function tanh to obtain the lighting adjustment map in the range of [-1,1]:

[0016] L Adjust =Φ L (|I t -I s→t |) (2)

[0017] Through the rigid flow extracted from the distortion transformation module, we can obtain A s→t , S s→t , K s→t ,Adjustment With A s→t Synthetic reconstruction

[0018] Calculate the source image I s , target image I t and reconstruct the image The decomposition and synthesis loss between ds :

[0019]

[0020] Among them, α is the weight factor, SSIM is the structural similarity loss function, and ‖·‖1 is the L1 loss function.

[0021] Calculate reflectivity map A t and the reconstructed A s→t The reflection loss between a :

[0022] L a (A t ,A s→t )=‖A t -A s→t ‖1 (4)

[0023] Calculate the target image I t and reconstruct the synthetic image The mapping synthesis loss L between ms :

[0024]

[0025] Calculate the depth map Dt and the target image I t The edge-aware depth smoothing loss L between es :

[0026]

[0027] in, and is the first-order derivative of the depth map along the x and y directions, and are the first-order derivatives of the target image along the x and y directions.

[0028] The total loss function is:

[0029]

[0030] Among them, λ ds , a , ms and λ as is the weight factor, M a is an automatic mask used to exclude lost areas during the distortion transformation, and its expression is M a =[I s→t >0], [·] is the Iverson bracket, Represents element-wise product.

[0031] Inputting the patient's endoscopic image into the network model will output the corresponding depth estimation map.

[0032] The beneficial effects of the present invention are as follows:

[0033] The present invention applies a new large model fine-tuning strategy based on the depth estimation model, which enhances the adaptability of the model to different scales and improves the model's ability to capture local features, greatly improving the depth estimation performance, thereby assisting in guiding endoscopic minimally invasive surgery and improving the safety, accuracy and efficiency of endoscopic minimally invasive surgery.

[0034] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the following briefly introduces the drawings required for use in the specific implementation or the prior art description. Obviously, the drawings in the following description are Figure 1 This is an implementation mode of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 This is a schematic flow chart of a method for unsupervised monocular endoscope depth estimation based on a large model according to an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of the structure of a depth estimation network model combined with a large model technology of the present invention;

[0038] Figure 3 RV-LoRA and Res-DSC modules of the present invention are exemplified;

[0039] Figure 4 This is a comparative display diagram of the visual results of the monocular endoscope depth estimation of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the drawings. The described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments.

[0041] Figure 1 FIG. 1 is a flow chart of an unsupervised monocular endoscope depth estimation method based on a large model according to an embodiment of the present invention. Figure 1 As shown, the method includes the following 6 steps:

[0042] Step 1: Build a distortion transformation module including a depth estimation network and a posture estimation network to estimate the depth information and camera posture information of the patient's endoscopic image respectively. The depth estimation network model structure is as follows Figure 2 As shown. Build the posture estimation network Φ T , the source image I s and the target image I t Input pose estimation network Φ T , get the estimated camera relative posture T t→s .

[0043] Step 2: Build a depth estimation network Φ D , the existing depth estimation model (Depth Anything Model) is fine-tuned using a model fine-tuning method based on low-rank decomposition. The fine-tuning strategy consists of two parts: low-rank adaptation based on random vectors (RV-LoRA) and residual block based on depthwise separable convolution (Res-DSC). t Input depth estimation network Φ D , get the estimated depth map D t .

[0044] The random vector based low rank adaptation (RV-LoRA) fine-tuning strategy is as follows Figure 3As shown in the figure, based on Low-Rank Adaptation (LoRA), the low-rank matrices B and A are scaled by adding randomly initialized vectors b and a to enhance the model's ability to adapt to different scales. Kaiming uniform initialization is used for a, b and A, and the low-rank matrix B is initialized to 0 to ensure that the weight matrix is ​​not affected during the first forward propagation. This method can be expressed as:

[0045] h=W0x+ΔWx=W0x+Λ b BΛ a Ax (1)

[0046] in, Rank r<<min(m,n). In this method, the low-rank matrices B and A are trainable, while the scaling vectors b and a are frozen, random, and shared between layers, formally represented as a diagonal matrix Λ b and Λ a , randomly scale the rows of the low-rank matrices B and A.

[0047] The fine-tuning strategy of the residual block based on depthwise separable convolution (Res-DSC) is to integrate the convolution module into the Transformer module of the DepthAnything Model. Figure 3 As shown in the figure, the module consists of three convolutional layers. The first layer is a 1×1 convolution to reduce the number of channels; the second layer is a 3×3 depth-separable convolution; and the last layer is a 1×1 convolution to restore the number of channels. Considering the number of model parameters and the accuracy of depth estimation, Res-DSC is only added after the 3rd, 6th, 9th, and 12th Transformer blocks.

[0048] Step 3: Build the intrinsic decomposition module. Using the intrinsic image decomposition theory, the source image I s and the target image I t Input eigendecomposition network Φ I ,reconstruction and The implementation method is: using a U-shaped structure with jump connections, decomposing diffuse reflection S, reflectivity A and specular reflection K, as shown in formula (2)

[0049] {A s ,S s ,K s}=Φ I (I s ); t ,S t ,K t}=Φ I (I t ) (2)

[0050] Then, according to the eigendecomposition theory, the reconstructed synthetic image is obtained by calculating the formula I(u,λ)=L(λ)(S(u)A(u,λ)+K(u)), where u is the pixel position, λ is the wavelength, and L is the illumination map. and

[0051] Build a synthetic reconstruction module. Build a lighting adjustment networkΦ L , adjust the diffuse and specular reflection maps and reconstruct The specific implementation method is: input the original target image I t and the synthesized target image I s→t , a four-layer convolutional network is used, and the last layer uses the activation function tanh to obtain the lighting adjustment map in the range of [-1,1]:

[0052] L Adjust =Φ L (|I t -I s→t |) (3)

[0053] Through the rigid flow extracted from the distortion transformation module, we can obtain A s→t , S s→t , K s→t ,Adjustment With A s→t Synthetic reconstruction

[0054] Step 4: Using the source image I s , target image I t and reconstruct the image Calculate the decomposition and synthesis loss L ds , as shown in formula (4). Using the reflectivity map A t and the reconstructed A s→t Calculate the reflection loss L a , as shown in formula (5). Using the target image I t and reconstruct the synthetic image Calculate the mapping synthesis loss L ms , as shown in formula (6). Using the depth map D t and the target image I t Compute edge-aware depth smoothing loss L es , as shown in formula (7). Based on the weighted four loss functions above, network training is performed and iterated until the model converges.

[0055]

[0056] Among them, α is the weight factor, SSIM is the structural similarity loss function, and ‖·‖1 is the L1 loss function.

[0057] L a (A t ,A s→t )=‖A t -A s→t ‖1 (5)

[0058]

[0059] in, and is the first-order derivative of the depth map along the x and y directions, and are the first-order derivatives of the target image along the x and y directions.

[0060] The total loss function is:

[0061]

[0062] Among them, λ ds , a , ms and λ es is the weight factor, M a is an automatic mask used to exclude lost areas during the distortion transformation, and its expression is M a =[I s→t >0], [·] is the Iverson bracket, Represents element-wise product.

[0063] Step 5: Input the patient's endoscopic image into the network model to output the corresponding depth estimation map. The example monocular endoscope depth estimation visual result comparison is shown below Figure 4 shown.

[0064] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. An unsupervised monocular endoscope depth estimation method based on a large model, characterized in that: The following steps are involved: Step 1: Build a distortion transformation module including a depth estimation network and a posture estimation network to estimate the depth information and camera posture information of the patient's endoscopic image respectively. Build the posture estimation network Φ T , the source image I s and the target image I t Input pose estimation network Φ T , get the estimated camera relative posture T t→s . Step 2: Build a depth estimation network Φ D , the existing depth estimation model (Depth Anything Model) is fine-tuned using a model fine-tuning method based on low-rank decomposition. The fine-tuning strategy consists of two parts: low-rank adaptation based on random vectors (RV-LoRA) and residual block based on depthwise separable convolution (Res-DSC). t Input depth estimation network Φ D , get the estimated depth map D t . Step 3: Build the intrinsic decomposition module and the synthetic reconstruction module. s and the target image I t Input eigendecomposition network Φ I , decompose to obtain diffuse reflection S, reflectivity A and specular reflection K, and reconstruct and A s , S s , K s Through the rigid flow extracted from the distortion transformation module, we can obtain A s→t , S s→t , K s→t . Build a lighting adjustment networkΦ L , adjust the diffuse and specular reflection maps and reconstruct Step 4: Using the source image I s , target image I t and reconstruct the image Calculate the decomposition and synthesis loss L ds , using reflectivity map A t and the reconstructed A s→t Calculate the reflection loss L a , using the target image I t and reconstruct the synthetic image Calculate the mapping synthesis loss L ms , using the depth map D t and the target image I t Compute edge-aware depth smoothing loss L es The network training is weighted based on the above four loss functions and iterated until the model converges. Step 5: Input the patient's endoscopic image into the network model to output the corresponding depth estimation map.

2. The method according to claim 1, characterized in that Random Vector Based Low Rank Adaptation (RV-LoRA) fine-tuning strategy, including: Based on Low-Rank Adaptation (LoRA), the low-rank matrices B and A are scaled by adding randomly initialized vectors b and a to enhance the model's ability to adapt to different scales. Kaiming uniform initialization is used for a, b, and A, and the low-rank matrix B is initialized to 0 to ensure that the weight matrix is ​​not affected during the first forward propagation. This method can be expressed as: h=W0x+ΔWx=W0x+Λ b BL a Axe (1) in, Rank r<<min(m,n). In this method, the low-rank matrices B and A are trainable, while the scaling vectors b and a are frozen, random, and shared between layers, formally represented as a diagonal matrix Λ b and Λ a , randomly scale the rows of the low-rank matrices B and A.

3. The method according to claim 1, characterized in that: The Res-DSC fine-tuning strategy based on depthwise separable convolution includes: The convolution module is integrated into the Transformer module of the Depth Anything Model. The module consists of three convolution layers. The first layer is a 1×1 convolution to reduce the number of channels; the second layer is a 3×3 depth-separable convolution; and the last layer is a 1×1 convolution to restore the number of channels. Considering the number of model parameters and the accuracy of depth estimation, Res-DSC is only added after the 3rd, 6th, 9th, and 12th Transformer blocks.

4. The method according to claim 1, characterized in that: By eigendecomposing the network Φ I , input source image I s and the target image I t Decomposition and Reconstruction and include: Using a jump-connected U-shaped structure, diffuse reflection S, reflectivity A, and specular reflection K are decomposed. Then, according to the eigendecomposition theory, the formula I(u,λ)=L(λ)(S(u)A(u,λ)+K(u)) is used to calculate, where u is the pixel position, λ is the wavelength, and L is the illumination map. Finally, the reconstructed synthetic image is obtained. and 5. The method according to claim 1, characterized in that: Adjusting the network Φ by illumination L , adjust the diffuse and specular reflection maps and reconstruct include: Input original target image I t and the synthesized target image I s→t , a four-layer convolutional network is used, and the last layer uses the activation function tanh to obtain the lighting adjustment map in the range of [-1,1]: L Adjust =Φ L (|I t -AND s→t |) (2) Through the rigid flow extracted from the distortion transformation module, we can obtain A s→t , S s→t , K s→t ,Adjustment With A s→t Synthetic reconstruction 6. The method according to claim 1, characterized in that Calculate the source image I s , target image I t and reconstruct the image The decomposition and synthesis loss between ds : Among them, α is the weight factor, SSIM is the structural similarity loss function, and ‖·‖1 is the L1 loss function.

7. The method according to claim 1, characterized in that Calculate reflectivity map A t and the reconstructed A s→t The reflection loss between a : L a (A t ,A s→t )=‖A t -A s→t ‖1 (4)。 8. The method according to claim 1, characterized in that: Calculate the target image I t and reconstruct the synthetic image The mapping synthesis loss L between ms :

9. The method according to claim 1, characterized in that: Calculate the depth map D t and the target image I t The edge-aware depth smoothing loss L between es : in, and is the first-order derivative of the depth map along the x and y directions, and are the first-order derivatives of the target image along the x and y directions.

10. The method according to claim 1, characterized in that The total loss function is: Among them, λ ds , a , ms and λ es is the weight factor, M a is an automatic mask used to exclude lost areas during the distortion transformation, and its expression is M a =[I s→t >0], [·] is the Iverson bracket, Represents element-wise product.

Images