Three-dimensional reconstruction method, device and equipment for digestive endoscope scene and medium

Abstract

The application discloses a three-dimensional reconstruction method, device, equipment and medium for a digestive endoscope scene, relates to the technical field of 3D modeling, and comprises the following steps: mapping a semantic embedding vector of a historical digestive endoscope image into LoRA modulation coefficients; constructing a current prediction model for depth pose prediction and comprising a backbone network and a hybrid expert multi-task head; the backbone network is a Depth Anything 3 network with a LoRA branch; adjusting the weight of the LoRA branch in the current prediction model based on the modulation coefficients, and extracting target image features fused with semantic information; generating a depth pose prediction result by using the multi-task head to determine a total loss value; updating the parameters of the low-rank incremental matrix of the LoRA branch to obtain a new current prediction model, performing the next round of iteration, and obtaining a target prediction model; and predicting the depth pose information by using the target prediction model and performing three-dimensional reconstruction. The accuracy and robustness of three-dimensional reconstruction in the digestive endoscope scene are improved.

Description

Technical Field

[0001] This invention relates to the field of 3D modeling technology, and in particular to a method, apparatus, equipment and medium for three-dimensional reconstruction of digestive endoscopy scenes. Background Technology

[0002] Digestive endoscopy is a core medical technique that uses a monocular camera to acquire continuous two-dimensional images of the body cavity, and it is widely used in the diagnosis of gastrointestinal diseases and minimally invasive surgery. Because monocular endoscopy only provides two-dimensional visual information, it lacks depth perception and three-dimensional spatial structure information, severely limiting the perception capabilities of surgical navigation, quantitative analysis, and robot-assisted systems. Compared to binocular vision or structured light systems, which can directly obtain depth information, monocular endoscopy is physically difficult to directly measure depth; therefore, three-dimensional reconstruction and depth estimation have become key technical challenges.

[0003] Traditional 3D reconstruction methods based on geometric feature matching rely on stable and rich texture features in images, which makes it difficult to cope with challenges such as weak texture, specular reflection, high dynamic lighting and non-rigid deformation commonly found in endoscopic images. This leads to unstable depth estimation and low reconstruction accuracy.

[0004] In summary, improving the accuracy and robustness of 3D reconstruction in gastrointestinal endoscopy scenarios is a problem that needs to be solved in this field. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a method, apparatus, device, and medium for three-dimensional reconstruction of digestive endoscopy scenes, thereby improving the accuracy and robustness of three-dimensional reconstruction in digestive endoscopy scenes. The specific solution is as follows: In a first aspect, this application discloses a three-dimensional reconstruction method for a digestive endoscopy scene, including: Semantic embedding vectors are generated based on the semantic description text of historical digestive endoscopy images, and the semantic embedding vectors are mapped to LoRA modulation coefficients using a multilayer perceptron. A current prediction model for deep pose prediction is constructed, comprising a backbone network and a hybrid expert multi-task head; wherein the backbone network is a Depth Anything 3 network with LoRA branch introduced. Based on the LoRA modulation coefficients, the target weights of the LoRA branch in the current prediction model are adjusted so that the backbone network can extract target image features with fused semantic information from the historical digestive endoscopy images using the adjusted weights. The hybrid expert multi-task head is used to generate a depth pose prediction result based on the target image features, and the total loss value of the current prediction model is determined based on the depth pose prediction result. The parameters of the low-rank increment matrix of the LoRA branch in the current prediction model are updated based on the total loss value to obtain a new current prediction model. Then, the process jumps back to the step of adjusting the target weight of the LoRA branch in the current prediction model based on the LoRA modulation coefficient until the preset stopping iteration condition is met, and the new current prediction model is determined as the target prediction model. The target prediction model is used to predict the depth and pose information of the current digestive endoscopy image, and three-dimensional reconstruction is performed based on the depth and pose information.

[0006] Optionally, generating semantic embedding vectors based on semantic description text of historical digestive endoscopy images includes: Semantic description text for historical digestive endoscopy images is generated using a multimodal large model; wherein, the semantic description text is used to describe the organ structure, cavity morphology, surface texture and light reflection characteristics in the historical digestive endoscopy images; The semantic description text is input into the CLIP text encoder so that the CLIP text encoder generates a semantic embedding vector based on the semantic description text.

[0007] Optionally, the LoRA branch is located in the linear transform layer of the Depth Anything 3 network, and the target weight of the LoRA branch is the sum of the original weight matrix and the low-rank increment matrix; adjusting the target weight of the LoRA branch in the current prediction model based on the LoRA modulation coefficient includes: The low-rank increment matrix is ​​adjusted based on the LoRA modulation coefficients to obtain the adjusted low-rank increment matrix; wherein the adjusted low-rank increment matrix is ​​the product of the LoRA modulation coefficient adjustment and the low-rank increment matrix. The sum of the original weight matrix and the adjusted low-rank increment matrix is ​​determined as the adjusted weight.

[0008] Optionally, the hybrid expert multi-task head includes multiple depth expert heads and multiple pose expert heads, and the depth pose prediction result includes a target depth prediction result and a target pose prediction result; the step of generating a depth pose prediction result based on the target image features using the hybrid expert multi-task head includes: The target image features are respectively input into each of the depth expert heads and each of the pose expert heads to obtain each initial depth prediction result generated by each of the depth expert heads and each initial pose prediction result generated by each of the pose expert heads; The depth weights of each depth expert head and the pose weights of each pose expert head are generated using a gating network based on the semantic embedding vector and the target image features, respectively. The initial depth prediction results are weighted and summed using the depth weights to obtain the target depth prediction result, and the initial pose prediction results are weighted and summed using the pose weights to obtain the target pose prediction result.

[0009] Optionally, the historical digestive endoscopy images are a continuous video sequence, and different target pose prediction results correspond to different historical digestive endoscopy images; determining the total loss value of the current prediction model based on the depth pose prediction results includes: The first and second frame images are determined from the historical digestive endoscopy images in adjacent frames; The first frame image is reprojected onto the viewpoint of the second frame image to obtain the reprojected image; The relative pose between the first frame image and the second frame image is determined based on the target pose prediction result of the first frame image and the target pose prediction result of the second frame image. The total loss value of the current prediction model is determined based on the second frame image, the target depth prediction result of the second frame image, the relative pose, and the reprojected image.

[0010] Optionally, the step of reprojecting the first frame image onto the viewpoint of the second frame image includes: The camera pose transformation, camera intrinsic parameter matrix, first frame image, and target depth prediction results of second frame image are input into a differentiable projection function based on pinhole camera model to reproject the first frame image onto the viewpoint of the second frame image. The camera pose transformation is used to describe the transformation relationship of the camera from the shooting pose of the second frame image to the shooting pose of the first frame image during the gastrointestinal endoscopy imaging process.

[0011] Optionally, determining the total loss value of the current prediction model based on the second frame image, the target depth prediction result of the second frame image, the relative pose, and the reprojected image includes: The photometric loss value of the current prediction model is obtained based on the second frame image and the reprojected image. The smoothing loss value of the current prediction model is obtained based on the second frame image and the target depth prediction result of the second frame image; The regularized loss value of the current prediction model is obtained based on the target depth prediction result and the relative pose of the second frame image. The total loss value of the current prediction model is obtained based on the photometric loss value, the smoothing loss value, and the regularization loss value.

[0012] Secondly, this application discloses a three-dimensional reconstruction device for a digestive endoscopy scene, comprising: The vector mapping module is used to generate semantic embedding vectors based on the semantic description text of historical digestive endoscopy images, and to map the semantic embedding vectors into LoRA modulation coefficients using a multilayer perceptron. The model building module is used to build a current prediction model for deep pose prediction, which includes a backbone network and a hybrid expert multi-task head; wherein the backbone network is a Depth Anything 3 network with LoRA branch introduced. The feature extraction module is used to adjust the target weights of the LoRA branch in the current prediction model based on the LoRA modulation coefficients, so that the backbone network can extract target image features with fused semantic information from the historical digestive endoscopy images using the adjusted weights; The loss determination module is used to generate a depth pose prediction result based on the target image features using the hybrid expert multi-task head, and to determine the total loss value of the current prediction model based on the depth pose prediction result. The parameter update module is used to update the parameters of the low-rank increment matrix of the LoRA branch in the current prediction model based on the total loss value to obtain a new current prediction model. Then, it jumps back to the step of adjusting the target weight of the LoRA branch in the current prediction model based on the LoRA modulation coefficient until the preset stopping iteration condition is met, and the new current prediction model is determined as the target prediction model. The three-dimensional reconstruction module is used to predict the depth and pose information of the current digestive endoscopy image using the target prediction model, and to perform three-dimensional reconstruction based on the depth and pose information.

[0013] Thirdly, this application discloses an electronic device, including: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the steps of the aforementioned disclosed method for three-dimensional reconstruction of a digestive endoscopy scene.

[0014] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed method for three-dimensional reconstruction of a digestive endoscopy scene.

[0015] The beneficial effects of this application are as follows: This application generates semantic embedding vectors based on the semantic description text of historical digestive endoscopy images, and maps the semantic embedding vectors to LoRA modulation coefficients using a multilayer perceptron; it constructs a current prediction model for depth pose prediction, including a backbone network and a hybrid expert multi-task head; wherein, the backbone network is a Depth Anything network with a LoRA branch. 3. Network: Based on the LoRA modulation coefficients, the target weights of the LoRA branch in the current prediction model are adjusted so that the backbone network can extract target image features with fused semantic information from the historical gastrointestinal endoscopy images using the adjusted weights; the hybrid expert multi-task head is used to generate depth pose prediction results based on the target image features, and the total loss value of the current prediction model is determined according to the depth pose prediction results; the parameters of the low-rank increment matrix of the LoRA branch in the current prediction model are updated based on the total loss value to obtain a new current prediction model, and then the process jumps back to the step of adjusting the target weights of the LoRA branch in the current prediction model based on the LoRA modulation coefficients until a preset stopping iteration condition is met, and the new current prediction model is determined as the target prediction model; the depth pose information of the current gastrointestinal endoscopy image is predicted using the target prediction model, and three-dimensional reconstruction is performed based on the depth pose information.Therefore, this application utilizes LoRA modulation coefficients to adjust the weights of the Depth Anything 3 backbone network introduced into the LoRA branch by transforming multimodal semantic features. This allows the backbone network to extract target image features with fused semantic information from historical gastrointestinal endoscopy images, achieving adaptive adaptation of the model to specific semantic features of gastrointestinal endoscopy under conditions without real depth annotation. This improves the model's adaptability to complex and diverse gastrointestinal endoscopy scenarios and the accuracy of depth pose feature extraction. Simultaneously, the LoRA branch only updates the low-rank increment matrix parameters, avoiding large-scale weight updates and significantly reducing training costs and parameter increments. By constructing a prediction model containing a hybrid expert multi-task head, and relying on this multi-task head to generate depth pose prediction results based on the target image features with fused semantics, the separation and adaptive selection of depth estimation and pose prediction tasks are achieved. This effectively reduces task interference during multi-task training and improves the stability and robustness of model training. Based on the depth pose prediction results, the total loss value is determined and L is iteratively updated. The low-rank incremental matrix parameters of the oRA branch are continuously optimized through self-supervision, ensuring consistently accurate depth pose prediction and further improving the model's accuracy in gastrointestinal endoscopy scenarios. Finally, the converged target prediction model is used to predict the depth pose of the current gastrointestinal endoscopy image and complete 3D reconstruction. Because the model is specifically adapted for gastrointestinal endoscopy scenarios and has low training costs and small parameter increments, it ensures both the accuracy and robustness of the 3D reconstruction results and makes the model more suitable for efficient deployment and application in practical engineering and clinical scenarios. Furthermore, the overall process combines semantic guidance and multi-task decoupling, giving the model's depth pose prediction stronger scenario specificity and training stability. This effectively solves the problems of insufficient depth estimation accuracy, performance degradation in multi-task training, high annotation costs, and difficult deployment associated with traditional methods in gastrointestinal endoscopy scenarios. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0017] Figure 1 This is a flowchart of a three-dimensional reconstruction method for a digestive endoscopy scene disclosed in this application; Figure 2 This is a schematic diagram of a specific prediction model disclosed in this application; Figure 3 This is a schematic diagram of a three-dimensional reconstruction device for a digestive endoscopy scene disclosed in this application; Figure 4This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0019] Digestive endoscopy is a core medical technique that uses a monocular camera to acquire continuous two-dimensional images of the body cavity, and it is widely used in the diagnosis of gastrointestinal diseases and minimally invasive surgery. Because monocular endoscopy only provides two-dimensional visual information, it lacks depth perception and three-dimensional spatial structure information, severely limiting the perception capabilities of surgical navigation, quantitative analysis, and robot-assisted systems. Compared to binocular vision or structured light systems, which can directly obtain depth information, monocular endoscopy is physically difficult to directly measure depth; therefore, three-dimensional reconstruction and depth estimation have become key technical challenges.

[0020] Traditional 3D reconstruction methods based on geometric feature matching rely on stable and rich texture features in images, which makes it difficult to cope with challenges such as weak texture, specular reflection, high dynamic lighting and non-rigid deformation commonly found in endoscopic images. This leads to unstable depth estimation and low reconstruction accuracy.

[0021] Therefore, this application provides a three-dimensional reconstruction scheme for digestive endoscopy scenes, which improves the accuracy and robustness of three-dimensional reconstruction in digestive endoscopy scenes.

[0022] See Figure 1 As shown in the figure, this application discloses a three-dimensional reconstruction method for a digestive endoscopy scene, including: Step S11: Generate a semantic embedding vector based on the semantic description text of historical digestive endoscopy images, and map the semantic embedding vector to LoRA modulation coefficients using a multilayer perceptron.

[0023] In this embodiment, the step of generating semantic embedding vectors based on semantic description text of historical digestive endoscopy images includes: generating semantic description text of historical digestive endoscopy images using a multimodal large model; wherein, the semantic description text is used to describe organ structures, cavity morphology, surface texture, and light reflection characteristics in the historical digestive endoscopy images; and inputting the semantic description text into a CLIP text encoder so that the CLIP text encoder generates semantic embedding vectors based on the semantic description text.

[0024] First, a multimodal large model was used to analyze historical digestive endoscopy images. Perform semantic analysis and feature interpretation, where t represents the t-th frame, and generate a semantic description text that can accurately represent the organ structure, lumen morphology, surface texture, and light reflection characteristics in the historical digestive endoscopy image , that is , this semantic description text can completely extract the visual semantic features of the historical digestive endoscopy image, providing a text basis for converting language information into feature vectors recognizable by the model in the subsequent process.

[0025] Then input the generated semantic description text into the CLIP (Contrastive Language-Image Pre-training) text encoder. With the feature mapping and embedding functions of the CLIP text encoder, perform feature extraction and vectorization transformation on the semantic description text, and convert the semantic description in natural language form into a fixed-length numerical semantic embedding vector , that is , this semantic embedding vector can accurately carry the semantic information of the historical digestive endoscopy image, can be recognized and utilized by subsequent model modules, realizes the transformation from the visual features of the digestive endoscopy image to the language semantic features, and then to the vector features that can be processed by the machine, laying a core semantic feature foundation for modulating the LoRA (Low-Rank Adaptation) branch parameters through language guidance later, enabling the model to capture the scene-specific semantic information of the digestive endoscopy image based on this semantic embedding vector, and then realizing the adaptive adaptation to the digestive endoscopy scene.

[0026] Input the semantic embedding vector carrying the semantic information of the digestive endoscopy image into a multi-layer perceptron for feature transformation and dimension adaptation processing. After the hierarchical feature mapping of the multi-layer perceptron, then perform non-linear transformation and numerical normalization through the Sigmoid activation function, and finally generate a LoRA modulation coefficient that can be used to regulate the update of the LoRA branch parameters. The specific formula is as follows: ; In the formula, represents the LoRA modulation coefficient, represents the multi-layer perceptron, represents the semantic embedding vector, represents the Sigmoid activation function.

[0027] This modulation coefficient can be in scalar or vector form, and can adaptively determine the update direction and update amplitude of the LoRA branch parameters according to different semantic features of the digestive endoscopy image in the subsequent process.

[0028] Step S12: Construct a current prediction model for depth pose prediction, which includes a backbone network and a mixture-of-experts multi-task head; wherein, the backbone network is the Depth Anything 3 network with a LoRA branch introduced.

[0029] In this embodiment, the LoRA branch is located in the linear transformation layer of the Depth Anything 3 network, and the target weight of the LoRA branch is the sum of the original weight matrix and the low-rank increment matrix.

[0030] Perform customized modification on the Depth Anything 3 network to construct the current prediction model for depth pose prediction in the digestive endoscopy scenario. For example Figure 2 As shown, the overall model includes a backbone network and a mixture-of-experts multi-task head. The backbone network is the Depth Anything 3 network with a LoRA branch introduced. Specifically, fine-tune the 3rd, 5th, 8th, and 11th layers of the Depth Anything 3 network and freeze the weights of the remaining network layers. At the same time, add a residual convolution module to this network to specifically compensate for the lack of detailed characterization ability of the original Depth Anything 3 network due to the adoption of the Transformer architecture, so that the backbone network can not only retain the basic feature extraction ability of the original model but also adapt to the feature extraction requirements of digestive endoscopy images and improve the ability to capture image details.

[0031] The LoRA branch is accurately set in the linear transformation layer of the Depth Anything 3 network. This branch determines the target weight by using the low-rank increment modeling method, and its target weight is the sum of the original weight matrix and the low-rank increment matrix of the linear transformation layer of the Depth Anything 3 network. Specifically, it is as follows: ; ; In the formula, represents the target weight, represents the frozen original weight, represents the low-rank increment matrix, and B and A represent two low-rank matrices. .

[0032] During the training process, only the parameters constituting the low-rank increment matrix are updated, and the original weight matrix is frozen without adjustment. In this way, while realizing the adaptive fine-tuning of the model for the digestive endoscopy scenario, the amount of parameter update is greatly reduced, and the training cost of the model is reduced. The mixture-of-experts multi-task head, as the exclusive output module of the model, cooperates with the modified backbone network to achieve depth estimation and pose prediction of digestive endoscopy images.

[0033] Step S13: Adjust the target weights of the LoRA branch in the current prediction model based on the LoRA modulation coefficients, so that the backbone network can extract target image features that fuse semantic information from the historical endoscopic images using the adjusted weights.

[0034] In this embodiment, the adjusting the target weights of the LoRA branch in the current prediction model based on the LoRA modulation coefficients includes: adjusting the low-rank increment matrix based on the LoRA modulation coefficients to obtain an adjusted low-rank increment matrix; wherein, the adjusted low-rank increment matrix is the product of the LoRA modulation coefficient adjustment and the low-rank increment matrix; determining the sum of the original weight matrix and the adjusted low-rank increment matrix as the adjusted weight.

[0035] Specifically adjust the target weights of the LoRA branch in the current prediction model based on the LoRA modulation coefficients, so that the Depth Anything 3 backbone network with the LoRA branch introduced can accurately extract target image features that fuse the semantic information of endoscopic images from historical endoscopic images based on the adjusted weights That is, shared features, where is the backbone network of Depth Anything 3. This adjustment process enables the feature extraction ability of the backbone network to fit specific semantic features such as the organ structure and lumen morphology of endoscopic images, so that the extracted target image features not only have the feature representation basis of the original model but also incorporate scene-specific semantic information, providing accurate feature support for the depth pose prediction of the subsequent mixture-of-experts multi-task head.

[0036] Specifically, when adjusting the target weights of the LoRA branch, first adjust the low-rank increment matrix with the LoRA modulation coefficients, perform a multiplication operation on the LoRA modulation coefficient and the low-rank increment matrix to obtain an adjusted low-rank increment matrix adapted to the semantic conditions of endoscopic images, and then sum the original weight matrix of the linear transformation layer of the Depth Anything 3 network and the adjusted low-rank increment matrix, and determine the sum result as the adjusted weight of the LoRA branch. The adjusted weight is This adjustment method enables the weight update of the LoRA branch to be guided by language semantic information, realizes the adaptive adjustment of weights according to different semantic scenarios of endoscopic images, and only adjusts the low-rank increment matrix to achieve weight changes without updating the original weight matrix, ensuring the efficiency and low parameter quantity characteristics of fine-tuning.

[0037] Step S14: Use the mixture-of-experts multi-task head to generate a depth pose prediction result based on the target image features, and determine the total loss value of the current prediction model according to the depth pose prediction result.

[0038] In this embodiment, the mixture-of-experts multi-task head includes a plurality of depth expert heads and a plurality of pose expert heads, and the depth and pose prediction result includes a target depth prediction result and a target pose prediction result; the step of generating a depth and pose prediction result based on the target image feature by using the mixture-of-experts multi-task head includes: inputting the target image feature into each of the depth expert heads and each of the pose expert heads respectively to obtain each initial depth prediction result generated by each of the depth expert heads and each initial pose prediction result generated by each of the pose expert heads; using a gating network to generate a depth weight for each of the depth expert heads and a pose weight for each of the pose expert heads respectively based on the semantic embedding vector and the target image feature; using the depth weight to perform weighted summation on each of the initial depth prediction results to obtain a target depth prediction result, and using the pose weight to perform weighted summation on each of the initial pose prediction results to obtain a target pose prediction result.

[0039] The mixture-of-experts multi-task head (Head) includes N depth expert heads dedicated to depth estimation tasks and M pose expert heads corresponding to pose prediction tasks. The generated depth and pose prediction results are divided into a target depth prediction result and a target pose prediction result. When generating a depth and pose prediction result based on the target image feature with fused semantic information extracted by the backbone network by relying on this mixture-of-experts multi-task head, first input the target image feature into each of the depth expert heads and each of the pose expert heads respectively. Each depth expert head independently generates a corresponding initial depth prediction result, and each pose expert head independently generates a corresponding initial pose prediction result , that is, the initial depth prediction result is , and the initial pose prediction result is , where represents the i-th depth expert head, represents the initial depth prediction result output by the i-th depth expert head, represents the j-th pose expert head, represents the initial pose prediction result output by the j-th pose expert head.

[0040] Then use the gating network Combined with the semantic embedding vector generated by the CLIP text encoder and the target image feature before, generate a depth weight matching its prediction ability for each depth expert head and a pose weight matching its prediction ability for each pose expert head. Both the depth weight and the pose weight meet the normalization requirements. The acquisition formula for the depth weight is: ; where is the depth weight vector, is the depth weight of the i-th depth expert head; Similarly, the formula for obtaining the pose weight is as follows: ; where is the pose weight vector, is the pose weight of the i-th pose expert head.

[0041] Finally, perform a weighted summation operation on all initial depth prediction results according to the depth weights corresponding to each depth expert head, and determine the operation result as the final target depth prediction result , and the specific formula is as follows: ; At the same time, perform a weighted summation operation on all initial pose prediction results according to the pose weights corresponding to each pose expert head, and determine the operation result as the final target pose prediction result , and the specific formula is as follows: ; This process allows the depth and pose tasks to be initially predicted by dedicated expert heads, and then dynamically allocate the expert head weights by combining semantic information and image features through a gating network, achieving effective decoupling of the depth and pose tasks, avoiding mutual interference in multi-task training, and enabling the model to adaptively select a more matching expert head according to different semantic features and image features of the digestive endoscopy images, improving the accuracy and robustness of the depth and pose prediction results, and adapting to the complex and variable characteristics of organ structures, textures, illuminations, etc. in the digestive endoscopy scenario.

[0042] In this embodiment, the historical digestive endoscopy images are a video sequence of consecutive frames, and different target pose prediction results correspond to different historical digestive endoscopy images; determining the total loss value of the current prediction model according to the depth and pose prediction results includes: determining a first frame image and a second frame image from the historical digestive endoscopy images of adjacent frames; reprojection the first frame image to the perspective of the second frame image to obtain a reprojection image; determining the relative pose between the first frame image and the second frame image according to the target pose prediction result of the first frame image and the target pose prediction result of the second frame image; determining the total loss value of the current prediction model according to the second frame image, the target depth prediction result of the second frame image, the relative pose, and the reprojection image.

[0043] It can be understood that the historical digestive endoscopy images for model training are a video sequence of consecutive frames, and each frame of historical digestive endoscopy image corresponds to its own target pose prediction result. When determining the total loss value of the current prediction model according to the depth and pose prediction results, first select two adjacent frames from the consecutive frames of historical digestive endoscopy images and respectively determine them as the first frame image and the second frame image , and then reproject the first frame image to the perspective of the second frame image to obtain the corresponding reprojected image . Meanwhile, calculate the relative pose between two adjacent frames by combining the target pose prediction results of the first frame image and the second frame image , that is , where can be implemented by the Mixture of Experts Pose Head

[0044] Finally, relying on the second frame image itself , the target depth prediction result corresponding to the second frame image , the calculated relative pose between the first frame image and the second frame image [[ID=十九]]and the reprojected image obtained by reprojection , determine the total loss value of the current prediction model through the fusion calculation of multi-dimensional constraints .

[0045] This process is based on a continuous frame endoscopic video sequence. By means of the pose correlation and cross-frame reprojection between adjacent frames, it realizes the self-supervised loss calculation without real depth annotation, enabling the determination of the total loss value to form effective supervision relying on the inter-frame geometric relationship and photometric features of endoscopic images. At the same time, by combining the associated calculation of depth prediction results and relative poses, the total loss value can accurately reflect the accuracy of the model's depth pose prediction, providing an effective optimization basis for the subsequent iterative update of model parameters that fits the endoscopic scenario. Moreover, through the inter-frame constraints of adjacent frames, the loss calculation is more in line with the actual continuous imaging scenario of endoscopic shooting, improving the rationality and effectiveness of loss supervision

[0046] In this embodiment, the reprojecting the first frame image to the perspective of the second frame image includes: inputting the camera pose transformation, the camera intrinsic matrix, the first frame image, and the target depth prediction result of the second frame image into a differentiable projection function based on the pinhole camera model to reproject the first frame image to the perspective of the second frame image; wherein, the camera pose transformation is used to describe the transformation relationship of the camera from the shooting pose of the second frame image to the shooting pose of the first frame image during the endoscopic shooting process

[0047] The camera pose transformation that depicts the spatial transformation relationship of the camera from the shooting pose of the second frame image to the shooting pose of the first frame image during endoscopic shooting , the fixed camera intrinsic matrix K calibrated by the endoscopic imaging device, the first frame image as the reprojection source image and the target depth prediction result of the second frame image that characterizes the spatial depth information of the second frame image , are jointly input into a differentiable projection function constructed based on the pinhole camera model In it, the first frame image is reprojected to the perspective of the second frame image to obtain a reprojected image. The specific formula is as follows: ; Relying on the differentiable projection function to follow the imaging principle of a pinhole camera and the spatial projection rules, the pixel points of the first frame image are combined with depth information, camera pose transformation, and intrinsic parameter characteristics to perform projection transformation calculations from three-dimensional space to a two-dimensional pixel plane, converting the first frame image from its own shooting perspective to the shooting perspective of the second frame image, and completing the reprojection operation of the first frame image to the perspective of the second frame image. The design of this differentiable projection function makes the reprojection process differentiable and capable of participating in the backpropagation of model training, providing an optimizable projection calculation basis for subsequent self-supervised loss terms such as calculating photometric consistency loss based on the reprojected image. At the same time, combined with multi-dimensional parameter inputs of camera pose transformation, intrinsic matrix, and depth prediction results, the reprojection result can accurately fit the actual imaging scene captured by the endoscope, ensuring the accuracy of cross-frame reprojection, and thus providing a reliable supervision basis for model self-supervised training.

[0048] In this embodiment, determining the total loss value of the current prediction model according to the second frame image, the target depth prediction result of the second frame image, the relative pose, and the reprojected image includes: obtaining the photometric loss value of the current prediction model according to the second frame image and the reprojected image; obtaining the smooth loss value of the current prediction model according to the second frame image and the target depth prediction result of the second frame image; obtaining the regularization loss value of the current prediction model according to the target depth prediction result of the second frame image and the relative pose; and obtaining the total loss value of the current prediction model based on the photometric loss value, the smooth loss value, and the regularization loss value.

[0049] When determining the total loss value of the current prediction model according to the second frame image, the target depth prediction result of the second frame image, the relative pose between adjacent frames, and the reprojected image, first, based on the second frame image and the reprojected image obtained by reprojecting the first frame image to the perspective of the second frame image, the photometric loss value of the current prediction model is obtained by calculating the pixel difference and structural similarity between the two frames of images , and the specific formula is as follows: ; In the formula, represents the L1 norm, SSIM represents the structural similarity, , represent weighting coefficients. This loss value is used to constrain the photometric consistency of cross-frame reprojection and reflects the matching degree between the reprojected image and the original second frame image.

[0050] Combined with the second-frame image and its corresponding target depth prediction result, the smooth loss value of the current prediction model is obtained by calculating and constraining the spatial gradient of the depth map , and the specific formula is as follows: ; In the formula, represents calculating the gradient along the x direction, represents the gradient along the y direction, and this loss value is used to suppress the noise in the depth map and improve the spatial continuity and rationality of the depth prediction result.

[0051] Meanwhile, relying on the target depth prediction result of the second-frame image and the relative pose between adjacent frames, the parameter update process of the depth and pose tasks is constrained through the stop-gradient operator to obtain the regularization loss value of the current prediction model, that is, the multi-task regularization term loss value , and the specific formula is as follows: ; This loss value is used to reduce the mutual interference in the multi-task training process of depth estimation and pose prediction and enhance the stability of co-training between tasks.

[0052] Finally, the calculated photometric loss value, smooth loss value, and regularization loss value are fused and calculated, and the fused result is determined as the total loss value of the current prediction model, as follows: ; In the formula, represents the weight of the smooth term, represents the weight of the regularization loss term; this total loss value combines photometric consistency and geometric consistency constraints, that is, multiple constraints of photometric consistency, depth map smoothness, and multi-task co-training, and can comprehensively reflect the accuracy and training stability of the model's depth and pose prediction results, providing a scientific and endoscopy-scene-appropriate optimization goal for the parameter iterative update of the LoRA branch low-rank increment matrix of the subsequent model, enabling the parameter update of the model to be in the direction of improving photometric matching, depth smoothness, and multi-task training robustness, and ensuring the continuous optimization of the model in self-supervised training without real depth annotations.

[0053] Step S15: Update the parameters of the low-rank increment matrix of the LoRA branch in the current prediction model based on the total loss value to obtain a new current prediction model, and then jump back to the step of adjusting the target weight of the LoRA branch in the current prediction model based on the LoRA modulation coefficient until the preset stop iteration condition is satisfied, and determine the new current prediction model as the target prediction model.

[0054] Taking the total loss value of the current prediction model as the optimization basis, the parameters of the low-rank increment matrix in the LoRA branch of the model are iteratively updated through backpropagation. The low-rank increment matrix is the product of two low-rank matrices. The update process only adjusts the parameters of the low-rank matrices that make up the low-rank increment matrix. The original weight matrix of the linear transformation layer of the Depth Anything 3 network remains frozen and is not updated throughout the process. In this way, under the premise of updating with a small number of parameters, the model parameters are optimized and a new current prediction model is obtained. Subsequently, it returns to the step of adjusting the target weights of the LoRA branch in the current prediction model based on the LoRA modulation coefficient. Continuing to use historical endoscopic images as training samples, the updated model extracts the target image features that fuse semantic information again, generates depth pose prediction results, and calculates the total loss value. By repeating the complete training process of weight adjustment, feature extraction, result prediction, loss calculation, and parameter update until the model training process meets the preset stop iteration condition, which can be set according to indicators such as model convergence status, loss value change, and number of iterations. After reaching the stop iteration condition, the new current prediction model that finally completes the parameter iteration optimization is determined as the target prediction model. Throughout the iterative update process, the weight is adaptively adjusted relying on the language-guided LoRA modulation coefficient, and only the parameters of the low-rank increment matrix in the LoRA branch are updated. This not only ensures that the model can continuously fit the semantic features and image characteristics of the endoscopic scene, continuously improving the accuracy and robustness of depth pose prediction, but also avoids large-scale weight updates of the model, greatly reducing the training cost and computational complexity of the model. At the same time, combined with the self-supervised loss optimization method, the model can complete efficient scene adaptation training under the condition of no real depth annotation, so that the finally obtained target prediction model can accurately adapt to the actual application scenario of the endoscopic.

[0055] In a specific embodiment of the prediction model training, the model can be trained for 150,000 steps on 4 A6000 GPUs, adopting a warm-up strategy of 5,000 steps, and the peak learning rate is set to 5×10 -4 ; The basic resolution is 504×504, which can be divided evenly by 2, 3, 4, 6, 9, and 14, and can better accommodate common photo aspect ratios such as 2:3, 3:4, and 9:16; The training image resolution is randomly sampled from 504×504, 504×378, 504×336, 504×280, 336×504, 896×504, 756×504, 672×504; For the 504×504 resolution, the number of views is uniformly sampled from the interval [2, 15]; The batch size is dynamically adjusted to ensure that the number of tokens per step is roughly constant; During the training process, pose condition adjustment is randomly enabled with a probability of 0.3.

[0056] Step S16: Use the target prediction model to predict the depth pose information of the current endoscopic image, and perform three-dimensional reconstruction based on the depth pose information.

[0057] Apply the trained target prediction model that meets the preset iteration conditions to the actual depth pose prediction scenario of endoscopic images. Input the current endoscopic image to be processed into this target prediction model. Relying on the Depth Anything 3 backbone network with LoRA branches adapted and trained for the endoscopic scenario in the model, extract image features fused with endoscopic scene semantic information from the current endoscopic image. Then, decouple and predict the depth and pose of this feature through the mixture-of-experts multi-task head of the model. After weighted fusion by the gating network by the dedicated depth expert head and pose expert head, accurately output the depth information and pose information corresponding to the current endoscopic image. Subsequently, taking the predicted depth pose information as the core basis, combined with the camera intrinsics of the endoscopic imaging device, the pinhole camera projection model, and the three-dimensional reconstruction method of geometric projection and point cloud fusion, map the two-dimensional pixel information of the current endoscopic image to the three-dimensional space by means of depth pose parameters. Through the calculation of the spatial coordinates of pixel points, the generation and fusion processing of dense point clouds, complete the three-dimensional reconstruction corresponding to the current endoscopic image. In this process, due to the language-guided LoRA fine-tuning and the training optimization of the mixture-of-experts multi-task head, the target prediction model can accurately adapt to the scene features such as the organ structure, cavity morphology, surface texture, and light reflection of endoscopic images. The output depth pose information has high accuracy and robustness. The three-dimensional reconstruction based on this can truly restore the three-dimensional space structure inside the body cavity under the endoscope. Moreover, since the model itself only updates the parameters with low-rank increments, it has a small number of parameters and high inference efficiency, enabling the efficient implementation of depth pose prediction and three-dimensional reconstruction of endoscopic images, providing accurate three-dimensional space structure support for surgical navigation, quantitative analysis, and robotic assistance systems in endoscopic examinations.

[0058] The reconstructed three-dimensional structure can be directly applied to depth estimation and scene reconstruction in endoscopic robotic surgery, providing accurate spatial references for surgical instrument positioning, operation path planning, and intraoperative tissue morphology perception. At the same time, it can also serve the automatic endoscopic examination process, assisting the system to complete tasks such as lesion recognition, lesion range measurement, and mucosal morphology analysis, improving the objectivity and efficiency of the examination.

[0059] The beneficial effects of this application are as follows: This application generates a semantic embedding vector based on the semantic description text of historical endoscopic images, and uses a multi-layer perceptron to map the semantic embedding vector into LoRA modulation coefficients; constructs a current prediction model for depth pose prediction, which includes a backbone network and a mixture-of-experts multi-task head; where the backbone network is the Depth Anything 3 network with a LoRA branch introduced; adjusts the target weights of the LoRA branch in the current prediction model based on the LoRA modulation coefficients, so that the backbone network extracts target image features that fuse semantic information from the historical endoscopic images using the adjusted weights; uses the mixture-of-experts multi-task head to generate a depth pose prediction result based on the target image features, and determines the total loss value of the current prediction model according to the depth pose prediction result; updates the parameters of the low-rank increment matrix of the LoRA branch in the current prediction model based on the total loss value to obtain a new current prediction model, and then jumps back to the step of adjusting the target weights of the LoRA branch in the current prediction model based on the LoRA modulation coefficients until a preset stop iteration condition is met, and determines the new current prediction model as the target prediction model; uses the target prediction model to predict the depth pose information of the current endoscopic image, and performs 3D reconstruction based on the depth pose information.It can be seen that this application adjusts the weights of the Depth Anything 3 backbone network introduced into the LoRA branch by means of the LoRA modulation coefficient transformation of multimodal semantic features, enabling the backbone network to extract target image features that fuse semantic information from historical endoscopic images, achieving the adaptive adaptation of the model to specific semantic features of endoscopic images under the condition of no real depth annotation, improving the adaptability of the model to complex and diverse endoscopic scenarios and the accuracy of depth pose feature extraction. At the same time, the LoRA branch only updates the parameters of the low-rank increment matrix, avoiding large-scale weight updates of the model and greatly reducing the training cost and parameter increment of the model. By constructing a prediction model containing a multi-expert multi-task head, relying on this multi-task head to generate depth pose prediction results based on the target image features of fused semantics, the separation and adaptive selection of depth estimation and pose prediction tasks are achieved, effectively reducing task interference in the multi-task training process and improving the stability and robustness of model training. Based on the depth pose prediction results, the total loss value is determined and the parameters of the low-rank increment matrix of the LoRA branch are iteratively updated. By means of self-supervision, the model parameters are continuously optimized to make the depth pose prediction results of the model continuously accurate, further improving the accuracy of depth pose prediction of the model in the endoscopic scenario. Finally, the trained target prediction model is used to perform depth pose prediction on the current endoscopic image and complete 3D reconstruction. Since the model is adapted to the endoscopic scenario, has a low training cost and a small parameter increment, it not only ensures the accuracy and robustness of the 3D reconstruction results, but also makes the model more suitable for efficient deployment and practical application in actual engineering and clinical scenarios. At the same time, the overall process combines semantic guidance and multi-task decoupling, making the depth pose prediction of the model have stronger scene specificity and training stability, effectively solving the problems of insufficient accuracy of depth estimation, degradation of multi-task training performance, high annotation cost and large deployment difficulty in the endoscopic scenario by traditional methods.

[0060] See Figure 3 As shown, an embodiment of this application discloses a 3D reconstruction device for an endoscopic scenario, including: A vector mapping module 11, configured to generate a semantic embedding vector according to the semantic description text of historical endoscopic images, and map the semantic embedding vector into a LoRA modulation coefficient by using a multi-layer perceptron; A model construction module 12, configured to construct a current prediction model for depth pose prediction, including a backbone network and a multi-expert multi-task head; wherein, the backbone network is a Depth Anything 3 network with a LoRA branch introduced; A feature extraction module 13, configured to adjust the target weights of the LoRA branch in the current prediction model based on the LoRA modulation coefficient, so that the backbone network extracts target image features that fuse semantic information from the historical endoscopic images by using the adjusted weights; A loss determination module 14 is configured to generate a depth pose prediction result based on the target image features by using the mixture-of-experts multi-task head, and determine the total loss value of the current prediction model according to the depth pose prediction result; A parameter update module 15 is configured to update the parameters of the low-rank increment matrix of the LoRA branch in the current prediction model based on the total loss value to obtain a new current prediction model, and then jump back to the step of adjusting the target weight of the LoRA branch in the current prediction model according to the LoRA modulation coefficient until a preset stop iteration condition is satisfied, and determine the new current prediction model as the target prediction model; A three-dimensional reconstruction module 16 is configured to predict the depth pose information of the current endoscopy image by using the target prediction model, and perform three-dimensional reconstruction based on the depth pose information.

[0061] Furthermore, an embodiment of the present application also provides an electronic device. Figure 4 It is a structural diagram of an electronic device 20 shown according to an exemplary embodiment, and the content in the figure should not be considered as any limitation to the scope of use of the present application.

[0062] Figure 4 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Specifically, it may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the three-dimensional reconstruction method for the endoscopy scenario executed by the electronic device disclosed in any of the foregoing embodiments.

[0063] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and external devices, and the communication protocol it follows is any communication protocol applicable to the technical solution of the present application, and no specific limitation is imposed on it here; the input / output interface 25 is used to obtain external input data or output data to the outside, and its specific interface type can be selected according to specific application needs, and no specific limitation is made here.

[0064] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for the rendering and drawing of the content to be displayed on the display screen. In some embodiments, the processor 21 may further include an AI (Artificial Intelligence) processor, and the AI processor is used to process computing operations related to machine learning.

[0065] In addition, as a carrier for resource storage, the memory 22 may be a read-only memory, a random access memory, a magnetic disk, or an optical disc, etc. The resources stored thereon include an operating system 221, a computer program 222, data 223, etc., and the storage method may be short-term storage or permanent storage.

[0066] Among them, the operating system 221 is used to manage and control each hardware device and the computer program 222 on the electronic device, so as to enable the processor 21 to perform operations and processing on the massive data 223 in the memory 22. It may be Windows, Unix, Linux, etc. In addition to the computer program that can be used to complete the three-dimensional reconstruction method of the endoscopy scene executed by the electronic device disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program that can be used to complete other specific tasks. The data 223 may include not only the data transmitted by external devices received by the electronic device, but also the data collected by its own input / output interface 25, etc.

[0067] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the three-dimensional reconstruction method of the endoscopy scene disclosed above. For the specific steps of this method, reference may be made to the corresponding content disclosed in the foregoing embodiments, and details will not be repeated here.

[0068] In the present specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference may be made to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple. For the relevant parts, reference may be made to the description in the method section.

[0069] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or the combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Skilled professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application. The steps of the methods or algorithms described in combination with the embodiments disclosed herein can be directly implemented by hardware, software modules executed by a processor, or the combination of the two. The software modules can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable EPROM (Erasable Programmable Read Only Memory), electrically erasable programmable EEPROM (Electrically Erasable Programmable read only memory), registers, hard disks, removable disks, CD-ROM (Compact Disc Read-Only Memory), or any other form of storage medium well-known in the technical field.

[0070] Finally, it should also be noted that in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the element.

[0071] The above has introduced in detail a three-dimensional reconstruction method, device, equipment and medium for a digestive endoscopy scenario. In this article, specific examples are used to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only for helping to understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A method for three-dimensional reconstruction of a digestive endoscopy scene, characterized in that, include: Semantic embedding vectors are generated based on the semantic description text of historical digestive endoscopy images, and the semantic embedding vectors are mapped to LoRA modulation coefficients using a multilayer perceptron. A current prediction model for deep pose prediction is constructed, comprising a backbone network and a hybrid expert multi-task head; wherein the backbone network is a Depth Anything 3 network with LoRA branch introduced. Based on the LoRA modulation coefficients, the target weights of the LoRA branch in the current prediction model are adjusted so that the backbone network can extract target image features with fused semantic information from the historical digestive endoscopy images using the adjusted weights. The hybrid expert multi-task head is used to generate a depth pose prediction result based on the target image features, and the total loss value of the current prediction model is determined based on the depth pose prediction result. The parameters of the low-rank increment matrix of the LoRA branch in the current prediction model are updated based on the total loss value to obtain a new current prediction model. Then, the process jumps back to the step of adjusting the target weight of the LoRA branch in the current prediction model based on the LoRA modulation coefficient until the preset stopping iteration condition is met, and the new current prediction model is determined as the target prediction model. The target prediction model is used to predict the depth and pose information of the current digestive endoscopy image, and three-dimensional reconstruction is performed based on the depth and pose information. The hybrid expert multi-task head includes multiple depth expert heads and multiple pose expert heads, and the depth pose prediction result includes a target depth prediction result and a target pose prediction result; the step of generating a depth pose prediction result based on the target image features using the hybrid expert multi-task head includes: The target image features are input into each of the depth expert heads and each of the pose expert heads to obtain initial depth prediction results generated by each of the depth expert heads and initial pose prediction results generated by each of the pose expert heads. A gating network is used to generate depth weights for each of the depth expert heads and pose weights for each of the pose expert heads based on the semantic embedding vector and the target image features. The initial depth prediction results are weighted and summed using the depth weights to obtain the target depth prediction result, and the initial pose prediction results are weighted and summed using the pose weights to obtain the target pose prediction result.

2. The three-dimensional reconstruction method for a digestive endoscopy scene according to claim 1, characterized in that, The step of generating semantic embedding vectors based on semantic description text of historical digestive endoscopy images includes: Semantic description text for historical digestive endoscopy images is generated using a multimodal large model; wherein, the semantic description text is used to describe the organ structure, cavity morphology, surface texture and light reflection characteristics in the historical digestive endoscopy images; The semantic description text is input into the CLIP text encoder so that the CLIP text encoder generates a semantic embedding vector based on the semantic description text.

3. The three-dimensional reconstruction method for a digestive endoscopy scene according to claim 1, characterized in that, The LoRA branch is located in the linear transform layer of the Depth Anything 3 network, and the target weight of the LoRA branch is the sum of the original weight matrix and the low-rank increment matrix; adjusting the target weight of the LoRA branch in the current prediction model based on the LoRA modulation coefficients includes: The low-rank increment matrix is ​​adjusted based on the LoRA modulation coefficients to obtain the adjusted low-rank increment matrix; wherein the adjusted low-rank increment matrix is ​​the product of the LoRA modulation coefficient adjustment and the low-rank increment matrix. The sum of the original weight matrix and the adjusted low-rank increment matrix is ​​determined as the adjusted weight.

4. The three-dimensional reconstruction method for a digestive endoscopy scene according to claim 1, characterized in that, The historical digestive endoscopy images are a continuous video sequence, and different target pose prediction results correspond to different historical digestive endoscopy images; determining the total loss value of the current prediction model based on the depth pose prediction results includes: The first and second frame images are determined from the historical digestive endoscopy images in adjacent frames; The first frame image is reprojected onto the viewpoint of the second frame image to obtain the reprojected image; The relative pose between the first frame image and the second frame image is determined based on the target pose prediction result of the first frame image and the target pose prediction result of the second frame image. The total loss value of the current prediction model is determined based on the second frame image, the target depth prediction result of the second frame image, the relative pose, and the reprojected image.

5. The three-dimensional reconstruction method for a digestive endoscopy scene according to claim 4, characterized in that, The viewpoint of reprojecting the first frame image onto the second frame image includes: The camera pose transformation, camera intrinsic parameter matrix, first frame image, and target depth prediction results of second frame image are input into a differentiable projection function based on pinhole camera model to reproject the first frame image onto the viewpoint of the second frame image. The camera pose transformation is used to describe the transformation relationship of the camera from the shooting pose of the second frame image to the shooting pose of the first frame image during the gastrointestinal endoscopy imaging process.

6. The three-dimensional reconstruction method for a digestive endoscopy scene according to claim 4, characterized in that, The step of determining the total loss value of the current prediction model based on the second frame image, the target depth prediction result of the second frame image, the relative pose, and the reprojected image includes: The photometric loss value of the current prediction model is obtained based on the second frame image and the reprojected image. The smoothing loss value of the current prediction model is obtained based on the second frame image and the target depth prediction result of the second frame image; The regularized loss value of the current prediction model is obtained based on the target depth prediction result and the relative pose of the second frame image. The total loss value of the current prediction model is obtained based on the photometric loss value, the smoothing loss value, and the regularization loss value.

7. A three-dimensional reconstruction device for a digestive endoscopy scene, characterized in that, The steps for implementing the three-dimensional reconstruction method of a digestive endoscopy scene as described in any one of claims 1 to 6 include: The vector mapping module is used to generate semantic embedding vectors based on the semantic description text of historical digestive endoscopy images, and to map the semantic embedding vectors into LoRA modulation coefficients using a multilayer perceptron. The model building module is used to build a current prediction model for deep pose prediction, which includes a backbone network and a hybrid expert multi-task head; wherein the backbone network is a Depth Anything 3 network with LoRA branch introduced. The feature extraction module is used to adjust the target weights of the LoRA branch in the current prediction model based on the LoRA modulation coefficients, so that the backbone network can extract target image features with fused semantic information from the historical digestive endoscopy images using the adjusted weights; The loss determination module is used to generate a depth pose prediction result based on the target image features using the hybrid expert multi-task head, and to determine the total loss value of the current prediction model based on the depth pose prediction result. The parameter update module is used to update the parameters of the low-rank increment matrix of the LoRA branch in the current prediction model based on the total loss value to obtain a new current prediction model. Then, it jumps back to the step of adjusting the target weight of the LoRA branch in the current prediction model based on the LoRA modulation coefficient until the preset stopping iteration condition is met, and the new current prediction model is determined as the target prediction model. The three-dimensional reconstruction module is used to predict the depth and pose information of the current digestive endoscopy image using the target prediction model, and to perform three-dimensional reconstruction based on the depth and pose information.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the three-dimensional reconstruction method for a digestive endoscopy scene as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when executed by a processor, the computer program implements the steps of the three-dimensional reconstruction method for a digestive endoscopy scene as described in any one of claims 1 to 6.

Images