A Method and System for Geometric Estimation of Colonoscopy Based on a 3D Basic Model

Abstract

This invention relates to a method and system for geometric estimation of colonoscopy based on a three-dimensional basic model, applicable to the fields of medical image processing and computer vision technology. The method includes: acquiring adjacent frames of colonoscopy images and inputting them into a geometric basic model to output dense point clouds and confidence maps; aligning adjacent frames of colonoscopy images, performing brightness correction, and estimating camera pose; performing image reconstruction, calculating the loss between the target image and the reconstructed image, and driving the geometric basic model through self-supervised training; and achieving three-dimensional geometric estimation of the colonoscopy using the trained geometric basic model. Utilizing the dense point cloud and confidence map output by the geometric basic model enhances the ability to express low-level texture information; after image reconstruction, the model undergoes self-supervised training based on the loss, resulting in higher stability and consistency in cross-frame predictions during training, thereby improving the accuracy of three-dimensional colonoscopy geometric estimation.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and computer vision technology, and in particular to a method and system for geometric estimation of colonoscopy based on a three-dimensional basic model. Background Technology

[0002] Colorectal cancer is one of the most common and serious malignant tumors worldwide, ranking among the top in both incidence and mortality, placing a huge burden on public health. Colonoscopy is widely used and considered the "gold standard" in the screening and treatment of colorectal cancer. However, traditional colonoscopy relies on monocular video images for observation, which is limited by a narrow field of view and a lack of spatial information. During the procedure, doctors often need to rely on experience to identify and assess lesions in complex anatomical environments. This increases the difficulty of the procedure and may affect the accuracy of clinical examination and diagnosis. To compensate for the shortcomings of monocular imaging in acquiring spatial information, researchers have proposed a method to assist colonoscopy navigation through three-dimensional image reconstruction. This method uses temporal information between video frames to estimate the three-dimensional geometric structure, helping doctors obtain more intuitive spatial information about the cavities under limited field of view.

[0003] Early 3D reconstruction methods primarily relied on feature-point-based frameworks, such as Simultaneous Localization and Mapping (SLAM) and Structure-of-Motion (SfM) techniques. These methods estimate camera pose and scene structure by detecting, matching, and tracking key feature points in the image. However, in colonoscopy scenarios, the mucosal surface typically has a simple texture, sparse and unevenly distributed feature points, coupled with complex lighting and frequent instrument movements. These factors make it difficult for traditional feature-point methods to achieve stable results in clinical settings, often leading to poor reconstruction or even failure. With the development of deep learning, learning-based methods have gradually gained attention in the field of 3D reconstruction of medical images. However, most of these methods assume that the camera's intrinsic parameters are known and require a multi-stage pipeline during training, which often leads to noise accumulation and error propagation, making it difficult to maintain stable results in complex clinical environments.

[0004] Traditional image-based 3D colonoscopy geometry estimation methods suffer from several challenges. The large, textureless areas encountered during colonoscopy imaging make it difficult for the model to distinguish between foreground and background. Dynamic shadows cast by the light source as the scope moves are often misinterpreted as geometric changes. Furthermore, the non-Lambertian reflectivity of the colonic surface, uneven illumination, and highlights make it difficult to apply photometric constraints. The complex anatomical structure of the colon, with its frequent bends, folds, and occlusions, further complicates the model's generalization. Therefore, traditional image-based 3D colonoscopy geometry estimation methods often suffer from low accuracy in estimating the 3D geometric structure. Summary of the Invention

[0005] Therefore, in order to solve the above-mentioned technical problems, a method and system for geometric estimation of colonoscopy based on a three-dimensional basic model is provided, which can improve the accuracy of three-dimensional colonoscopy geometric estimation.

[0006] A method for geometric estimation of colonoscopy based on a three-dimensional basic model, the method comprising:

[0007] Acquire adjacent colonoscopy images and input them into the geometric basis model. The geometric basis model then outputs a dense point cloud and a confidence map.

[0008] The adjacent colonoscopy images are aligned, the processed images are brightness corrected and the camera pose is estimated to generate the corrected image and relative camera pose information.

[0009] The dense point cloud and confidence map, the corrected image, and the relative camera pose information are used to perform image reconstruction to obtain the reconstructed image.

[0010] Determine the target image, calculate the loss between the target image and the reconstructed image, and drive the geometric basis model to perform self-supervised training based on the loss;

[0011] The trained geometric foundation model is used to generate a 3D point cloud based on monocular colonoscopy images to achieve 3D geometric estimation of colonoscopy.

[0012] In one embodiment, the geometrical foundation model includes a convolutional neural network and a visual Transformer encoder; the method further includes:

[0013] The adjacent colonoscopy images are input into the convolutional neural network to extract multi-layer features;

[0014] The adjacent colonoscopy images are segmented into individual patches and input into the encoder of the visual Transformer. Global modeling is performed through a self-attention mechanism to output high-level semantic features.

[0015] The multi-layer features are projected and aligned, and then their size is adjusted by upsampling or spatial transformation before being added to the high-level semantic features to obtain fused features.

[0016] In one embodiment, the dense point cloud and confidence map are output through the geometric foundation model, including:

[0017] Each query vector is obtained, and cross-attention calculation is performed between each query vector and the high-level semantic feature to obtain the context information corresponding to the query vector.

[0018] The three-dimensional spatial coordinates corresponding to the query vector are obtained by a multilayer perceptron based on the context information, and the three-dimensional spatial coordinates are output and aggregated to form a dense point cloud.

[0019] Calculate the confidence score corresponding to each query vector, and arrange the confidence scores according to their spatial location to obtain a confidence score map.

[0020] In one embodiment, the alignment of the adjacent colonoscopy images includes:

[0021] The adjacent colonoscopy images are input into an optical flow network, and the encoder in the optical flow network extracts multi-scale features of the two images.

[0022] Calculate the feature correlation between multi-scale features of two frames of images, and obtain the dense optical flow field based on the feature correlation;

[0023] Image alignment is performed based on the dense optical flow field to obtain the processed image.

[0024] In one embodiment, the processed image undergoes brightness correction and camera pose estimation to generate a corrected image and relative camera pose information, including:

[0025] The processed image is input into a brightness correction network to predict correction parameters, and a differentiable image transformation is performed on the processed image based on the predicted correction parameters to obtain the corrected image.

[0026] The processed image is input into the pose estimation network to extract the translation vector and rotation vector. The relative pose transformation matrix is ​​calculated based on the translation vector and rotation vector, and the relative camera pose information is obtained based on the relative pose transformation matrix.

[0027] In one embodiment, an image reconstruction operation is performed using the dense point cloud and confidence map, the corrected image, and relative camera pose information to obtain a reconstructed image, including:

[0028] Based on the relative camera pose information and confidence map, a 3D geometric transformation is used to transform each 3D point in the dense point cloud into a camera coordinate system to obtain the transformed 3D point.

[0029] Based on the camera intrinsic parameter matrix in the relative camera pose information, the transformed 3D points are projected onto the 2D image plane, and the projection coordinates are calculated.

[0030] The color values ​​are obtained from the corrected image based on the projection coordinates, and the image is filled using the color values ​​to obtain the filled image;

[0031] The filled image is subjected to differentiable image sampling processing to complete the image reconstruction operation.

[0032] In one embodiment, the loss between the target image and the reconstructed image includes confidence-weighted photometric loss and geometric consistency loss.

[0033] A colonoscopy geometric estimation system based on a three-dimensional basic model, the system comprising:

[0034] The image processing module is used to acquire adjacent frames of colonoscopy images and input them into the geometric basis model, and output dense point cloud and confidence map through the geometric basis model;

[0035] The alignment module is used to align adjacent colonoscopy images, perform brightness correction on the processed images, estimate camera pose, and generate the corrected image and relative camera pose information.

[0036] The image reconstruction module is used to perform image reconstruction operations using the dense point cloud and confidence map, the corrected image and the relative camera pose information to obtain the reconstructed image;

[0037] The training module is used to determine the target image, calculate the loss between the target image and the reconstructed image, and drive the geometric basis model to perform self-supervised training based on the loss;

[0038] The geometric estimation module is used to generate a 3D point cloud based on a monocular colonoscopy image using a trained geometric model, thereby achieving 3D geometric estimation of the colonoscopy.

[0039] In one embodiment, the geometric foundation model includes a convolutional neural network and a visual Transformer encoder. The image processing module is further configured to: input adjacent colonoscopy images into the convolutional neural network to extract multi-layer features; segment the adjacent colonoscopy images into individual patches and input them into the encoder of the visual Transformer, perform global modeling through a self-attention mechanism, and output high-level semantic features; project and align the multi-layer features, and adjust their size through upsampling or spatial transformation before adding them to the high-level semantic features to obtain fused features.

[0040] In one embodiment, the image processing module is further configured to: acquire each query vector, and perform cross-attention calculation on each query vector and the high-level semantic features respectively to obtain context information corresponding to the query vector; obtain the three-dimensional spatial coordinates corresponding to the query vector through a multilayer perceptron based on the context information, and output and aggregate the three-dimensional spatial coordinates to form a dense point cloud; calculate the confidence score corresponding to each query vector, and arrange the confidence scores according to spatial position to obtain a confidence score map.

[0041] The aforementioned colonoscopy geometric estimation method and system based on a three-dimensional basic model utilizes the geometric basic model as the backbone network to output dense point clouds and confidence maps, which can enhance the expressive power of low-level texture information. After reconstructing the image, self-supervised training is performed based on a loss-driven model, which enables the model to have higher stability and consistency in cross-frame prediction during the training process, thereby improving the accuracy of three-dimensional colonoscopy geometric estimation. Attached Figure Description

[0042] Figure 1 This is an application environment diagram of a colonoscopy geometric estimation method based on a three-dimensional basic model in one embodiment;

[0043] Figure 2 This is a flowchart illustrating a colonoscopy geometric estimation method based on a three-dimensional basic model in one embodiment.

[0044] Figure 3 This is a schematic diagram of the detail recovery module in one embodiment;

[0045] Figure 4 This is an application framework diagram of a colonoscopy geometric estimation method based on a three-dimensional basic model in one embodiment;

[0046] Figure 5 This is a depth map and a schematic diagram of the three-dimensional structure predicted in the experiment.

[0047] Figure 6 This is a schematic diagram of the reconstruction results on the EndoMapper dataset in the experiment;

[0048] Figure 7 This is a block diagram of a colonoscopy geometric estimation system based on a three-dimensional basic model in one embodiment.

[0049] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] The colonoscopy geometric estimation method based on a three-dimensional basic model provided in this application can be applied to, for example... Figure 1 The application environment shown. For example... Figure 1 As shown, the application environment includes a computer device 110. The computer device 110 can acquire adjacent frames of colonoscopy images and input them into a geometric basis model, outputting a dense point cloud and a confidence map through the geometric basis model. The computer device 110 can align adjacent frames of colonoscopy images, perform brightness correction on the processed images, and estimate camera pose, generating a corrected image and relative camera pose information. The computer device 110 can use the dense point cloud and confidence map, the corrected image, and the relative camera pose information to perform image reconstruction, obtaining a reconstructed image. The computer device 110 can determine the target image, calculate the loss between the target image and the reconstructed image, and perform self-supervised training of the geometric basis model based on the loss. The computer device 110 can use the trained geometric basis model to generate a three-dimensional point cloud based on the monocular colonoscopy image, achieving three-dimensional geometric estimation of the colonoscopy. The computer device 110 can be, but is not limited to, various personal computers, laptops, smartphones, robots, tablets, and other devices.

[0052] In one embodiment, such as Figure 2 As shown, a method for geometric estimation of colonoscopy based on a three-dimensional basic model is provided, including the following steps:

[0053] Step 202: Acquire adjacent colonoscopy images and input them into the geometric basis model. Output dense point cloud and confidence map through the geometric basis model.

[0054] Among them, the geometric foundation model is a pre-trained geometric foundation model. The geometric foundation model takes two adjacent colonoscopy images as input and outputs a dense point cloud and a confidence map.

[0055] In one embodiment, the provided colonoscopy geometric estimation method based on a three-dimensional basic model may further include a feature extraction and fusion process. The specific process includes: inputting adjacent colonoscopy images into a convolutional neural network to extract multi-layer features; segmenting adjacent colonoscopy images into individual image patches and inputting them into the encoder of a visual Transformer, performing global modeling through a self-attention mechanism, and outputting high-level semantic features; projecting and aligning the multi-layer features, and adjusting their size through upsampling or spatial transformation before adding them to the high-level semantic features to obtain fused features.

[0056] To improve the reconstruction capability of the geometric foundation model in low-texture and complex structure regions, in this embodiment, the following can be introduced into the geometric foundation model: Figure 3 The detail recovery module shown is an encoder with a convolutional neural network and a visual Transformer set up in the geometric foundation model. For example... Figure 3 As shown, the detail restoration module uses multi-layer features extracted by a convolutional neural network and fuses them with features from a visual Transformer. The fusion method employs a combination of channel projection, spatial projection, and zero-convolution to ensure that low-level detail information is effectively injected into the Transformer's decoding layer, thereby improving the accuracy of the reconstruction. The convolutional neural network can be a ResNet-18.

[0057] In one embodiment, a colonoscopy geometric estimation method based on a three-dimensional basic model further includes the process of outputting a dense point cloud and a confidence map through the geometric basic model. The specific process includes: obtaining each query vector and performing cross-attention calculation on each query vector with high-level semantic features to obtain context information corresponding to the query vector; obtaining the three-dimensional spatial coordinates corresponding to the query vector through a multilayer perceptron based on the context information, and outputting and aggregating the three-dimensional spatial coordinates to form a dense point cloud; calculating the confidence score corresponding to each query vector, and arranging the confidence scores according to spatial position to obtain a confidence map.

[0058] Specifically, the geometric foundation model can prepare a set of learnable query vectors. Each query vector can be understood as responsible for the 3D information of a specific pixel location in the final output point cloud. The query vectors can undergo cross-attention calculation with high-level semantic features, collecting corresponding information from these global features through the attention mechanism. Then, each query vector that has collected information passes through a multilayer perceptron, which, based on the enhanced contextual information carried by the query vector, directly regresses the 3D spatial coordinates corresponding to the query point. When the outputs of all query vectors are combined, a dense point cloud is formed.

[0059] During the generation of dense point clouds, after cross-attention, a scalar value can be output as a confidence score for each query vector. All confidence scores are arranged according to their spatial location to obtain a confidence map.

[0060] Step 204: Align adjacent colonoscopy images, perform brightness correction on the processed images, and estimate camera pose to generate the corrected image and relative camera pose information.

[0061] During the training phase of the geometric foundation model, this embodiment can introduce auxiliary modules to improve illumination consistency and robustness of pose estimation. These auxiliary modules may include optical flow networks, brightness correction networks, and pose estimation networks.

[0062] In one embodiment, a colonoscopy geometric estimation method based on a three-dimensional basic model may further include an image alignment process, specifically including: inputting adjacent colonoscopy images into an optical flow network, extracting multi-scale features of the two images through an encoder in the optical flow network; calculating the feature correlation between the multi-scale features of the two images, and obtaining a dense optical flow field based on the feature correlation; and performing image alignment processing based on the dense optical flow field to obtain the processed image.

[0063] Optical flow describes the instantaneous motion vector of each pixel in an image between two consecutive frames, reflecting the apparent motion of objects in the scene. This motion is caused by the relative motion between the camera and objects in the scene. In a colonoscopy scene, because the camera moves inside the intestine, the same tissue point will appear at different pixel locations in two frames. The optical flow network is used to align the previous frame to the target frame and accurately estimate the displacement field.

[0064] Specifically, the optical flow network extracts multi-scale features from two frames of images through the encoder, and then in the iterative refinement part of the decoder, it infers the dense optical flow field by calculating the correlation between features; then, it uses the dense optical flow field to distort or deform the image frames so that they are geometrically aligned.

[0065] In one embodiment, the provided colonoscopy geometric estimation method based on a three-dimensional basic model may further include image correction and pose estimation processes. The specific processes include: inputting the processed image into a brightness correction network to predict correction parameters, and performing a differentiable image transformation on the processed image based on the predicted correction parameters to obtain a corrected image; inputting the processed image into a pose estimation network to extract translation and rotation vectors, and calculating a relative pose transformation matrix based on the translation and rotation vectors, and obtaining relative camera pose information based on the relative pose transformation matrix.

[0066] Among them, the brightness correction network can eliminate the brightness difference caused by the movement of the light source and obtain the corrected image; the pose estimation network uses adjacent frames to directly regress the relative pose of the camera, avoiding the failure of the traditional PnP algorithm in the low confidence region.

[0067] Computer equipment can input the processed image into a brightness correction network for correction parameter prediction. The encoder in the brightness correction network extracts features to understand the global illumination difference between two frames. For global correction of the processed image, the brightness correction network can output correction parameters to perform image transformation and obtain the corrected image.

[0068] A pose estimation network is typically a convolutional neural network followed by a regression head. In this embodiment, the computer device can extract and fuse features through the encoder in the pose estimation network, and finally output translation and rotation vectors through the regression head to obtain the relative pose transformation matrix, thereby obtaining the relative camera pose information.

[0069] Step 206: Perform image reconstruction using dense point cloud and confidence map, corrected image and relative camera pose information to obtain reconstructed image.

[0070] In one embodiment, the provided colonoscopy geometric estimation method based on a three-dimensional basic model may further include an image reconstruction process, specifically including: based on relative camera pose information and a confidence map, using three-dimensional geometric transformation to transform each 3D point in the dense point cloud into a camera coordinate system to obtain transformed 3D points; according to the camera intrinsic parameter matrix in the relative camera pose information, projecting the transformed 3D points onto a 2D image plane and calculating the projection coordinates; obtaining color values ​​from the corrected image based on the projection coordinates and using the color values ​​to fill the image to obtain a filled image; and performing differentiable image sampling processing on the filled image to complete the image reconstruction operation.

[0071] Computer equipment can transform the camera coordinate system of each 3D point in a dense point cloud, primarily using a predicted relative pose matrix. The equipment can then determine which pixel position the transformed 3D point will fall on in the image, i.e., by projecting the 3D point onto the 2D image plane using a camera intrinsic matrix and calculating the projected coordinates. The camera intrinsics can be pre-calibrated.

[0072] Next, the computer device can obtain color values ​​from the corrected image based on the calculated projection coordinates to fill in the reconstructed image. In this embodiment, since the coordinates calculated by projection are mostly floating-point numbers rather than integer pixel coordinates, a differentiable bilinear sampler can be used to calculate the final pixel values, thereby completing image reconstruction.

[0073] Step 208: Determine the target image, calculate the loss between the target image and the reconstructed image, and perform self-supervised training based on the loss-driven geometric foundation model.

[0074] To ensure the stability and accuracy of 3D reconstruction of colonoscopy images, two key loss functions were designed in one embodiment: confidence-weighted photometric loss and geometric consistency loss.

[0075] In this embodiment, a confidence-weighted photometric loss is used: the invention utilizes the point cloud and confidence map output by the geometric foundation model to obtain a reconstructed image through reprojection, and then compares its photometric value with the target image. Traditional photometric loss is prone to unstable supervision in low-texture areas or areas with uneven illumination, leading to oscillations or even degradation during training. Therefore, this embodiment introduces a pixel-level confidence map, using the confidence score of each pixel as a weight in the photometric difference calculation. In the loss calculation, the geometric foundation model processes the image bidirectionally. and Generate four sets of point cloud images: In the calculation of confidence-weighted photometric loss, the following is used: To complement the confidence plot ,and It is only used to calculate standard photometric loss. Specifically, photometric loss is defined as: ,in These are the target image and the reconstructed image, respectively. The value is 0.85. Therefore, the confidence-weighted photometric loss is defined as: By defining a confidence-weighted photometric loss, supervision of high-confidence regions can be emphasized during training, while avoiding the complete neglect of low-confidence regions. Furthermore, bidirectional frame supervision is employed, simultaneously considering both the reconstruction error from the current frame to the previous frame and the reconstruction error from the previous frame to the current frame, ensuring temporal consistency. In this way, the model maintains stable training performance under varying illumination and low-texture regions, thereby improving the final geometric prediction quality.

[0076] Since both point cloud prediction and pose estimation suffer from non-unique scales, inconsistencies in geometric results across frames may occur. To address this issue, this embodiment introduces a geometric consistency loss to constrain point clouds and poses across frames. Specifically, optical flow alignment is used to temporally match point clouds from different frames and calculate the differences between aligned point clouds, thereby ensuring consistency in scale and geometric structure across frame point clouds. During training, the auxiliary module predicts optical flow maps. and and occlusion map and Using optical flow diagrams, we can... and Transform into and The first part of the calculated geometric consistency loss can be expressed as: Next, in this embodiment, the predicted camera pose is applied to the point cloud coordinate transformation to ensure that the point cloud results remain geometrically consistent across different coordinate systems. Through the combined effect of these two types of constraints, the distortion of the 3D structure caused by scale drift or pose error is effectively eliminated, resulting in a more stable and coherent reconstructed cavity geometry. In this part of the loss calculation, the pose information predicted by the auxiliary module is utilized. and Will Transformation and The loss function for the second part can be expressed as: The final geometric consistency loss is defined as: ;in, , Both represent constants, and represent the weights of the first part of the loss and the second part of the loss, respectively.

[0077] Step 210: Use the trained geometric basis model to generate a three-dimensional point cloud based on the monocular colonoscopy image to achieve three-dimensional geometric estimation of the colonoscopy.

[0078] This application provides a colonoscopy geometry estimation method based on a three-dimensional basic model. The geometric basic model is used as the backbone network to output dense point clouds and confidence maps, which can enhance the expressive power of low-level texture information. After reconstructing the image, self-supervised training is performed based on a loss-driven model, which makes the model more stable and consistent in cross-frame prediction during the training process, thereby improving the accuracy of three-dimensional colonoscopy geometry estimation.

[0079] In one embodiment, a colonoscopy geometric estimation method based on a three-dimensional basic model can be applied to, for example... Figure 4 The structure shown includes: an improved base module, auxiliary modules, and image reconstruction and loss calculation. The improved base module consists of a pre-trained geometric foundation model and a detail recovery module. Figure 4As shown, the basic module uses an existing geometric foundation model as its backbone network and fine-tunes it using low-rank adaptation technology to generate dense point cloud representations and corresponding confidence maps. During the training phase, the auxiliary module handles illumination changes and pose estimation, generating corrected images and relative camera pose information. Finally, the reconstructed image is obtained through image reconstruction, and photometric loss is calculated between it and the target image, achieving end-to-end self-supervised training. Specifically, during the model training phase, two adjacent frames are input, the geometric foundation model predicts point clouds and confidence maps, and the auxiliary module performs brightness correction and pose estimation. Subsequently, image reconstruction is performed using the prediction results, and photometric loss, confidence-weighted photometric loss, and geometric consistency loss are calculated. After training, during the inference phase, only the basic module is needed to directly generate 3D point clouds and camera parameters from monocular images, thus achieving 3D reconstruction without external calibration.

[0080] This application provides a colonoscopy geometry estimation method based on a 3D basic model, which can effectively solve problems such as large areas of textureless regions, illumination variations, and non-Lambertian reflections in colonoscopy images. It utilizes a pre-trained geometric basic model as the backbone network and improves its adaptability in medical scenarios through low-rank adaptation. It introduces a detail recovery module to enhance the expressive power of low-level texture information. By combining confidence-weighted photometric loss and geometric consistency loss, the model has higher stability and consistency in cross-frame predictions during training.

[0081] In one embodiment, to verify the effectiveness of the colonoscopy geometry estimation method based on a 3D basic model provided in this application, experimental evaluations were conducted on various synthetic and real colonoscopy datasets, including the SimCol3D dataset, C3VD dataset, CSD dataset, and EndoMapper dataset. The experiments comprehensively tested three aspects: camera pose estimation, monocular depth estimation, and 3D point cloud reconstruction. The experimental results are shown in the table below:

[0082]

[0083] Therefore, in the camera pose estimation task, the proposed method outperforms the comparative methods on the SimCol3D dataset, including self-supervised joint depth and pose estimation methods and existing basic model adaptation methods. The results show that the proposed method significantly reduces both absolute translation error and relative rotation error, and maintains stable pose estimation performance even without using camera intrinsics.

[0084] In the monocular depth estimation task, the method in this application demonstrates excellent results on both the SimCol3D and CSD datasets, as shown in the table below:

[0085]

[0086] In the SimCol3D dataset, the proposed method outperforms the comparative methods in terms of absolute relative error and root mean square error, maintaining stable predictions in large, textureless regions. In the CSD dataset, the proposed method's error is slightly higher than some traditional methods that rely on a fixed depth range, but its overall performance is still better than most comparative methods, indicating strong adaptability to real-world complex environments. Furthermore, experimental results on the C3VD dataset show that while the proposed method's absolute relative error is slightly higher than some comparative methods, it significantly outperforms existing schemes in all other metrics, demonstrating more stable prediction capabilities.

[0087] like Figure 5 As shown, qualitative experiments on real colonoscopy videos using EndoMapper further demonstrate the advantages of the proposed method. Compared with existing methods, the proposed method can predict more complete depth maps and 3D structures without obvious artifacts under conditions of low-texture regions, bubbles, and high light reflection, significantly reducing erroneous reconstructions caused by illumination variations and non-Lambertian surfaces.

[0088] In the point cloud reconstruction task, the proposed method was quantitatively compared with the contrasting methods on the SimCol3D dataset. The results are shown in the table below:

[0089]

[0090] The method described in this application achieves optimal results in terms of both accuracy and completeness of point clouds, demonstrating that high-quality 3D point cloud representations can still be obtained without the need for camera intrinsic parameters. Furthermore, as... Figure 6 As shown, the reconstruction results on the EndoMapper dataset demonstrate that the method in this application can generate a well-structured and coherent 3D scene in real colonoscopy images, while the contrasting methods often exhibit large areas of planarization or geometric distortion.

[0091] Furthermore, ablation experiments were conducted on the method proposed in this application to verify the effectiveness of the proposed detail recovery module, confidence-weighted photometric loss, and geometric consistency loss. Experiments show that removing the confidence-weighted photometric loss leads to unstable model training and decreased accuracy; removing the geometric consistency loss makes the model prone to scale inconsistencies, resulting in biased geometric predictions across frames; and removing the detail recovery module significantly weakens the model's ability to recover low-level texture structures. The combined results demonstrate that each innovative module proposed in this application plays a crucial role in improving the final performance.

[0092] Through experimental validation on various synthetic and real colonoscopy datasets, the proposed method outperforms existing methods in tasks such as camera pose estimation, monocular depth prediction, and 3D point cloud reconstruction. In particular, it maintains stable performance even in low-texture and complex lighting scenarios. It not only eliminates the dependence on camera intrinsics and ground truth depth values, but also enables reliable 3D reconstruction in real clinical environments, demonstrating strong practicality and promotional value.

[0093] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0094] In one embodiment, such as Figure 7 As shown, a colonoscopy geometric estimation system based on a three-dimensional basic model is provided, including: an image processing module 710, a correction and alignment module 720, an image reconstruction module 730, a training module 740, and a geometric estimation module 750, wherein:

[0095] Image processing module 710 is used to acquire adjacent frames of colonoscopy images and input them into the geometric basis model, and output dense point cloud and confidence map through the geometric basis model;

[0096] The alignment module 720 is used to align adjacent colonoscopy images, perform brightness correction on the processed image and estimate camera pose, and generate the corrected image and relative camera pose information.

[0097] The image reconstruction module 730 is used to perform image reconstruction operations using dense point cloud and confidence map, corrected image and relative camera pose information to obtain reconstructed image;

[0098] Training module 740 is used to determine the target image, calculate the loss between the target image and the reconstructed image, and perform self-supervised training based on the loss-driven geometric foundation model;

[0099] The geometric estimation module 750 is used to generate a 3D point cloud based on a monocular colonoscopy image using a trained geometric basis model, thereby realizing 3D geometric estimation of colonoscopy.

[0100] In one embodiment, the geometric foundation model includes a convolutional neural network and a visual Transformer encoder; the image processing module 710 is further configured to: input adjacent colonoscopy images into the convolutional neural network to extract multi-layer features; segment adjacent colonoscopy images into individual patches and input them into the encoder of the visual Transformer, perform global modeling through a self-attention mechanism, and output high-level semantic features; project and align the multi-layer features, and add them to the high-level semantic features after adjusting their size through upsampling or spatial transformation to obtain fused features.

[0101] In one embodiment, the image processing module 710 is further configured to: acquire each query vector, and perform cross-attention calculation on each query vector and high-level semantic features respectively to obtain context information corresponding to the query vector; obtain the three-dimensional spatial coordinates corresponding to the query vector through a multilayer perceptron based on the context information, and output and aggregate the three-dimensional spatial coordinates to form a dense point cloud; calculate the confidence level corresponding to each query vector, and arrange the confidence levels according to their spatial positions to obtain a confidence map.

[0102] In one embodiment, the correction alignment module 720 is further configured to: input adjacent colonoscopy images into an optical flow network, extract multi-scale features of the two images through an encoder in the optical flow network; calculate the feature correlation between the multi-scale features of the two images, and obtain a dense optical flow field based on the feature correlation; perform image alignment processing based on the dense optical flow field to obtain the processed image.

[0103] In one embodiment, the correction alignment module 720 is further configured to: input the processed image into a brightness correction network to predict correction parameters, and perform a differentiable image transformation on the processed image based on the predicted correction parameters to obtain a corrected image; input the processed image into a pose estimation network to extract translation vectors and rotation vectors, and calculate a relative pose transformation matrix based on the translation vectors and rotation vectors, and obtain relative camera pose information based on the relative pose transformation matrix.

[0104] In one embodiment, the image reconstruction module 730 is further configured to: transform each 3D point in the dense point cloud into a camera coordinate system based on the relative camera pose information and confidence map using a three-dimensional geometric transformation to obtain the transformed 3D point; project the transformed 3D point onto a 2D image plane according to the camera intrinsic parameter matrix in the relative camera pose information, and calculate the projection coordinates; obtain color values ​​from the corrected image according to the projection coordinates, and use the color values ​​to fill the image to obtain the filled image; and perform differentiable image sampling processing on the filled image to complete the image reconstruction operation.

[0105] In one embodiment, the loss between the target image and the reconstructed image includes confidence-weighted photometric loss and geometric consistency loss.

[0106] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a geometric estimation method for colonoscopy based on a three-dimensional model. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0107] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0108] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of a colonoscopy geometric estimation method based on a three-dimensional basic model.

[0109] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of a colonoscopy geometric estimation method based on a three-dimensional basic model.

[0110] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0111] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0112] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for geometric estimation of colonoscopy based on a three-dimensional basic model, characterized in that, The method includes: Adjacent colonoscopy images are acquired and input into a geometrical model, which outputs a dense point cloud and a confidence map. The geometrical model includes a convolutional neural network and a visual Transformer encoder. Adjacent colonoscopy images are input into the convolutional neural network to extract multi-layer features. The adjacent colonoscopy images are then segmented into patches and input into the visual Transformer encoder. Global modeling is performed using a self-attention mechanism to output high-level semantic features. These multi-layer features are then projected and aligned, and their size is adjusted through upsampling or spatial transformation before being added to the high-level semantic features to obtain fused features. Outputting a dense point cloud and a confidence map through the geometric foundation model includes: acquiring each query vector, and performing cross-attention calculation on each query vector and the high-level semantic features to obtain context information corresponding to the query vector; obtaining the three-dimensional spatial coordinates corresponding to the query vector through a multilayer perceptron based on the context information, and outputting and aggregating the three-dimensional spatial coordinates to form a dense point cloud; calculating the confidence score corresponding to each query vector, and arranging the confidence scores according to spatial position to obtain a confidence map; The adjacent colonoscopy images are aligned, the processed images are brightness corrected and the camera pose is estimated to generate the corrected image and relative camera pose information. The dense point cloud and confidence map, the corrected image, and the relative camera pose information are used to perform image reconstruction to obtain the reconstructed image. Determine the target image, calculate the loss between the target image and the reconstructed image, and drive the geometric basis model to perform self-supervised training based on the loss; The trained geometric foundation model is used to generate a 3D point cloud based on monocular colonoscopy images to achieve 3D geometric estimation of colonoscopy.

2. The colonoscopy geometric estimation method based on a three-dimensional basic model according to claim 1, characterized in that, Alignment processing of adjacent colonoscopy images includes: The adjacent colonoscopy images are input into an optical flow network, and the encoder in the optical flow network extracts multi-scale features of the two images. Calculate the feature correlation between multi-scale features of two frames of images, and obtain the dense optical flow field based on the feature correlation; Image alignment is performed based on the dense optical flow field to obtain the processed image.

3. The colonoscopy geometric estimation method based on a three-dimensional basic model according to claim 1, characterized in that, The processed image undergoes brightness correction and camera pose estimation to generate a corrected image and relative camera pose information, including: The processed image is input into a brightness correction network to predict correction parameters, and a differentiable image transformation is performed on the processed image based on the predicted correction parameters to obtain the corrected image. The processed image is input into the pose estimation network to extract the translation vector and rotation vector. The relative pose transformation matrix is ​​calculated based on the translation vector and rotation vector, and the relative camera pose information is obtained based on the relative pose transformation matrix.

4. The colonoscopy geometric estimation method based on a three-dimensional basic model according to claim 1, characterized in that, Image reconstruction is performed using the dense point cloud and confidence map, the corrected image, and the relative camera pose information to obtain a reconstructed image, including: Based on the relative camera pose information and confidence map, a 3D geometric transformation is used to transform each 3D point in the dense point cloud into a camera coordinate system to obtain the transformed 3D point. Based on the camera intrinsic parameter matrix in the relative camera pose information, the transformed 3D points are projected onto the 2D image plane, and the projection coordinates are calculated. The color values ​​are obtained from the corrected image based on the projection coordinates, and the image is filled using the color values ​​to obtain the filled image; The filled image is subjected to differentiable image sampling processing to complete the image reconstruction operation.

5. The colonoscopy geometric estimation method based on a three-dimensional basic model according to claim 1, characterized in that, The loss between the target image and the reconstructed image includes confidence-weighted photometric loss and geometric consistency loss.

6. A colonoscopy geometric estimation system based on a three-dimensional basic model, characterized in that, The system includes: The image processing module is used to acquire adjacent colonoscopy images and input them into a geometric base model, which outputs a dense point cloud and a confidence map. The geometric base model includes a convolutional neural network and a visual Transformer encoder. The image processing module is also used to: input the adjacent colonoscopy images into the convolutional neural network to extract multi-layer features; segment the adjacent colonoscopy images into individual image patches and input them into the visual Transformer encoder, performing global modeling through a self-attention mechanism to output high-level semantic features; project and align the multi-layer features, and adjust their size through upsampling or spatial transformation before adding them to the high-level semantic features to obtain fused features. The image processing module is further configured to: acquire each query vector, and perform cross-attention calculation on each query vector and the high-level semantic features respectively to obtain context information corresponding to the query vector; obtain the three-dimensional spatial coordinates corresponding to the query vector through a multilayer perceptron based on the context information, and output and aggregate the three-dimensional spatial coordinates to form a dense point cloud; calculate the confidence score corresponding to each query vector, and arrange the confidence scores according to spatial position to obtain a confidence score map; The alignment module is used to align adjacent colonoscopy images, perform brightness correction on the processed images, estimate camera pose, and generate the corrected image and relative camera pose information. The image reconstruction module is used to perform image reconstruction operations using the dense point cloud and confidence map, the corrected image and the relative camera pose information to obtain the reconstructed image; The training module is used to determine the target image, calculate the loss between the target image and the reconstructed image, and drive the geometric basis model to perform self-supervised training based on the loss; The geometric estimation module is used to generate a 3D point cloud based on a monocular colonoscopy image using a trained geometric model, thereby achieving 3D geometric estimation of the colonoscopy.

Images