A real-time tracking method and system for three-dimensional pose of monocular endoscope based on deep learning

Abstract

The present invention discloses a method and system for real-time three-dimensional pose tracking of a monocular endoscope based on deep learning, comprising: stitching two input images to obtain a stitched image, using an optical flow estimation network to output a corresponding optical flow map based on the two input images; extracting scene features of the two input images and motion features of the optical flow map using a feature extractor; simultaneously, extracting joint features of the stitched image using a multidimensional feature extractor; after stitching all the extracted features, estimating the relative pose transformation vector of the endoscope at corresponding moments between the two frames of image using a pose decoder; iteratively determining the absolute three-dimensional pose of the endoscope at the corresponding moment of each frame of image based on the estimated relative pose transformation vector and the absolute three-dimensional pose of the previous frame, thereby achieving real-time tracking of the endoscope. Through deep learning and multi-feature fusion technology, the present invention significantly improves the accuracy and real-time performance of three-dimensional pose tracking of a monocular endoscope.

Description

Technical Field

[0001] The present invention belongs to the fields of medical imaging technology and endoscope posture tracking technology, and in particular relates to a method and system for real-time tracking of the three-dimensional posture of a monocular endoscope based on deep learning. Background Art

[0002] Endoscopic technology uses a high-definition camera to transmit internal body images to an external display, enabling the operator to clearly observe the target area and perform procedures. In recent years, with the rapid development of technology, new endoscopic surgical robotic systems based on endoscopy technology, such as Medtronic's Hugo RAS and CMR Surgical's Versius, have emerged. Robotic-assisted endoscopic surgery has become a mainstream method for minimally invasive endoscopic surgery. Simultaneously, the rise of embodied intelligence technology has also promoted the application of artificial intelligence models in robotic systems.

[0003] However, endoscopic surgery faces many challenges. The natural cavity space is narrow, the instrument operation space and degrees of freedom are limited, and the two-dimensional images obtained by the endoscope are difficult to reflect the spatial depth and layer information, and are often accompanied by image distortion. Therefore, endoscopic three-dimensional scene perception and navigation have become key tasks for achieving precise and safe surgery. In recent years, the application of extended reality (XR) technology in surgical navigation has gradually increased. Through registration technology, preoperative imaging data is superimposed with intraoperative endoscopic images to provide more intuitive three-dimensional visual information. However, the accuracy of the registration of endoscopic video with preoperative scanning data is highly dependent on intraoperative three-dimensional scene perception.

[0004] Endoscopic pose tracking technology is an important means of achieving three-dimensional scene perception during surgery. The endoscope pose refers to the three-dimensional spatial coordinates and posture of the endoscope in the reference frame of the surgical object. Currently, pose tracking technologies are mainly divided into three categories: pose tracking based on optical positioning, pose tracking based on electromagnetic positioning, and pose tracking based on vision. Optical positioning technology often fails to track due to obstruction by personnel and instruments, while electromagnetic positioning technology is susceptible to interference from metal objects and electromagnetic equipment in the scene. Therefore, vision-based endoscope pose tracking technology has become a feasible way to obtain endoscope pose information due to its adaptability to complex surgical scenarios.

[0005] In summary, there is an urgent need to propose a real-time tracking method and system for the three-dimensional pose of a monocular endoscope based on deep learning, so as to achieve accurate estimation of the endoscope pose through deep learning technology. Summary of the Invention

[0006] In order to solve the above technical problems, the present invention proposes a method and system for real-time tracking of the three-dimensional posture of a monocular endoscope based on deep learning to solve the problems existing in the above-mentioned prior art.

[0007] To achieve the above objectives, the present invention provides a method for real-time tracking of the three-dimensional pose of a monocular endoscope based on deep learning, comprising the following steps:

[0008] Set the initial posture of the endoscope and collect endoscopic image sequences in real time;

[0009] For the current frame, the two input images are spliced ​​together to obtain a spliced ​​image, and the optical flow estimation network is used to output the corresponding optical flow map based on the two input images;

[0010] Constructing a deep learning framework, which includes a feature extractor based on a graph convolutional neural network, a multi-dimensional feature extractor based on an attention mechanism, and a pose decoder based on multi-scale depthwise separable convolution;

[0011] A feature extractor based on a graph convolutional neural network extracts scene features from the two input images and motion features from the optical flow map. Meanwhile, a multi-dimensional feature extractor based on an attention mechanism extracts joint features from the spliced ​​images.

[0012] After concatenating all the extracted features, a pose decoder based on multi-scale depthwise separable convolution estimates the relative pose transformation vector of the endoscope at the corresponding moment in the two frames of image.

[0013] Based on the estimated relative pose transformation vector and the absolute 3D pose of the previous frame, the absolute 3D pose of the endoscope at the corresponding moment of each frame image is iteratively determined to achieve real-time tracking of the endoscope.

[0014] Optionally, the optical flow estimation network adopts a FlowNet network, and the architecture is an encoder-decoder structure.

[0015] Optionally, the process of using the optical flow estimation network to output corresponding optical flow maps based on two input images includes:

[0016] The encoder performs multi-layer convolution and pooling on the two input images to obtain the corresponding features. The decoder performs feature fusion and upsampling on the obtained features and then maps them back to the optical flow field to obtain the corresponding optical flow map.

[0017] Optionally, the graph convolutional neural network uses a model pre-trained on the ImageNet dataset.

[0018] Optionally, the process of extracting joint features of the spliced ​​images by the multi-dimensional feature extractor based on the attention mechanism includes:

[0019] The spliced ​​image is input into the self-attention module, and multiple feature maps with different channel dimension information are obtained through flipping operation; the flipped feature map is input into the channel attention mechanism module, and the channel attention mechanism module simultaneously performs global average pooling and global maximum pooling operations on the flipped feature map, and then calculates the attention weights of the results of global average pooling and global maximum pooling, and finally obtains the joint features of the spliced ​​image.

[0020] Optionally, based on the relative pose transformation vector and the absolute 3D pose of the previous frame The iterative formula for iteratively determining the absolute three-dimensional pose of the endoscope at each frame of image is as follows:

[0021]

[0022] in, is the relative translation transformation vector, is the relative rotation transformation vector, is the pose vector at time i, is the known initial pose vector, is the position coordinate vector at the moment ik, is the rotation vector at moment ik.

[0023] The present invention also provides a real-time tracking system for three-dimensional posture of a monocular endoscope based on deep learning, which comprises: an image acquisition module, an image preprocessing module, a learning framework construction module, a feature extraction module, a posture decoding module and a posture tracking module based on the method described above;

[0024] The image acquisition module is used to set the initial posture of the endoscope and acquire endoscopic image sequences in real time;

[0025] The image preprocessing module is used to splice the two input images to obtain a spliced ​​image for the current frame, and output a corresponding optical flow map based on the two input images using an optical flow estimation network;

[0026] The learning framework construction module is used to construct a deep learning framework based on a feature extractor of a graph convolutional neural network, a multidimensional feature extractor based on an attention mechanism, and a pose decoder based on multi-scale depth-separable convolution;

[0027] The feature extraction module is used to extract scene features of the two input images and motion features of the optical flow map using a feature extractor based on a graph convolutional neural network; at the same time, a multi-dimensional feature extractor based on an attention mechanism extracts joint features of the spliced ​​images;

[0028] The posture decoding module is used to splice all the extracted features and estimate the relative posture transformation vector of the endoscope at the corresponding moments of two frames of images based on the posture decoder of multi-scale depth-separable convolution;

[0029] The posture tracking module is used to iteratively determine the absolute three-dimensional posture of the endoscope at the corresponding moment of each frame image based on the estimated relative posture transformation vector and the absolute three-dimensional posture of the previous frame, thereby realizing real-time tracking of the endoscope.

[0030] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.

[0031] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method when executed by a processor.

[0032] The present invention also provides a computer program product, comprising a computer program, which implements the steps of the method when executed by a processor.

[0033] Compared with the prior art, the present invention has the following advantages and technical effects:

[0034] 1. Improve the accuracy of endoscope posture tracking:

[0035] By combining an optical flow estimation network with a deep learning framework, this method can extract rich scene and motion features from endoscopic images. The optical flow map provides pixel-level motion information, while a graph convolutional neural network and attention mechanism further enhance the feature representation capabilities. This multi-feature fusion approach can significantly improve the accuracy of endoscopic pose tracking, especially in complex human cavities.

[0036] 2. Realize real-time pose updates:

[0037] This method uses an iterative formula to update the absolute 3D position of the endoscope in real time, enabling rapid response to changes in the endoscope's motion. This real-time performance is crucial for endoscopic navigation, providing the system with stable and reliable position information, enabling more precise endoscope operation and reducing the risk of misoperation.

[0038] 3. Adapt to complex cavity environment:

[0039] This method automatically learns the features of endoscopic images through a deep learning network, adapting to different cavity environments and tissue types. Furthermore, the optical flow estimation network and attention mechanism can handle noise and distortion in the images, further enhancing the system's robustness in complex environments.

[0040] 4. Reduce dependence on traditional positioning technology:

[0041] Compared to pose tracking technologies based on optical or electromagnetic positioning, this method relies entirely on the endoscope image itself and is unaffected by occlusion or electromagnetic interference in the inspection scene. This makes this method more advantageous in real-world inspection environments, providing more stable and reliable pose tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0043] Figure 1 Schematic diagram of absolute pose tracking based on the initial pose and the trained deep learning network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0045] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0046] Example 1

[0047] Traditional methods for endoscopic navigation can be divided into optical positioning tracking and electromagnetic positioning tracking, both of which have been widely used in navigation systems. However, optical positioning tracking is easily affected by occlusions from the human body and equipment, while electromagnetic positioning tracking is easily interfered with by the electromagnetic fields of certain medical instruments. At the same time, the deployment of optical positioning tracking or electromagnetic positioning tracking systems with multiple sensors and computing devices in actual scenarios is expensive and complex. Therefore, vision-based real-time endoscope pose tracking has lower cost and more convenient configuration, and has become an important task in endoscopic navigation. The goal of pose estimation is to estimate the pose of an object from the scene image containing the object, which is impossible for closed and narrow human cavities.

[0048] To solve this problem, Figure 1 As shown, this embodiment provides a method for real-time tracking of the three-dimensional posture of a monocular endoscope based on deep learning, comprising the following steps:

[0049] Set the initial posture of the endoscope and collect endoscopic image sequences in real time;

[0050] For the current frame, the two input images are spliced ​​together to obtain a spliced ​​image, and the optical flow estimation network is used to output the corresponding optical flow map based on the two input images;

[0051] Constructing a deep learning framework, which includes a feature extractor based on a graph convolutional neural network, a multi-dimensional feature extractor based on an attention mechanism, and a pose decoder based on multi-scale depthwise separable convolution;

[0052] A feature extractor based on a graph convolutional neural network extracts scene features from the two input images and motion features from the optical flow map. Meanwhile, a multi-dimensional feature extractor based on an attention mechanism extracts joint features from the spliced ​​images.

[0053] After concatenating all the extracted features, a pose decoder based on multi-scale depthwise separable convolution estimates the relative pose transformation vector of the endoscope at the corresponding moment in the two frames of image.

[0054] Based on the estimated relative pose transformation vector and the absolute 3D pose of the previous frame, the absolute 3D pose of the endoscope at the corresponding moment of each frame image is iteratively determined to achieve real-time tracking of the endoscope.

[0055] As a specific implementation method, first, the initial position of the endoscope is given as p0 = [t0, q0], and the image I obtained by the i-th frame is i (i=1,2,…), apply the deep learning network framework, and use the stored ikth frame of the last sampling to obtain the image I i-k and the corresponding estimated pose Estimate the relative pose transformation of the endoscope corresponding to the two frames of image Then, according to the iterative formula (1), the absolute three-dimensional pose of the endoscope at each frame of image can be determined: Realize the effect of real-time tracking.

[0056]

[0057] in, is the relative translation transformation vector, is the relative rotation transformation vector, is the pose vector at time i, is the known initial pose vector, is the position coordinate vector at the moment ik, is the rotation vector at moment ik.

[0058] As a specific implementation, the constructed deep learning framework includes multiple feature extractors and a pose decoder for extracting the pose of two adjacent frames I k frames apart in the endoscopic video. i-k and Ii To predict the relative posture change of the endoscope First, in the preprocessing stage, the two input images are spliced ​​to obtain the spliced ​​image I c At the same time, the efficient optical flow estimation network outputs the corresponding optical flow map I based on the two input images o In the feature extraction stage, the feature extractor based on graph convolutional neural network For the corresponding scene features f in the two input images i-k and f i , and the motion features f contained in the optical flow map o At the same time, based on the spliced ​​image I c , multi-dimensional feature extractor based on attention mechanism Extract the joint feature f of the two frames of observation images c All features are concatenated and input into the pose decoder based on multi-scale depth-separable convolution In the decoder, the output of the endoscope’s relative pose transformation vector at the corresponding moment of the two frames of image is

[0059] Furthermore, the optical flow estimation network in the preprocessing stage uses the FlowNet network, employing an encoder-decoder architecture to estimate optical flow. The specific process involves taking adjacent image frames I1 and I2 as input, the encoder extracting features through multiple layers of convolution and pooling to produce F1 and F2, and the decoder fusing F1 and F2 and mapping the features back to the optical flow field through operations such as upsampling, outputting the corresponding optical flow map.

[0060] Furthermore, the graph convolutional neural network-based feature extractor uses a model pre-trained on the ImageNet dataset.

[0061] Furthermore, the process of extracting the joint features of two observed images by the multi-dimensional feature extractor based on the attention mechanism includes:

[0062] The extracted features are input into the self-attention mechanism to calculate the relationship between the features. First, the input feature map is flipped to obtain three feature maps with different channel dimension information. and The three flipped feature maps are respectively passed through the channel attention mechanism module Calculate the attention weight. Specifically, the proposed channel attention mechanism module first performs global average pooling on the input feature map at the same time. and global max pooling operation, and then calculate the attention weights on the results of global average pooling and global maximum pooling to obtain the final joint features.

[0063] Example 2

[0064] This embodiment also provides a real-time tracking system for three-dimensional posture of a monocular endoscope based on deep learning, which is based on the method described above and includes: an image acquisition module, an image preprocessing module, a learning framework construction module, a feature extraction module, a posture decoding module, and a posture tracking module;

[0065] The image acquisition module is used to set the initial posture of the endoscope and acquire endoscopic image sequences in real time;

[0066] The image preprocessing module is used to splice the two input images to obtain a spliced ​​image for the current frame, and output a corresponding optical flow map based on the two input images using an optical flow estimation network;

[0067] The learning framework construction module is used to construct a deep learning framework based on a feature extractor of a graph convolutional neural network, a multidimensional feature extractor based on an attention mechanism, and a pose decoder based on multi-scale depth-separable convolution;

[0068] The feature extraction module is used to extract scene features of the two input images and motion features of the optical flow map using a feature extractor based on a graph convolutional neural network; at the same time, a multi-dimensional feature extractor based on an attention mechanism extracts joint features of the spliced ​​images;

[0069] The posture decoding module is used to splice all the extracted features and estimate the relative posture transformation vector of the endoscope at the corresponding moments of two frames of images based on the posture decoder of multi-scale depth-separable convolution;

[0070] The posture tracking module is used to iteratively determine the absolute three-dimensional posture of the endoscope at the corresponding moment of each frame image based on the estimated relative posture transformation vector and the absolute three-dimensional posture of the previous frame, thereby realizing real-time tracking of the endoscope.

[0071] Example 3

[0072] This embodiment further provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.

[0073] Example 4

[0074] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method when executed by a processor.

[0075] Example 5

[0076] This embodiment also provides a computer program product, including a computer program, which implements the steps of the method when executed by a processor.

[0077] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for real-time tracking of three-dimensional pose of a monocular endoscope based on deep learning, characterized in that: The following steps are involved: Set the initial posture of the endoscope and collect endoscopic image sequences in real time; For the current frame, the two input images are spliced ​​together to obtain a spliced ​​image, and the optical flow estimation network is used to output the corresponding optical flow map based on the two input images; Constructing a deep learning framework, which includes a feature extractor based on a graph convolutional neural network, a multi-dimensional feature extractor based on an attention mechanism, and a pose decoder based on multi-scale depthwise separable convolution; A feature extractor based on a graph convolutional neural network extracts scene features from the two input images and motion features from the optical flow map. Meanwhile, a multi-dimensional feature extractor based on an attention mechanism extracts joint features from the spliced ​​images. After concatenating all the extracted features, a pose decoder based on multi-scale depthwise separable convolution estimates the relative pose transformation vector of the endoscope at the corresponding moment in the two frames of image. Based on the estimated relative pose transformation vector and the absolute 3D pose of the previous frame, the absolute 3D pose of the endoscope at the corresponding moment of each frame image is iteratively determined to achieve real-time tracking of the endoscope.

2. The method according to claim 1, characterized in that The optical flow estimation network adopts the FlowNet network, and the architecture is an encoder-decoder structure.

3. The method according to claim 2, characterized in that The process of using the optical flow estimation network to output the corresponding optical flow map based on two input images includes: The encoder performs multi-layer convolution and pooling on the two input images to obtain the corresponding features. The decoder performs feature fusion and upsampling on the obtained features and then maps them back to the optical flow field to obtain the corresponding optical flow map.

4. The method according to claim 1, wherein The graph convolutional neural network uses a model pre-trained on the ImageNet dataset.

5. The method according to claim 1, wherein The process of extracting joint features of the spliced ​​images by the multi-dimensional feature extractor based on the attention mechanism includes: The spliced ​​image is input into the self-attention module, and multiple feature maps with different channel dimension information are obtained through flipping operation; the flipped feature map is input into the channel attention mechanism module, and the channel attention mechanism module simultaneously performs global average pooling and global maximum pooling operations on the flipped feature map, and then calculates the attention weights of the results of global average pooling and global maximum pooling, and finally obtains the joint features of the spliced ​​image.

6. The method according to claim 1, wherein Based on the relative pose transformation vector and the absolute 3D pose of the previous frame, the iterative formula for iteratively determining the absolute 3D pose of the endoscope at each frame of image at the corresponding moment is as follows: in, is the relative translation transformation vector, is the relative rotation transformation vector, is the pose vector at time i, is the known initial pose vector, is the position coordinate vector at the moment ik, is the rotation vector at the ik moment, and the relative posture transformation vector is The absolute 3D pose of the previous frame is 7. A real-time tracking system for three-dimensional pose of a monocular endoscope based on deep learning, characterized in that: The method according to any one of claims 1 to 6, comprising: an image acquisition module, an image preprocessing module, a learning framework construction module, a feature extraction module, a posture decoding module, and a posture tracking module; The image acquisition module is used to set the initial posture of the endoscope and acquire endoscopic image sequences in real time; The image preprocessing module is used to splice the two input images to obtain a spliced ​​image for the current frame, and output a corresponding optical flow map based on the two input images using an optical flow estimation network; The learning framework construction module is used to construct a deep learning framework based on a feature extractor of a graph convolutional neural network, a multidimensional feature extractor based on an attention mechanism, and a pose decoder based on multi-scale depth-separable convolution; The feature extraction module is used to extract scene features of the two input images and motion features of the optical flow map using a feature extractor based on a graph convolutional neural network; at the same time, a multi-dimensional feature extractor based on an attention mechanism extracts joint features of the spliced ​​images; The posture decoding module is used to splice all the extracted features and estimate the relative posture transformation vector of the endoscope at the corresponding moments of two frames of images based on the posture decoder of multi-scale depth-separable convolution; The posture tracking module is used to iteratively determine the absolute three-dimensional posture of the endoscope at the corresponding moment of each frame image based on the estimated relative posture transformation vector and the absolute three-dimensional posture of the previous frame, thereby realizing real-time tracking of the endoscope.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images