A soft tissue real-time tracking and navigation method based on inter-frame deformation field guidance and related device

Abstract

The application provides a soft tissue real-time tracking and navigation method based on inter-frame deformation field guidance and a related device. The method comprises the following steps: S1. acquiring an intraoperative endoscope image sequence, and extracting a current frame and a historical frame; S2. performing dense optical flow estimation on the current frame and the historical frame to obtain an inter-frame deformation field; S3. performing spatial transformation on a historical mask based on the inter-frame deformation field to obtain a deformation compensation segmentation prior; S4. performing prior guided fusion on the segmentation prior and a current frame feature to obtain a refined anatomical segmentation mask; S5. performing key point matching based on the refined mask and a pre-stored atlas to obtain a spatial registration parameter; and S6. rendering a safety boundary based on the registration parameter and the deformation field, and outputting a safety navigation view. The application also provides a related device corresponding to the method, and the related device comprises a device, an electronic device, a computer readable storage medium and a computer program product.

Description

Technical Field

[0001] This application relates to the fields of surgical navigation and medical image processing technology, and in particular to a method and related apparatus for real-time tracking and navigation of soft tissue based on inter-frame deformation field guidance. Background Technology

[0002] In laparoscopic, gastrointestinal, and other soft tissue surgeries, surgeons need to understand the relative position of instrument tips to surrounding anatomical structures in real time to avoid accidental injury to blood vessels and nerves. Current intraoperative navigation methods are mostly based on the registration between preoperative CT or MRI three-dimensional images and intraoperative endoscopic images. However, soft tissues undergo continuous non-rigid deformation during surgery due to factors such as pneumoperitoneum inflation, instrument traction, and respiratory movements. This causes errors in navigation methods based on rigid registration to accumulate rapidly, failing to meet the precision requirements of soft tissue surgery.

[0003] First, existing rigid registration methods directly align preoperative static 3D images with intraoperative images without modeling and compensating for the non-rigid deformation of soft tissues. Under pneumoperitoneum conditions, the deformation of abdominal organs can reach 20 to 50 millimeters, far exceeding the acceptable error range of rigid registration methods. Second, existing methods based on pre-calculated deformation models usually rely on finite element simulation or biomechanical models, which have excessive computational delays that cannot meet the real-time requirements during surgery, and require accurate organ elastic parameters, which are difficult to obtain clinically. Third, existing pose tracking methods based on visual odometry are not robust enough in cavity environments with weak soft tissue surface texture and frequent instrument occlusion, and inter-frame tracking errors are prone to accumulate over time, leading to drift.

[0004] Therefore, there is an urgent need for an intraoperative navigation method that can estimate soft tissue deformation in real time, does not require complex biomechanical models, and can maintain the accuracy of anatomical segmentation through deformation compensation. Summary of the Invention

[0005] To address the above technical issues, this application provides a method and related apparatus for real-time tracking and navigation of soft tissue based on inter-frame deformation field guidance.

[0006] Firstly, a real-time tracking and navigation method for soft tissue based on inter-frame deformation field guidance is provided, comprising the following steps: S1. Acquire an intraoperative endoscopic image sequence, and extract the current frame and at least one historical frame from it, wherein the time interval between the current frame and the historical frame does not exceed a preset maximum inter-frame interval; S2. Perform dense optical flow estimation on the current frame and the historical frames to obtain an inter-frame deformation field. The inter-frame deformation field contains the displacement vector of each pixel from the historical frame to the current frame. The dense optical flow estimation is performed based on a neural network architecture. S3. Based on the inter-frame deformation field, perform spatial transformation on the anatomical segmentation mask of the historical frame to obtain the deformation compensation segmentation prior of the current frame. The spatial transformation maps the anatomical labels of each pixel in the historical frame to the corresponding positions in the current frame along the displacement vector. S4. Perform prior guided feature fusion with the deformation compensation segmentation prior and the depth features of the current frame to obtain a refined anatomical segmentation mask. The prior guided feature fusion constrains the attention weight with confidence. In regions with high prior confidence, the prior map has a higher guiding weight for the segmentation features, while in regions with low prior confidence, the current frame image features have a higher autonomous segmentation weight. S5. Based on the refined anatomical segmentation mask and the pre-stored anatomical atlas, perform sparse keypoint matching to obtain the spatial registration parameters of the current frame. The pre-stored anatomical atlas contains the three-dimensional coordinates of the standard anatomical structure and a feature descriptor subset corresponding to the surgical type. S6. Based on the spatial registration parameters and the inter-frame deformation field, render the anatomical safety boundary in the surgical navigation interface and output a safety navigation view. The safety navigation view overlays colored semi-transparent anatomical segmentation region markings and distance warning information based on the instrument position on the current frame. The distance warning information is calculated based on the Euclidean distance from the instrument tip to the nearest high-risk anatomical structure.

[0007] In any embodiment of this application, the dense optical flow estimation and the prior-guided segmentation refinement shared feature encoder backbone are jointly trained on two tasks to reduce computation and enhance cross-task consistency of feature representations.

[0008] In any embodiment of this application, the deformation compensation segmentation prior generation also includes the calculation of the cumulative deformation field spanning K frames, where K ranges from 2 to 5, in order to handle deformation scenes with larger amplitudes.

[0009] In any embodiment of this application, the sparse keypoint registration further includes Kalman filtering temporal smoothing, and the process noise covariance matrix is ​​dynamically adjusted according to the inter-frame deformation field amplitude.

[0010] In conjunction with any embodiment of this application, the pre-stored anatomical atlas includes specialized atlases for surgical types such as laparoscopic radical resection of colorectal cancer and radical resection of gastric cancer, each containing a set of three-dimensional key points of high-risk anatomical structures.

[0011] In any embodiment of this application, the safety navigation view also displays a deformation intensity heat map thumbnail in the corner area, and the near-field warning threshold ranges from 3 mm to 8 mm.

[0012] In a second aspect, a real-time soft tissue tracking and navigation device based on inter-frame deformation field guidance is provided, comprising: an image sequence acquisition unit 11, a dense optical flow estimation unit 12, a deformation compensation prior generation unit 13, a prior guidance segmentation unit 14, a sparse key point registration unit 15, and a safe boundary rendering output unit 16, which respectively execute steps S1 to S6 of the method described in the first aspect.

[0013] Thirdly, an electronic device is provided, including a processor and a storage unit, the storage unit being used to store computer program code containing computer instructions, wherein when the processor executes these instructions, the electronic device performs the method described in the first aspect.

[0014] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in the first aspect.

[0015] Fifthly, a computer program product is provided, the computer program product comprising a computer program or instructions that, when the computer program or instructions are run on a computer, cause the computer to perform the method described in the first aspect.

[0016] It should be understood that the above general descriptions and subsequent specific descriptions are for illustrative and explanatory purposes only and do not impose any limitations on this application.

[0017] In this application, compared with the prior art, the technical problems to be solved and the non-obvious aspects of this application include: First, this application estimates the inter-frame deformation field of soft tissue based on dense optical flow and transforms the historical frame segmentation mask along the deformation field space to obtain the deformation compensation segmentation prior. This design achieves real-time compensation for non-rigid deformation of soft tissue without establishing a biomechanical model, which is not seen in existing methods based solely on rigid registration; Second, this application uses the deformation compensation segmentation prior as a priori guiding weight to constrain the segmentation attention of the current frame features. The priori guiding weight is increased in regions with high prior confidence, and the image autonomous segmentation weight is increased in regions with low prior confidence. This adaptive prior guiding mechanism is not seen in existing methods with fixed fusion weights; Third, in this application, dense optical flow estimation and anatomical segmentation share the feature encoder backbone and are jointly trained, so that the feature representations of the two tasks are mutually constrained. The spatial structure of the deformation field helps the temporal consistency of the segmentation results, and the segmentation results, in turn... To constrain the semantic rationality of deformation field estimation, the dual-task joint design is not seen in existing methods that independently handle two tasks; fourth, this application extracts sparse keypoints from the high-confidence region of the refined anatomical segmentation mask for registration, making the registration rely only on the reliable segmentation region to establish a 2D-3D correspondence, eliminating interference from instrument occlusion and optical flow estimation failure regions. The selective keypoint matching based on segmentation confidence is not seen in existing methods with uniform sampling across the entire region; fifth, this application renders a deformation field amplitude heatmap in real time in the safe navigation view, enabling the surgeon to perceive the current deformation intensity and adjust their confidence in the navigation results accordingly. The deformation visualization mechanism is not seen in navigation systems; sixth, this application explicitly designs a special technical path for the non-rigid deformation problem in soft tissue surgery, which is fundamentally different from the technical route of existing mainstream navigation methods applicable to rigid registration in orthopedics. It achieves frame-level deformation compensation navigation for the first time in the abdominal soft tissue environment without the need for real-time preoperative CT registration. The related devices described in this application include devices, electronic devices, computer-readable storage media, and computer program products. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be explained below.

[0019] Figure 1 This is a flowchart illustrating a real-time tracking and navigation method for soft tissue based on inter-frame deformation field guidance, provided as an embodiment of this application.

[0020] Figure 2 This is a schematic diagram of the structure of a real-time soft tissue tracking and navigation device based on inter-frame deformation field guidance, provided in an embodiment of this application.

[0021] Figure 3 This is a schematic diagram of the hardware architecture of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] To enable those skilled in the art to more fully understand the technical solution of this application, the technical solution of this application will be explained in detail and clearly with reference to the accompanying drawings.

[0023] Please see Figure 1 , Figure 1 This is a flowchart illustrating a real-time tracking and navigation method for soft tissue based on inter-frame deformation field guidance, provided as an embodiment of this application. The method includes steps S1 to S6.

[0024] In this embodiment, the intraoperative endoscopic image sequence is acquired through a cavity endoscopic imaging system, with a frame rate ranging from 25 Hz to 60 Hz. The historical frames include single historical frames adjacent to the current frame, and may also include a temporal window composed of N historical frames, where N ranges from 1 to 5. The preset maximum inter-frame interval is derived from engineering experience that tissue deformation remains within the optical flow estimation's processing range (single pixel displacement does not exceed 20% of the image width) at normal surgical operation speeds. When a sudden change in image brightness between frames is detected exceeding a preset threshold, the system automatically resets to a repositioning mode based on anatomical atlases to avoid mistracking after lens wiping or rapid movement.

[0025] In this embodiment, the dense optical flow estimation employs an optical flow estimation method based on convolutional neural networks. This method includes, but is not limited to, lightweight variants of FlowNet, PWC-Net, or RAFT. This application does not rely on a specific optical flow network architecture, but requires its inference latency to not exceed 30 milliseconds to meet intraoperative real-time requirements. The inter-frame deformation field is stored in the form of a dense displacement vector field, with each pixel location corresponding to a two-dimensional displacement vector (dx, dy), in pixels. The inter-frame deformation field also includes the optical flow estimation confidence value for each location, obtained based on the uncertainty estimation of the optical flow network. Regions with low confidence (such as instrument obstruction areas) have their weights reduced in subsequent steps.

[0026] In this embodiment, the spatial transformation is performed based on bilinear interpolation: for each category channel of the historical frame dissection segmentation mask, a forward mapping is performed based on the displacement vector in the inter-frame deformation field to generate a prior probability map of that category for the current frame; the prior probability maps of each category constitute the deformation compensation segmentation prior for the current frame. Each position of the deformation compensation segmentation prior also stores the transformation confidence at that position, which is given by the product of the segmentation confidence at that position in the historical frame and the optical flow estimation confidence at that position. The prior probability of regions with transformation confidence below a low transformation confidence threshold (such as instrument occlusion areas or optical flow estimation failure areas) is set to zero to avoid erroneous priors interfering with subsequent refinement steps; the low transformation confidence threshold ranges from 0.2 to 0.4.

[0027] In this embodiment, the depth features of the current frame are extracted through a segmentation branch that shares a backbone network with the dense optical flow estimation. The prior-guided feature fusion is performed based on the following steps: using the confidence map of the deformation-compensated segmentation prior as a spatial attention weight mask; calculating the fusion feature F_fusion = mask * F_prior + (1-mask) * F_image, where F_prior is the prior feature obtained by mapping the prior probability map through the embedding layer, and F_image is the current frame image feature; inputting F_fusion into a lightweight segmentation head, and outputting a refined anatomical segmentation mask. The refined anatomical segmentation mask includes at least one of the following anatomical categories: vascular region, neural region, fascia region, muscle region, and fat region, with each pixel outputting the probability distribution of each category and the overall segmentation confidence.

[0028] In this embodiment, the extraction of the high-confidence region is based on a segmentation confidence threshold, which ranges from 0.7 to 0.9. Key feature points (no fewer than 8 points) are extracted within the high-confidence region. Each feature point contains two-dimensional pixel coordinates and a feature descriptor based on local texture. Nearest-neighbor matching 3D key points are searched in a pre-stored anatomical atlas to establish a 2D-3D point correspondence set. A RANSAC-based PnP algorithm is used to robustly estimate the six-degree-of-freedom spatial registration parameters (rotation matrix R and translation vector t) of the current frame relative to the anatomical atlas from the point correspondence set. The estimation accuracy of the registration parameters is based on reprojection residual evaluation; relocalization is triggered when the average reprojection residual exceeds 5 pixels.

[0029] In this embodiment, the rendering of the safety navigation view is performed based on the following steps: The three-dimensional anatomical structures in the pre-stored anatomical atlas are projected onto the current frame image plane using spatial registration parameters to obtain the projected outline of the anatomical structure; the refined anatomical segmentation mask is fused with the projected outline to generate an anatomical overlay view of the current frame; based on the three-dimensional coordinates of the instrument tip in the anatomical atlas coordinate system, the Euclidean distance to various high-risk structures in the vascular and neural regions is calculated, and distance warning information is output; when the distance from the instrument tip to any high-risk structure is less than the proximity warning threshold, a red warning border is rendered, the proximity warning threshold ranging from 3 mm to 8 mm; the safety navigation view is rendered and output in real time through the endoscope display system, with a rendering delay of no more than 50 milliseconds.

[0030] In laparoscopic total mesorectal resection for rectal cancer, the described method provides real-time tracking and navigation throughout the entire mesentery dissection phase. Approximately 6000 images were acquired during the surgery, with an intraoperative pneumoperitoneum pressure of 12 mmHg and abdominal organ deformation of approximately 30 mm compared to preoperative CT. Step S2 estimated the average amplitude of the inter-frame deformation field at 2.3 pixels / frame. Step S3, with its deformation-compensated segmentation prior, improved the inter-frame consistency of the mesentery segmentation results from 62% to 88% without compensation. Step S4 refined the segmentation mask with an average confidence level of 0.82, achieving a high-confidence region coverage of 75%. Step S5, based on keypoint matching in high-confidence regions, achieved an average reprojection residual of 3.1 pixels, reducing the registration parameter error by 67% compared to rigid registration on preoperative CT. Step S6 rendered the anatomical safety boundary between the rectal mesentery and rectal fascia proper in real time, helping the surgeon accurately identify the protected area of ​​the nerve zone. The entire navigation process was uninterrupted by rendering.

[0031] In laparoscopic radical right hemicolectomy, the method provides focused navigation for the treatment of the ileocolic vessel root. Intraoperative instrument-induced local soft tissue deformation was detected. Step S2 estimated the local deformation field with a maximum amplitude of 15 pixels within approximately 35 frames, corresponding to a physical deformation of approximately 3.5 mm. Step S3's cumulative deformation field (K=3) correctly tracked the segmentation prior of the superior mesenteric vessel region to the new deformed position without drift. Step S4's refined segmentation mask accurately labeled the ileocolic artery course with a segmentation confidence of 0.79. Step S5 extracted 14 keypoints in the high-confidence vessel region, achieving successful registration with an average reprojection residual of 2.8 pixels. Step S6's safe navigation view triggered a distance warning (proximity warning threshold of 8 mm) when the instrument tip was approximately 5 mm from the ileocolic artery, prompting the surgeon to slow down the operation and effectively avoiding the risk of vessel injury.

[0032] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0033] The methods of the embodiments of this application have been described in detail above, and the apparatus of the embodiments of this application is provided below.

[0034] Please see Figure 2 , Figure 2 This application provides a schematic diagram of the structure of a real-time soft tissue tracking and navigation device based on inter-frame deformation field guidance. The device 1 includes: The image sequence acquisition unit 11 is used to acquire intraoperative endoscopic image sequences and extract the current frame and at least one historical frame from them.

[0035] The dense optical flow estimation unit 12 is used to perform dense optical flow estimation on the current frame and the historical frame to obtain the inter-frame deformation field.

[0036] The deformation compensation prior generation unit 13 is used to perform spatial transformation on the anatomical segmentation mask of the historical frame based on the inter-frame deformation field to obtain the deformation compensation segmentation prior of the current frame.

[0037] The prior guidance segmentation unit 14 is used to perform prior guidance feature fusion with the deformation compensation segmentation prior and the depth features of the current frame to obtain a refined anatomical segmentation mask.

[0038] The sparse keypoint registration unit 15 is used to perform sparse keypoint matching based on the refined anatomical segmentation mask and the pre-stored anatomical atlas to obtain the spatial registration parameters of the current frame.

[0039] The safety boundary rendering output unit 16 is used to render the anatomical safety boundary in the surgical navigation interface based on the spatial registration parameters and the inter-frame deformation field, and output the safety navigation view.

[0040] Please see Figure 3 , Figure 3 This is a schematic diagram of the hardware architecture of an electronic device provided in an embodiment of this application. The electronic device 2 includes a processor 21, a memory 22, an input device 23, and an output device 24. The processor 21, the memory 22, the input device 23, and the output device 24 are communicatively connected via a bus. The memory 22 stores computer program code, which includes computer instructions. When the processor 21 executes the instructions, the electronic device 2 executes the aforementioned method for real-time tracking and navigation of soft tissue based on inter-frame deformation field guidance.

[0041] The method described in this application can be implemented based on hardware, software, or a combination of hardware and software.

Claims

1. A real-time tracking and navigation method for soft tissue based on inter-frame deformation field guidance, characterized in that, Includes the following steps: S1. Acquire an intraoperative endoscopic image sequence, extract the current frame and at least one historical frame from it, wherein the time interval between the current frame and the historical frame does not exceed a preset maximum inter-frame interval. S2. Perform dense optical flow estimation on the current frame and the historical frames to obtain an inter-frame deformation field. The inter-frame deformation field contains the displacement vector of each pixel from the historical frame to the current frame. The dense optical flow estimation is performed based on a neural network architecture. S3. Based on the inter-frame deformation field, perform spatial transformation on the anatomical segmentation mask of the historical frame to obtain the deformation compensation segmentation prior of the current frame. The spatial transformation maps the anatomical labels of each pixel in the historical frame to the corresponding positions in the current frame along the displacement vector. S4. Perform prior guided feature fusion with the deformation compensation segmentation prior and the depth features of the current frame to obtain a refined anatomical segmentation mask. The prior guided feature fusion constrains the attention weight with confidence. In regions with high prior confidence, the prior map has a higher guiding weight for the segmentation features, while in regions with low prior confidence, the current frame image features have a higher autonomous segmentation weight. S5. Based on the refined anatomical segmentation mask and the pre-stored anatomical atlas, perform sparse keypoint matching to obtain the spatial registration parameters of the current frame. The pre-stored anatomical atlas contains the three-dimensional coordinates of the standard anatomical structure and a feature descriptor subset corresponding to the surgical type. S6. Based on the spatial registration parameters and the inter-frame deformation field, render the anatomical safety boundary in the surgical navigation interface and output a safety navigation view. The safety navigation view overlays colored semi-transparent anatomical segmentation region markings and distance warning information based on the instrument position on the current frame. The distance warning information is calculated based on the Euclidean distance from the instrument tip to the nearest high-risk anatomical structure.

2. The method according to claim 1, characterized in that, The dense optical flow estimation is fused with the prior guided features to share a feature encoder backbone. The multi-granularity features extracted by the shared feature encoder backbone are used for both optical flow estimation and segmentation refinement. The parameters of the shared feature encoder backbone are jointly trained on the two tasks to reduce computation and enhance the cross-task consistency of feature representation.

3. The method according to claim 1, characterized in that, Step S3 further includes performing temporal accumulation on the inter-frame deformation field: when the image sequence contains K historical frames, the deformation fields from K frames to the current frame are combined and transformed sequentially to obtain the accumulated deformation field spanning K frames, and the mapping from the historical segmentation mask to the current frame is performed based on the accumulated deformation field; the value of K ranges from 2 to 5.

4. The method according to claim 1, characterized in that, The sparse keypoint matching also includes temporal filtering of the registration parameters: the estimated registration parameters of the current frame are fused with the predicted values ​​of the registration parameters of the previous frame using Kalman filtering to eliminate the impact of single-frame registration errors on navigation stability; the noise covariance matrix of the Kalman filtering process is dynamically adjusted according to the amplitude of the inter-frame deformation field, and the larger the amplitude, the greater the process noise.

5. The method according to claim 1, characterized in that, The pre-stored anatomical atlas includes at least one of the following surgical types: laparoscopic total mesorectal resection, laparoscopic radical right hemicolectomy, and laparoscopic radical distal gastrectomy; each atlas contains a set of three-dimensional key points and spatial topological relationships of the corresponding high-risk anatomical structures in the surgery.

6. The method according to claim 1, characterized in that, The rendering of the safe navigation view also includes visualizing the deformation compensation error: generating a deformation intensity heatmap based on the amplitude of the inter-frame deformation field, and displaying the deformation intensity heatmap in the corner area of ​​the safe navigation view in the form of a thumbnail, so that the operator can perceive the deformation intensity of the current frame in real time and its potential impact on registration accuracy.

7. The method according to claim 1, characterized in that, The method is used for at least one of the following downstream applications: real-time visual navigation for neurovascular protection during laparoscopic radical resection of colorectal cancer, layered depth labeling assistance for endoscopic mucosal dissection of soft tissue in cavities, and continuous distance monitoring and early warning from the instrument tip to important anatomical structures during gastrointestinal endoscopy.

8. A real-time soft tissue tracking and navigation device based on inter-frame deformation field guidance, characterized in that, include: Image sequence acquisition unit (11) is used to perform step S1 as described in claim 1; Dense optical flow estimation unit (12) is used to perform step S2 as described in claim 1; Deformation compensation prior generation unit (13) is used to perform step S3 as described in claim 1; A priori guidance segmentation unit (14) is used to perform step S4 as described in claim 1; A sparse keypoint registration unit (15) is used to perform step S5 as described in claim 1; A security boundary rendering output unit (16) is used to perform step S6 as described in claim 1.

9. An electronic device, characterized in that, include: A processor and a storage unit for storing computer program code, the code containing computer instructions, wherein when the processor executes these instructions, the electronic device performs the method of any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program containing program instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1 to 6.

11. A computer program product, characterized in that, The computer program product includes a computer program or instructions that, when executed on a computer, cause the computer to perform the method described in any one of claims 1 to 6.

Images