A Surgical Navigation Method and System for Dynamic Soft Tissue

By reconstructing local spatiotemporal windows and quantitatively managing depth confidence, combined with optical flow constraints and biomechanical constraints, and optimizing Gaussian anchor point parameters, the problems of low reconstruction efficiency and low accuracy in dynamic soft tissue surgical navigation are solved. This enables real-time and accurate navigation and linked updates of deep risk structures, improving the safety and real-time performance of minimally invasive surgery.

CN123058832APending Publication Date: 2026-11-10WANLAN ZHIYING (CHENGDU) TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611228601.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-13
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing surgical navigation technologies are inefficient and inaccurate in dynamic soft tissue reconstruction, failing to meet the real-time and safety requirements of minimally invasive surgery. Furthermore, they struggle to achieve synchronized updates between soft tissue surface deformation and deep risk structures, and lack effective occlusion assessment and display strategies, leading to increased navigation errors and risks.

Method used

By reconstructing local spatiotemporal windows, quantifying depth confidence, and managing anchor points in a refined manner, combined with optical flow constraints, instrument contact events, and soft tissue biomechanical prior constraints, Gaussian anchor point parameters are optimized to achieve real-time updates and visualization rendering of surface displacement fields and deep risk structures, and adaptive adjustment of occlusion states.

Benefits of technology

It improves the accuracy and real-time performance of surgical navigation, meets the comprehensive needs of dynamic soft tissue navigation and deep risk protection in minimally invasive surgery, and enhances the scenario adaptability and continuous navigation effect of the navigation system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123058832A_ABST
    Figure CN123058832A_ABST
Patent Text Reader

Abstract

This invention discloses a surgical navigation method and system for dynamic soft tissue, relating to the fields of medical image processing and surgical navigation technology. The method includes: real-time generation of Gaussian anchor points; identifying regions within a local spatiotemporal window where the depth confidence of corresponding pixels exceeds a preset threshold as valid reliable regions; generating new Gaussian anchor points based on these regions to obtain updated Gaussian anchor points; iteratively optimizing the parameters of the Gaussian anchor points based on optical flow constraints, instrument contact event constraints, and soft tissue biomechanical prior constraints to obtain a Gaussian field; registering the Gaussian field with an organ surface model to obtain a surface displacement field; updating the three-dimensional spatial position of deep risk structures based on the surface displacement field using radial basis function interpolation; and visually rendering the updated deep risk structures to generate enhanced navigation images. This invention meets the comprehensive needs of minimally invasive surgery for dynamic soft tissue navigation, deep risk protection, and continuous navigation under occlusion conditions.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of medical image processing and surgical navigation technology, and in particular to a surgical navigation method and system for dynamic soft tissues. Background Technology

[0002] Minimally invasive surgery, with its advantages of small incisions, rapid recovery, and fewer complications, has been widely used in clinical fields such as abdominal, thoracic, and orthopedic procedures. Precise intraoperative navigation is crucial for improving surgical safety and reducing intraoperative damage. In minimally invasive surgery, the surgical field of view relies on the monocular video stream from an endoscope or laparoscope. However, soft tissues exhibit dynamic deformation, are susceptible to interference from instrument contact, and their surgical field of view is easily obstructed by instruments, posing numerous challenges to intraoperative navigation.

[0003] In existing surgical navigation technologies, 3D reconstruction of dynamic soft tissues often adopts a global reconstruction approach, which fails to consider the local characteristics of surgical operations. This results in low reconstruction efficiency, high resource consumption, and difficulty in adapting to the dynamic deformation of soft tissues in real time. In the pseudo-depth estimation process, there is a lack of an effective depth confidence assessment mechanism, which makes it impossible to quantify reconstruction uncertainty and easily leads to navigation errors. Gaussian anchor point management often adopts a global maintenance strategy without establishing local spatiotemporal windows. The anchor point lifecycle management is unreasonable, which easily leads to the accumulation of redundant anchor points or the loss of effective anchor points, affecting the accuracy of deformation field optimization.

[0004] Meanwhile, existing technologies struggle to achieve synchronized updates between soft tissue surface deformation and deep risk structures, often focusing only on surface reconstruction while neglecting the dynamic positional changes of deep risk structures (such as blood vessels and nerves), thus increasing the risk of surgical injury. When surgical instruments obstruct the field of vision, the lack of an effective obstruction assessment and adaptive adjustment mechanism for display strategies leads to missing or reduced navigation information in the obstructed area, failing to provide reliable navigation guidance for the surgeon.

[0005] Furthermore, existing navigation systems have independently designed functional modules that lack coordination and linkage, making it impossible to achieve integrated video preprocessing, depth estimation, deformation optimization, navigation rendering, and display control. This makes it difficult to simultaneously address core pain points such as dynamic soft tissue deformation, instrument occlusion, and lack of deep risk information, further restricting the real-time performance and accuracy of navigation.

[0006] Therefore, a surgical navigation method and system for dynamic soft tissue was developed to solve the above problems. Summary of the Invention

[0007] This invention proposes a surgical navigation method and system for dynamic soft tissues to solve the problems of low reconstruction efficiency and low navigation accuracy in existing technologies, which cannot meet the comprehensive requirements of minimally invasive surgery for navigation accuracy, real-time performance and safety.

[0008] The present invention achieves the above objectives through the following technical solutions:

[0009] This invention provides a surgical navigation method for dynamic soft tissue, comprising:

[0010] Acquire video streams of the intraoperative surgical field and preoperative 3D medical images;

[0011] The surface model of the organ is segmented from the preoperative three-dimensional medical image, and the deep risk structures inside the organ are marked.

[0012] Preoperative 3D medical images were segmented using the ITK-SNAP / 3D Slicer model to create organ surface models. The surgeon then used enhanced CT / MRI to grow and delineate the liver outline layer by layer / region by region (described in S41 of the embodiment). Deep risk structures can also be annotated using ITK-SNAP / 3D Slicer and delineated on preoperative images, with each anatomical structure assigned an integer label value.

[0013] Perform reflection suppression, smoke removal, and instrument instance segmentation on each frame of the video stream, and output an enhanced tissue surface image and an instrument region mask;

[0014] Each tissue surface image is input frame by frame into a monocular depth estimation network, which outputs a corresponding dense pseudo-depth map. A depth confidence map is generated based on the tissue surface image and the dense pseudo-depth map.

[0015] Extract the instrument tip from the instrument region mask and construct a local spatiotemporal window centered on the instrument tip in the three-dimensional pixel space;

[0016] Within a local spatiotemporal window, Gaussian anchor points are generated in real time based on the dense pseudo-depth map and depth confidence map corresponding to each frame of the image. Regions within the local spatiotemporal window whose depth confidence of the corresponding pixel is greater than a preset threshold are identified as valid reliable regions. New Gaussian anchor points are generated based on the valid reliable regions to obtain the updated Gaussian anchor points of the corresponding image.

[0017] The parameters of the Gaussian anchor point are iteratively optimized based on optical flow constraints, instrument contact event constraints, and soft tissue biomechanical prior constraints to obtain the Gaussian field.

[0018] The Gaussian field is registered with the organ surface model to obtain the surface displacement field. Based on the surface displacement field, the three-dimensional spatial position of the deep risk structure is updated using the radial basis function interpolation method. The updated deep risk structure is then visualized and rendered to generate an enhanced navigation image.

[0019] Furthermore, based on the tissue surface image and the dense pseudo-depth map, a depth confidence map is generated, including:

[0020] Calculate the confidence scores for photometric consistency and texture richness based on tissue surface images;

[0021] Calculate the network uncertainty confidence of the monocular depth estimation network based on the dense pseudo-depth map;

[0022] The depth confidence map is obtained by weighted fusion of the photometric consistency confidence, texture richness confidence, and network uncertainty confidence.

[0023] Furthermore, the instrument tip is extracted from the instrument region mask, and a local spatiotemporal window centered on the instrument tip is constructed in three-dimensional space, including:

[0024] Two-dimensional pixel coordinates of the instrument tip are extracted from the instrument region mask based on connected component analysis;

[0025] The two-dimensional pixel coordinates of the instrument tip are converted into a three-dimensional spatial position based on the dense pseudo-depth map;

[0026] A local spatiotemporal window is established in three-dimensional space, centered on the tip of the surgical instrument, and based on the preset radius of the spatial interest region.

[0027] Furthermore, when the distance between the deep risk structure and the instrument tip is less than a preset safety threshold, the radius of the spatial domain of interest is adaptively adjusted. The expression for this adaptive adjustment is:

[0028] in, The radius of the spatial interest domain preset for the current operation phase. The minimum radius required to cover deep-seated risk structures.

[0029] Furthermore, after generating Gaussian anchor points in real time, the process also includes maintaining the Gaussian anchor points within the local spatiotemporal window, specifically including:

[0030] Configure independent age and maturity parameters for all Gaussian anchor points within a local spatiotemporal window;

[0031] When the Gaussian anchor point is within the effective reliable region of the local spatiotemporal window, the age parameter of the Gaussian anchor point is forcibly reset to the maximum value of the preset age parameter, and the maturity parameter is set to increase iteratively frame by frame according to the preset formula.

[0032] When the Gaussian anchor point leaves the local spatiotemporal window, the age parameter is set to decrease automatically frame by frame, and the anchor point enters a dormant standby state.

[0033] When the age parameter of an anchor point decays to less than or equal to 0, the anchor point is determined to be invalid and has no observation value in the long term, and a permanent deletion operation is performed on the anchor point.

[0034] Furthermore, the construction process based on optical flow constraints, instrument contact event constraints, and soft tissue biomechanical prior constraints is as follows:

[0035] Optical flow constraints are constructed, which include projecting Gaussian anchor points onto a two-dimensional image plane based on the Lucas-Kanade optical flow algorithm and tracking the corresponding position of the projected Gaussian anchor points in the next frame to obtain the displacement vector of the Gaussian anchor points. When the optical flow brightness residual or forward and backward optical flow consistency error of the projected point exceeds the preset optical flow error threshold, the displacement vector is determined to be invalid and discarded. The corresponding Gaussian anchor points are then updated by interpolation of adjacent valid anchor points or soft tissue biomechanical prior constraints.

[0036] The instrument contact event constraints are constructed, which include calculating the directed distance d between the instrument tip and the tissue surface based on the tissue surface image and the position of the instrument tip, and calculating the maximum depth gradient magnitude G_max within a preset small region Ω centered on the instrument tip. The calculation formula is as follows:

[0037]

[0038] in, This represents the depth value at position p. For position Depth gradient at that location;

[0039] When both conditions are met < and > When a contact event between the instrument and tissue is determined, among which... Distance threshold Gradient threshold;

[0040] After a contact event is triggered, all Gaussian anchor points within a preset range around the contact point are marked as pending refresh. The anchor point weights are updated, and the maturity parameters are reset. The expression for updating the Gaussian anchor point weights is:

[0041]

[0042] The weight of the Gaussian anchor point after the contact event is triggered. The weight of the Gaussian anchor point prior to the contact event being triggered. For the opacity increment;

[0043] The weight of a Gaussian anchor is the anchor's opacity / rendering contribution weight. It's used for image rendering. When the instrument presses on the tissue, the local deformation is large, and the original anchor point is no longer reliable, so these anchor points are... Adding a fixed increment Δw_contact allows them to obtain larger gradients in subsequent iterations and be refreshed and reconstructed faster, thus quickly adapting to local deformations.

[0044] Constructing biomechanical prior constraints for soft tissue, including introducing a hybrid physical prior model as a regularization term. The specific calculation formula is as follows:

[0045]

[0046] in, The elastic smoothing term penalizes the second-order gradient of the displacement vector between adjacent Gaussian anchor points, and is defined as:

[0047]

[0048] Where N is the set of spatially adjacent Gaussian anchor pairs, Let i be the displacement vector of the i-th Gaussian anchor point. As a volume conservation term, the rate of change of volume in the constrained local tissue region approaches zero, and the calculation formula is:

[0049]

[0050] in, Let $e$ be the deformation gradient Jacobian matrix of the local element $e$. As a low-dimensional motion basis constraint, the current deformation field is projected onto a low-dimensional motion basis extracted from historical deformation data, penalizing the projection reconstruction error. The calculation formula is as follows:

[0051]

[0052] in, It is a low-dimensional motion basis matrix, extracted from historical deformation data by principal component analysis or autoencoder;

[0053] , , These are the weights of the elastic smoothing term, the volume conservation term, and the low-dimensional motion basis constraints, respectively.

[0054] Furthermore, the parameters of the Gaussian anchor point are iteratively optimized based on optical flow constraints, instrument contact event constraints, and soft tissue biomechanical prior constraints, including:

[0055] Construct the loss function for the gradient descent iterative strategy:

[0056]

[0057] For optical flow constraint loss, Constraint loss for instrument contact events, For soft tissue biomechanical prior constraint loss;

[0058] A gradient descent iterative strategy is used to perform frame-by-frame reverse correction of the position, covariance, color and opacity parameters of Gaussian anchor points and to update anchor point parameters in real time.

[0059] In each iteration, the following operations are performed:

[0060] The Lucas–Kanade optical flow algorithm is used to project Gaussian anchor points onto a two-dimensional image plane and track their corresponding positions in the next frame to obtain the displacement vector. ,by Substitute the data items into the loss function of the gradient descent iterative strategy. ;

[0061] When it is determined that the instrument has come into contact with the tissue, the opacity of the anchor points within a preset range around the contact point is increased and their corresponding weights are raised. This allows for priority reconstruction of local deformation at the anchor points in the contact region, and the local deformation is then substituted as a data term into the loss function of the gradient descent iterative strategy. ;

[0062] regularization terms Substituting into the loss function of the gradient descent iterative strategy ;

[0063] The process is iterated until the loss converges, and the optimal Gaussian anchor point parameters corresponding to each frame are obtained. Based on the optimized optimal Gaussian anchor point parameters, continuous frame-to-frame temporal iterative accumulation is performed using 4D Gaussian splashing technology to obtain the four-dimensional deformation field.

[0064] Furthermore, when visualizing the updated deep risk structure, the visualization of the deep risk structure is based on reconstruction uncertainty, including:

[0065] Based on the depth confidence map and the optimized residual of the four-dimensional deformation field, the reconstruction uncertainty of each region in space is calculated using the following formula:

[0066] σ = √(1-C) × Res / Res_max;

[0067] C represents the depth of confidence. To optimize residuals for a four-dimensional deformation field, The upper limit threshold of the residual is σ, and the range of the reconstruction uncertainty σ is [0,1]. The larger the σ is, the higher the reconstruction uncertainty.

[0068] Transparency Modulation: Opacity in Deep Risk Structure Rendering With Reconstruction Uncertainty The following relationship must be satisfied:

[0069] ;

[0070] in, The preset minimum visible transparency has a value range of (0, 0.3]. The upper limit threshold for uncertainty; k is the kurtosis control index of the modulation curve, k≥1; when As the risk increases, the risk structure becomes more transparent but still retains a level no lower than [a certain percentage]. Visibility;

[0071] Boundary diffusion display: The outline of deep risk structures or safety boundaries is rendered as a diffusion band, and the width W of the diffusion band is proportional to the reconstruction uncertainty, calculated as follows:

[0072] ;

[0073] Where W0 is the base width, η is the diffusion coefficient, and σ is the reconstruction uncertainty; the color saturation within the diffusion band decreases linearly or non-linearly from the inside to the outside, and the higher the uncertainty, the more blurred the visual effect of the boundary.

[0074] Color coding: In the risk heatmap mode, a mapping function is established from cool colors with high uncertainty to warm colors with low uncertainty. The specific determination rule is as follows:

[0075] when < When this occurs, a highly saturated warning color will be used for display;

[0076] when ≥ and ≤ At that time, color saturation increased with Increase linearly decreasing;

[0077] when > When the saturation drops to a preset minimum or gradually changes to a cool color;

[0078] in, For low uncertainty threshold, For high uncertainty threshold, < .

[0079] Furthermore, it also includes occlusion status assessment and adaptive adjustment of display strategy, specifically including:

[0080] The occlusion area assessment includes calculating the proportion of the occluded area to the total area of ​​the current frame based on the instrument area mask. When it exceeds a preset threshold, the enhanced navigation image display mode is automatically adjusted.

[0081] The overall reconstruction quality assessment includes the construction of overall reconstruction quality indicators. Overall reconstruction quality indicators The calculation formula is:

[0082]

[0083] in, The total number of pixels in the image. For all pixel-level depth confidence, To optimize residuals for a four-dimensional deformation field, The upper limit threshold for residuals. , For the weighting coefficients, satisfying + =1;

[0084] Display strategy adaptively adjusts: when Below the preset threshold At that time, the safety boundary is offset outward by a conservative bias amount. The calculation formula is:

[0085]

[0086] in, Based on the bias, This is the scaling factor.

[0087] The present invention also provides a system for a surgical navigation method for dynamic soft tissue, comprising:

[0088] The acquisition module is used to acquire video streams of the surgical field during surgery and preoperative three-dimensional medical images;

[0089] The segmentation module is used to segment the surface model of organs from preoperative three-dimensional medical images and label the deep risk structures within the organs.

[0090] An enhancement module is used to perform reflection suppression, smoke removal, and instrument instance segmentation on each frame of the video stream, and output an enhanced tissue surface image and an instrument region mask.

[0091] The confidence module is used to input the surface images of each tissue frame by frame into the monocular depth estimation network, output the corresponding dense pseudo-depth map, and generate a depth confidence map based on the tissue surface images and the dense pseudo-depth map.

[0092] A construction module is used to extract the instrument tip from the instrument region mask and construct a local spatiotemporal window centered on the instrument tip in a three-dimensional pixel space.

[0093] The generation module is used to generate Gaussian anchor points in real time within a local spatiotemporal window based on the dense pseudo-depth map and depth confidence map corresponding to each frame of the image. The region with the depth confidence of the corresponding pixel within the local spatiotemporal window that is greater than a preset threshold is determined as a valid reliable region. New Gaussian anchor points are generated based on the valid reliable regions to obtain the updated Gaussian anchor points of the corresponding image.

[0094] The optimization module is used to iteratively optimize the parameters of the Gaussian anchor point based on optical flow constraints, instrument contact event constraints, and soft tissue biomechanical prior constraints to obtain a Gaussian field.

[0095] The rendering module is used to register the Gaussian field with the organ surface model to obtain the surface displacement field. Based on the surface displacement field, the three-dimensional spatial position of the deep risk structure is updated using the radial basis function interpolation method. The updated deep risk structure is then visualized and rendered to generate an enhanced navigation image.

[0096] The beneficial effects of this invention are as follows:

[0097] This invention proposes a surgical navigation method and system for dynamic soft tissues, which achieves the synergistic integration of local spatiotemporal window reconstruction, depth confidence quantification, and anchor point fine management. It breaks through the bottlenecks of existing navigation technologies in dynamic soft tissue adaptation, occlusion scenario response, and deep risk structure linkage updates, thereby improving the accuracy, real-time performance, and scenario adaptability of surgical navigation. It meets the comprehensive needs of minimally invasive surgery for dynamic soft tissue navigation, deep risk protection, and continuous navigation under occlusion conditions. Attached Figure Description

[0098] Figure 1 This is a flowchart of the surgical navigation method for dynamic soft tissue described in this application. Detailed Implementation

[0099] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0100] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0101] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0102] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0103] like Figure 1 As shown, this embodiment provides a surgical navigation method for dynamic soft tissue, including:

[0104] S1, Intraoperative Video Stream Acquisition and Preprocessing, is used to acquire laparoscopic video streams during liver surgery and perform image enhancement and instrument segmentation to provide high-quality input for subsequent depth estimation and reconstruction. Specifically, it includes the following steps:

[0105] S11, Video Acquisition: Using an Olympus OTV-S190 monocular laparoscope, the resolution was set to 1920×1080P, the frame rate to 30fps, and the lens focal length to 25mm. During the operation, the video stream of liver tissue, surgical instruments (ultrasonic scalpel, dissecting forceps) and surrounding tissues was acquired in real time and transmitted to a medical image processing server (configured with an Intel Core i9-13900K processor and an NVIDIA RTX 4090 GPU, supporting CUDA acceleration). The transmission was carried out using an HDMI 2.1 interface, and the latency was controlled within 50ms to ensure the real-time performance of the video stream.

[0106] S12, Image Enhancement: To address the issues of liver tissue reflection and image blurring caused by intraoperative smoke interference during liver surgery, each acquired image frame is enhanced. Reflection suppression employs a homomorphic filtering algorithm with a Gaussian filter kernel size of 5×5, a low-frequency component weight of 0.3, and a high-frequency component weight of 0.7, effectively suppressing surgical light reflection on the liver capsule surface. Smoke removal utilizes a dark channel prior algorithm with 10 iterations and a filter radius of 3, restoring liver tissue details in smoke-occluded areas and outputting an enhanced liver tissue surface image.

[0107] S13, Instrument Instance Segmentation: The U-Net++ deep learning network is used to segment surgical instruments in the enhanced image. The segmentation categories include ultrasonic scalpels and dissecting forceps. The network input is a 1080P image, and the output is an instrument region mask (binary image, with the instrument region having a pixel value of 255 and the background region having a value of 0). The segmentation accuracy is ≥95%, the inference speed is ≥30fps, and the position of the surgical instruments is marked in real time, providing a basis for subsequent instrument contact detection and occlusion assessment.

[0108] S2, a pseudo-depth estimation, depth confidence generation, and local spatiotemporal window-based Gaussian anchor point management system, combined with an optimized 4DGS model, achieves high-fidelity real-time reconstruction of liver soft tissue while simultaneously initializing and maintaining anchor points.

[0109] In S2, after obtaining the dense pseudo-depth map, the confidence level of the depth confidence map is determined based on at least one of the following factors: the photometric consistency calculated from the enhanced tissue surface image, the texture richness calculated from the enhanced tissue surface image, and the network uncertainty estimate output by the monocular depth estimation network. A weighted summation method is then used to generate pixel-level confidence levels, specifically as follows:

[0110] (1) Calculate the confidence level of photometric consistency:

[0111]

[0112] in, This represents an image block of a preset size centered at pixel coordinates (x, y) in the enhanced tissue surface image. and Let represent the mean values ​​of the left and right image patches, respectively, and d represent the disparity of the dense pseudo-depth map. The value ranges from [0,1]. A larger value indicates better photometric consistency and higher confidence.

[0113] (2) Texture richness confidence calculation: The local texture intensity is quantified by gradient magnitude analysis and used as the texture richness confidence. The calculation formula is as follows:

[0114]

[0115] Where ∂I / ∂u and ∂I / ∂v represent the partial derivatives of the enhanced tissue surface image in the u and v directions, respectively. The larger the G value, the richer the texture and the higher the confidence. After normalization, the value range is [0,1].

[0116] (3) Calculation of confidence level for network uncertainty: Monte Carlo Dropout technique is used to obtain the deep prediction variance through multiple forward propagations. The confidence level is inversely proportional to the variance. The calculation formula is as follows:

[0117]

[0118] Where σ is the standard deviation of multiple depth predictions from the dense pseudo-depth map. The value ranges from [0,1]. A larger value indicates lower uncertainty and higher confidence in network prediction.

[0119] (4) Multi-factor weighted fusion: The three confidence levels mentioned above are weighted and summed to obtain the final pixel-level depth confidence level. The calculation formula is as follows:

[0120]

[0121] Among them, w1, w2, and w3 are weighting coefficients that satisfy w1 + w2 + w3 = 1. w1, w2, and w3 can be dynamically adjusted according to the surgical scenario. Calculated from enhanced tissue surface images, This is the normalized result of the gradient magnitude G. .

[0122] The specific construction process of the local spatiotemporal window in S2 is as follows:

[0123] Window center determination: A binary mask of the instrument region in the current frame is output by the instrument instance segmentation network. This mask undergoes multi-level image post-processing, including morphological denoising, skeletal midline extraction, distal pixel filtering, and sub-pixel precision correction, to accurately obtain the high-precision two-dimensional coordinates of the surgical instrument tip. Combined with the dense pseudo-depth map output in step 2, the two-dimensional pixel coordinates of the instrument tip are mapped back to a three-dimensional position. A spatial region of interest is established with this three-dimensional position as the center and radius R, serving as the local spatiotemporal window. When no instrument is detected within the field of view, the process reverts to a global window centered on the center of the field of view or a preset anatomical region of interest, ensuring continuous and uninterrupted reconstruction. This method accurately locks the center of the local spatiotemporal window, providing a precise spatial reference for subsequent localized Gaussian anchor point initialization and temporal optimization.

[0124] The formula for mask morphology denoising preprocessing is as follows:

[0125]

[0126] in, A clean instrument mask after noise reduction; For erosion calculation, Dilation operation, It is a 3×3 square structural element (adapted to the noise scale of 1080P intraoperative images).

[0127] Ultrasonic scalpel skeleton central axis extraction: The ultrasonic scalpel is a slender, rigid instrument, with its tip located at the farthest point of the instrument's central axis. An image thinning algorithm is used to extract the skeleton from the denoised mask, compressing the instrument region to a single-pixel width of the central axis. This accurately represents the instrument's extension direction and eliminates interference from the instrument's contour width on tip positioning. The instrument tip is the endpoint pixel of the instrument skeleton that is farthest from the image origin and is free and unobstructed. By calculating the Euclidean distance of all skeleton pixels, the pixel with the largest distance is selected as the initial coordinates of the tip. The formula for calculating the single-pixel Euclidean distance is:

[0128]

[0129] in,( , The reference origin for intraoperative images is set as the top left corner of the image or the center of the surgical field of view, which is fixed globally. For skeleton pixels ( , Distance weights are assigned. The process iterates through all valid skeleton pixels and filters for the initial pixel coordinates of the tip.

[0130]

[0131] in, The initial two-dimensional coordinates of the pixel-level discrete tip.

[0132] To overcome the positioning error of integer pixel coordinates and improve the positioning accuracy of the window center, a neighborhood gray-level weighted algorithm is used to perform sub-pixel optimization on the initial tip coordinates, resulting in high-precision continuous two-dimensional coordinates. The formula is as follows:

[0133]

[0134] in, The grayscale value of the enhanced tissue image output from step 1; the value neighborhood is a 3×3 pixel range from the initial coordinates of the tip; the final output This represents the two-dimensional coordinates of the ultrasonic scalpel tip with sub-pixel-level high precision.

[0135] The specific implementation of anchor lifecycle management in S2 is as follows:

[0136] Each Gaussian anchor point carries an age parameter A and a maturity parameter M, used to characterize the stability of the Gaussian anchor point in time-series observations. A higher M value indicates that the anchor point's characterization of tissue surface geometry and appearance is more mature and stable. Specific management rules are as follows:

[0137]

[0138]

[0139]

[0140] Specifically, when an anchor point is within the spatial region of interest and is effectively observed in the current frame, A is reset to its maximum value Amax, and M increases according to the above formula; when an anchor point leaves the spatial region of interest, A decreases frame by frame and enters a dormant state; if A≤0, the anchor point is permanently deleted; the initial value of M for newly initialized anchor points is 0, the covariance matrix of Gaussian anchor points is enlarged to Σinit, and the weights are reduced to... , Let ΔM be the maximum maturity value, ΔM be the maturity increment, and Σbase be the base covariance matrix of the Gaussian anchor point. β is the base weight of the Gaussian anchor point, β is the covariance amplification factor used to amplify the covariance matrix of the newly initialized anchor point, and γ is the weight reduction factor used to reduce the weight of the newly initialized anchor point and reduce its interference with the initial rendering result. The base covariance matrix of the Gaussian anchor point is preset according to the accuracy requirements of the surgical scene.

[0141] The adaptive adjustment of the spatial interest domain radius in step S2 is specifically as follows:

[0142] When the distance between the deep risk structure and the instrument tip is less than a preset safety threshold, the radius of the spatial domain of interest is adaptively adjusted. The expression for the adaptive adjustment is:

[0143]

[0144] in, The radius of the spatial interest domain preset for the current operation phase. The minimum radius required to cover deep-seated risk structures.

[0145] In this embodiment, S2 specifically includes the following steps:

[0146] S21, Depth Estimation: Based on the monocular depth estimation network MiDaS v3, the appearance parameter iteration strategy of Gaussian anchor points is adjusted, the weight of liver tissue texture features is increased, and the anchor point covariance update rate is optimized to adapt to the characteristics of complex deformation and uneven texture of liver soft tissue. Pseudo-depth estimation is performed on the enhanced liver tissue image output by S1, and a dense pseudo-depth map with a resolution of 1920×1080P is output. The depth measurement range is 0-10cm, and the depth error is ≤0.3mm. At the same time, through the optimized 4D Gaussian sputtering model, the small deformations and motion trajectories of liver tissue are accurately captured.

[0147] S22, Confidence Fusion: Calculate pixel-level depth confidence, specifically: use the ZNCC method to calculate photometric consistency confidence, set the image block N{x,y} size to 7×7, and the disparity d range to 0-5px; use gradient magnitude analysis to calculate texture richness confidence, calculate the partial derivatives of the image in the x and y directions using the Sobel operator, and after normalization, the Gnorm value range is [0,1]; use Monte Carlo Dropout technology to calculate the network's own uncertainty confidence, set the Dropout probability to 0.2, the number of forward propagation times to 10, and calculate the standard deviation σ; during weighted fusion, adjust the weight coefficients w1=0.4 (photometric consistency), w2=0.3 (texture richness), and w3=0.3 (network uncertainty) in conjunction with the liver disease surgery scenario, satisfying w1+w2+w3=1, and output the pixel-level depth confidence map C;

[0148] S23, Window establishment: Extract the two-dimensional coordinates (m,n) of the ultrasonic scalpel tip, combine with the pseudo-depth map output from S21, restore the three-dimensional position (m,n,z) of the tip, take this three-dimensional position as the center to establish a local spatiotemporal window (spatial region of interest), and adaptively adjust the radius R according to the liver disease surgical operation phase: take R2=10mm in the exploration phase (liver tissue exposure), take R1=5mm in the fine separation phase (liver parenchyma separation, blood vessel exposure), where R1<R2; when no ultrasonic scalpel is detected in the field of view, fall back to the global window centered at the center of the liver surgical region (preset anatomical region of interest), with the radius set to 15mm; when the distance between the deep risk structure (right branch of portal vein) and the instrument tip is <Dsafe=3mm, follow the formula to adjust the window radius, where the minimum radius required to cover the right branch of portal vein Rrisk=8mm;

[0149] S24, Anchor point management: Maintain Gaussian anchor points within the local spatiotemporal window, set the maximum anchor point age Amax=10 frames, maturity increment ΔM=0.1, maximum maturity Mmax=1.0, covariance amplification factor β=1.5, weight reduction coefficient γ=0.8, base covariance matrix Σbase=0.01mm², base weight wbase=1.0; the initial M value of a newly initialized anchor point is 0, the covariance matrix is amplified to Σinit=β·Σbase=0.015mm², and the weight is reduced to winit=γ·wbase=0.8; when the anchor point is located within the window and is effectively observed by the current frame (confidence C≥0.5), A is reset to 10, and M follows the formula to increment; when the anchor point leaves the window, A decreases frame by frame (subtract 1 per frame) and enters a dormant state; if A≤0, the anchor point is permanently deleted; through the optimized 4DGS model, the anchor point update frequency is increased to 30fps, which effectively solves the problem of anchor point observation failure caused by liver tissue degradation and local occlusion, and preserves the structural details of liver tissue.

[0150] S3, Four-dimensional deformation field optimization and linkage update of surface and deep structures: Combine the biomechanical properties of liver soft tissue to achieve accurate optimization of the deformation field and dynamic update of deep risk structures.

[0151] The construction process based on optical flow constraint, instrument contact event constraint and soft tissue biomechanical prior constraint in step S3 is as follows:

[0152] Construct the instrument contact event constraint, which includes calculating the directed distance d between the instrument tip and the tissue surface according to the tissue surface image and the position of the instrument tip, calculating the maximum depth gradient amplitude G_max in the preset small region Ω centered on the instrument tip, and the calculation formula is:

[0153]

[0154] in, This represents the depth value at position p. For position Depth gradient at that location;

[0155] When both conditions are met < and > When a contact event between the instrument and tissue is determined, among which... Distance threshold Gradient threshold;

[0156] After a contact event is triggered, all Gaussian anchor points within a preset range around the contact point are marked as pending refresh. The anchor point weights are updated, and the maturity parameters are reset. The expression for updating the Gaussian anchor point weights is:

[0157]

[0158] The weight of the Gaussian anchor point after the contact event is triggered. The anchor point weight is the anchor point's opacity / rendering contribution weight. The weight of the Gaussian anchor point prior to the contact event being triggered. For the opacity increment;

[0159] Its maturity parameters Reset to The formula is:

[0160]

[0161] in, The preset maturity level has a value range of [0.5]. , ], This is an opacity increment used to quickly adapt to localized tissue deformation caused by contact.

[0162] Constructing biomechanical prior constraints for soft tissue, including introducing a hybrid physical prior model as a regularization term. The specific calculation formula is as follows:

[0163]

[0164] in, The elastic smoothing term penalizes the second-order gradient of the displacement vector between adjacent Gaussian anchor points, and is defined as:

[0165]

[0166] Where N is the set of spatially adjacent Gaussian anchor pairs, Let i be the displacement vector of the i-th Gaussian anchor point. As a volume conservation term, the rate of change of volume in the constrained local tissue region approaches zero, and the calculation formula is:

[0167]

[0168] in, Let $e$ be the deformation gradient Jacobian matrix of the local element $e$. As a low-dimensional motion basis constraint, the current deformation field is projected onto a low-dimensional motion basis extracted from historical deformation data, penalizing the projection reconstruction error. The calculation formula is as follows:

[0169]

[0170] in, It is a low-dimensional motion basis matrix, extracted from historical deformation data by principal component analysis or autoencoder;

[0171] , , These are the weights of the elastic smoothing term, the volume conservation term, and the low-dimensional motion basis constraints, respectively.

[0172] In this embodiment, S3 specifically includes the following steps:

[0173] S31, Constraint Fusion: Within a local spatiotemporal window, an iterative optimization of Gaussian anchor point parameters (position, covariance, and appearance) is achieved by constructing and minimizing a multi-factor weighted objective function. Specifically, the calculation involves: first, using the Lucas-Kanade optical flow algorithm to calculate the motion trend of feature points between adjacent frames; projecting the 3D Gaussian anchor point onto the 2D image plane; and then tracking the corresponding position of the projected point in the next frame using the Lucas-Kanade optical flow algorithm to obtain the displacement vector. The optical flow error threshold is set to 0.5px to ensure the anchor point motion conforms to the continuity of visual observation. Second, the interaction between the instrument and the tissue is monitored in real time. This is achieved by calculating the directed distance d between the ultrasonic scalpel tip and the liver tissue surface, as well as the local depth gradient. The system makes a judgment. A contact distance threshold dthr = 0.5 mm is set, the preset small region Ω size is 3×3 pixels, and the gradient threshold Gthr = 2.0. When d < 0.5 mm and Gmax > 2.0, it is determined that the instrument has made contact with the liver tissue. The system then improves the loss function of the gradient descent iterative strategy. In And reduce the weight of the optical flow constraint term. The parameters (position, covariance, opacity α) of the anchor points in the contact area are re-optimized to enable the deformation field to quickly adapt to the local deformation caused by mechanical contact. Specifically, anchor points within a 5mm radius of the contact point are given a higher update weight increment. The maturity level is then reset. Finally, to prevent non-physical distortions in the optimization results, a hybrid physical prior model is introduced as a regularization term. The weight coefficients of the regularization term are set as follows: λ1=0.4 (elastic smoothing term), λ2=0.3 (volume conservation term), and λ3=0.3 (low-dimensional motion basis constraint), satisfying λ1+λ2+λ3=1.

[0174] S32, Deformation Field Optimization: Temporal optimization of the position, covariance, and appearance parameters of Gaussian anchor points within the window. This temporal optimization relies on constructing temporal series correlations based on 30fps continuous video frames during surgery. Within the local spatiotemporal window, cross-frame motion correspondences of Gaussian anchor points are established through multi-frame optical flow constraints to ensure visual temporal continuity. Instrument contact event constraints are fused to capture local tissue deformations caused by surgical interactions in real time. Soft tissue biomechanical regularization constraints, including elastic smoothing, volume conservation, and low-dimensional motion basis priors, are superimposed to jointly constitute the total temporal loss function.

[0175]

[0176] A gradient descent iterative strategy with 20 iterations and a convergence threshold of 1e-5 was adopted to correct the frame-by-frame and update the anchor point parameters in real time until the loss converged and the optimal anchor point parameters for a single frame were obtained. The optimized 4DGS model was used to complete the continuous frame-by-frame temporal iterative accumulation to obtain the four-dimensional deformation field of the liver tissue surface in three-dimensional space plus time dimension (time dimension frame rate 30fps, spatial resolution 0.1mm).

[0177] S33, Post-contact processing: After a contact event is triggered, Gaussian anchor points within a 5mm radius of the contact point are marked as pending refresh. Δwcontact = 0.2 (weight increment) is set, and the anchor point weights are determined according to the formula... Update, and simultaneously reset the anchor maturity M to Mreset=0.8 (value range [0.5Mmax, Mmax], i.e. [0.5, 1.0]), the formula is as follows. Localized liver deformation caused by rapid adapter mechanical contact.

[0178] S4, with coordinated updates of surface and deep structures, enables simultaneous updates of liver surface deformation and deep risk structures, ensuring navigation safety.

[0179] Specifically, the optimized intraoperative surface Gaussian field is registered with the organ surface model segmented from the preoperative three-dimensional medical image to obtain the surface displacement field; the surface displacement field is used to drive the body deformation model to diffuse the surface deformation to the deep risk structure in the preoperative three-dimensional medical image, update the three-dimensional spatial position of the deep risk structure, and realize the linkage between surface deformation and deep risk structure.

[0180] In S4, surface deformation diffusion to deep risk structures in preoperative 3D medical images is performed using radial basis function interpolation, and non-physical distortions are corrected through biomechanical constraints. The deformation diffusion interpolation formula is:

[0181]

[0182] Among them, x represents the three-dimensional coordinates of any deep voxel to be solved in the preoperative three-dimensional image space; and These represent the coordinates and displacement of the j-th effective control point obtained from the surface registration, respectively. This indicates the total number of effective control points at the surface level; It is a radial basis kernel function; The weighting coefficients for interpolation are adaptively adapted to the local deformation amplitude. These are low-order polynomial terms, ensuring that the overall deformation has affine invariance and avoiding diffusion distortion caused by overall offset and rotation.

[0183] The parameters of the Gaussian anchor point are iteratively optimized based on optical flow constraints, instrument contact event constraints, and soft tissue biomechanical prior constraints, including:

[0184] Construct the loss function for the gradient descent iterative strategy:

[0185]

[0186] For optical flow constraint loss, Constraint loss for instrument contact events, For soft tissue biomechanical prior constraint loss;

[0187] A gradient descent iterative strategy is used to perform frame-by-frame reverse correction of the position, covariance, color and opacity parameters of Gaussian anchor points and to update anchor point parameters in real time.

[0188] In each iteration, the following operations are performed:

[0189] The Lucas–Kanade optical flow algorithm is used to project Gaussian anchor points onto a two-dimensional image plane and track their corresponding positions in the next frame to obtain the displacement vector. ,by Substitute the data items into the loss function of the gradient descent iterative strategy. ;

[0190] When it is determined that the instrument has come into contact with the tissue, the opacity of the anchor points within a preset range around the contact point is increased and their corresponding weights are raised. This allows for priority reconstruction of local deformation at the anchor points in the contact region, and the local deformation is then substituted as a data term into the loss function of the gradient descent iterative strategy. ;

[0191] regularization terms Substituting into the loss function of the gradient descent iterative strategy ;

[0192] The process is iterated until the loss converges, and the optimal Gaussian anchor point parameters corresponding to each frame are obtained. Based on the optimized optimal Gaussian anchor point parameters, continuous frame-to-frame temporal iterative accumulation is performed using 4D Gaussian splashing technology to obtain the four-dimensional deformation field.

[0193] In this embodiment, S4 specifically includes the following steps:

[0194] S41, Model Registration: The optimized intraoperative Gaussian field of the liver surface is registered with the liver surface model segmented from the preoperative CT image (segmented using ITK-SNAP software, resolution 0.5mm×0.5mm×0.5mm). The Iterative Closest Point (ICP) algorithm is used, with 50 registration iterations and a registration error ≤0.4mm to obtain the liver surface displacement field.

[0195] S42, Deep Structure Update: Based on the surface displacement field-driven thin plate spline deformation model (i.e., a specific preferred form of the deformation diffusion model based on radial basis function interpolation), the liver surface deformation is diffused to the deep risk structures (right branch of portal vein, hepatic artery branch, hepatic vein) in the preoperative CT image, updating the three-dimensional spatial position of these deep risk structures, realizing the linkage update of liver surface deformation and deep risk structures, with an update frequency consistent with the four-dimensional deformation field (30fps), ensuring that the position of deep risk structures is synchronized with the actual deformation of the liver.

[0196] S5, visualization rendering, achieves high-fidelity visualization of liver tissue and deep risk structures based on reconstruction uncertainty, generating enhanced navigation images. Specifically, it includes the following steps:

[0197] S51, Uncertainty Calculation: Based on the depth confidence map C output by S22 and the four-dimensional deformation field optimization residual Res output by S32, calculate the reconstruction uncertainty σ of each region of the liver tissue space. Set Res_max=0.2mm (residual upper limit threshold). The uncertainty σ is calculated as σ=√(1-C)×Res / Res_max, with a value range of [0,1]. The larger the σ, the higher the reconstruction uncertainty.

[0198] S52, High-fidelity visualization rendering: Based on the reconstruction uncertainty σ, the updated deep risk structures (right branch of the portal vein, hepatic artery branches) are visualized and rendered to generate an enhanced navigation image. Specific parameter settings: α_min=0.1 (minimum visible transparency), σ_max=0.8 (uncertainty upper limit threshold), k=2 (modulation curve steepness index), transparency is calculated according to the formula. Modulation; boundary diffusion is shown in the display. =1mm (base width), η=0.5 (diffusion coefficient), the diffusion band width is calculated according to the formula In the calculation and color coding, σ_low=0.3 (low uncertainty threshold) and σ_high=0.6 (high uncertainty threshold) are set. When σ<0.3, a highly saturated red (warning color) is used for display. When 0.3≤σ≤0.6, the color saturation decreases linearly with the increase of σ. When σ>0.6, it gradually changes to blue (cool color). At the same time, the rendering accuracy of liver tissue texture is improved by using the optimized 4DGS model to achieve high-fidelity real-time visualization.

[0199] S6, with its occlusion assessment and adaptive adjustment of display strategy, addresses instrument occlusion scenarios during liver surgery, ensuring navigation continuity and reliability. Specifically, it includes the following steps:

[0200] S61, Occlusion Assessment: Calculate the proportion of the occluded area to the total area of ​​the current frame, Oratio, based on the instrument region mask. Set Othr=0.3 (occlusion threshold). When Oratio>0.3, it is judged as severe occlusion, triggering display strategy adjustment;

[0201] S62, Reconstruction Quality Assessment: Calculate the overall reconstruction quality index Qrecon, set ωC=0.6 (depth confidence weight) and ωRes=0.4 (residual weight), satisfy ωC+ωRes=1, Qrecon takes the value range [0,1], the larger the value, the better the reconstruction quality, set the reconstruction quality threshold to Qthr=0.6;

[0202] S63, Strategy Adjustment: When Qrecon < 0.6, shift the safety boundary of the deep risk structure outward by a conservative bias. Set Δd0 = 0.3mm (base offset), κ = 0.5 (scaling factor), and follow the formula. The system calculates and simultaneously uses a flashing red prompt box to visually alert the operator to the decrease in current navigation accuracy. It increases the refresh frequency of safety boundaries and risk hotspots in high uncertainty areas (σ>0.6) to twice the normal frequency. When Oratio>0.3, it automatically adjusts the display level of the enhanced navigation image, raising the rendering level of deep risk structures to the highest level and increasing the overlay transparency to 0.7 to ensure that the operator can still clearly observe the location of deep risk structures in occluded scenes.

[0203] The present invention also provides Embodiment 2: a local spatiotemporal window reconstruction and occlusion navigation display system for dynamic soft tissues.

[0204] This embodiment uses liver surgery (laparoscopic partial hepatectomy) as an application background to provide a local spatiotemporal window reconstruction and occlusion navigation display system for dynamic soft tissues. Each module has a clear function and works collaboratively. The specific structure is as follows:

[0205] Video acquisition and preprocessing module. This module is used to acquire and optimize high-quality intraoperative images, providing reliable input for subsequent steps such as depth estimation and 3D reconstruction. It acquires video streams of the surgical area using intraoperative image acquisition equipment. Addressing common image interference issues during surgery, it employs targeted image enhancement algorithms to suppress interference and improve image quality. Simultaneously, it uses a deep learning network to segment surgical instruments and mark their locations.

[0206] Depth Estimation and Anchor Point Management Module. This module enables high-fidelity real-time 3D reconstruction of dynamic soft tissues, quantifies the reliability of depth estimation, and finely manages Gaussian anchor points to adapt to dynamic soft tissue deformation. Combining a monocular depth estimation network with an optimized 4DGS model, pseudo-depth estimation is performed on preprocessed images, generating a depth confidence map. A local spatiotemporal window centered on the surgical instrument tip is established, within which Gaussian anchor points are initialized, updated, dormant, and deleted, achieving fine-grained management of the anchor point lifecycle. This is used to accurately capture minute soft tissue deformations, solve reconstruction failures caused by occlusion and tissue degradation, and provide high-quality data support for subsequent deformation field optimization.

[0207] Deformation Field Optimization Module. This module is used to accurately acquire the four-dimensional deformation field of dynamic soft tissue, capturing the deformation patterns of soft tissue in spatial and temporal dimensions. Within a local spatiotemporal window, it combines optical flow constraints, instrument contact event constraints, and prior biomechanical constraints of soft tissue to perform temporal optimization of the core parameters of Gaussian anchor points, and leverages the advantages of the 4DGS model to achieve real-time parameter updates.

[0208] The module features a synchronized update and rendering mechanism. This module enables the synchronous updating of soft tissue surface deformation and deep risk structures, while providing clear navigation guidance to the surgeon through visual rendering. It registers the intraoperatively reconstructed Gaussian field with the organ model from preoperative medical images to obtain the surface displacement field and drive the volumetric deformation model, achieving dynamic updates of deep risk structures. Based on reconstruction uncertainties, a targeted visualization strategy is employed to render deep risk structures, generating enhanced navigation images.

[0209] The display strategy control module addresses instrument occlusion during surgery, ensuring the continuity and reliability of navigation information and adapting to the navigation needs of different surgical scenarios. It assesses the occlusion area based on instrument segmentation results, quantifies the overall reconstruction quality by considering reconstruction uncertainty, and adaptively adjusts the navigation image display strategy, including display hierarchy, transparency, and refresh rate, according to the degree of occlusion and reconstruction quality. This provides surgeons with continuous and stable navigation guidance, improving the safety and consistency of surgical procedures.

[0210] Processor and storage module. This module supports the stable operation of the entire system, enabling instruction execution and data security management. It employs high-performance processing equipment to execute the computational instructions of each module, ensuring real-time system response; and uses highly reliable storage media to store surgical-related data, system programs, and user health records, achieving secure data storage and rapid retrieval. It provides hardware and storage support for the collaborative work of various system modules, ensuring the real-time performance and stability of the navigation system, while also guaranteeing the traceability of surgical data and user privacy, providing data support for postoperative analysis and system optimization.

[0211] The present invention also provides Embodiment 3: a system for the aforementioned surgical navigation method for dynamic soft tissue, comprising:

[0212] The acquisition module is used to acquire video streams of the surgical field during surgery and preoperative three-dimensional medical images;

[0213] The segmentation module is used to segment the surface model of organs from preoperative three-dimensional medical images and label the deep risk structures within the organs.

[0214] An enhancement module is used to perform reflection suppression, smoke removal, and instrument instance segmentation on each frame of the video stream, and output an enhanced tissue surface image and an instrument region mask.

[0215] The confidence module is used to input the surface images of each tissue frame by frame into the monocular depth estimation network, output the corresponding dense pseudo-depth map, and generate a depth confidence map based on the tissue surface images and the dense pseudo-depth map.

[0216] A construction module is used to extract the instrument tip from the instrument region mask and construct a local spatiotemporal window centered on the instrument tip in a three-dimensional pixel space.

[0217] The generation module is used to generate Gaussian anchor points in real time within a local spatiotemporal window based on the dense pseudo-depth map and depth confidence map corresponding to each frame of the image. The region with the depth confidence of the corresponding pixel within the local spatiotemporal window that is greater than a preset threshold is determined as a valid reliable region. New Gaussian anchor points are generated based on the valid reliable regions to obtain the updated Gaussian anchor points of the corresponding image.

[0218] The optimization module is used to iteratively optimize the parameters of the Gaussian anchor point based on optical flow constraints, instrument contact event constraints, and soft tissue biomechanical prior constraints to obtain a Gaussian field.

[0219] The rendering module is used to register the Gaussian field with the organ surface model to obtain the surface displacement field. Based on the surface displacement field, the three-dimensional spatial position of the deep risk structure is updated using the radial basis function interpolation method. The updated deep risk structure is then visualized and rendered to generate an enhanced navigation image.

[0220] This invention provides a surgical navigation method and system that is adaptable to dynamic soft tissue, balances reconstruction efficiency and navigation accuracy, and can handle occlusion scenarios. By introducing local window reconstruction, depth confidence quantification, anchor point fine management, and multi-module collaborative linkage mechanism, it achieves three-dimensional reconstruction of dynamic soft tissue surface, linkage updating of deep risk structures, and adaptive navigation display under occlusion conditions, thereby meeting the comprehensive requirements of minimally invasive surgery for navigation accuracy, real-time performance, and safety.

[0221] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A surgical navigation method for dynamic soft tissue, characterized in that, include: Acquire video streams of the intraoperative surgical field and preoperative 3D medical images; The surface model of the organ is segmented from the preoperative three-dimensional medical image, and the deep risk structures inside the organ are marked. Perform reflection suppression, smoke removal, and instrument instance segmentation on each frame of the video stream, and output an enhanced tissue surface image and an instrument region mask; Each tissue surface image is input frame by frame into a monocular depth estimation network, which outputs a corresponding dense pseudo-depth map. A depth confidence map is generated based on the tissue surface image and the dense pseudo-depth map. Extract the instrument tip from the instrument region mask and construct a local spatiotemporal window centered on the instrument tip in the three-dimensional pixel space; Within a local spatiotemporal window, Gaussian anchor points are generated in real time based on the dense pseudo-depth map and depth confidence map corresponding to each frame of the image. Regions within the local spatiotemporal window whose depth confidence of the corresponding pixel is greater than a preset threshold are identified as valid reliable regions. New Gaussian anchor points are generated based on the valid reliable regions to obtain the updated Gaussian anchor points of the corresponding image. The parameters of the Gaussian anchor point are iteratively optimized based on optical flow constraints, instrument contact event constraints, and soft tissue biomechanical prior constraints to obtain the Gaussian field. The Gaussian field is registered with the organ surface model to obtain the surface displacement field. Based on the surface displacement field, the three-dimensional spatial position of the deep risk structure is updated using the radial basis function interpolation method. The updated deep risk structure is then visualized and rendered to generate an enhanced navigation image.

2. The surgical navigation method for dynamic soft tissue according to claim 1, characterized in that, Based on the tissue surface image and dense pseudo-depth map, a depth confidence map is generated, including: Calculate the confidence scores for photometric consistency and texture richness based on tissue surface images; Calculate the network uncertainty confidence of the monocular depth estimation network based on the dense pseudo-depth map; The depth confidence map is obtained by weighted fusion of the photometric consistency confidence, texture richness confidence, and network uncertainty confidence.

3. The surgical navigation method for dynamic soft tissue according to claim 1, characterized in that, Extract the instrument tip from the instrument region mask, and construct a local spatiotemporal window centered on the instrument tip in three-dimensional space, including: Two-dimensional pixel coordinates of the instrument tip are extracted from the instrument region mask based on connected component analysis; The two-dimensional pixel coordinates of the instrument tip are converted into a three-dimensional spatial position based on the dense pseudo-depth map; A local spatiotemporal window is established in three-dimensional space, centered on the tip of the surgical instrument, and based on the preset radius of the spatial interest region.

4. The surgical navigation method for dynamic soft tissue according to claim 3, characterized in that, When the distance between the deep risk structure and the instrument tip is less than a preset safety threshold, the radius of the spatial domain of interest is adaptively adjusted. The expression for the adaptive adjustment is: ; in, The radius of the spatial interest domain preset for the current operation phase. The minimum radius required to cover deep-seated risk structures.

5. A surgical navigation method for dynamic soft tissue according to claim 3, characterized in that, After generating Gaussian anchor points in real time, the process also includes maintaining the Gaussian anchor points within the local spatiotemporal window, specifically including: Configure independent age and maturity parameters for all Gaussian anchor points within a local spatiotemporal window; When the Gaussian anchor point is within the effective reliable region of the local spatiotemporal window, the age parameter of the Gaussian anchor point is forcibly reset to the maximum value of the preset age parameter, and the maturity parameter is set to increase iteratively frame by frame according to the preset formula. When the Gaussian anchor point leaves the local spatiotemporal window, the age parameter is set to decrease automatically frame by frame, and the anchor point enters a dormant standby state. When the age parameter of an anchor point decays to less than or equal to 0, the anchor point is determined to be invalid and has no observation value in the long term, and a permanent deletion operation is performed on the anchor point.

6. The surgical navigation method for dynamic soft tissue according to claim 3, characterized in that, The construction process based on optical flow constraints, instrument contact event constraints, and soft tissue biomechanical prior constraints is as follows: Optical flow constraints are constructed, which include projecting Gaussian anchor points onto a two-dimensional image plane based on the Lucas-Kanade optical flow algorithm and tracking the corresponding position of the projected Gaussian anchor points in the next frame to obtain the displacement vector of the Gaussian anchor points. When the optical flow brightness residual or forward and backward optical flow consistency error of the projected point exceeds the preset optical flow error threshold, the displacement vector is determined to be invalid and discarded. The corresponding Gaussian anchor points are then updated by interpolation of adjacent valid anchor points or soft tissue biomechanical prior constraints. The instrument contact event constraints are constructed, which include calculating the directed distance d between the instrument tip and the tissue surface based on the tissue surface image and the position of the instrument tip, and calculating the maximum depth gradient magnitude G_max within a preset small region Ω centered on the instrument tip. The calculation formula is as follows: in, This represents the depth value at position p. For position Depth gradient at that location; When both conditions are met < and > When a contact event between the instrument and tissue is determined, among which... Distance threshold Gradient threshold; After a contact event is triggered, all Gaussian anchor points within a preset range around the contact point are marked as pending refresh. The anchor point weights are updated, with the weight determined by the anchor point's opacity / rendering contribution. The maturity parameter is then reset. The expression for updating the Gaussian anchor point weights is: The weight of the Gaussian anchor point after the contact event is triggered. The weight of the Gaussian anchor point prior to the contact event being triggered. For the opacity increment; Constructing biomechanical prior constraints for soft tissue, including introducing a hybrid physical prior model as a regularization term. The specific calculation formula is as follows: in, The elastic smoothing term penalizes the second-order gradient of the displacement vector between adjacent Gaussian anchor points, and is defined as: Where N is the set of spatially adjacent Gaussian anchor pairs, Let i be the displacement vector of the i-th Gaussian anchor point. As a volume conservation term, the rate of change of volume in the constrained local tissue region approaches zero, and the calculation formula is: in, Let $e$ be the deformation gradient Jacobian matrix of the local element $e$. As a low-dimensional motion basis constraint, the current deformation field is projected onto a low-dimensional motion basis extracted from historical deformation data, penalizing the projection reconstruction error. The calculation formula is as follows: in, It is a low-dimensional motion basis matrix, extracted from historical deformation data by principal component analysis or autoencoder; , , These are the weights of the elastic smoothing term, the volume conservation term, and the low-dimensional motion basis constraints, respectively.

7. A surgical navigation method for dynamic soft tissue according to claim 6, characterized in that, The parameters of the Gaussian anchor point are iteratively optimized based on optical flow constraints, instrument contact event constraints, and soft tissue biomechanical prior constraints, including: Construct the loss function for the gradient descent iterative strategy: For optical flow constraint loss, Constraint loss for instrument contact events, For soft tissue biomechanical prior constraint loss; A gradient descent iterative strategy is used to perform frame-by-frame reverse correction of the position, covariance, color and opacity parameters of Gaussian anchor points and to update anchor point parameters in real time. In each iteration, the following operations are performed: The Lucas–Kanade optical flow algorithm is used to project Gaussian anchor points onto a two-dimensional image plane and track their corresponding positions in the next frame to obtain the displacement vector. ,by Substitute the data items into the loss function of the gradient descent iterative strategy. ; When it is determined that the instrument has come into contact with the tissue, the opacity of the anchor points within a preset range around the contact point is increased and their corresponding weights are raised. This allows for priority reconstruction of local deformation at the anchor points in the contact region, and the local deformation is then substituted as a data term into the loss function of the gradient descent iterative strategy. ; regularization terms Substituting into the loss function of the gradient descent iterative strategy ; The process is iterated until the loss converges, and the optimal Gaussian anchor point parameters corresponding to each frame are obtained. Based on the optimized optimal Gaussian anchor point parameters, continuous frame-to-frame temporal iterative accumulation is performed using 4D Gaussian splashing technology to obtain the four-dimensional deformation field.

8. A surgical navigation method for dynamic soft tissue according to claim 7, characterized in that, When visualizing the updated deep risk structure, the visualization of the deep risk structure is based on reconstruction uncertainty, including: Based on the depth confidence map and the optimized residual of the four-dimensional deformation field, the reconstruction uncertainty of each region in space is calculated using the following formula: σ = √(1-C) × Res / Res_max; C represents the depth of confidence. To optimize residuals for a four-dimensional deformation field, The upper limit threshold of the residual is σ, and the range of the reconstruction uncertainty σ is [0,1]. The larger the σ is, the higher the reconstruction uncertainty. Transparency Modulation: Opacity in Deep Risk Structure Rendering With Reconstruction Uncertainty The following relationship must be satisfied: ; in, The preset minimum visible transparency has a value range of (0, 0.3]. The upper limit threshold for uncertainty; k is the kurtosis control index of the modulation curve, k≥1; when As the risk increases, the risk structure becomes more transparent but still retains a level no lower than [a certain percentage]. Visibility; Boundary diffusion display: The outline of deep risk structures or safety boundaries is rendered as a diffusion band, and the width W of the diffusion band is proportional to the reconstruction uncertainty, calculated as follows: ; Where W0 is the base width, η is the diffusion coefficient, and σ is the reconstruction uncertainty; the color saturation within the diffusion band decreases linearly or non-linearly from the inside to the outside, and the higher the uncertainty, the more blurred the visual effect of the boundary. Color coding: In the risk heatmap mode, a mapping function is established from cool colors with high uncertainty to warm colors with low uncertainty. The specific determination rule is as follows: when < When this occurs, a highly saturated warning color will be used for display; when ≥ and ≤ At that time, color saturation increased with Increase linearly decreasing; when > When the saturation drops to a preset minimum or gradually changes to a cool color; in, For low uncertainty threshold, For high uncertainty threshold, < .

9. A surgical navigation method for dynamic soft tissue according to claim 7, characterized in that, This also includes occlusion status assessment and adaptive adjustment of display strategy, specifically including: The occlusion area assessment includes calculating the proportion of the occluded area to the total area of ​​the current frame based on the instrument area mask. When it exceeds a preset threshold, the enhanced navigation image display mode is automatically adjusted. The overall reconstruction quality assessment includes the construction of overall reconstruction quality indicators. Overall reconstruction quality indicators The calculation formula is: in, The total number of pixels in the image. For all pixel-level depth confidence, To optimize residuals for a four-dimensional deformation field, The upper limit threshold for residuals. , For the weighting coefficients, satisfying + =1; Display strategy adaptively adjusts: when Below the preset threshold At that time, the safety boundary is offset outward by a conservative bias amount. The calculation formula is: in, Based on the bias, This is the scaling factor.

10. A system for a surgical navigation method for dynamic soft tissue as described in any one of claims 1-9, characterized in that, include: The acquisition module is used to acquire video streams of the surgical field during surgery and preoperative three-dimensional medical images; The segmentation module is used to segment the surface model of organs from preoperative three-dimensional medical images and label the deep risk structures within the organs. An enhancement module is used to perform reflection suppression, smoke removal, and instrument instance segmentation on each frame of the video stream, and output an enhanced tissue surface image and an instrument region mask. The confidence module is used to input the surface images of each tissue frame by frame into the monocular depth estimation network, output the corresponding dense pseudo-depth map, and generate a depth confidence map based on the tissue surface images and the dense pseudo-depth map. A construction module is used to extract the instrument tip from the instrument region mask and construct a local spatiotemporal window centered on the instrument tip in a three-dimensional pixel space. The generation module is used to generate Gaussian anchor points in real time within a local spatiotemporal window based on the dense pseudo-depth map and depth confidence map corresponding to each frame of the image. The region with the depth confidence of the corresponding pixel within the local spatiotemporal window that is greater than a preset threshold is determined as a valid reliable region. New Gaussian anchor points are generated based on the valid reliable regions to obtain the updated Gaussian anchor points of the corresponding image. The optimization module is used to iteratively optimize the parameters of the Gaussian anchor point based on optical flow constraints, instrument contact event constraints, and soft tissue biomechanical prior constraints to obtain a Gaussian field. The rendering module is used to register the Gaussian field with the organ surface model to obtain the surface displacement field. Based on the surface displacement field, the three-dimensional spatial position of the deep risk structure is updated using the radial basis function interpolation method. The updated deep risk structure is then visualized and rendered to generate an enhanced navigation image.