Mixed reality surgical navigation method and system

Abstract

The invention provides a mixed reality operation navigation method and system, relates to the technical field of mixed reality, and is used for solving the problem that an existing navigation system is low in positioning accuracy of organ key structures in a soft tissue operation. According to the technical key points, the method comprises the steps of obtaining a preoperative organ 3D model; tracking intraoperative organ surface data by using mixed reality equipment; performing fusion processing on the organ 3D model and the organ surface data to generate an optimal matching deformation correction model; performing holographic projection on the optimal matching deformation correction model to display the corrected internal structure of the organ; wherein hologram display is stabilized by arranging an external reference tool; tools and organ surface points are tracked in real time by using a head-mounted depth camera, so that the tracking precision is improved; the soft tissue deformation is accurately compensated by combining a weighted iteration nearest point algorithm and a linear iteration boundary reconstruction algorithm. The method is wide in applicability, and can be expanded to open surgery, potential laparoscopic workflow and other soft tissue interventions.

Description

Technical Field

[0001] This invention relates to the field of mixed reality technology, and in particular to a mixed reality surgical navigation method and system. Background Technology

[0002] When evaluating radical interventions for liver malignancies, hepatectomy requires meticulous preoperative planning to precisely locate the subsurface tumor and key vascular structures. For patients with unresectable disease, localized treatment—especially thermal ablation—serves as an important transitional intervention prior to final resection. Comprehensive preoperative planning is essential to ensure tumor margins, optimize functional liver reserve, and mitigate surgical complications, including bleeding and bile duct injury.

[0003] A major technical challenge in liver surgery involves intraoperative organ deformation caused by surgical manipulations, including perihepatic packing in open surgery, pneumoperitoneum in laparoscopic surgery, traction manipulations, and displacement from supporting ligamentous structures. These deformations significantly reduce the fidelity of preoperative imaging data when used intraoperatively. Quantitative analysis has revealed significant morphological differences between preoperative and intraoperative liver conformation in both open and minimally invasive surgical approaches. Recent advances in biomechanical modeling, particularly linearized iterative boundary reconstruction (LIBR) methods, have shown great potential in compensating for these intraoperative deformations.

[0004] Image-guided surgery (IGS) systems offer significant clinical utility by spatially correlating preoperative data with intraoperative anatomy. However, traditional IGS platforms exhibit inherent limitations in their visualization paradigm. Specifically, the navigation display is typically located considerably distanced from the surgical area, requiring clinicians to repeatedly shift their attention from the surgical site to the guidance system. This constant context switching places a substantial cognitive burden on surgeons, as they must mentally switch between different reference frames and integrate spatial information. Mixed reality (MR) technology offers a promising solution to this fundamental limitation. MR environments facilitate bidirectional interaction between virtual and physical elements, enabling context-responsive digital objects to be dynamically integrated with the surgical area. This technological approach establishes a unified perceptual framework where digital and physical components coexist and interact synchronously.

[0005] However, existing navigation systems have the following limitations: 1) Existing image-guided systems typically use rigid registration methods, which cannot accurately compensate for soft tissue deformation; 2) Navigation displays are usually located far from the surgical area, requiring surgeons to repeatedly shift their attention between the surgical area and the navigation display; 3) Existing mixed reality systems are mainly applied to rigid anatomical structures, such as dental implants, skull and spine surgeries, lacking quantitative verification of soft tissue deformation; 4) Existing technologies do not integrate precise tool tracking, biomechanical model deformation correction, and intuitive mixed reality visualization; 5) The spatial positioning stability of mixed reality devices is insufficient, causing virtual content to drift in the field of view, affecting surgical accuracy. These limitations may lead to patients requiring secondary surgery after surgery.

[0006] In summary, existing navigation systems for soft tissue surgery suffer from high computational complexity, high data density requirements, and insufficient real-time performance, making them unsuitable for clinical applications. Therefore, developing a navigation system capable of processing soft tissue deformation in real time and displaying information intuitively through mixed reality technology is of significant value. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention provides a mixed reality surgical navigation method and system.

[0008] According to one aspect of the present invention, a mixed reality surgical navigation method is proposed, the method comprising:

[0009] S1. Obtain a preoperative 3D model of the organ;

[0010] S2. Use mixed reality devices to track intraoperative organ surface data;

[0011] S3. The organ 3D model and the organ surface data are fused to generate the best matching deformation correction model;

[0012] S4. The best-matching deformation correction model is holographically projected to display the corrected internal structure of the organ.

[0013] Furthermore, the use of mixed reality devices to track intraoperative organ surface data in S2 includes:

[0014] S21. Pre-setting retroreflective markers includes: setting multiple retroreflective markers on an external reference tool fixedly placed near the organ, and setting multiple retroreflective markers on each organ surface data acquisition tool, wherein the organ surface data acquisition tool includes a stylus and surgical instruments;

[0015] S22. The shape of the organ surface data acquisition tool is defined based on the position of multiple retroreflective markers, wherein the shape is the relative positional relationship between the geometric center of the tool and the multiple retroreflective markers on it;

[0016] S23. During the operation, an organ surface data acquisition tool is slid across the organ surface to acquire intraoperative organ surface data; a camera in a mixed reality device is used to acquire continuous reflectance images of the organ surface data acquisition tool sliding across the organ surface in real time; the reflectance images include an external reference tool;

[0017] S24. Detect the 3D position of each retroreflection marker in the continuous reflectivity image and separate the retroreflection marker from other objects;

[0018] S25. Identify the organ surface data acquisition tool based on the 3D position of each retroreflective marker and the shape of the predefined organ surface data acquisition tool, and then determine the 3D position of the tool's geometric center.

[0019] S26. Determine organ surface data based on the 3D position of the organ surface data acquisition tool in the continuous reflectance image, wherein the organ surface data is the 3D coordinates of a series of points on the organ surface.

[0020] Furthermore, in S25, based on the 3D position of each retroreflective marker and the shape of the predefined organ surface data acquisition tool, a depth-first graph search algorithm is used to find a subset of retroreflective markers that match the shape of the predefined organ surface data acquisition tool, thereby identifying the organ surface data acquisition tool.

[0021] Furthermore, in step S25, when multiple organ surface data acquisition tools are identified, conditional filtering is performed on the multiple organ surface data acquisition tools to remove mismatched tools; the conditions include a first condition and a second condition, wherein the first condition is:

[0022]

[0023] Among them, t shape Indicates a preset threshold for the shape; This represents a standard spatial distribution template of all retroreflection marks in the l-th tool relative to the tool's geometric center; N represents the k-th subset of candidate retroreflection tags that may match the l-th tool definition; l Representation tool T l The number of retroreflection markers included in the definition; This represents the Euclidean distance between the i-th and j-th retroreflection markers in the l-th tool definition. This represents the actual measured distance between the i-th and j-th retroreflective markers in the candidate retroreflective marker subset;

[0024] The second condition is: Among them, t side This indicates a preset threshold for the side length; the recognition results that simultaneously satisfy the first and second conditions will be retained.

[0025] Furthermore, in step S3, the fusion processing of the organ 3D model and the organ surface data to generate the optimal matching deformation correction model includes:

[0026] S31. Initially align the point cloud data corresponding to the organ 3D model and the organ surface data, including: pre-setting multiple salient features on the organ; obtaining preoperative point clouds of multiple salient features based on the organ 3D model; obtaining intraoperative point clouds of multiple salient features based on the organ surface data; calculating the centroids of the preoperative point cloud and the intraoperative point cloud respectively; and registering the centroids of the preoperative point cloud and the intraoperative point cloud for initial alignment.

[0027] S32. Use the improved iterative nearest point algorithm to re-align the point cloud after the initial alignment;

[0028] S33. The point cloud data after re-alignment is processed using the linear iterative boundary reconstruction method to compensate for soft tissue deformation.

[0029] Furthermore, in step S32, the point cloud alignment process after initial alignment using the improved iterative nearest neighbor algorithm includes: for each point in the source point cloud, in the process of finding the nearest neighbor in the target point cloud, a weighting factor w is introduced. PC A weighted nearest-point operation is performed, meaning that if the source point and the target point both belong to the same salient feature region, the distance between the source point and the target point is multiplied by a weighting factor w. PC After determining the corresponding points in the source and target point clouds, the weighted rigid point base registration method is used to calculate the rigid body transformation that minimizes the distance between the corresponding points. The objective function is:

[0030]

[0031] Where, N M This represents the total number of points in a point cloud of data on the surface of an organ. This indicates a weighted nearest-point operation, s m Let T represent the source point cloud, and T represent the set of point clouds corresponding to all salient feature regions; if Then w m =1, if Then w m =w PBR w PBR This represents the dynamic weighting factor used for bias point base registration. It is the label that identifies which salient feature region the m-th point in the source point cloud belongs to; Θ represents the rigid body transformation matrix.

[0032] Furthermore, in S32, the dynamic weighting factor w used for bias point base registration PBR The calculation formula is:

[0033] w PBR (t)=w PBR,max ×e -λ(t-1) +w PBR,base ×(1-e -λ(t-1) );

[0034] Among them, w PBR,max This represents the maximum value of the preset bias point base registration weight; w PBR,base λ represents the base weight at the end of the iteration; λ represents the relaxation factor; and t represents the current iteration number.

[0035] Furthermore, in step S33, processing the re-aligned point cloud data using a linear iterative boundary reconstruction method to compensate for soft tissue deformation includes:

[0036] S331. The point cloud corresponding to the 3D model of the organ is rasterized in advance, and each raster vertex is perturbed on three axes respectively; according to Saint-Venant's principle, each perturbation is solved by applying millimeter displacement to each raster vertex and applying zero displacement at other raster vertices, and the response includes the displacement response matrix.

[0037] S332. Construct a displacement basis according to the superposition principle. The displacement basis is the product of the displacement response matrix and the weight vector of each disturbance. The length of the weight vector is three times the number of grid boundary vertices.

[0038] S333. The displacement base is superimposed on the point cloud corresponding to the organ 3D model after the point cloud is aligned again to obtain the reconstructed best matching deformation correction model.

[0039] Furthermore, in step S332, the weight vector for each perturbation is iteratively solved according to the following process:

[0040] S3321. Superimpose the displacement basis onto the point cloud corresponding to the organ 3D model after the point cloud alignment, to obtain the reconstructed deformation correction model; S3322. Minimize the objective function formula of the following design to solve for the objective function value:

[0041]

[0042] Where β = [α, τ, θ], α represents the weight vector for each perturbation, and τ, θ represent the parameters of the rigid body transformation matrix; w F N represents the weight of the salient feature F; F f represents the number of points representing the salient feature F;i w represents the model error for the i-th data point; E f represents the regularization coefficient; E The energy penalty function represents the overall 3D model of the organ;

[0043] S3323, If |φ(β) t )-φ(β t-1 )|>10 -12 Then return to S3321 and continue iterating, where φ(β) t ) represents the objective function value obtained in the t-th iteration, β t Let represent the value of β obtained after the t-th iteration; and update the initial value of β for the next iteration according to the following formula:

[0044] β t+1 -β t =(J T WJ+χdiag(J T WJ)) -1 J T Wf;

[0045] Where, f = [f i ,f E ] represents the error vector; The Jacobian matrix representing the error f; W = diag(w F / N F ,w E ) represents the weighting factor for each type of error; χ is the regularization parameter used to balance the weights of fitting error and system stiffness;

[0046] S3324, If |φ(β) t )-φ(β t-1 )|<10 -12 Then stop iterating and obtain the value of β after the t-th iteration. t Then, the weight vector α for each perturbation is obtained.

[0047] According to another aspect of the present invention, a mixed reality surgical navigation system is proposed, the system comprising:

[0048] The organ model creation module is configured to acquire preoperative 3D organ models.

[0049] The organ real-time coordinate calculation module is configured to use a mixed reality device to track intraoperative organ surface data;

[0050] An organ deformation correction module is configured to fuse the organ 3D model and the organ surface data to generate an optimal matching deformation correction model.

[0051] The projection display module is configured to holographically project the best-matched deformation correction model to display the corrected internal structure of the organ.

[0052] The present invention has the following technical effects:

[0053] This invention proposes a mixed reality surgical navigation method and system, comprising: acquiring a preoperative 3D model of an organ; tracking intraoperative organ surface data using a mixed reality device; fusing the 3D organ model and the organ surface data to generate an optimal matching deformation correction model; and holographically projecting the optimal matching deformation correction model to display the corrected internal structure of the organ. Specifically, by stabilizing the hologram content using an external reference tool, the hologram drift can be reduced from an average of 4.0±1.2 mm to 1.1±0.3 mm; and by using the depth camera of the head-mounted display to track the tool and surface points in real time, the tracking accuracy can be improved from 3.6±1 mm. The accuracy was improved from 0.3mm to 2.3±0.8mm; by combining the weighted iterative nearest point algorithm and the linear iterative boundary reconstruction algorithm, the surface target registration error was reduced from 7.4±4.8mm to 3.0±2.7mm, with an average error reduction of 57%, achieving a guidance accuracy of sub-4mm and accurately compensating for soft tissue deformation; by directly integrating the navigation display into the surgical field of view, the problem of doctors needing to repeatedly switch their attention between the surgical area and the remote monitor was eliminated; by superimposing the predicted subcutaneous structures (such as tumors and blood vessels) in the form of holograms on the corresponding locations in the patient's body, doctors were able to "see" the subcutaneous structures.

[0054] This invention has broad applicability, as long as it has a depth sensor and an accessible depth data stream, it can be adapted to different mixed reality devices; and it can be extended to open surgery and potential laparoscopic workflows as well as other soft tissue interventions. Attached Figure Description

[0055] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0056] Figure 1 A flowchart of a mixed reality surgical navigation method according to the present invention is shown.

[0057] Figure 2 A framework diagram of a mixed reality surgical navigation method according to an embodiment of the present invention is shown.

[0058] Figure 3A diagram showing the coordinate transformation relationship between the acquisition tool and the external reference tool in an embodiment of the present invention is provided.

[0059] Figure 4 The diagram shows a comparison of the tracking performance of mixed reality headsets with and without external reference tools in an embodiment of the present invention.

[0060] Figure 5 A schematic diagram of the linear iterative boundary reconstruction method in an embodiment of the present invention is shown.

[0061] Figure 6 Examples of anatomical structures of a liver model from multiple angles are shown in embodiments of the present invention.

[0062] Figure 7 A schematic diagram of the structure of a mixed reality surgical navigation system according to the present invention is shown.

[0063] Figure 8 The deformation process of the liver phantom in an embodiment of the present invention is shown; where (a) represents the undeformed liver phantom; (b) to (f) represent different deformation states from deformation 1 to deformation 5.

[0064] Figure 9 The following is an illustration of the visualization measurement of surface target points on a liver phantom in an embodiment of the present invention; wherein (a) represents the baseline registration result of the liver phantom in the absence of deformation; (b) represents the equivalent physical target points obtained by measurement using an optical tracking stylus; (c) represents the virtual target points measured using a rigid registration method in deformation state 1; and (d) represents the virtual target points measured using a non-rigid registration method in deformation state 1.

[0065] Figure 10 The illustration shows the surface target visualization results obtained using rigid registration and non-rigid registration methods under different deformation states in embodiments of the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0067] This invention proposes a mixed reality surgical navigation method, such as... Figures 1-2 As shown, the method includes:

[0068] S1. Obtain a preoperative 3D model of the organ;

[0069] S2. Use mixed reality devices to track intraoperative organ surface data;

[0070] S3. The organ 3D model and the organ surface data are fused to generate the best matching deformation correction model;

[0071] S4. The best-matching deformation correction model is holographically projected to display the corrected internal structure of the organ.

[0072] The method begins with S1, in which a 3D model of the patient's organs is obtained before surgery.

[0073] According to an embodiment of the present invention, the patient undergoes an imaging examination (CT or MRI) before surgery. Based on the CT or MRI images, image segmentation software is used to segment organs (e.g., liver), blood vessels inside the organs, and tumors to obtain a detailed preoperative 3D model of the organs.

[0074] Then, in S2, intraoperative organ surface data is tracked using a mixed reality device; including: S21, pre-setting retroreflective markers, including: setting multiple retroreflective markers on an external reference tool fixed near the organ, and setting multiple retroreflective markers on each organ surface data acquisition tool, wherein the organ surface data acquisition tool includes a stylus and surgical instruments; S22, defining the shape of the organ surface data acquisition tool based on the positions of the multiple retroreflective markers, wherein the shape is the relative positional relationship between the geometric center of the tool and the multiple retroreflective markers on it; S23, during the operation, the organ surface data acquisition tool is slid across the organ surface to acquire intraoperative organ surface data; using mixed reality... A camera in a real-world device acquires a continuous reflectance image of an organ surface data acquisition tool sliding on the organ surface in real time; the reflectance image includes an external reference tool; S24, the 3D position of each retroreflection mark in the continuous reflectance image is detected, and the retroreflection mark is separated from other objects; S25, the organ surface data acquisition tool is identified based on the 3D position of each retroreflection mark and the predefined shape of the organ surface data acquisition tool, thereby determining the 3D position of the tool's geometric center; S26, organ surface data is determined based on the 3D position of the organ surface data acquisition tool in the continuous reflectance image, wherein the organ surface data is the 3D coordinates of a series of points on the organ surface.

[0075] According to an embodiment of the present invention, after a routine surgical procedure, the soft tissue of the organ deforms. A continuous reflectivity image is acquired using the AHAT (Articulated Hand Tracking) camera of a mixed reality head-mounted display, and intraoperative soft tissue surface data is collected using a tracking stylus. Specifically, the AHAT camera in the mixed reality head-mounted display device (HMD), particularly... 2. Construct a robust tool tracking framework. This framework includes: 3D marker center detection, tool definition, tool identification and localization, and external reference tool detection.

[0076] 1) Three-dimensional marker center detection: Multiple (greater than or equal to 4) retroreflective markers are placed on each organ surface data acquisition tool, including a stylus and surgical instruments. These retroreflective markers exhibit extremely high intensity values ​​in the reflectivity images acquired by the AHAT camera, which are represented in 16-bit unsigned integer format. The values ​​of the retroreflective markers are significantly higher than the surrounding environment, typically exceeding 2000 at the center. The retroreflective markers can be reflective spheres. Figure 3 An example of the arrangement of retroreflective markings on a stylus is shown.

[0077] To detect these retroreflective markers, an intensity-based thresholding method is applied to separate them from other objects in the scene. Then, a connected component detection algorithm is used to isolate individual retroreflective markers, taking into account environmental noise such as reflections from planes or random noise. An additional threshold for the area of ​​connected components helps to accurately extract individual markers. Then, based on the depth information of the AHAT camera, the 3D position of each retroreflective marker is calculated by backprojecting the center pixel of each retroreflective marker onto a unit plane using the camera's intrinsic parameters.

[0078] 2) Tool Definition: The shape of the organ surface data acquisition tool is defined based on the positions of multiple retroreflective markers, whereby the shape is the relative positional relationship between the tool's geometric center and the multiple retroreflective markers on it. This involves extracting the 3D distribution of the retroreflective markers from a single frame to generate the tool's geometry. The tool definition is generated by removing the average translation of the retroreflective marker set, ensuring the tool center is located at the geometric center of all the retroreflective markers. The shape of the organ surface data acquisition tool is calculated using the retroreflective marker positions extracted from each frame. This step involves minimizing the root mean square error between the marker positions observed in different frames.

[0079] 3) Tool Identification and Localization: The organ surface data acquisition tool is identified based on the 3D position of each retroreflective marker and the predefined shape of the tool, thereby determining the 3D position of the tool's geometric center. This involves identifying multiple tools from a single AHAT frame by estimating the Euclidean distance between the detected retroreflective markers and the predefined tool shape. A depth-first graph search algorithm is used to find a subset of detected retroreflective markers that match the defined tool shape.

[0080] After extracting all possible solutions for each tool, the next step is to process redundant information in these solutions. For example, some solutions may reuse the same marker point, or multiple solutions may use different index orders to estimate the pose of the same tool. When multiple solutions use the same set of marker points (but in different index orders) to detect the same tool, the solution with the smallest loss value L is retained, and the rest are deleted. Then, T is defined for each solution and its tool. lThe loss values ​​L are sorted in ascending order, and the solution with the smallest loss is taken as the reference value. If other solutions conflict with the reference solution or attempt to detect the same tool, they are deleted. This process is repeated until all defined tools have been detected or no more solutions are available.

[0081] Furthermore, to ensure the accuracy of tool identification, when multiple tools are identified for acquiring data from the organ surface, two thresholds are applied to remove mismatched tools: one threshold is used to identify mismatches between corresponding edges of the tool (corresponding to the second condition), and the other threshold is used to identify the overall tool shape (corresponding to the first condition). These thresholds are calculated based on the standard error of the detected retroreflective marker positions, which varies with depth. This method enables the framework to accurately identify and locate multiple tools simultaneously in the scene. The first condition is:

[0082]

[0083] Among them, t shape Indicates a preset threshold for the shape; This represents a standard spatial distribution template of all retroreflection marks in the l-th tool relative to the tool's geometric center; N represents the k-th subset of candidate retroreflection tags that may match the l-th tool definition; l Representation tool T l The number of retroreflection markers included in the definition; This represents the Euclidean distance between the i-th and j-th retroreflection markers in the l-th tool definition. This represents the actual measured distance between the i-th and j-th retroreflective markers in the candidate retroreflective marker subset; the second condition is: Where t side This indicates a preset threshold for the side length; the recognition results that simultaneously satisfy the first and second conditions will be retained.

[0084] In practice, the preset thresholds for shape and side length are set as follows. Consider the detection marker point {P}. n The three-dimensional positioning uncertainty of} is due to the standard error σ of depth detection. p This value varies with the depth of observation, therefore it can be used to estimate the maximum standard error of one side. If a detection probability of 95% is desired, a preset threshold for the side length can be set as: t side =2σ side The preset shape threshold is:

[0085] 4) External Reference Tool Detection: Tracking requires a stable holographic reference; therefore, an external reference tool is introduced as the world origin. This external reference tool, equipped with retroreflective markers, is crucial for maintaining spatial stability and solves the problem... 2. Inherent limitations of built-in SLAM functionality.

[0086] The external fixation reference tool is an asymmetrical cross-shaped structure with four reflective spheres. Theoretically, it can be placed anywhere near the organ, but its position must be fixed after placement. In practice, it is attached directly to the organ surface using surgical adhesive to counteract shape fluctuations caused by breathing during surgery. This external reference tool anchors the mixed reality environment to a fixed coordinate system, ensuring that the virtual object remains aligned with the physical workspace under dynamic conditions.

[0087] The system tracks two tools simultaneously: an organ surface data acquisition tool and an external reference tool. The stylus, acting as the data acquisition tool, captures the precise spatial information needed for tasks such as surface measurements and intraoperative point acquisition. Furthermore, a tip transformation is assigned in Unity to represent the stylus tip, enabling accurate surface data measurement. Meanwhile, the external reference tool, serving as the world origin, provides a stable foundation for overlaying all location data and virtual models. By binding all measurements and visualizations to this external anchor point, such as… Figure 3 As shown, the transformation relationships between the AHAT coordinate system, the world coordinate system, and the image coordinate system are illustrated, ensuring a robust and unified coordinate framework. This dual-tool tracking method not only enhances holographic stability but also forms the backbone of the entire navigation system, enabling seamless integration of tracking, data collection, and visualization, and facilitating accurate and reliable augmented reality applications in surgical workflows.

[0088] This external reference tool serves as a stable world origin, enabling 2. It can maintain consistent spatial alignment of the hologram regardless of environmental or user-induced interference. This integration utilizes... The tracking capability of 2 ensures that the virtual model remains anchored within the physical workspace. Experimental results show that the stability of the hologram is significantly improved after placing an external reference tool, with minimal drift and consistent virtual object positioning. Figure 4 A comparison chart of the tracking performance of a mixed reality head-mounted display with and without an external reference tool is shown, where (a) represents the tracking accuracy measured by the standard deviation of the point cloud, and (b) represents the tracking accuracy measured by the reference registration error.

[0089] In addition, voice commands for "record point" and "stop recording point" were implemented to facilitate the recording process. 2. The tracking results are integrated with the preoperative 3D model. The system establishes a UDP connection, which enables the recorded tracking data to be transmitted to the PC. This setting enhances the overall efficiency of the data collection process.

[0090] Then, in S3, the organ 3D model and the organ surface data are fused to generate the best matching deformation correction model; including: S31, initially aligning the point cloud data corresponding to the organ 3D model and the organ surface data; S32, using an improved iterative nearest point algorithm to re-align the point cloud after the initial alignment; S33, using a linear iterative boundary reconstruction method to process the point cloud data after the re-alignment to compensate for soft tissue deformation.

[0091] According to an embodiment of the present invention, firstly, in S31, the point cloud data corresponding to the organ 3D model and the organ surface data are initially aligned, i.e., point-based registration (PBR) is used to generate an initial pose estimate using anatomical landmarks that are identifiable in both preoperative imaging data and intraoperative presentation. Specifically, multiple salient features are labeled on the preoperative organ 3D model to obtain multiple corresponding point clouds. For example, the salient features of the liver can be the falciform ligament, the left inferior ridge, and the right inferior ridge. These features are consistently identifiable in both datasets (the point cloud dataset corresponding to the organ 3D model and the organ surface data). Subsequently, the centroid of each point cloud is calculated. Next, the intraoperative organ surface data corresponding to the multiple salient features is acquired to generate another set of three-dimensional point clouds, and its centroid is calculated similarly. Then, point-based registration is performed between these two sets of multiple centroid points as an initial alignment step between the preoperative model and the intraoperative data. The above process of obtaining the corresponding preoperative point cloud based on the organ 3D model can be performed using existing software for point cloud extraction.

[0092] Then, in S32, the improved Iterative Nearest Point (PBR) algorithm is used to align the initially aligned point cloud again. Specifically, homologous anatomical feature regions (i.e., labeled salient features) are used to simultaneously guide the point correspondence determination process and play a more important role in point base registration in each iteration. The weighting mechanism used for biased PBR changes dynamically throughout the algorithm execution: in the initial stage, homologous regions dominate the registration process to ensure they align first; while in subsequent iterations, their influence gradually weakens. Let the source point cloud set be S = {s} m ), where m = 1, ..., N S The target point set is T = {t} n}, where n = 1, ..., N T Assume that point clouds S and T each contain N points. p A salient subset of features is used to represent homologous anatomical feature regions; further, a subset of features is defined. and It is an integer array whose values ​​are in the range [0, N].p The value is taken from ], where 0 indicates that the point does not belong to any significant feature region, and a value greater than 0 indicates that the point belongs to a specific significant anatomical feature region (for example, a value of 1 indicates that it belongs to the first significant anatomical feature region); let the weight set be {w m}, where At that time, w m =1; At that time, w m =w PBR w PBR This is the dynamic weighting factor used for bias point base registration in each iteration.

[0093] For each point in the source point cloud, in the process of finding the nearest neighbor in the target point cloud, a weighting factor w is introduced to determine the correspondence of the biased salient feature point set. PC Its value range is: 0 <w PC <1. This weighting factor is used for C operations at the nearest point. m If the Euclidean distance d between salient feature point pairs is significantly reduced, then the weighted nearest point operation... The calculation formula is as follows:

[0094]

[0095] In other words, if the source point and the target point both belong to the same salient anatomical feature region, their distance is multiplied by a smaller weight w. PC This increases the likelihood that a point will be selected as the nearest point. This approach is more general and flexible: it does not restrict non-salient feature points from corresponding to salient feature regions, nor does it impose hard constraints; it even allows salient feature points to correspond to non-salient feature regions in certain situations.

[0096] After determining the corresponding points, the weighted rigid point basis registration method is used to find the rigid body transformation matrix Θ, which minimizes the following objective function f(Θ):

[0097]

[0098] Where, N M This represents the total number of points in a point cloud within the acquired organ surface data. This indicates a weighted nearest-point operation, s m Let T represent the source point cloud, and T represent the set of point clouds corresponding to all salient feature regions; if Then w m =1, if Then w m =w PBR w PBR This represents the dynamic weighting factor used for bias point base registration to enhance the influence of salient feature regions in the transformation calculation; Θ is the label that identifies which salient feature region the m-th point in the source point set belongs to (0 indicates a normal point, >0 indicates a specific anatomical feature region); Θ represents the rigid body transformation matrix.

[0099] Furthermore, considering that registration error (FLE) of anatomical points and soft tissue deformation may lead to very poor initial alignment, the algorithm initially allows salient anatomical feature regions to play a dominant role in PBR to avoid getting trapped in erroneous local optima. However, since segmentation errors and source-target salient anatomical feature regions are not perfectly one-to-one correspondences, excessive emphasis on salient anatomical feature regions may cause registration to converge to erroneous results. Therefore, it is necessary to gradually reduce the dependence on salient anatomical feature regions, allowing the overall surface data to play a greater role in registration. Accordingly, this embodiment of the invention defines a dynamic weighting factor w. PBR The formula for how (t) changes with the number of iterations t is as follows:

[0100] w PBR (t)=w PBR,max ×e -λ(t-1) +w PBR,base ×(1-e -λ(t-1) )

[0101] Among them, w PBR,max This represents the maximum value of the preset bias point base registration weight; w PBR,base The base weights at the end of the iteration are represented by λ, which represents the relaxation factor, controlling the rate of weight descent. A smaller λ value indicates a longer period where significant anatomical features dominate registration. t represents the current iteration number; when t is large, w... PBR (t)→w PBR,base This mechanism ensures that regions with significant anatomical features provide anchoring in the early stages, gradually handing over control to global surface registration in later stages, thus improving robustness.

[0102] Then, in S33, the point cloud data after re-alignment is processed using the linear iterative boundary reconstruction method to compensate for soft tissue deformation.

[0103] The Linearized Iterative Boundary Reconstruction (LIBR) method transcends rigid registration by integrating a biomechanical model that accounts for complex soft tissue deformation. During preoperative planning, a basis for potential deformation is first established on the liver surface by systematically distributing control points. These control points are then perturbed along orthogonal directions to generate a linear basis that contains the fundamental elastic response to the boundary conditions of local spatial deformation within the tissue model. During the intraoperative phase, the actual deformation state is reconstructed by simultaneously estimating boundary condition parameters and rigidity transformation parameters. This process is carried out through an optimization framework utilizing a closed-form gradient function that minimizes a loss function consisting of two key components: the discrepancy between observed intraoperative data and deformation model predictions, and the overall strain energy of the system. This dual optimization ensures that the resulting deformation not only matches available intraoperative measurements but also maintains physically plausible tissue behavior governed by biomechanical model constraints. The result is a volumetric deformation correction model that accurately represents the intraoperative state of the organ, capturing surface alignment and internal structural relationships.

[0104] The specific steps are as follows: S331, the point cloud corresponding to the organ 3D model is rasterized in advance, and each raster vertex is perturbed on three axes respectively; according to Saint-Venant's principle, each perturbation is solved by applying millimeter displacement to each raster vertex and applying zero displacement at other raster vertices, and the response includes the displacement response matrix.

[0105] S332. Construct a displacement basis according to the superposition principle. The displacement basis is the product of the displacement response matrix and the weight vector of each perturbation. The length of the weight vector is three times the number of grid boundary vertices. The weight vector of each perturbation is solved iteratively according to the following process:

[0106] S3321. The displacement base is superimposed on the point cloud corresponding to the organ 3D model after the point cloud is aligned again to obtain the reconstructed deformation correction model.

[0107] S3322. Solve for the objective function value by minimizing the following design objective function formula:

[0108]

[0109] Where β = [α, τ, θ], α represents the weight vector for each perturbation, and τ, θ represent the parameters of the rigid body transformation matrix; w F N represents the weight of the salient feature F; F f represents the number of points representing the salient feature F; i w represents the model error for the i-th data point; E f represents the regularization coefficient; E The energy penalty function represents the overall 3D model of the organ;

[0110] S3323, If |φ(β) t )-φ(β t-1 )|>10 -12 Then return to S3321 and continue iterating, where φ(β) t ) represents the objective function value obtained in the t-th iteration, β t Let represent the value of β obtained after the t-th iteration; and update the initial value of β for the next iteration according to the following formula:

[0111] β t+1 -β t =(J T WJ+χdiag(J T WJ)) -1 J T Wf

[0112] Where, f = [f i ,f E ] represents the error vector; The Jacobian matrix representing the error f; W = diag(w F / N F ,w E ) represents the weighting factor for each type of error; χ is the regularization parameter used to balance the weights of fitting error and system stiffness;

[0113] S3324, If |φ(β) t )-φ(β t-1 )|<10 -12 Then stop iterating and obtain the value of β after the t-th iteration. t Then, the weight vector α for each perturbation is obtained;

[0114] S333. The displacement base is superimposed on the point cloud corresponding to the organ 3D model after the point cloud is aligned again to obtain the reconstructed best matching deformation correction model.

[0115] like Figure 5 As shown, S33 will be described in detail below.

[0116] Blood vessels and organs were modeled using relevant software, and tetrahedral finite element meshes of the organs were generated. As an example, the mesh edge length was discretized to 4 mm; a typical liver, after rasterization, has approximately 25,000 vertices. An isotropic linear elastic finite element model was used to simulate organ deformation. Under static equilibrium, the linear elasticity is governed by the three-dimensional Navier-Cauchy equations:

[0117]

[0118] Where E is Young's modulus, v is Poisson's ratio, u is displacement, and F is applied force. Using the Galerkin weighted residual method and linear Lagrange basis functions, the above partial differential equation can be transformed into: Ku = f, where K is the global stiffness matrix and f is the known external force vector. In the forward boundary value problem, the displacement vector u in the domain can only be solved when the displacement and force at the boundary are known; while the goal of the reverse boundary reconstruction problem is to estimate the displacement response by using the partially observed true displacement vector u, under the premise that the exact spatial correspondence is unknown. Distributed boundary loads.

[0119] According to the superposition principle, a displacement basis can be constructed such that: in Let α be the displacement response matrix generated by a unit perturbation in all three spatial directions for all boundary nodes. * The weight vector for each perturbation mode has a length three times the number of boundary nodes. Solve for α. * This is equivalent to a full-resolution boundary reconstruction problem, allowing each boundary node to deform independently in three directions. However, due to computational cost and insufficient information, directly solving this problem is not feasible in practice.

[0120] To reduce dimensionality, the reconstructed degrees of freedom are restricted to a set of control points uniformly distributed on the boundary. The loads at these control points are equivalent to a linear combination of concentrated point loads. According to the Saint-Venant principle, the difference in response between the actual and approximate loads decays rapidly with distance. Therefore, k-means clustering can be used to uniformly distribute k = 45 control points on the surface, each independently perturbed in three directions, generating 3k deformation modes. Each perturbation is solved by applying a 5 mm displacement to the control point and zero displacement boundary conditions at other control points. The resulting displacements, stresses, and strains are calculated using linear elasticity theory. Point loads introduce non-smoothing artifacts locally, thus requiring Saint-Venant relaxation, i.e., redistributing the loads to the boundary nodes in the neighborhood of the control points. The relaxation response of the Voronoi region (half the radius of the control point spacing) near the control point is obtained by constructing it into static equilibrium, serving as the final response mode for that control point. This set of responses constitutes a linear basis for spatial local deformation, suitable for reconstructing the deformation state from sparse data.

[0121] The displacement response matrix J is constructed from these responses. u Stress matrix J σ and strain matrix J ∈ satisfy:

[0122]

[0123] Where α is a deformable state vector of length 3k (i.e., the weight vector of each perturbation mode).

[0124] The Levenberg-Marquardt (LM) optimization algorithm is used to jointly minimize model data errors and system strain energy, thereby solving for the deformation state, including the rigid transformation. To reduce the inherent rotational error of linear elasticity and optimize the rigid transformation parameters (including translation τ and rotation θ), the total parameter vector is defined as: β = [α, τ, θ]; the objective function is:

[0125]

[0126] Where, w F N represents the weight of feature F; F f represents the number of points representing feature F; i w represents the model error for the i-th data point; E It is the regularization coefficient; f E This represents the average strain energy density of the system. M is the total number of nodes in the raster vertex grid, f E This energy penalty term is added to the objective function as a regularization term to control the rationality and smoothness of deformation during registration. The weight w of this energy penalty term is adjusted accordingly. E This can balance registration accuracy and the physical rationality of deformation. It should be noted that different feature types (such as organ surfaces, blood vessel centerlines, single-point markers, etc.) correspond to different model error expressions (i.e., f). i (The specific expression).

[0127] The error at each iteration is: β t+1 -β t =(J T WJ+χdiag(J T WJ)) -1 J T Wf; where f = [f i ,f E ] represents the error vector; The Jacobian matrix representing the error f; W = diag(w F / N F ,w E ) represents the weighting factor for each type of error; χ is the regularization parameter used to balance the weights of fitting error and system stiffness; the termination condition is: error change |φ(β) t )-φ(β t-1 )|<10 -12 .

[0128] It is important to note that although the model's displacement and strain solutions are independent of material stiffness (only displacement boundary conditions are applied), the strain energy is proportional to the stiffness. Therefore, this can be achieved by adjusting w. E This is used to indirectly control the "stiffness" of the model in order to adapt to the actual organ characteristics of different patients.

[0129] Then in S4, the best-matched deformation correction model is holographically projected to display the corrected internal structure of the organ. Figure 6 A holographic projection of the liver's anatomical structure is shown.

[0130] According to embodiments of the present invention, these optimally matched deformation correction models are then projected as virtual anatomical representations of the patient's organs in the real physical space, forming an interactive mixed reality, all facilitated by the MR helmet, thereby enhancing intraoperative visualization and surgical precision.

[0131] The HoloLens tracking system serves as the world origin for displaying virtual models. It employs a server-client architecture designed for use with... 2. Efficient transmission and display of 3D models in mixed reality environments. The server, implemented in Python, manages the distribution of model files and transformation data, categorizing them into rigid and non-rigid types and storing them in separate directories. Once a client connection is established, the server transmits a batch of files, including the 3D model and transformation matrices; the client receives and processes these files. The model is rendered in a 2-view configuration. The client maintains a persistent connection with the server, handling potential disconnections and automatically attempting to reconnect. Once the model data is received, the client saves the file locally, reads the transformation matrix, loads the 3D model, and correctly scales, transforms, and positions it. An external reference tool is set as the world origin, ensuring that the display and tracking scenes share a unified coordinate system. Careful attention must be paid to coordinate system conventions when implementing the HoloLens display system to ensure proper alignment and visualization of the virtual model. Because components of the HoloLens ecosystem use different conventions—such as the AHAT camera, the HoloLens built-in world space, Unity, and external visualization tools like ParaView—the HoloLens ecosystem involves transformations between multiple coordinate systems. The AHAT camera operates in a left-handed coordinate system, which is automatically converted to a right-handed coordinate system when the data is processed by the HoloLens built-in world space via the Mixed Reality Toolkit (MRTK).

[0132] To enhance visual differentiation, each model is assigned a unique color from a predefined color palette. The client organizes the models into two containers: one for rigid models and another for non-rigid models. Users can easily switch visibility between them. A key feature of the system is the ability to switch between displaying rigid and non-rigid models using voice commands, providing users with an intuitive and hands-free method to switch between different anatomical representations. The system also includes a visual indicator displaying the current connection status with the server, enhancing user awareness of the application's state. After the model is successfully loaded and displayed, the client removes temporary files for efficient storage management.

[0133] In summary, this invention provides a mixed reality surgical navigation method, comprising three main stages: tool tracking, deformation correction, and dynamic 3D model display in a mixed reality environment. Microsoft [the system / technology] is used in this method. 2. As a mixed reality helmet. In the clinical workflow, patients first undergo preoperative imaging to acquire data for segmentation of organs, internal blood vessels, and tumors, thereby creating a detailed preoperative 3D model; after routine surgical procedures cause soft tissue deformation of organs, the helmet is used... A two-dimensional articulated hand-tracking (AHAT) camera and a tracking stylus collect intraoperative organ surface data; this intraoperative surface data is then integrated with a preoperative 3D model to generate a deformable correction model that is optimally aligned with the observed intraoperative anatomy; the deformed virtual model is then projected onto the appropriate physical location within the patient, thereby providing visualization of predicted subsurface anatomy and occult metastases. This invention aims to enhance intraoperative decision-making by accurately representing the anatomical location of key subsurface features.

[0134] This invention also proposes a mixed reality surgical navigation system, such as... Figure 7 As shown, the system includes: an organ model building module 710, configured to acquire a preoperative 3D organ model; an organ real-time coordinate calculation module 720, configured to track intraoperative organ surface data using a mixed reality device; an organ deformation correction module 730, configured to fuse the organ 3D model and organ surface data to generate an optimal matching deformation correction model; and a projection display module 740, configured to holographically project the optimal matching deformation correction model to display the corrected internal structure of the organ.

[0135] The function of the mixed reality surgical navigation system described in this embodiment can be explained by the aforementioned mixed reality surgical navigation method. Therefore, for the parts of the system embodiment not described in detail, please refer to the above method embodiment, and they will not be repeated here.

[0136] The system architecture of this invention can be extended to various soft tissue surgeries, such as laparoscopic liver surgery, breast tumor ablation surgery, and minimally invasive spinal surgery.

[0137] The technical effects of the present invention were further verified through experiments.

[0138] To evaluate the accuracy of the navigation system, target points were created on the surface of a liver phantom to systematically compare the positions of real and virtual target points. Ten cross-shaped target points were drawn on the surface of the liver phantom using a marker pen as physical target points. These target points were then digitized and integrated into the system for further analysis.

[0139] First, the surface of the liver phantom was reconstructed using Adobe Substance 3D Sampler, with 55 photographs taken from different angles to cover the entire surface. The software generated a 3D model at a relative scale, which was then scaled down to full size using a sphere with a known diameter of 2.5 cm as a reference. After scaling, 3D cross-shaped target points were created at each target location using Blender software. Each cross-shaped target point consisted of small cylinders with a diameter of 0.5 mm and an axial length of 5 mm, facilitating visualization in a virtual environment. Then, these 3D cross-shaped target points were registered onto the preoperative liver phantom model using the weighted ICP (wICP) algorithm proposed in this invention. In this invention, these cross-shaped target points are referred to as soft tissue "target points" and superimposed on the actual liver phantom surface as virtual surface target points to evaluate system accuracy.

[0140] The system's accuracy is evaluated by measuring the target registration error (TRE) between the actual target location and the corresponding virtual target location:

[0141] TRE(q) = ||q v -q p ||

[0142] Where, q p q represents the physical target coordinates in the tracker coordinate system. v These correspond to the coordinates of the virtual target points in the tracker's coordinate system. The stylus, tracked using the Polaris Vega optical tracking system, precisely acquires the positions of the physical crosshair target points. After the system is applied and the virtual target points are displayed in the HoloLens' field of view, these virtual target points remain fixed relative to the reference tool. However, some virtual target points may appear below the phantom. To accurately measure these positions, the phantom can be temporarily moved, as the positions of all virtual target points relative to the reference tool remain unchanged.

[0143] To comprehensively evaluate system performance, the procedure was repeated in five different deformable states of the liver phantom, yielding five sets of data (10 soft tissue target points per set). Comparing the real and virtual target point locations provides the system's accuracy characteristics under different conditions. To ensure comprehensiveness of the evaluation, three sets of data were recorded for each deformable state: the physical cross-shaped target point location, the virtual cross-shaped target point location after using rigid registration (RRM) (S32), and the virtual cross-shaped target point location after using non-rigid registration (NRM) (S33). By comparing the TRE obtained from RRM and NRM, the analysis highlights the higher accuracy achieved by NRM, thus emphasizing its potential in mixed reality applications.

[0144] The experiment placed the liver phantom in five different deformable states, such as Figure 8 As shown, this simulates the deformation caused by manipulation or liver packing during surgery. Figure 9 An example of the experimental setup and visualization of the virtual target points are shown. (a) shows the baseline registration results in the undeformed state, (b) shows the equivalent physical target point positions measured by an optical tracking stylus, and (c) and (d) present a deformed state, showing the virtual target points superimposed using rigid registration (RRM) and non-rigid registration (NRM), respectively. In each deformed state, virtual surface target points are superimposed on the deformed phantom surface using rigid and non-rigid registration methods. The overall system performance is evaluated by calculating the mean target registration error (TRE) in each case. To reduce target selection bias, the target points are uniformly distributed on the phantom surface, such as... Figure 9 As shown in (a). Through careful observation Figure 9 The results show that, in the presence of deformation, the non-rigid registration method ( Figure 9 (d) is significantly better than the rigid registration method. Figure 9 (c)).

[0145] Figure 10 The TRE results of the two registration methods under all test conditions were summarized, showing that the error compensation capability of non-rigid registration ranged from 47% to 69% under five deformation states. The mean TRE of rigid registration was 7.4 ± 4.8 mm, while that of non-rigid registration was significantly improved, with a mean TRE of 3.0 ± 2.7 mm (paired t-test, p = 0.00172).

[0146] The above experimental results demonstrate the effectiveness of the present invention and further indicate its application potential in clinical scenarios where accurate registration is crucial.

[0147] It should be noted that the terminology used in this invention is for describing specific embodiments only and is not intended to limit the scope of this application. As shown in this specification, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include the plural. The terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, or apparatus.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A mixed reality surgical navigation method, characterized in that, include: S1. Obtain a preoperative 3D model of the organ; S2. Use mixed reality devices to track intraoperative organ surface data; S3. The organ 3D model and the organ surface data are fused to generate the best matching deformation correction model; S4. The best-matching deformation correction model is holographically projected to display the corrected internal structure of the organ.

2. The mixed reality surgical navigation method according to claim 1, characterized in that, The use of mixed reality devices in S2 to track intraoperative organ surface data includes: S21. Pre-setting retroreflective markers includes: setting multiple retroreflective markers on an external reference tool fixedly placed near the organ, and setting multiple retroreflective markers on each organ surface data acquisition tool, wherein the organ surface data acquisition tool includes a stylus and surgical instruments; S22. The shape of the organ surface data acquisition tool is defined based on the position of multiple retroreflective markers, wherein the shape is the relative positional relationship between the geometric center of the tool and the multiple retroreflective markers on it; S23. During the operation, an organ surface data acquisition tool is slid across the organ surface to acquire intraoperative organ surface data; a camera in a mixed reality device is used to acquire continuous reflectance images of the organ surface data acquisition tool sliding across the organ surface in real time; the reflectance images include an external reference tool; S24. Detect the 3D position of each retroreflection marker in the continuous reflectivity image and separate the retroreflection marker from other objects; S25. Identify the organ surface data acquisition tool based on the 3D position of each retroreflective marker and the shape of the predefined organ surface data acquisition tool, and then determine the 3D position of the tool's geometric center. S26. Determine organ surface data based on the 3D position of the organ surface data acquisition tool in the continuous reflectance image, wherein the organ surface data is the 3D coordinates of a series of points on the organ surface.

3. The mixed reality surgical navigation method according to claim 2, characterized in that, In step S25, based on the 3D position of each retroreflective marker and the shape of a predefined organ surface data acquisition tool, a depth-first graph search algorithm is used to find a subset of retroreflective markers that match the shape of the predefined organ surface data acquisition tool, thereby identifying the organ surface data acquisition tool.

4. The mixed reality surgical navigation method according to claim 3, characterized in that, In step S25, when multiple organ surface data acquisition tools are identified, conditional filtering is performed on these tools to remove mismatched tools. The conditions include a first condition and a second condition, wherein the first condition is: Among them, t shape Indicates a preset threshold for the shape; This represents a standard spatial distribution template of all retroreflection marks in the l-th tool relative to the tool's geometric center; N represents the k-th subset of candidate retroreflection tags that may match the l-th tool definition; l Representation tool T l The number of retroreflection markers included in the definition; This represents the Euclidean distance between the i-th and j-th retroreflection markers in the l-th tool definition. This represents the actual measured distance between the i-th and j-th retroreflective markers in the candidate retroreflective marker subset; The second condition is: Among them, t side This indicates a preset threshold for the side length; Recognition results that satisfy both the first and second conditions will be retained.

5. The mixed reality surgical navigation method according to claim 4, characterized in that, The step S3 involves fusing the organ's 3D model and surface data to generate an optimal matching deformation correction model, including: S31. Initially align the point cloud data corresponding to the organ 3D model and the organ surface data, including: pre-setting multiple salient features on the organ; obtaining preoperative point clouds of multiple salient features based on the organ 3D model; obtaining intraoperative point clouds of multiple salient features based on the organ surface data; calculating the centroids of the preoperative point cloud and the intraoperative point cloud respectively; and registering the centroids of the preoperative point cloud and the intraoperative point cloud for initial alignment. S32. Use the improved iterative nearest point algorithm to re-align the point cloud after the initial alignment; S33. The point cloud data after re-alignment is processed using the linear iterative boundary reconstruction method to compensate for soft tissue deformation.

6. The mixed reality surgical navigation method according to claim 5, characterized in that, In step S32, the process of aligning the initially aligned point cloud again using the improved iterative nearest-point algorithm includes: For each point in the source point cloud, a weighting factor w is introduced during the process of finding the nearest neighbor in the target point cloud. PC A weighted nearest-point operation is performed, meaning that if the source point and the target point both belong to the same salient feature region, the distance between the source point and the target point is multiplied by a weighting factor w. PC ; After determining the corresponding points in the source and target point clouds, the weighted rigid point base registration method is used to calculate the rigid body transformation that minimizes the distance between the corresponding points. The objective function is: Where, N M This represents the total number of points in a point cloud of data on the surface of an organ. This indicates a weighted nearest-point operation, s m Let T represent the source point cloud, and T represent the set of point clouds corresponding to all salient feature regions; if Then w m =1, if Then w m =w PBR w PBR This represents the dynamic weighting factor used for bias point base registration. It is the label that identifies which salient feature region the m-th point in the source point cloud belongs to; Θ represents the rigid body transformation matrix.

7. A mixed reality surgical navigation method according to claim 6, characterized in that, In S32, the dynamic weighting factor w used for bias point base registration PBR The calculation formula is: w PBR (t)=w PBR,max ×e -λ(t-1) +w PBR,base ×(1-e -λ(t-1) ); Among them, w PBR,max This represents the maximum value of the preset bias point base registration weight; w PBR,base λ represents the base weight at the end of the iteration; λ represents the relaxation factor; and t represents the current iteration number.

8. A mixed reality surgical navigation method according to claim 7, characterized in that, In step S33, the point cloud data after re-alignment is processed using a linear iterative boundary reconstruction method to compensate for soft tissue deformation, including: S331. The point cloud data corresponding to the 3D model of the organ is rasterized in advance, and each raster vertex is perturbed on three axes respectively; according to Saint-Venant's principle, each perturbation is solved by applying millimeter displacement to each raster vertex and applying zero displacement to other raster vertices, and the response includes the displacement response matrix. S332. Construct a displacement basis according to the superposition principle. The displacement basis is the product of the displacement response matrix and the weight vector of each disturbance. The length of the weight vector is three times the number of grid boundary vertices. S333. The displacement base is superimposed on the point cloud corresponding to the organ 3D model after the point cloud is aligned again to obtain the reconstructed best matching deformation correction model.

9. A mixed reality surgical navigation method according to claim 8, characterized in that, In step S332, the weight vector for each perturbation is iteratively solved according to the following process: S3321. The displacement base is superimposed on the point cloud corresponding to the organ 3D model after the point cloud is aligned again to obtain the reconstructed deformation correction model. S3322. Solve for the objective function value by minimizing the formula for the objective function of the following design: Where β = [α, τ, θ], α represents the weight vector for each perturbation, and τ, θ represent the parameters of the rigid body transformation matrix; w F N represents the weight of the salient feature F; F f represents the number of points representing the salient feature F; i w represents the model error for the i-th data point; E f represents the regularization coefficient; E The energy penalty function represents the overall 3D model of the organ; S3323, If |φ(β) t )-φ(β t-1 )|>10 -12 Then return to S3321 and continue iterating, where φ(β) t ) represents the objective function value obtained in the t-th iteration, β t Let represent the value of β obtained after the t-th iteration; and update the initial value of β for the next iteration according to the following formula: β t+1 -β t =(J T WJ+χdiag(J T WJ)) -1 J T Wf: Where, f = [f i ,f E ] represents the error vector; The Jacobian matrix representing the error f; W = diag(w F / N F ,w E ) represents the weighting factor for each type of error; χ is the regularization parameter used to balance the weights of fitting error and system stiffness; S3324, If |φ(β) t )-φ(β t-1 )|<10 -12 Then stop iterating and obtain the value of β after the t-th iteration. t Then, the weight vector α for each perturbation is obtained.

10. A mixed reality surgical navigation system, characterized in that, include: The organ model creation module is configured to acquire preoperative 3D organ models. The organ real-time coordinate calculation module is configured to use a mixed reality device to track intraoperative organ surface data; An organ deformation correction module is configured to fuse the organ 3D model and the organ surface data to generate an optimal matching deformation correction model. The projection display module is configured to holographically project the best-matched deformation correction model to display the corrected internal structure of the organ.

Images