A mixed reality guided neurosurgical procedure navigation method

Abstract

The application discloses a mixed reality guided neurosurgery operation navigation method, steps are as follows: S1, obtaining preoperative medical image of patient, and constructing to obtain three-dimensional model of operation site and surface reconstruction model of operation site; S2, constructing and training multi-scale label-free registration network; S3, designing hierarchical bidirectional iterative closest point algorithm, to successively carry out rigid structure alignment and fine registration to surface reconstruction model of operation site and real-time model of operation site surface; S4, through mixed reality equipment, three-dimensional model after registration is superimposed in real operation visual field in multilevel visual presentation mode; the method of the application innovatively label-free registration method, while avoiding invasive operation, can realize high-precision and high-stability registration; compared with the traditional ICP method, the registration error is low, and the registration error has stronger robustness to initial position change, and in repeatability test, the registration error shows consistency and reliability.

Description

Technical Field

[0001] This invention relates to the fields of intelligent medical care and mixed reality technology, and in particular to a mixed reality-guided neurosurgical navigation method. Background Technology

[0002] Precise surgical navigation is a crucial prerequisite for preoperative planning and accurate lesion location in neurosurgical craniotomy. The complex and intricate intracranial anatomy, with its hidden locations of deep lesions and surrounding vital blood vessels and nerve function areas, makes preoperative localization difficult and intraoperative procedures extremely risky. To improve surgical safety, accuracy, and smoothness of operation, mixed reality-based navigation technology has gradually become a research hotspot. Through virtual-real fusion and 3D interaction, it holds the promise of constructing an intuitive, contactless intraoperative guidance system. However, in practical applications, existing systems suffer from low registration accuracy and recognition rates in complex scenarios such as changes in patient position or intraoperative occlusion, making it difficult to effectively achieve precise preoperative path planning and real-time intraoperative image guidance.

[0003] Traditional optical imaging techniques, such as optical coherence tomography (OCT) and fluorescence imaging, provide surgeons with tissue information by delivering high-resolution images, playing a crucial role, especially in tumor boundary identification, vascular localization, and minimally invasive surgery. The minimally invasive common-path OCT system developed by Evans et al. utilizes a forward-looking endoscope to achieve high-resolution imaging of deep, small lesions, but its insufficient penetration depth limits its stability in complex anatomical regions. Black et al.'s deep learning-based hyperspectral image correction and unmixing method significantly improves the tissue visualization capabilities of intraoperative fluorescence imaging for brain tumors; however, its algorithm relies on a large amount of labeled data, and its real-time performance in dynamic scenarios remains challenging. Furthermore, intraoperative brain tissue displacement or deformation can easily lead to registration errors between optical imaging and preoperative images. Pang et al. point out that current optical technologies are limited by insufficient penetration of deep tissues, the complexity of optical signal processing, and limitations in contrast agent selection, which restrict their universality in complex surgical scenarios.

[0004] To overcome the inherent limitations of traditional optical technologies, mixed reality (MR) technology, with its characteristics of virtual-real fusion and three-dimensional interaction, has gradually become a hot topic in neurosurgical navigation research. MR technology, by overlaying preoperative three-dimensional reconstruction models with the real intraoperative field of view, can directly project the spatial relationships of lesions, blood vessels, and functional areas into the surgical area, effectively compensating for the shortcomings of optical imaging in dynamic registration and spatial perception. Montemurro et al. concluded through case studies that augmented reality head-mounted displays have the potential to become a useful auxiliary tool for tumor resection, craniotomy for craniotomy of skull vault lesions, and skull base surgery. Gestel et al. developed an augmented reality-based workflow for intracranial tumor resection planning on Microsoft HoloLens II, significantly shortening the overall surgical time. The University Hospital of Angeles in Mexico City was the first in Latin America to combine VisAR software with Microsoft HoloLens 2 for craniocerebral surgery, demonstrating the significant advantages of MR navigation in improving tumor localization accuracy and reducing damage to key structures, providing a model for the promotion of cross-regional medical technologies. The mixed reality spinal navigation system developed by Elmi-Terander et al. achieved sub-millimeter-level positioning accuracy, validating the feasibility of MR in spinal canal lesion surgery. Benmahdjoub et al. proposed a multimodal labeling method that matches augmented reality devices with tracking systems, significantly improving the alignment accuracy between 3D skull models and their physical counterparts. Furthermore, Saemann et al.'s research indicates that the application of 3D virtual reality (VR) technology can significantly improve the reliability of intracranial aneurysm morphology measurements, thereby optimizing surgical approach selection and enhancing anatomical understanding. Mishra et al. explored the potential applications of VR technology in various aspects of neurosurgery, including preoperative planning, training, simulation, intraoperative usability, neuronavigation, postoperative rehabilitation, and pain management. However, mixed reality navigation is currently less commonly used in neurosurgical craniotomy. Domestic research teams focus on combining MR technology with local medical needs, emphasizing independent control and technological innovation. For example, Shen et al. explored the application of augmented reality technology in perioperative visual guidance, broadening the application avenues of AR in various fields; Chinese research teams developed a dedicated dataset for mixed reality navigation of intracranial lesions, combining artificial intelligence algorithms to optimize the accuracy and efficiency of three-dimensional reconstruction of brain tumors; the synergistic application of MR technology and neurosurgical robotics has also become a key focus, with some teams launching systems capable of autonomously achieving real-time intraoperative path planning and precise robotic arm control through MR navigation. Furthermore, to address the challenge of alignment between virtual scenes and physical reality, Sun et al.'s team proposed a fast and accurate OST-HMD calibration method.The exploration of combining wearable devices with metaverse technology has opened up new scenarios for MR navigation: intraoperative monitoring platforms based on immersive head displays can integrate patient vital signs data with virtual anatomical models in real time, thereby assisting surgeons in making multi-dimensional decisions. However, problems still exist, such as intraoperative tissue displacement easily leading to marked registration failure, and sparse anatomical features or intraoperative visual field obstruction easily leading to a decrease in the accuracy of unmarked registration. Summary of the Invention

[0005] The purpose of this invention is to provide a mixed reality-guided neurosurgical navigation method that solves the above-mentioned technical problems.

[0006] Therefore, the technical solution of the present invention is as follows:

[0007] A mixed reality-guided neurosurgical navigation method, comprising the following steps:

[0008] Step 1: Obtain the patient's preoperative medical images and construct a 3D model of the surgical site and a surface reconstruction model of the surgical site;

[0009] Step 2: Construct and train a multi-scale label-free registration network, which includes: an anatomical feature acquisition module, used to extract local and global features from medical images; an anatomical awareness attention module, whose input is connected to the output of the anatomical feature acquisition module to extract and enhance key anatomical landmark features from local and global features; a multi-scale feature fusion module, which is connected to the outputs of the anatomical feature acquisition module and the anatomical awareness attention module respectively, to fuse multi-scale features and output multi-scale enhanced features; and a Transformer module, which is connected to the output of the multi-scale feature fusion module to output registration features.

[0010] Step 3: Design a hierarchical bidirectional iterative nearest point algorithm. First, by introducing spatial consistency constraints, high-confidence regions are matched to achieve rigid structural alignment between the surgical site surface reconstruction model and the real-time surgical site surface model. Then, the Bayesian optimization framework is used for adaptive adjustment to achieve fine registration between the surgical site surface reconstruction model and the real-time surgical site surface model.

[0011] Step 4: Using mixed reality equipment, the registered 3D model is superimposed onto the real surgical field of view in a multi-layered visual presentation manner.

[0012] Furthermore, the specific implementation steps of step S1 are as follows:

[0013] Step 1.1: Standardize and spatially isotropically resample the sequence of preoperative medical images of the patient to form input volume data;

[0014] Step 1.2: Input the volume data into the 3D U-Net network to obtain the initial three-dimensional binary mask of the surgical site;

[0015] Step 1.3: Use the Monte Carlo Dropout layer to perform multiple random voxel samplings on the initial 3D binary mask, and generate an uncertainty quantization map by calculating the variance between the voxel intensity value of the initial 3D binary mask and the voxel intensity value in the corresponding input volume data in each sampling. Each uncertainty quantization map marks the voxel coverage area where the variance value is lower than the variance threshold.

[0016] Step 1.4: Set a probability threshold. The voxel coverage area where the probability of occurrence in multiple uncertainty quantization maps exceeds the probability threshold is taken as the region to be corrected. The voxels of the region to be corrected are extracted from the input volume data and the initial 3D binary mask, respectively.

[0017] Step 1.5: Input the voxels of the region to be corrected extracted from the input volume data and the initial 3D binary mask into the conditional generative adversarial network to output high-quality geometric blocks;

[0018] Step 1.6: Using the implicit surface fusion method based on the Poisson equation, high-quality geometric blocks are seamlessly integrated into the initial three-dimensional binary mask to generate a three-dimensional reconstruction model of the surgical site. Based on the three-dimensional reconstruction model of the surgical site, the surface reconstruction model of the surgical site is extracted.

[0019] Furthermore, in the anatomical features, the local feature branch consists of a first branch and a second branch. The first branch is composed of a first 3D convolutional module, a second 3D convolutional module, and a third 3D convolutional module connected in sequence. The second branch is composed of a 2×2×2 downsampling module, a first 3D convolutional module, and a second 3D convolutional module connected in sequence. The global feature branch consists of a 4×4×4 downsampling module, a first 3D convolutional module, and a second 3D convolutional module connected in sequence. Each convolutional module consists of a 3D convolutional layer, a ReLU activation function, and a BN layer. The number of channels in the three 3D convolutional modules of the first branch are set to 32, 64, and 128, respectively. The number of channels in the two 3D convolutional modules of the second branch are set to 64 and 128, respectively. The number of channels in the two 3D convolutional modules of the global feature branch are set to 64 and 128, respectively.

[0020] Furthermore, in step 2, the anatomical perception attention module consists of a 3D convolution module, an SE attention module, a multi-head attention module, and an adaptive feature fusion module connected in sequence, and the input of the adaptive feature fusion module is also connected to the output of the SE attention module.

[0021] Further, in step 2, the fusion processing steps of the multi-scale feature fusion module are as follows: 1) First, use a multi-head attention mechanism or a self-attention mechanism to process local features, global features and adaptive fusion features, and then use splicing or weighted summation to fuse features from different scales into multi-scale fusion features; 2) Use additive fusion or splicing fusion methods to fuse local features and multi-scale fusion features to obtain multi-scale enhanced features.

[0022] Furthermore, in step 2, the training of the multi-scale label-free registration network is achieved using a training dataset containing registration labels; wherein, the method for constructing the training dataset containing registration labels is as follows:

[0023] 1) Acquire similar medical images from several patients and standardize the size of each image sequence;

[0024] 2) Set registration labels for each sequence image to obtain a set of labeled images including adjustment labels and deviation labels; the adjustment labels include the translation matrix, rotation matrix and voxel offset set that convert the sequence images into a front view state; the deviation labels include the Dice similarity coefficient, Hausdorff distance and root mean square error that characterize the surgical site in the sequence image and the patient's actual surgical site.

[0025] 3) Enhance the label image set and combine it with the label image set to construct a training dataset containing registration labels.

[0026] Furthermore, in step 3, by introducing spatial consistency constraints and matching high-confidence regions, the specific implementation steps for achieving rigid structural alignment between the surgical site surface reconstruction model and the real-time surgical site surface model are as follows:

[0027] S3.2.1 Define the morphologically stable and feature-distinct rigid structural regions in the surgical site as high-confidence regions, and extract feature points of the high-confidence region contours from the surgical site surface reconstruction model and the surgical site surface real-time model respectively to obtain two sets of high-confidence region feature point sets to be matched.

[0028] S3.2.2 Calculate the descriptor similarity of each feature point pair in the two sets of high-confidence region feature point sets to be matched, and according to the preset similarity threshold, select feature point pairs with descriptor similarity less than the similarity threshold as high-confidence matching point pairs; based on all high-confidence matching point pairs, solve the optimal rigid transformation matrix through singular value decomposition to initially align the surgical site surface reconstruction model to the real-time surgical site surface model, and obtain the rigid registration model;

[0029] S3.2.3 Calculate the overlap between the reconstructed surgical site surface model and the real-time surgical site surface model after initial alignment. When the overlap is lower than the preset overlap threshold, use the iterative nearest point algorithm to optimize the rigid transformation parameters until the overlap between the two models is higher than or equal to the overlap threshold.

[0030] Furthermore, in step 3, the specific implementation steps for achieving refined registration between the surgical site surface reconstruction model and the real-time surgical site surface model through adaptive adjustment using a Bayesian optimization framework are as follows:

[0031] S3.3.1 Determine the set of registration parameters θ to be optimized, which consists of image similarity loss weights, deformation field smoothing loss weights, and feature consistency loss weights; randomly select a set of initial values ​​for the registration parameters.

[0032] S3.3.2 Construct the Gaussian process surrogate model, whose expression is:

[0033] ,

[0034] In the formula, The similarity coefficient is Dice. The root mean square error, Let θ be the cosine similarity of the feature vectors, and θ be the set of registration parameters. , , These are the weighting coefficients for the Dice similarity coefficient, root mean square error, and eigenvector cosine similarity, respectively.

[0035] S3.3.2. Starting with the initial combination of registration parameters selected in step 3.3.1, the registration parameter values ​​are updated through the acquisition function in each iteration, and the registration operation is performed to obtain new accuracy indicators until the prediction confidence of the Gaussian process surrogate model reaches the convergence criterion or the number of iterations reaches the preset upper limit, thus obtaining the optimal combination of registration parameters.

[0036] S3.3.3 Based on the rigid registration model obtained in step S3.2.3 and the optimal registration parameter combination obtained in step S3.3.2, the optimal fine registration model is obtained.

[0037] Furthermore, in step 3, patient position changes are continuously monitored, and a re-registration process is triggered when registration quality is too low. The specific steps are as follows:

[0038] S3.4.1 Periodically generate a real-time model of the patient's surgical position surface and calculate the displacement vector of the feature points of the high-confidence region contour extracted from the real-time model of the surgical position surface generated at the previous moment.

[0039] S3.4.2 Extract the depth feature vector of the specified anatomical structure in the three-dimensional reconstruction model of the surgical position. Based on the displacement vector, calculate the depth feature vector of the same anatomical structure located in the real-time model of the surgical position surface, and calculate the cosine similarity of the depth feature vectors.

[0040] S3.4.3 Calculate the mean square error and overlap between the reconstructed surface model of the surgical location and the real-time surface model of the surgical location at the current moment;

[0041] S3.4.3 Set the cosine similarity threshold, mean square error threshold, and overlap threshold respectively. If the cosine similarity is lower than the cosine similarity threshold, the mean square error is higher than the mean square error threshold, or the overlap is lower than the overlap threshold, return to step S3.1 to re-register.

[0042] Furthermore, the specific implementation steps of step 4 are as follows:

[0043] Step 4.1: Import the 3D model of the surgical site into Unity software and create a virtual probe; the virtual probe is controlled by gestures, and by reading the real-time spatial coordinates of the virtual probe, a one-to-one correspondence is formed with the 3D model of the surgical site to complete the construction of the virtual surgical scene;

[0044] Step 4.2: Connect the mixed reality device and import the virtual surgical scene from Step 4.1;

[0045] Step 4.3: Use the registration method in Step 3 to perform fine registration between the virtual surgical scene and the real surgical scene;

[0046] Step 4.4: Test the fine registration effect between the virtual surgical scene and the real surgical scene, and adjust the visual parameters of the mixed reality device to obtain the best visualization effect.

[0047] Compared to existing technologies, this mixed reality-guided neurosurgical navigation method achieves high-precision and high-stability registration while avoiding invasive procedures through an innovative markerless registration method. Compared to traditional ICP methods, this invention exhibits lower registration error under the same registration task weight and demonstrates excellent performance in various markerless registration tasks. Furthermore, in addition to good performance under standard orientation, this invention maintains low registration error under random orientation shifts, proving that the registration error is more robust to changes in initial position. In repeatability tests under the same registration task weight, the registration error exhibits consistency and reliability. Attached Figure Description

[0048] Figure 1 A flowchart of the mixed reality-guided neurosurgical navigation method of the present invention;

[0049] Figure 2 This is a schematic diagram of the structure of the multi-scale label-free registration network of the present invention;

[0050] Figure 3 This is a schematic diagram of the visualization interface and multimodal interaction interface of the method of the present invention in conjunction with the HoloLens mixed reality device;

[0051] Figure 4 This is a comparison chart of registration errors when using the method of the present invention and the traditional ICP method to register 44 sets of high-precision 3D printed skull models in an embodiment of the present invention. Detailed Implementation

[0052] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the following embodiments are by no means intended to limit the present invention.

[0053] See Figure 1 The following describes the specific implementation steps of the mixed reality-guided neurosurgical navigation method, taking craniotomy as an example.

[0054] Step 1: Obtain the patient's preoperative medical images and construct a 3D model and surface model of the surgical site. Specifically, the patient's preoperative medical images can be MRI or CT images.

[0055] Using the following patient's cranial CT images as an example, the specific implementation steps of step S1 are described below.

[0056] Step 1.1: Standardize and spatially isotropically resample the patient's cranial CT image sequence to obtain input volume data.

[0057] Specifically, the essence of cranial CT images is a three-dimensional data set composed of many consecutive two-dimensional tomographic images arranged in the vertical direction (axial direction) of the head. The original cranial CT image sequences with different specifications are standardized, that is, the intensity values ​​of the images are normalized to make the intensity data of each image have a consistent numerical range. The cranial CT image sequences are spatially isotropically resampled, that is, the geometric spatial information of the images is interpolated to make the voxel spacing of the volume data the same in the X, Y, and Z directions.

[0058] Step 1.2: Input the input volume data obtained in Step 1.1 into the 3D U-Net network to obtain an initial three-dimensional binary mask of the brain through three-dimensional segmentation.

[0059] Step 1.3: The Monte Carlo Dropout layer is used to perform multiple random voxel samplings on the initial three-dimensional binary mask of the brain. By calculating the variance between the voxel intensity value of the initial three-dimensional binary mask of the brain and the voxel intensity value in the corresponding input volume data in each sampling, multiple uncertainty quantization maps are generated. Voxels with variance values ​​below the variance threshold on each uncertainty quantization map are identified as low-confidence voxels, so as to identify multiple regions covering low-confidence voxels in the uncertainty quantization map.

[0060] Step 1.4: Set a probability threshold. Regions with a probability exceeding the probability threshold in multiple uncertainty quantization images are designated as regions to be corrected. Voxels of the regions to be corrected are extracted from the input volume data and used as reference blocks for the three-dimensional image. Voxels of the regions to be corrected are also extracted from the initial three-dimensional binary mask of the cranium and used as defect geometry blocks.

[0061] Step 1.5: Input the 3D image reference block and the defective geometric block into the conditional generative adversarial network to refine the defective geometric block and output the repaired high-quality geometric block.

[0062] Step 1.6: Using the implicit surface fusion method based on the Poisson equation, high-quality geometric blocks are seamlessly integrated into the initial three-dimensional binary mask of the brain to generate a three-dimensional reconstruction model of the brain, and a reconstruction model of the brain surface is extracted based on the three-dimensional reconstruction model of the brain.

[0063] Step 2: Construct a multi-scale label-free registration network.

[0064] This invention innovatively constructs a multi-scale label-free registration network, which consists of an anatomical feature acquisition module, an anatomical perception attention module, a multi-scale feature fusion module, and a Transformer module. This multi-scale label-free registration network aims to abandon the traditional marker-based registration method, thereby achieving precise alignment between the 3D reconstructed model and the anatomical structure of the real model, simplifying the clinical workflow while ensuring high registration accuracy.

[0065] Specifically, the implementation steps of step 2 are described below.

[0066] Step 2.1: Construct an anatomical feature acquisition module to extract anatomical features from preoperative three-dimensional images; specifically, the anatomical feature acquisition module consists of local feature branches and global feature branches.

[0067] The local feature branch is used to capture fine-grained details and preserve the original local structure of the data to the greatest extent. It consists of a first branch and a second branch. The first branch connects a first 3D convolutional module, a second 3D convolutional module, and a third 3D convolutional module in sequence. The second branch connects a 2×2×2 downsampling module, a first 3D convolutional module, and a second 3D convolutional module in sequence. The first convolutional module of the first branch consists of a 3D convolutional layer, a ReLU activation function, and a BN layer connected in sequence, with a 3×3×3 kernel and 32 channels. The second 3D convolutional module consists of a 3D convolutional layer, a ReLU activation function, and a BN layer connected in sequence. The first 3D convolutional module consists of a 3D convolutional layer with a 3×3×3 kernel and 64 channels. The second 3D convolutional module uses a 3D max pooling layer. The third 3D convolutional module consists of a 3D convolutional layer, a ReLU activation function, and a BN layer connected in sequence, with the 3D convolutional layer having a 3×3×3 kernel and 128 channels. The second 3D convolutional module consists of a 3D convolutional layer, a ReLU activation function, and a BN layer connected in sequence, with the 3D convolutional layer having a 3×3×3 kernel and 64 channels. The second 3D convolutional module consists of a 3D convolutional layer, a ReLU activation function, and a BN layer connected in sequence, with the 3D convolutional layer having a 3×3×3 kernel and 128 channels. In the local feature branch, the outputs of the first and second branches together constitute the local features.

[0068] The global feature branch is used to control the overall spatial layout and enhance feature robustness, and outputs global features. The global feature branch is specifically composed of a 4×4×4 downsampling module, a first 3D convolutional module and a second 3D convolutional module connected in sequence. The downsampling module uses a 3D max pooling layer. The first 3D convolutional module is composed of a 3D convolutional layer, a ReLU activation function and a BN layer connected in sequence, and the convolutional kernel of the 3D convolutional layer is 3×3×3 and the number of channels is set to 64. The second 3D convolutional module is composed of a 3D convolutional layer, a ReLU activation function and a BN layer connected in sequence, and the convolutional kernel of the 3D convolutional layer is 3×3×3 and the number of channels is set to 128.

[0069] Step 2.2: Construct the anatomical perception attention module, which is used to enhance features related to key anatomical landmarks. The anatomical perception attention module is specifically composed of a 3D convolutional layer, an SE attention module, a multi-head attention module, and an adaptive feature fusion module connected in sequence. The input of the adaptive feature fusion module is also connected to the output of the SE attention module. The 3D convolutional layer has a 3×3×3 kernel and 128 channels. The multi-head attention module adopts an eight-head attention mechanism.

[0070] This anatomical awareness attention module employs a two-stream architecture. One stream extracts local texture and shape features through a 3D convolutional module and enhances key responses using the channel attention mechanism of the SE attention module. Simultaneously, the other stream, based on enhanced key responses, calculates long-distance dependencies between voxels using the multi-head attention mechanism of a multi-head attention module. Furthermore, the features output from the SE and multi-head attention modules are adaptively fused in an adaptive feature fusion module using learnable weights, outputting enhanced key anatomical landmark features. In this embodiment, after training on a training dataset constructed based on cranial CT images, the anatomical awareness attention module can dynamically recognize key anatomical landmark features such as skull contours and vascular branch points.

[0071] Step 2.3: Construct a multi-scale feature fusion module, which is used to fuse the local and global features output by the anatomical feature acquisition module and the adaptive fusion features output by the anatomical perception attention module. The specific fusion operation steps are described below.

[0072] 2.3.1 To achieve adaptive fusion, a multi-head attention mechanism or self-attention mechanism is first used to process the local and global features output by the anatomical feature acquisition module, as well as the adaptive fusion features output by the anatomical perception attention module. Then, the features from different scales are fused by splicing or weighted summation to obtain multi-scale fusion features. The preferred multi-head attention mechanism is an eight-head attention mechanism.

[0073] 2.3.2. Using additive fusion or splicing fusion methods, the local features are fused with the multi-scale fusion features obtained in step 2.3.1 to obtain features that can enhance the ability to capture detailed information, thereby outputting multi-scale enhanced features.

[0074] Step 2.4: Connect the anatomical feature acquisition module, anatomical perception attention module, multi-scale feature fusion module and Transformer module in sequence, and connect the input of the multi-scale feature fusion module to the output of the anatomical feature acquisition module at the same time to construct a multi-scale label-free registration network.

[0075] The Transformer Block is the basic building block of the Transformer architecture, consisting of a multi-head self-attention mechanism and a feed-forward network connected sequentially. Specifically, the multi-head self-attention mechanism employs an eight-head attention mechanism. In practical applications, the multi-scale enhancement features obtained in step 2.3 are input into the Transformer Block to output the registered features, which have 256 channels. The registered features are specifically the accurate dataset of the three-dimensional reconstruction model of the cranium.

[0076] In practical applications, based on a precise dataset of a 3D model of the surgical site, key anatomical landmarks are extracted and used for subsequent registration. This enables precise spatial alignment of CT images acquired at different times and with different devices, ensuring that corresponding anatomical structures in the images are in consistent positions. Key features, or anatomical landmarks, include surface contours and key nodes. Key nodes include vascular contours, vascular branch points, nerves, and surgically removed tissues (e.g., tumors).

[0077] Step 2.3: Construct a training dataset containing registration labels and train the multi-scale unlabeled registration network. The specific steps are described below.

[0078] 2.3.1 Construct a training dataset containing registration labels. The construction steps include:

[0079] ① Acquire CT images of several patients and standardize the size of each sequence of images. The image size is 224×224 or higher. In this embodiment, the CT images can be directly taken from existing medical image datasets, such as the BRATS dataset and / or the IXI dataset.

[0080] ② Set registration labels for each sequence image to obtain a set of labeled images including adjustment labels and deviation labels; the adjustment labels include the translation matrix, rotation matrix and voxel offset set that convert the sequence images into a front view state; the deviation labels include the Dice similarity coefficient, Hausdorff distance and root mean square error that characterize the surgical site in the sequence image and the patient's actual surgical site.

[0081] ③ Apply random cropping, horizontal flipping, color dithering and other methods to enhance the label image set to obtain an enhanced image set, thereby improving the generalization ability of the training dataset;

[0082] ④ The labeled image set and the augmented image set together constitute the training dataset containing the registration labels.

[0083] 2.3.2. Set network training parameters.

[0084] During network training, the optimizer uses either the Adam optimizer or the SGD optimizer; the learning rate employs a cosine annealing strategy, which gradually decays from a high initial value to a minimum value to avoid local optima and achieve stable convergence; gradient pruning is used to prevent gradient explosion.

[0085] The loss function for network training is set as follows:

[0086] ,

[0087] In the formula, For image similarity loss, For deformation field smoothing loss, For feature consistency loss, For regularization loss multiplication, These are the image similarity loss weight coefficients. The deformation field smoothing loss weighting coefficient is used. The feature consistency loss weight coefficients are... This is the regularization loss multiplied by the weighting coefficient.

[0088] 2.3.3 Training the multi-scale label-free registration network.

[0089] The training dataset containing registration labels was randomly divided into training, validation, and test sets in an 8:1:1 ratio. The multi-scale unlabeled registration network was then trained, and the loss curve was monitored during the training process to ensure that the loss on the training and validation sets decreased synchronously, thus avoiding overfitting.

[0090] Step 3: Design a hierarchical bidirectional iterative nearest point algorithm. First, by introducing spatial consistency constraints, high-confidence regions are matched to achieve rigid structural alignment between the cranial surface reconstruction model and the real-time cranial surface model. Then, the Bayesian optimization framework is used for adaptive adjustment to achieve fine registration between the cranial surface reconstruction model and the real-time cranial surface model.

[0091] The specific implementation steps of step 3 are described below.

[0092] Step 3.1: Using mixed reality equipment, the patient's brain is perceived in real time during the operation, and a real-time model of the brain surface is generated.

[0093] In this embodiment, the mixed reality device can specifically be the HoloLens mixed reality device, which has a stereo vision sensor with an MR module to acquire the patient's head depth information in real time and generate a real-time model of the brain surface through the Kinect Fusion algorithm built into the device.

[0094] Step 3.2: Introduce spatial consistency constraints, match high-confidence regions, and achieve rigid structural alignment between the reconstructed cranial surface model and the real-time cranial surface model.

[0095] The specific steps in step 3.2 are described below.

[0096] S3.2.1 For the cranium, the skull surface contour is a rigid structural region with stable morphology and significant features. Therefore, in this step, the skull surface contour is defined as a high-confidence region. Then, using a medical image segmentation algorithm, feature points of the skull surface contour are extracted from both the reconstructed cranium surface model and the real-time cranium surface model, resulting in two sets of high-confidence region feature points to be matched. Two feature points at the same location in the two models constitute a feature point pair. In this embodiment, the skull surface can be defined as a typical high-confidence region; the medical image segmentation algorithm uses a 3D U-Net segmentation model based on deep learning.

[0097] S3.2.2 Calculate the descriptor similarity of each feature point pair in the two sets of high-confidence region feature point sets to be matched, and select feature point pairs with descriptor similarity less than the similarity threshold as high-confidence matching point pairs according to a preset similarity threshold (e.g., 0.8). Based on all high-confidence matching point pairs, solve for the optimal rigid transformation matrix through singular value decomposition (SVD) to initially align the cranial surface reconstruction model to the real-time cranial surface model, thus obtaining a rigid registration model. In this embodiment, the descriptor similarity is obtained based on the feature descriptor matching algorithm, which may specifically use, but is not limited to, the ORB algorithm.

[0098] S3.2.3 Calculate the overlap between the reconstructed cranial surface model and the real-time cranial surface model after initial alignment. If the overlap is lower than a preset overlap threshold (e.g., 0.85), the Iterative Closest Point (ICP) algorithm is used to optimize the rigid transformation parameters until the overlap between the two models is higher than the overlap threshold. This indicates that the spatial consistency requirement has been met, providing a precise initial alignment basis for subsequent non-rigid registration. In this embodiment, the overlap is specifically represented by the Dice similarity coefficient.

[0099] Step 3.3: Through adaptive adjustment of the Bayesian optimization framework, the fine registration of the cranial surface reconstruction model and the real-time cranial surface model is achieved, and the registration efficiency is improved while ensuring accuracy.

[0100] The specific steps for step 3.3 are described below.

[0101] S3.3.1 Determine the set of registration parameters θ to be optimized, which consists of image similarity loss weights, deformation field smoothing loss weights, and feature consistency loss weights; within the reasonable range of the above registration parameters, randomly select a set of initial values ​​of registration parameters as the starting point of Bayesian optimization.

[0102] S3.3.2 Construct the Gaussian process surrogate model, whose expression is:

[0103] ,

[0104] In the formula, The similarity coefficient is Dice. The root mean square error, Let θ be the cosine similarity of the feature vectors, and θ be the set of registration parameters. , , These are the weighting coefficients for the Dice similarity coefficient, root mean square error, and eigenvector cosine similarity, respectively.

[0105] S3.3.2. Starting with the initial combination of registration parameters selected in step 3.3.1, the iterative operation is performed. The specific operation of each iteration is as follows: input the initial combination of registration parameters and the rigid registration model after preliminary alignment into the registration tool to obtain the fine registration model; then calculate the accuracy index Dice similarity coefficient, root mean square error and eigenvector cosine similarity between the fine registration model and the rigid registration model to obtain F(θ); the eigenvector cosine similarity is any specified vector on the skull surface contour.

[0106] In each iteration, the registration parameter values ​​are updated by the acquisition function, and the Gaussian process model is continuously updated by performing registration operations and obtaining accuracy indicators until the prediction confidence of the Gaussian process surrogate model reaches the convergence criterion or the number of iterations reaches the preset upper limit, and finally the optimal combination of registration parameters is obtained; in this embodiment, the acquisition function specifically adopts the expectation improvement (EI) function.

[0107] S3.3.3 Based on the rigid registration model obtained in step S3.2.3 and the optimal registration parameter combination obtained in step S3.3.2, the optimal fine registration model is obtained.

[0108] In step S3.2, the optimal parameter combination obtained by Bayesian optimization is applied to the actual registration process, while changes in registration accuracy are monitored in real time during training. When fluctuations in registration accuracy are detected, the local optimization mechanism of Bayesian optimization is triggered to quickly adjust the parameters, avoiding a decrease in registration effect due to fixed parameters, and achieving a dynamic balance between accuracy and efficiency.

[0109] S3.4 Considering that the patient's position may move during the entire operation, it is also necessary to continuously monitor the patient's position changes through feature point matching and depth consistency verification. When the registration quality is lower than the threshold, the re-registration process is automatically triggered to ensure the reliability of the registration accuracy throughout the entire operation.

[0110] S3.4.1 During the operation, a real-time model of the patient's cranial surface is generated periodically, and the feature points of the skull surface contour are extracted using the same method as in step 3.2.1. The displacement vector of the skull surface contour feature points extracted relative to the real-time model of the cranial surface generated at the previous moment is calculated.

[0111] S3.4.2 Extract the depth feature vector of the specified anatomical structure in the three-dimensional reconstruction model of the cranium. Based on the displacement vector obtained in step S3.3.1, calculate the depth feature vector of the same anatomical structure located in the real-time model of the cranium surface, and calculate the cosine similarity of the depth feature vector as the depth consistency index.

[0112] S3.4.3. Using the same method as step S3.2.3, calculate the mean square error and overlap between the reconstructed brain surface model and the real-time brain surface model at the current moment (in this embodiment, the Dice similarity coefficient is used).

[0113] S3.4.3 Set the cosine similarity threshold (e.g., 0.9), mean square error threshold (e.g., 0.8), and overlap threshold (e.g., 0.85) respectively. Set that if the cosine similarity calculated in step S3.3.2 is lower than the cosine similarity threshold, the mean square error is higher than the mean square error threshold, or the overlap is lower than the overlap threshold, return to step S3.1 to re-register, so as to ensure that the registration accuracy meets the surgical requirements. Then continue to monitor in real time to form a closed-loop registration quality assurance mechanism.

[0114] Step 4: Using mixed reality equipment, the registered 3D model is superimposed onto the real surgical field of view in a multi-layered visual presentation manner.

[0115] Specifically, the implementation steps for step 4 are as follows:

[0116] Step 4.1: Import the 3D reconstruction model of the brain into Unity software and create a virtual probe; the virtual probe is controlled by gestures, and by reading the real-time spatial coordinates of the virtual probe, a one-to-one correspondence is formed with the imaging of the 3D reconstruction model of the brain, thus completing the construction of the virtual surgical scene;

[0117] Step 4.2: Connect the HoloLens mixed reality device and import the virtual surgical scene from Step 4.1;

[0118] Step 4.3: Use the registration method in Step 3 to perform fine registration between the virtual surgical scene and the real surgical scene;

[0119] Step 4.4: Test the fine registration effect between the virtual surgical scene and the real surgical scene, and adjust the visual parameters of the mixed reality device (such as field of view, brightness, etc.) to obtain the best visualization effect.

[0120] like Figure 3 The core visualization navigation layer, located in the central field of view, presents image information of the probe's location and employs semi-transparent stereoscopic projection technology (40% transparency) to ensure visual fusion. The dynamic reference layer, located in the sub-central region, displays the CT transverse, coronal, and sagittal planes where the probe is situated, dynamically adjusting the display using an adaptive transparency algorithm. The environmental awareness layer, located in the peripheral field of view, presents the system status using the principle of minimal interference. The gaze area is rendered with high precision, while the surrounding area is represented using a simplified representation.

[0121] Furthermore, in practical applications, the method of this invention, in conjunction with the HoloLens mixed reality device, can receive user instructions and update the visual navigation interface by recognizing the user's gaze, gestures, and voice commands. The interaction employs a multimodal contactless design, combining gaze tracking (600ms gaze threshold), gesture recognition (seven dedicated gestures), and voice commands (supporting 24 neurosurgical terms).

[0122] To ensure safety, the system employs a three-tiered confirmation mechanism, requiring combined eye-to-speech confirmation for critical operations. An adaptive learning system based on physician habits is also incorporated, automatically adjusting response sensitivity and operational preferences by recording interaction patterns to continuously optimize the user experience. All designs have undergone human factors engineering evaluation in simulated surgical environments to minimize interference with normal surgical procedures.

[0123] Furthermore, in order to evaluate the effectiveness of the system for navigation in neurosurgical craniotomy, the method of the present invention was combined with the HoloLens mixed reality device, and the overall performance, registration accuracy, visualization and interactive usability of the system were experimentally evaluated.

[0124] The specific operational steps of this experiment are as follows: First, the constructed 3D reconstruction model of the brain is imported into the HoloLens device to ensure that the virtual image can be accurately mapped to the real space. Then, the experimenters wear the HoloLens device and, in a simulated surgical scenario, use the device to accurately overlay the virtual neural image with the patient's actual anatomical structure or a simulated human body model. Through gestures such as pinching, dragging, and voice commands, they can observe and manipulate the holographic image from multiple angles, including rotation, scaling, and layering. At the same time, during the experiment, the accompanying data acquisition tools are used to record in real time the registration error between the virtual image and the real structure, the response speed of the interactive operation, the time it takes for the experimenters to complete the specified operation, and the user's satisfaction with the system.

[0125] The experimental results showed that the 17 novice medical students were extremely positive about the system's evaluation, with an overall satisfaction rate of 97.3%. This data reflects the system's outstanding performance in the field of medical education tools. Regarding the interface design, 94.1% of participants rated the system layout as intuitive and clear, with key functions readily apparent. Among them, 88.2% of participants particularly praised the system's adaptive interface design, which automatically adjusts the position of frequently used functions according to user habits, improving operational efficiency. The color scheme is harmonious and unified with the medical environment, effectively reducing visual fatigue caused by prolonged use. Ease of use was unanimously recognized by all participants (100%), with data showing that it only takes an average of 23 minutes to master the basic operating procedures. 92.3% of participants were able to independently complete the assigned clinical simulation tasks upon first use without additional guidance. Particularly noteworthy is the gesture control function, which received high praise from 94.7%. Visualization also performed excellently, with 96.5% satisfaction with the 3D model rendering quality; participants generally believed that the model accuracy fully met teaching needs. 88.9% of participants stated that the anatomical structure marking system was clear and concise, facilitating rapid identification of key structures. 91.2% of participants considered the real-time feedback mechanism to be one of the most valuable features, as it can immediately identify potential risks in operations.

[0126] To further verify the superior performance of the MS-Regnet method compared to the traditional ICP method in registration, as follows: Figure 4 The figure shows the experimental results of simulation operation tests based on 44 sets of high-precision 3D printed skull models. These 44 sets of skull models from different patients differ in shape. In the figure, the horizontal axis represents the serial number of the 3D printed skull model, and the vertical axis represents the registration error. In the bar chart corresponding to each set of 3D printed skull models, the double-headed arrows indicate the difference in registration error between the two methods.

[0127] from Figure 4The test comparison results show that, for the same registration task, the registration error using the method of this invention is lower than that using the traditional ICP method. Taking the first group of 3D printed skull models as an example, the registration error using the method of this invention is <1, while the registration error using the traditional ICP method is >1.6. Similarly, the test results are similar for the other 43 groups of 3D printed skull models, confirming the excellent performance of this method in label-free registration tasks.

[0128] In addition to exhibiting good performance under standard orientation (no initial offset), the method of this invention also demonstrated good performance in another set of clinical tests with random offset orientation conditions of 0-15° rotation and 0-10mm translation. The tests showed that the method of this invention also maintained a low registration error, with an average registration error of 1.404±0.24mm, significantly better than the 1.631±0.20mm of the traditional ICP method. This demonstrates stronger robustness to changes in initial position, which is particularly important in clinical settings. Furthermore, repeated tests were conducted to evaluate the stability of the algorithm. The maximum registration error deviation of the method of this invention in 10 consecutive registrations was only 0.22mm, while the maximum deviation of the traditional ICP method was as high as 0.64mm, demonstrating higher consistency and reliability.

[0129] In summary, this invention successfully combines mixed reality technology with high-precision label-free registration, multi-level visualization, and multimodal interaction, providing a complete and high-performance solution to address the pain points of traditional neurosurgical navigation systems. It has extremely high clinical value and promising prospects for widespread application.

Claims

1. A mixed reality-guided neurosurgical navigation method, characterized by the following steps: include: Step 1: Obtain the patient's preoperative medical images and construct a 3D model of the surgical site and a surface reconstruction model of the surgical site; Step 2: Construct and train a multi-scale label-free registration network, which includes: an anatomical feature acquisition module, used to extract local and global features from medical images; an anatomical awareness attention module, whose input is connected to the output of the anatomical feature acquisition module to extract and enhance key anatomical landmark features from local and global features; a multi-scale feature fusion module, which is connected to the outputs of the anatomical feature acquisition module and the anatomical awareness attention module respectively, to fuse multi-scale features and output multi-scale enhanced features; and a Transformer module, which is connected to the output of the multi-scale feature fusion module to output registration features. Step 3: Design a hierarchical bidirectional iterative nearest point algorithm. First, by introducing spatial consistency constraints, high-confidence regions are matched to achieve rigid structural alignment between the surgical site surface reconstruction model and the real-time surgical site surface model. Then, the Bayesian optimization framework is used for adaptive adjustment to achieve fine registration between the surgical site surface reconstruction model and the real-time surgical site surface model. Step 4: Using mixed reality equipment, the registered 3D model is superimposed onto the real surgical field of view in a multi-layered visual presentation manner.

2. The mixed reality-guided neurosurgical navigation method according to claim 1, characterized in that, The specific implementation steps of step S1 are as follows: Step 1.1: Standardize and spatially isotropically resample the sequence of preoperative medical images of the patient to form input volume data; Step 1.2: Input the volume data into the 3D U-Net network to obtain the initial three-dimensional binary mask of the surgical site; Step 1.3: Use the Monte Carlo Dropout layer to perform multiple random voxel samplings on the initial 3D binary mask, and generate an uncertainty quantization map by calculating the variance between the voxel intensity value of the initial 3D binary mask and the voxel intensity value in the corresponding input volume data in each sampling. Each uncertainty quantization map marks the voxel coverage area where the variance value is lower than the variance threshold. Step 1.4: Set a probability threshold. The voxel coverage area where the probability of occurrence in multiple uncertainty quantization maps exceeds the probability threshold is taken as the region to be corrected. The voxels of the region to be corrected are extracted from the input volume data and the initial 3D binary mask, respectively. Step 1.5: Input the voxels of the region to be corrected extracted from the input volume data and the initial 3D binary mask into the conditional generative adversarial network to output high-quality geometric blocks; Step 1.6: Using the implicit surface fusion method based on the Poisson equation, high-quality geometric blocks are seamlessly integrated into the initial three-dimensional binary mask to generate a three-dimensional reconstruction model of the surgical site. Based on the three-dimensional reconstruction model of the surgical site, the surface reconstruction model of the surgical site is extracted.

3. The mixed reality-guided neurosurgical navigation method according to claim 1, characterized in that, In the anatomical features, the local feature branch consists of a first branch and a second branch. The first branch is composed of a first 3D convolutional module, a second 3D convolutional module, and a third 3D convolutional module connected in sequence. The second branch is composed of a 2×2×2 downsampling module, a first 3D convolutional module, and a second 3D convolutional module connected in sequence. The global feature branch consists of a 4×4×4 downsampling module, a first 3D convolutional module, and a second 3D convolutional module connected in sequence. Each convolutional module consists of a 3D convolutional layer, a ReLU activation function, and a BN layer. The number of channels in the three 3D convolutional modules of the first branch are set to 32, 64, and 128, respectively. The number of channels in the two 3D convolutional modules of the second branch are set to 64 and 128, respectively. The number of channels in the two 3D convolutional modules of the global feature branch are set to 64 and 128, respectively.

4. The mixed reality-guided neurosurgical navigation method according to claim 1, characterized in that, In step 2, the anatomical perception attention module consists of a 3D convolution module, an SE attention module, a multi-head attention module, and an adaptive feature fusion module connected in sequence, and the input of the adaptive feature fusion module is also connected to the output of the SE attention module.

5. The mixed reality-guided neurosurgical navigation method according to claim 1, characterized in that, In step 2, the fusion processing steps of the multi-scale feature fusion module are as follows: 1) First, use a multi-head attention mechanism or a self-attention mechanism to process local features, global features, and adaptive fusion features, and then use splicing or weighted summation to fuse features from different scales into multi-scale fusion features; multi-scale fusion features; 2) Use additive fusion or splicing fusion methods to fuse local features and multi-scale fusion features to obtain multi-scale enhanced features.

6. The mixed reality-guided neurosurgical navigation method according to claim 1, characterized in that, In step 2, the multi-scale label-free registration network is trained using a training dataset containing registration labels; the method for constructing the training dataset containing registration labels is as follows: 1) Acquire similar medical images from several patients and standardize the size of each image sequence; 2) Set registration labels for each sequence image to obtain a set of labeled images including adjustment labels and deviation labels; the adjustment labels include the translation matrix, rotation matrix and voxel offset set that convert the sequence images into a front view state; the deviation labels include the Dice similarity coefficient, Hausdorff distance and root mean square error that characterize the surgical site in the sequence image and the patient's actual surgical site. 3) Enhance the label image set and combine it with the label image set to construct a training dataset containing registration labels.

7. The mixed reality-guided neurosurgical navigation method according to claim 1, characterized in that, In step 3, the specific implementation steps for achieving rigid structural alignment between the surgical site surface reconstruction model and the real-time surgical site surface model by introducing spatial consistency constraints and matching high-confidence regions are as follows: S3.2.1 Define the morphologically stable and feature-distinct rigid structural regions in the surgical site as high-confidence regions, and extract feature points of the high-confidence region contours from the surgical site surface reconstruction model and the surgical site surface real-time model respectively to obtain two sets of high-confidence region feature point sets to be matched. S3.2.2 Calculate the descriptor similarity of each feature point pair in the two sets of high-confidence region feature point sets to be matched, and according to the preset similarity threshold, select feature point pairs with descriptor similarity less than the similarity threshold as high-confidence matching point pairs; based on all high-confidence matching point pairs, solve the optimal rigid transformation matrix through singular value decomposition to initially align the surgical site surface reconstruction model to the real-time surgical site surface model, and obtain the rigid registration model; S3.2.3 Calculate the overlap between the reconstructed surgical site surface model and the real-time surgical site surface model after initial alignment. When the overlap is lower than the preset overlap threshold, use the iterative nearest point algorithm to optimize the rigid transformation parameters until the overlap between the two models is higher than or equal to the overlap threshold.

8. The mixed reality-guided neurosurgical navigation method according to claim 7, characterized in that, In step 3, the specific implementation steps for achieving refined registration between the surgical site surface reconstruction model and the real-time surgical site surface model through adaptive adjustment using a Bayesian optimization framework are as follows: S3.3.1 Determine the set of registration parameters θ to be optimized, which consists of image similarity loss weights, deformation field smoothing loss weights, and feature consistency loss weights; randomly select a set of initial values ​​for the registration parameters. S3.3.2 Construct the Gaussian process surrogate model, whose expression is: ， In the formula, The similarity coefficient is Dice. The root mean square error, Let θ be the cosine similarity of the feature vectors, and θ be the set of registration parameters. , , These are the weighting coefficients for the Dice similarity coefficient, root mean square error, and eigenvector cosine similarity, respectively. S3.3.

2. Starting with the initial combination of registration parameters selected in step 3.3.1, the registration parameter values ​​are updated through the acquisition function in each iteration, and the registration operation is performed to obtain new accuracy indicators until the prediction confidence of the Gaussian process surrogate model reaches the convergence criterion or the number of iterations reaches the preset upper limit, thus obtaining the optimal combination of registration parameters. S3.3.3 Based on the rigid registration model obtained in step S3.2.3 and the optimal registration parameter combination obtained in step S3.3.2, the optimal fine registration model is obtained.

9. The mixed reality-guided neurosurgical navigation method according to claim 8, characterized in that, In step 3, patient position changes are continuously monitored, and a re-registration process is triggered when registration quality is too low. The specific steps are as follows: S3.4.1 Periodically generate a real-time model of the patient's surgical position surface and calculate the displacement vector of the feature points of the high-confidence region contour extracted from the real-time model of the surgical position surface generated at the previous moment. S3.4.2 Extract the depth feature vector of the specified anatomical structure in the three-dimensional reconstruction model of the surgical position. Based on the displacement vector, calculate the depth feature vector of the same anatomical structure located in the real-time model of the surgical position surface, and calculate the cosine similarity of the depth feature vectors. S3.4.3 Calculate the mean square error and overlap between the reconstructed surface model of the surgical location and the real-time surface model of the surgical location at the current moment; S3.4.3 Set the cosine similarity threshold, mean square error threshold, and overlap threshold respectively. If the cosine similarity is lower than the cosine similarity threshold, the mean square error is higher than the mean square error threshold, or the overlap is lower than the overlap threshold, return to step S3.1 to re-register.

10. The mixed reality-guided neurosurgical navigation method according to claim 1, characterized in that, The specific implementation steps of step 4 are as follows: Step 4.1: Import the 3D model of the surgical site into Unity software and create a virtual probe; the virtual probe is controlled by gestures, and by reading the real-time spatial coordinates of the virtual probe, a one-to-one correspondence is formed with the 3D model of the surgical site to complete the construction of the virtual surgical scene; Step 4.2: Connect the mixed reality device and import the virtual surgical scene from Step 4.1; Step 4.3: Use the registration method in Step 3 to perform fine registration between the virtual surgical scene and the real surgical scene; Step 4.4: Test the fine registration effect between the virtual surgical scene and the real surgical scene, and adjust the visual parameters of the mixed reality device to obtain the best visualization effect.

Images