Mixed reality surgical navigation system and space registration method and device

Abstract

The invention relates to a mixed reality surgical navigation system and a spatial registration method and device thereof. The system comprises a mixed reality device and a registration marker, the mixed reality device comprises a processor and an optical tracker, and the registration marker comprises a registration mark sticker and a global marker. The registration mark sticker is attached to the head of a patient and comprises a plurality of coding mark points, and the global marker is detachably connected to the registration mark sticker. The optical tracker obtains the spatial position information of the coding mark point, and the processor carries out spatial registration according to the spatial position information and the image position information of the coding mark point. By combining the mixed reality technology and the accurate space registration method, the accuracy and reliability of surgical navigation are improved, and the method is suitable for the field of accurate surgery such as neurosurgery.

Description

Technical Field

[0001] This application relates to the field of medical navigation technology, and in particular to a mixed reality surgical navigation system, spatial registration method and device. Background Technology

[0002] Currently, mixed reality surgical navigation systems in neurosurgery commonly employ a patient registration method that combines optical positioning with probe tracking of marker points. While this method offers advantages such as being non-invasive and cost-effective, it suffers from the following significant technical drawbacks in practical clinical applications:

[0003] First, when the probe contacts the marker points on the patient's head, the elasticity of the scalp's soft tissue causes deformation of the marker point position due to contact pressure, introducing acquisition errors and directly affecting the accuracy of the registration results. Second, the system uses a headframe to rigidly connect the patient's head to the bed and the reference array to the headframe. This requirement necessitates that the patient be under general anesthesia, making this method unsuitable for minimally invasive surgeries under local anesthesia, significantly limiting its clinical application. More critically, after initial registration, operations that generate significant mechanical thrust, such as bone drilling, may cause relative displacement between the patient's head and the reference array. However, the existing system cannot detect and correct this displacement in real time, continuing to use outdated registration data for navigation, inevitably resulting in significant navigation errors and severely impacting surgical safety.

[0004] These technical deficiencies severely restrict the widespread application of neurosurgical mixed reality navigation systems in precision medicine and minimally invasive surgery, and urgently need to be addressed through technological innovation. Summary of the Invention

[0005] Therefore, it is necessary to provide a high-precision mixed reality surgical navigation system, spatial registration method, and device that can be applied to various clinical scenarios to address the above-mentioned technical problems.

[0006] A mixed reality surgical navigation system includes a mixed reality device and registration markers; the mixed reality device includes a processor and an optical tracker; the registration markers include registration marker patches and global markers; the registration marker patches are attached to the patient's head and include multiple coded marker points; the global markers are detachably connected to the registration marker patches; the optical tracker is used to acquire spatial location information of the coded marker points; the processor is used to perform spatial registration based on the spatial location information and the image location information of the coded marker points.

[0007] In one embodiment, the registration tag further includes a flexible substrate and an mounting interface; the coded marker points protrude from the flexible substrate; each of the coded marker points has the same thickness, or at least two of the coded marker points have different thicknesses; the coded marker points have a regular shape; and the global marker is detachably connected to the flexible substrate through the mounting interface.

[0008] In one embodiment, the registration tag further includes a tag outline; the coded tag points are located within the tag outline; the coded tag points and the tag outline are made of a developing material.

[0009] In one embodiment, the registration tag further includes a magnetic sensor, and the system further includes a magnetic field generator; the magnetic field generator is used to generate a magnetic field; the magnetic sensor is fixed in relative position to the coded tag point and is used to collect magnetic field information in the magnetic field; the processor is used to perform spatial registration based on the magnetic field information, the spatial location information, and the image location information.

[0010] In one embodiment, the registration tag further includes a pressure-sensitive sensor; the pressure-sensitive sensor is used to monitor patient skin deformation and obtain deformation information; the processor is used to perform spatial registration based on the deformation information, the spatial location information, and the image location information.

[0011] A spatial registration method is applied to the mixed reality surgical navigation system described in any of the above embodiments. The method includes: acquiring a medical image of a patient's head to determine the image position information of the coded marker point in the medical image coordinate system; the medical image includes the coded marker point; in response to a registration start command, controlling an optical tracker to capture an environmental image to obtain the spatial position information of the coded marker point in the global marker coordinate system; the environmental image includes the coded marker point and the global marker; and performing spatial registration based on the image position information and the spatial position information.

[0012] In one embodiment, the method further includes: determining whether the environmental image meets preset requirements; if not, generating adjustment prompt information based on the spatial location information.

[0013] A spatial registration device is applied to the mixed reality surgical navigation system described in any of the above embodiments. The device includes: a medical image acquisition module for acquiring medical images of a patient's head to determine the image position information of the coded marker points in the medical image coordinate system; the medical image includes the coded marker points; a registration trigger module for controlling the optical tracker to capture environmental images in response to a registration start command to acquire the spatial position information of the coded marker points in the global marker coordinate system; the environmental image includes the coded marker points and the global markers; and a registration calculation module for performing spatial registration based on the image position information and the spatial position information.

[0014] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described in any of the above embodiments.

[0015] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the above embodiments.

[0016] The aforementioned mixed reality surgical navigation system, spatial registration method, and device involve attaching a registration tag containing coded markers to the patient's head and using an optical tracker to acquire the spatial position information of these coded markers. This spatial registration is then performed by combining the image position information of the coded markers from medical images. First, the system employs non-contact optical tracking technology. By attaching a specially designed registration tag to the patient's head, high-precision spatial positioning is achieved. This design effectively avoids skin deformation errors caused by pressure during traditional contact probe registration, significantly improving registration accuracy. Second, the system uses a unique marker connection design, directly connecting global markers to the registration tag on the patient's head. This architecture not only simplifies system configuration but, more importantly, achieves seamless adaptation to both fixed and non-fixed patient head states, perfectly supporting various surgical scenarios from local to general anesthesia. Most importantly, by directly connecting the global markers to the registration tag on the patient's head, the relative positional relationship between the global markers and the patient's head remains constant. This design fundamentally solves the navigation error problem caused by patient movement in traditional systems, reliably ensuring navigation accuracy during surgery. Attached Figure Description

[0017] Figure 1 This is a functional component architecture diagram of the mixed reality surgical navigation system provided in this embodiment;

[0018] Figure 2 A flowchart of the space registration method provided in this embodiment;

[0019] Figure 3 This is a schematic diagram of the structure of the registration mark sticker provided in this embodiment;

[0020] Figure 4 This is a flowchart of the registration workflow provided in this embodiment;

[0021] Figure 5 A block diagram of the space registration device provided in this embodiment. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0023] This embodiment provides a mixed reality surgical navigation system. Figure 1 The functional component architecture diagram of the mixed reality surgical navigation system provided in this embodiment is as follows: Figure 1 As shown, the mixed reality surgical navigation system 10 includes a mixed reality device 11 and registration markers 12. The mixed reality device 11 includes a processor 111 and an optical tracker 112. The registration markers 12 include registration marker stickers 121 and global markers 122.

[0024] The registration tag 121 is applied to a patient's head. Application refers to the method of directly attaching it to the skin surface using an adhesive (such as medical tape or patch). The registration tag 121 includes multiple coded markers. Each coded marker is a specially designed mark that can be identified by the optical tracker 112 to obtain its spatial location information. These coded markers have unique geometric shapes or arrangements, enabling the optical tracker 112 to accurately identify the location of each marker.

[0025] The global marker 122 is detachably attached to the registration marker 121. The global marker 122 is used to establish a global coordinate system. The global marker 122 can have a stable geometry and can be continuously tracked by the optical tracker 112.

[0026] The optical tracker 112 is used to acquire the spatial position information of the coded marker points. The optical tracker 112 is a high-precision optical device capable of capturing marker points in the environment and calculating their three-dimensional spatial positions. The optical tracker 112 can be at least one of a monocular camera, a binocular camera, an infrared camera, or a depth camera; for example, it can be an infrared binocular camera, determining the spatial coordinates of the marker points through the principle of triangulation.

[0027] The processor 111 is used to perform spatial registration based on spatial location information and image location information of coded marker points. Spatial registration is the process of aligning the patient's medical image coordinate system with the actual physical spatial coordinate system, enabling doctors to accurately locate anatomical structures within the patient's body in a mixed reality environment.

[0028] Specifically, the processor 111 acquires medical images of the patient's head to determine the image location information of the coded marker points in the medical image coordinate system. The medical images contain coded marker points. The medical images can be image data acquired through computed tomography (CT), magnetic resonance imaging (MRI), or other three-dimensional imaging techniques. Before acquiring the medical images, the registration marker sticker 121 needs to be attached to the appropriate position on the patient's head to ensure that the coded marker points are clearly displayed in the medical images.

[0029] After the medical images are acquired, the processor 111 automatically or with the assistance of a doctor identifies coded markers in the medical images. For example, the processor 111 can use image processing algorithms, such as edge detection, shape matching, or deep learning methods, to extract the location information of the coded markers from the medical images.

[0030] In response to a registration start command, processor 111 controls optical tracker 112 to capture an environmental image to obtain spatial location information of coded marker points in the coordinate system of global marker 122. The environmental image includes coded marker points and global marker 122. The registration start command can be issued by a doctor via voice, gesture, or interface button.

[0031] After the optical tracker 112 captures an environmental image, the processor 111 automatically identifies coded marker points and global markers 122 in the image. The processor 111 can use computer vision algorithms, such as feature point detection, template matching, or deep learning methods, to extract the positional information of the coded marker points and global markers 122 from the environmental image. Using the global markers 122 as the origin of the reference coordinate system, the processor 111 calculates the three-dimensional spatial position of each coded marker point relative to the global markers 122.

[0032] Processor 111 performs spatial registration based on image location information and spatial location information. Spatial registration is the process of aligning the medical image coordinate system with the global marker 122 coordinate system. Processor 111 can use rigid body transformation algorithms, such as Singular Value Decomposition (SVD) or Iterative Closest Point (ICP) algorithms, to calculate the transformation matrix between the two coordinate systems. The transformation matrix includes rotation matrices and translation vectors, used to transform points in the medical image coordinate system to the global marker 122 coordinate system, or vice versa.

[0033] The mixed reality surgical navigation system provided in this embodiment attaches a registration tag containing coded markers to the patient's head and uses an optical tracker to acquire the spatial position information of the coded markers. This spatial registration is then combined with the image position information of the coded markers in medical images. First, the system employs non-contact optical tracking technology. By attaching a specially designed registration tag to the patient's head, high-precision spatial positioning is achieved. This design effectively avoids skin deformation errors caused by pressure during traditional contact probe registration, significantly improving registration accuracy. Second, the system uses a unique marker connection design, directly connecting global markers to the registration tag on the patient's head. This architecture not only simplifies system configuration but, more importantly, achieves seamless adaptation to both fixed and non-fixed patient head states, perfectly supporting various surgical scenarios from local anesthesia to general anesthesia. Most importantly, by directly connecting the global markers to the registration tag on the patient's head, the relative positional relationship between the global markers and the patient's head remains constant. This design fundamentally solves the navigation error problem caused by patient movement in traditional systems, reliably ensuring navigation accuracy during surgery.

[0034] Based on the above embodiments, this embodiment further describes the structure of the registration mark sticker. The registration mark sticker provided in this embodiment also includes a flexible substrate and an mounting interface; the coded marking points protrude from the flexible substrate; the thickness of each coded marking point is the same, or at least two coded marking points have different thicknesses; the shape of the coded marking points is a regular shape; the global marker is detachably connected to the flexible substrate through the mounting interface.

[0035] Specifically, the registration tag also includes a flexible substrate and an installation interface. The flexible substrate is the main body of the registration tag and is made of a soft material that conforms to the curvature of the patient's head. The flexible substrate can be made of medical-grade silicone, medical-grade polyurethane, or other biocompatible materials, and its thickness can be between 0.5 mm and 2 mm. The surface treatment of the flexible substrate is hypoallergenic, ensuring that prolonged application to the patient's skin will not cause discomfort or allergic reactions. The flexible substrate can be made of a transparent material, which facilitates the application of the registration tag to the patient's bony areas and ensures that the coded markings are aligned with the patient's bony physiological features, thereby improving data acquisition accuracy and thus registration accuracy.

[0036] The coded markers protrude from the flexible substrate. This raised design makes the coded markers easier for optical trackers to recognize, improving tracking accuracy and stability. The thickness of each coded marker can be the same, or at least two markers can have different thicknesses. When the coded markers are of the same thickness, they can range from 1 mm to 3 mm, for example, all being 2 mm. When the coded markers are of different thicknesses, specific thickness patterns can be formed, such as combinations of 0.5 mm, 1 mm, and 1.5 mm. This thickness difference can serve as additional coding information, enhancing the recognition accuracy of the coded markers.

[0037] The coded markers are regular shapes, such as circles, triangles, squares, pentagons, and other geometric shapes. These regular shapes facilitate image processing and feature extraction by the optical tracker. The diameter or side length of the coded markers can be between 3 mm and 10 mm. This size range ensures that the optical tracker can clearly identify the markers within a certain distance without causing significant discomfort to the patient.

[0038] The global marker is detachably attached to the flexible substrate via an installation interface. This interface is a specially designed connection structure on the flexible substrate and can be in the form of a snap-on, magnetic, or threaded connection. This detachable design allows the global marker to be installed or removed as needed, increasing the system's flexibility. The installation interface ensures a secure and reliable connection between the global marker and the registration tag, preventing accidental detachment during surgery.

[0039] The mixed reality surgical navigation system provided in this embodiment features a flexible substrate in the registration marker patch, ensuring that prolonged attachment to the patient's skin will not cause discomfort. The coded markers protruding from the flexible substrate are more easily recognized by the optical tracker, improving tracking accuracy and stability. The regular shape of the coded markers facilitates image processing and feature extraction by the optical tracker. This structural design enhances patient comfort, registration efficiency, and accuracy.

[0040] Based on the above embodiments, this embodiment further describes the structure of the registration mark sticker. The registration mark sticker provided in this embodiment also includes a mark outline; coded mark points are located within the mark outline; the coded mark points and the mark outline are made of a developing material.

[0041] Specifically, a marker profile is a boundary marker surrounding coded marker points, used to assist optical trackers in identifying marked regions. The marker profile can be designed as a closed curve, such as a circle, ellipse, or polygon.

[0042] The coded markers are located within the marker outline. This layout design allows the optical tracker to first identify the marker outline and then accurately locate each coded marker within it, improving recognition efficiency and accuracy. The arrangement of the coded markers within the marker outline can be a regular geometric arrangement, such as a ring, a matrix, or a specific asymmetric pattern; this arrangement itself also contains coded information.

[0043] The coded markers and their outlines are made of a contrasting material. A contrasting material is a special material that is clearly visible in medical images, such as compounds containing elements like barium, iodine, or gadolinium. These materials exhibit high contrast in medical images such as X-rays, CT scans, or MRI scans, making the coded markers and their outlines clearly visible. The choice of contrasting material depends on the type of medical image; for example, barium sulfate or iodides can be used for X-rays and CT images, while gadolinium-based contrast agents can be used for MRI images.

[0044] In some embodiments, when the optical tracker is an infrared camera: the surfaces of the coded marker points and marker contours are also covered with a highly reflective infrared material to efficiently reflect infrared light, making the coded marker points and marker contours clearly visible in the environmental images captured by the optical tracker; or the surfaces of the coded marker points and marker contours are directly integrated with infrared light-emitting diodes (LEDs) to actively emit infrared light, making the coded marker points and marker contours clearly visible in the environmental images captured by the optical tracker.

[0045] The mixed reality surgical navigation system provided in this embodiment uses the setting of the marker outline in the registration marker sticker, which enables the optical tracker to first identify the marker outline and then accurately locate each coded marker point within the outline, thereby improving the recognition efficiency and accuracy.

[0046] Based on the above embodiments, this embodiment further describes the structure of the registration label. The registration label provided in this embodiment also includes a magnetic sensor, and the system further includes a magnetic field generator. The magnetic sensor is an electronic component capable of detecting the strength and direction of a magnetic field; it can be a Hall effect sensor or a magnetoresistive sensor. The thickness of the magnetic sensor can be between 0.5 mm and 2 mm to ensure that it does not significantly increase the thickness and weight of the registration label.

[0047] A magnetic field generator is used to generate a magnetic field. It is a device capable of producing a stable magnetic field and may include multiple coils and control circuitry. The magnetic field strength generated by the generator can range from 0.1 Gauss to 10 Gauss, a range that ensures accurate detection by magnetic sensors without interfering with the patient or medical equipment. The magnetic field generator can be placed beside the bedside, within a specific distance from the patient's head.

[0048] A magnetic sensor, positioned relative to an coded marker, is used to acquire magnetic field information within a magnetic field. The magnetic sensor can be embedded in a flexible substrate, maintaining a fixed relative position with the coded marker. This fixed relative position allows the processor to infer the position and orientation of the coded marker from the magnetic field information. The magnetic field information acquired by the magnetic sensor includes magnetic field strength and direction, which can be converted into the sensor's position and orientation in three-dimensional space.

[0049] The processor is used to perform spatial registration based on magnetic field information, spatial location information, and image location information. Specifically, the processor can first enable optical tracking, determine whether the spatial location information of the coded marker points can be directly used for spatial registration based on the spatial location information, and if not, it will also enable magnetic field tracking. The magnetic field information collected by the magnetic sensor is combined with the spatial location information obtained by optical tracking and the image location information to perform registration, thereby achieving spatial registration.

[0050] It should be noted that if the number of spatial location information items is less than a predetermined value, or if the spatial location information contains data errors, it can be determined that the spatial location information cannot be directly used for spatial registration. If the number of spatial location information items is less than a predetermined value, the coded markers corresponding to the spatial location information (defined as the first coded marker) can be determined, and the spatial location information of the second coded marker (at least some of the coded markers other than the first coded marker) can be calculated based on the magnetic field information and the relative positional relationship between the magnetic sensor and the coded marker point. The spatial location information of the first coded marker and the second coded marker can be registered with the image location information of the coded marker to achieve spatial registration. In some other embodiments, the average location information of the spatial location information of the coded marker obtained by optical tracking and the spatial location information of the coded marker obtained by magnetic field tracking can also be calculated, and the average location information can be registered with the image location information to achieve spatial registration.

[0051] The mixed reality surgical navigation system provided in this embodiment utilizes a magnetic sensor in the registration marker, allowing the processor to fuse data from optical and magnetic field tracking, thus improving the accuracy and robustness of spatial registration. This multimodal fusion registration method maintains registration accuracy even under conditions where optical tracking is limited, such as occlusion or poor lighting.

[0052] Based on the above embodiments, this embodiment further describes the structure of the registration label. The registration label provided in this embodiment also includes a pressure-sensitive sensor. A pressure-sensitive sensor is an electronic component capable of detecting pressure changes, and can be a piezoresistive, capacitive, or piezoelectric sensor. The thickness of the pressure-sensitive sensor can be between 0.2 mm and 1 mm to ensure that the thickness and weight of the registration label are not significantly increased.

[0053] Pressure sensors are used to monitor patient skin deformation and obtain deformation information. Embedded in a flexible substrate, the pressure sensor can detect minute deformations of the patient's skin. These deformations may arise from changes in facial expressions, adjustments in head posture, or skin movement caused by surgical procedures. The pressure sensor's sensitivity can detect pressure changes ranging from 0.1 Newtons to 10 Newtons, a sensitivity range capable of capturing most clinically relevant skin deformations.

[0054] The processor is used to perform spatial registration based on deformation information, spatial location information, and image location information. Specifically, the processor can compensate the spatial location information involved in the registration based on the deformation information to obtain compensated spatial location information, and then register the compensated spatial location information with the image location information.

[0055] The mixed reality surgical navigation system provided in this embodiment utilizes magnetic sensors in the registration markers. This allows the processor to compensate and adjust the registration results in real time by taking into account the effects of skin deformation, thereby improving the accuracy and stability of the registration. This deformation compensation mechanism is particularly suitable for long surgeries, as skin deformation may gradually accumulate during prolonged procedures, affecting registration accuracy.

[0056] This embodiment provides a spatial registration method applied to the mixed reality surgical navigation system described above. The method is explained by its application to a processor in a mixed reality device. Figure 2 The flowchart of the space registration method provided in this embodiment is as follows: Figure 2 As shown, the method includes the following steps:

[0057] Step 201: Obtain medical images of the patient's head to determine the image location information of the coded markers in the medical image coordinate system; the medical images contain coded markers.

[0058] The processor acquires medical images of the patient's head to determine the image location information of the coded markers in the medical image coordinate system. The medical images contain coded markers. Medical images can be image data acquired through computed tomography (CT), MRI, or other 3D imaging techniques. Before acquiring the medical images, registration markers need to be affixed to the appropriate positions on the patient's head to ensure that the coded markers are clearly visible in the medical images.

[0059] After medical images are acquired, the processor automatically or with the assistance of a doctor identifies coded markers in the images. The system uses image processing algorithms, such as edge detection, shape matching, or deep learning methods, to extract the location information of the coded markers from the medical images.

[0060] Step 202: In response to the registration start command, control the optical tracker to capture an environmental image to obtain the spatial position information of the coded marker points in the global marker coordinate system; the environmental image contains the coded marker points and global markers.

[0061] The registration start command can be issued by the doctor via voice, gesture, or interface button.

[0062] In response to a registration start command, the processor controls the optical tracker to capture environmental images. After the optical tracker captures the images, the processor automatically identifies coded marker points and global markers in the images. The processor can use computer vision algorithms, such as feature point detection, template matching, or deep learning methods, to extract the positional information of the coded marker points and global markers from the environmental images. Using the global markers as the origin of the reference coordinate system, the system calculates the three-dimensional spatial position of each coded marker point relative to the global markers.

[0063] Step 203: Perform spatial registration based on image location information and spatial location information.

[0064] Spatial registration is the process of aligning the medical image coordinate system with the global marker coordinate system. The processor can use rigid body transformation algorithms, such as SVD or ICP, to compute the transformation matrix between the two coordinate systems. The transformation matrix includes rotation matrices and translation vectors, used to transform points in the medical image coordinate system to the global marker coordinate system.

[0065] After spatial registration is complete, the processor calculates the registration error, which is the distance between the transformed coded marker in the medical image and the corresponding coded marker in the environmental image. The registration error can be represented by the root mean square error (RMSE). Ideally, the registration error should be less than 1 millimeter, a level of precision sufficient for most neurosurgical procedures.

[0066] The spatial registration method provided in this embodiment involves attaching a registration tag containing coded markers to the patient's head and using an optical tracker to acquire the spatial position information of the coded markers. This spatial registration is then performed in conjunction with the image position information of the coded markers in medical images. First, the system employs non-contact optical tracking technology. By attaching a specially designed registration tag to the patient's head, high-precision spatial positioning is achieved. This design effectively avoids skin deformation errors caused by pressure during traditional contact probe registration, significantly improving registration accuracy. Second, the system uses a unique marker connection design, directly connecting global markers to the registration tag on the patient's head. This architecture not only simplifies system configuration but, more importantly, achieves seamless adaptation to both fixed and non-fixed patient head states, perfectly supporting various surgical scenarios from local anesthesia to general anesthesia. Most importantly, by directly connecting the global markers to the registration tag on the patient's head, the relative positional relationship between the global markers and the patient's head remains constant. This design fundamentally solves the navigation error problem caused by patient movement in traditional systems, reliably ensuring navigation accuracy during surgery.

[0067] Based on the above embodiments, the method provided in this embodiment further includes:

[0068] Determine whether the environmental image meets the preset requirements.

[0069] Preset requirements may include aspects such as image clarity, marker visibility, and the number of markers. The processor uses image quality assessment algorithms, such as contrast analysis, noise assessment, or feature point detection rate, to determine whether the environmental image meets the preset requirements. If the number of coded markers in the environmental image is less than a preset threshold (e.g., 80% of the total), or the recognition confidence of the markers is lower than a preset threshold (e.g., 0.8), the environmental image is considered not to meet the preset requirements.

[0070] If the conditions are not met, an adjustment prompt message will be generated based on the spatial location information.

[0071] If the environmental image does not meet preset requirements, the processor generates adjustment prompts based on currently visible coded markers. In some embodiments, the adjustment prompts are visual or audio feedback that guides the physician to adjust the position of the optical tracker to obtain a better environmental image. Visual prompts may be arrows, text, or color-coded indicators displayed on the mixed reality device's display interface. Audio prompts may be voice commands or tone variations indicating the direction and magnitude of the adjustment.

[0072] In other embodiments, the optical tracker is fixed to the housing of the mixed reality device via a control gimbal, which is communicatively connected to a processor. The processor automatically controls the optical tracker to adjust its pose (i.e., position or orientation) based on adjustment prompts to obtain a better image of the environment. For example, the adjustment prompts may include translation and rotation information. The processor sends the adjustment prompts to the control gimbal, causing it to perform corresponding movements to adjust the pose of the connected optical tracker.

[0073] The spatial registration method provided in this embodiment can provide prompts to doctors or automatically control the optical tracker to adjust the pose when the acquired environmental image does not meet the requirements. This allows for faster and more accurate pose adjustment to obtain an environmental image that meets the requirements, thereby improving the efficiency and accuracy of registration.

[0074] Based on the above embodiments, this embodiment provides a detailed description of the specific form of the registration marker and the registration workflow of the mixed reality surgical navigation system based on the registration marker.

[0075] Figure 3 This is a schematic diagram of the structure of the registration label provided in this embodiment, as shown below. Figure 3 As shown, the registration marker is in the form of an eye patch and is applied to the patient's eye. The registration marker patch includes a marking outline 301 and multiple coded marking points 302 (the form is not limited, for example, it can be dot-shaped or strip-shaped). Figure 3 (The center is dotted), one mounting interface 303 (form not limited), Figure 3 The center is recessed, and the fixing strap is made of 304 stainless steel. The registration label 121 has a designed tear-out opening on the edge. Figure 3 (Not shown in the image) This design allows doctors to easily peel and apply the registration marker 121 to the patient's eye. The main body of the registration marker 121 is transparent, while the coded marker points 302 and the marking contour 301 are coated with a reflective layer, allowing it to be identified by an optical tracker and visualized in medical imaging scans. The registration marker 121 features repositionable characteristics; it can be temporarily removed during preoperative waiting and precisely repositioned using the marking contour 301, supporting the replacement of a new registration marker and marking contour 301.

[0076] Figure 4 The flowchart of the registration workflow provided in this embodiment is as follows: Figure 4 As shown, the method includes:

[0077] Step 401, Registration Marker Installation: The doctor peels off the protective film of the registration marker and fixes it around the patient's eyes. The main body of the registration marker is transparent, while the marking outline and coded marking points are not transparent. Ensure that the coded marking point area is aligned with bony landmarks such as the brow bone and bridge of the nose. The marking outline of the edge of the registration marker can be aligned with facial anatomical landmarks for easy repositioning later.

[0078] Step 402, Medical Image Acquisition: The patient wears a registration tag to complete a CT / MRI scan. The image processing algorithm automatically identifies the outline area of ​​the registration tag, the spatial distribution characteristics of the coded marker points, and the marker point coding information.

[0079] Step 403, Patient Positioning Preparation: The doctor positions the patient, supporting various surgical scenarios. The anesthesia method can be general anesthesia or local anesthesia, the fixation method can be head frame fixation or no head frame, and the surgical position can be supine or lateral decubitus.

[0080] Step 404, Global Marker Installation: The doctor embeds the global marker into the installation interface of the registration marker patch, ensuring a rigid connection between the patient's head, the registration marker patch, and the global marker.

[0081] Step 405, Registration process starts: The doctor triggers the registration command through the interactive interface, and the processor executes optical tracking initialization, marker recognition program loading, and spatial coordinate system calibration.

[0082] Step 406, Optical Data Acquisition: The optical tracker performs multi-angle (≥3 viewpoints) environmental image capture to simultaneously capture global markers and at least 2 coded marker points.

[0083] Step 407, Spatial Registration Calculation: If the environmental image taken by the doctor meets the system requirements, the processor will call the algorithm to calculate the spatial location information of the coded marker points and match it with the image location information of the coded marker points in the patient's medical image, thereby realizing spatial registration.

[0084] Step 408, Accuracy Confirmation: After space registration is completed, the processor calculates the registration accuracy, and the doctor confirms whether the accuracy is appropriate and whether to proceed with subsequent operations.

[0085] It should be understood that, although Figure 2 , Figure 4 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 , Figure 4 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0086] This embodiment provides a spatial registration device applied to the mixed reality surgical navigation system described in the above embodiment. Figure 5 A block diagram of the space registration device provided in this embodiment, such as Figure 5 As shown, the spatial registration device 500 includes a medical image acquisition module 501, a trigger registration module 502, and a registration calculation module 503.

[0087] The medical image acquisition module 501 is used to acquire medical images of the patient's head to determine the image position information of coded marker points in the medical image coordinate system. The medical image contains coded marker points. The medical image acquisition module 501 can be directly connected to the hospital's imaging system to receive image data transmitted from CT, MRI, or other imaging equipment. The medical image acquisition module 501 also includes an image preprocessing unit for preprocessing the raw image data, such as denoising, contrast enhancement, and reconstructing a 3D model.

[0088] The medical image acquisition module 501 also includes a marker recognition unit for identifying and locating coded markers in medical images. The marker recognition unit uses various image processing algorithms, such as threshold segmentation, morphological operations, and connected component analysis, to extract features from the coded markers.

[0089] The trigger registration module 502, in response to a registration start command, controls the optical tracker to capture an environmental image to obtain the spatial position information of the coded marker points in the global marker coordinate system. The environmental image contains the coded marker points and global markers. The trigger registration module includes a command recognition unit for recognizing registration start commands from different input sources, such as voice commands, gestures, or interface buttons.

[0090] The trigger registration module 502 also includes a camera control unit, used to control the parameter settings and image acquisition of the optical tracker. The camera control unit can adjust parameters such as exposure time, gain, and white balance of the optical tracker to adapt to different lighting conditions. The camera control unit is also responsible for controlling the frame rate and resolution of the optical tracker, optimizing the real-time performance of the system while ensuring image quality.

[0091] The registration calculation module 503 is used to perform spatial registration based on image location information and spatial location information. The registration calculation module includes a coordinate transformation unit, which calculates the transformation relationship between the medical image coordinate system and the global marker coordinate system. The coordinate transformation unit implements various registration algorithms, such as point-to-point registration, surface registration, and hybrid registration, allowing the selection of the most suitable algorithm based on the specific application scenario.

[0092] The registration calculation module 503 also includes an error evaluation unit for assessing the accuracy and reliability of the registration results. The error evaluation unit calculates the distance error, angle error, and target point error between registration point pairs, generating a comprehensive error evaluation result. The error evaluation unit can also assign different weights to marker points at different locations, improving registration accuracy in key areas.

[0093] The registration calculation module 503 also includes a visualization unit to present the registration results to doctors in an intuitive way. The visualization unit generates an augmented reality overlay display, precisely aligning medical image data with the real-time view. The visualization unit supports multiple display modes, such as transparent overlay, outline display, and cross-sectional display, allowing doctors to switch between different modes as needed.

[0094] The spatial registration device provided in this embodiment attaches a registration tag containing coded markers to the patient's head and uses an optical tracker to acquire the spatial position information of the coded markers. This spatial registration is then performed by combining the image position information of the coded markers from medical images. First, the system employs non-contact optical tracking technology. By attaching a specially designed registration tag to the patient's head, high-precision spatial positioning is achieved. This design effectively avoids skin deformation errors caused by pressure during traditional contact probe registration, significantly improving registration accuracy. Second, the system uses a unique marker connection design, directly connecting global markers to the registration tag on the patient's head. This architecture not only simplifies system configuration but, more importantly, achieves seamless adaptation to both fixed and non-fixed patient head states, perfectly supporting various surgical scenarios from local anesthesia to general anesthesia. Most importantly, by directly connecting the global markers to the registration tag on the patient's head, the relative positional relationship between the global markers and the patient's head remains constant. This design fundamentally solves the navigation error problem caused by patient movement in traditional systems, reliably ensuring navigation accuracy during surgery.

[0095] For specific limitations regarding the device, please refer to the limitations of the method above, which will not be repeated here. Each module in the above device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0096] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: acquiring a medical image of a patient's head to determine image location information of coded markers in a medical image coordinate system; the medical image contains coded markers; in response to a registration start command, controlling an optical tracker to capture an environmental image to acquire spatial location information of coded markers in a global marker coordinate system; the environmental image contains coded markers and global markers; and performing spatial registration based on the image location information and the spatial location information.

[0097] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining whether the environmental image meets preset requirements; if not, generating adjustment prompt information based on spatial location information.

[0098] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the following steps: acquiring a medical image of a patient's head to determine image location information of coded markers in a medical image coordinate system; the medical image includes coded markers; in response to a registration start command, controlling an optical tracker to capture an environmental image to acquire spatial location information of coded markers in a global marker coordinate system; the environmental image includes coded markers and global markers; and performing spatial registration based on the image location information and the spatial location information.

[0099] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining whether the environmental image meets preset requirements; if not, generating adjustment prompt information based on spatial location information.

[0100] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0101] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0102] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A mixed reality surgical navigation system, characterized in that, The system includes a mixed reality device and registration markers; The mixed reality device includes a processor and an optical tracker; The registration markers include a registration marker patch and a global marker; the registration marker patch is used to be attached to the patient's head, and the registration marker patch includes multiple coded marker points; the global marker is detachably connected to the registration marker patch; The optical tracker is used to acquire the spatial location information of the coded marker points; The processor is used to perform spatial registration based on the spatial location information and the image location information of the coded marker points.

2. The system according to claim 1, characterized in that, The registration label also includes a flexible substrate and a mounting interface; The coded markers protrude from the flexible substrate; each coded marker has the same thickness, or at least two coded markers have different thicknesses; the coded markers have a regular shape. The global marker is detachably connected to the flexible substrate via the mounting interface.

3. The system according to claim 1, characterized in that, The registration marker sticker also includes a marker outline; The coded marker points are located within the marker outline; the coded marker points and the marker outline are made of developing material.

4. The system according to claim 1, characterized in that, The registration tag also includes a magnetic sensor, and the system also includes a magnetic field generator; The magnetic field generator is used to generate a magnetic field; The magnetic sensor, with its relative position to the coded marker point fixed, is used to collect magnetic field information in the magnetic field. The processor is used to perform spatial registration based on the magnetic field information, the spatial location information, and the image location information.

5. The system according to claim 1, characterized in that, The registration label also includes a pressure-sensitive sensor; The pressure-sensitive sensor is used to monitor the patient's skin deformation and obtain deformation information; The processor is used to perform spatial registration based on the deformation information, the spatial location information, and the image location information.

6. A spatial registration method, characterized in that, The method, applied to the mixed reality surgical navigation system according to any one of claims 1-5, comprises: Acquire medical images of the patient's head to determine the image location information of the coded marker points in the medical image coordinate system; the medical image contains the coded marker points; In response to a registration start command, the optical tracker is controlled to capture an environmental image to obtain the spatial location information of the coded marker point in the global marker coordinate system; the environmental image includes the coded marker point and the global marker. Spatial registration is performed based on the image location information and the spatial location information.

7. The method according to claim 6, characterized in that, The method further includes: Determine whether the environmental image meets the preset requirements; If the conditions are not met, an adjustment prompt message will be generated based on the spatial location information.

8. A space registration device, characterized in that, The device is applied to the mixed reality surgical navigation system according to any one of claims 1-5, the device comprising: A medical image acquisition module is used to acquire medical images of a patient's head to determine the image position information of the coded marker points in the medical image coordinate system; the medical image includes the coded marker points. The registration module is triggered to control the optical tracker to capture an environmental image in response to a registration start command, so as to obtain the spatial position information of the coded marker point in the global marker coordinate system; the environmental image includes the coded marker point and the global marker. The registration calculation module is used to perform spatial registration based on the image location information and the spatial location information.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 6 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 6 to 7.

Images