MR surgical navigation method and system based on multi-path camera acquisition

Abstract

The invention belongs to the field of MR surgical navigation, and particularly relates to an MR surgical navigation method based on multi-camera acquisition, which comprises the following steps: step 1, acquiring basic image positioning image data of a patient before a surgery, performing preprocessing, feature extraction and analysis to obtain basic reference data, and making a surgical plan; step 2, acquiring navigation positioning data after real-time image fusion processing in the operation of the patient, and performing preprocessing, threshold segmentation and feature extraction to obtain real-time reference data; step 3, carrying out feature matching and comparison on the basic reference data and the real-time reference data, and generating real-time navigation information according to a matching result; mR surgical navigation is achieved through head-mounted equipment, operation is convenient, space limitation is avoided, binocular infrared camera shooting is used for collecting data, the high depth perception and space positioning capacity is achieved, the real-time navigation and head tracking technology is combined, accurate and real-time surgical navigation information is provided, and the system can be applied to various surgical scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of MR surgery navigation, in particular to a MR surgery navigation method and system based on multi-camera acquisition. BACKGROUND

[0002] MR surgery navigation is a technology that uses mixed reality to visually display the patient's lesions, and three-dimensionally visualizes structures such as lesions, scalp, skull, neural fiber bundles, ventricles, and blood vessels. Doctors can navigate intraoperatively by measuring the distance between lesions and anatomical markers, thereby more intuitively and accurately understanding the patient's lesion condition and improving the accuracy and safety of surgery.

[0003] When MR surgery navigation is applied to surgery, the doctor's perspective is non-fixed, and when the perspective is switched, the navigation information of the surgical site needs to be updated synchronously. If the update is not timely, there is a risk of virtual image and actual surgical site error overlap, affecting the doctor's subsequent operation judgment. SUMMARY

[0004] (I) Invention purpose

[0005] To solve the technical problems in the background art, the present application provides a MR surgery navigation method and system based on multi-camera acquisition, which has multi-camera acquisition of navigation positioning image data after real-time image fusion processing, and the characteristics of quickly updating the visual display of real-time navigation information after the perspective is changed.

[0006] (II) Technical solutions

[0007] To solve the above technical problems, the present application provides a MR surgery navigation method based on multi-camera acquisition, comprising the following steps:

[0008] Step one, obtain the basic image positioning image data of the patient before surgery, and perform preprocessing, feature extraction and analysis to obtain basic reference data, and develop a surgical plan;

[0009] Step two, obtain the navigation positioning image data after real-time image fusion processing of the patient during surgery, and perform preprocessing, threshold segmentation and feature extraction to obtain real-time reference data;

[0010] Step three, perform feature matching and comparison of the basic reference data and the real-time reference data, and generate real-time navigation information according to the matching result;

[0011] Step four, visually display the real-time navigation information to assist the doctor's surgery navigation;

[0012] Step five, the real-time reference data is updated with the doctor's head and line of sight position, and the real-time navigation information in the visual display is updated and adjusted.

[0013] Preferably, in step one, the basic image data comes from preoperative imaging examination, such as MRI or CT scan, to obtain the patient's anatomical structure data.

[0014] Preferably, in step one, the pre-processing and feature extraction and analysis of the basic image positioning image data include:

[0015] The collected basic image positioning image data is pre-processed, including denoising, graying, and contrast enhancement, to obtain a pre-processed basic image positioning image;

[0016] The feature parameters of the pre-processed basic image positioning image are extracted, such as edge, texture, color, and other information;

[0017] The pre-processed basic image positioning image and the extracted feature parameters are analyzed and processed to develop a surgical plan and determine the surgical approach and operation target.

[0018] Preferably, in step two, the navigation positioning image data after real-time image fusion processing in the patient's surgery comes from real-time capture and recording of the RGB lens of the head-mounted MR device;

[0019] The navigation positioning image data after real-time image fusion processing is pre-processed by denoising, graying, and contrast enhancement;

[0020] The navigation positioning image after real-time image fusion processing is segmented by the best threshold value;

[0021] The best threshold value is determined by Otsu's algorithm, and the specific steps are as follows:

[0022] Calculate the gray histogram of the image: count the number of pixels at each gray level in the image;

[0023] Normalize each gray level: divide the number of pixels at each gray level by the total number of pixels to obtain the pixel proportion of each gray level;

[0024] Calculate the cumulative probability of each gray level: accumulate and sum the pixel proportion of each gray level to obtain the cumulative probability;

[0025] Calculate the global average gray value: calculate the global average gray value from the cumulative probability;

[0026] Calculate the intra-class variance: calculate the intra-class variance of each gray level based on the global average gray value;

[0027] Calculate the inter-class variance: calculate the inter-class variance of each gray level from the cumulative probability and intra-class variance;

[0028] Select the gray level corresponding to the maximum inter-class variance as the optimal threshold value;

[0029] The feature extraction includes:

[0030] The edge detection algorithm (such as Sobel, Canny, etc.) is used to extract the edge features of the image.

[0031] The gray level co-occurrence matrix (GLCM) is used to calculate the texture features of the image.

[0032] Preferably, the feature matching and comparison of the basic reference data and the real-time reference data in step three includes:

[0033] The basic reference data and the real-time reference data have feature parameters such as edge information and texture information.

[0034] The edge information and texture information are matched and compared to locate the surgical target, and the real-time navigation information is generated in combination with the surgical plan.

[0035] The real-time navigation information is a virtual marker or contour displayed on the surgical area to indicate the correct operation position and path of the doctor.

[0036] Or the navigation information is the real-time tracking of the position and direction of the surgical target.

[0037] Preferably, in step four, the real-time navigation information is visually displayed to assist the doctor in surgical navigation.

[0038] The real-time navigation information is digitally converted to generate graphics or text, which are displayed in the form of graphics or text on the equipment of the surgical navigation system for the doctor to refer to in real time.

[0039] Preferably, step four further includes human-computer interaction, and the interactive interface can adjust the display mode, zoom in or out, and other operations as needed.

[0040] Preferably, in step five, the real-time reference data is updated based on head tracking and adjustment as the position of the doctor's head and line of sight changes.

[0041] The line of sight direction is tracked by the eye tracking camera to capture the line of sight direction of the doctor in real time, and the infrared camera and sensor are used to detect the position and movement of the pupil to determine the position being observed by the doctor.

[0042] Obtaining the navigation positioning image data after real-time image fusion processing corresponding to the doctor's observation position, repeating the operations of steps two, three and four, updating the real-time reference data to obtain real-time navigation information;

[0043] Adjusting the content of the visual display according to the real-time navigation information to assist the surgery.

[0044] The application also provides an MR surgery navigation system based on multi-camera acquisition, comprising:

[0045] The preoperative preparation module obtains the basic image positioning image data of the patient before surgery, and performs preprocessing, feature extraction and analysis to obtain the basic reference data and develop a surgery plan;

[0046] The intraoperative processing module obtains the navigation positioning image data after real-time image fusion processing of the patient during surgery, and performs preprocessing, threshold segmentation and feature extraction to obtain real-time reference data;

[0047] The planning module performs feature matching and comparison of the basic reference data and the real-time reference data, and generates real-time navigation information according to the matching result;

[0048] The navigation module visualizes the real-time navigation information to assist the doctor's surgery navigation;

[0049] The real-time updating module updates the real-time reference data with the doctor's head position and line of sight position, and updates and adjusts the real-time navigation information in the visual display.

[0050] The head-mounted MR device is provided with a plurality of cameras and a light source; the plurality of cameras include an IR infrared camera, a Fish eye fish eye camera, a TOF lens, an RGB lens and an Eye tracking eye movement lens.

[0051] The above technical scheme of the application has the following beneficial technical effects: the MR surgery navigation is realized through the head-mounted device, which is convenient to operate and not limited by space; the data is collected by using binocular infrared cameras, which has high depth perception and spatial positioning capability; the real-time navigation and head tracking technology are combined to provide accurate and real-time surgery navigation information, significantly improving the accuracy and safety of surgery, and can be applied to various surgery scenes. BRIEF DESCRIPTION OF DRAWINGS

[0052] Fig. 1 The figure is a schematic diagram of the method of the application;

[0053] Fig. 2 The figure is a schematic diagram of the structure of the head-mounted MR device of the application;

[0054] Fig. 3An image preprocessing interaction schematic diagram of the present application. DETAILED DESCRIPTION

[0055] In order to make the objects, technical solutions and advantages of the present application clearer and more comprehensible, the present application will be further described in detail below with reference to the specific embodiments and the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present application. In addition, in the following description, the description of the known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present application.

[0056] As shown in Figs. 1-3 The MR surgery navigation method based on multi-camera acquisition proposed by the present application includes the following steps:

[0057] Step one, acquire the basic image positioning image data of the patient before surgery, and perform preprocessing, feature extraction and analysis to acquire basic reference data and develop a surgery plan;

[0058] The basic image data comes from the preoperative imaging examination, such as MRI or CT scan, to acquire the patient's anatomical structure data.

[0059] The preprocessing, feature extraction and analysis of the basic image positioning image data include:

[0060] The collected basic image positioning image data is preprocessed, including denoising, grayscale, and contrast enhancement, to obtain the preprocessed basic image positioning image;

[0061] The feature parameters of the preprocessed basic image positioning image are extracted, such as edge, texture, color and other information;

[0062] The preprocessed basic image positioning image and the extracted feature parameters are analyzed and processed to develop a surgery plan. The surgery plan is planned by the doctor before the operation to determine the surgical approach and operation target.

[0063] Step two, acquire the navigation positioning image data after real-time image fusion processing of the patient during surgery, and perform preprocessing, threshold segmentation and feature extraction to acquire real-time reference data;

[0064] The navigation positioning image data after real-time image fusion processing of the patient during surgery comes from the real-time snapshot and recording of the RGB lens of the head-mounted MR device;

[0065] The navigation positioning image data after real-time image fusion processing is preprocessed by denoising, grayscale and contrast enhancement;

[0066] Denoising: using a Gaussian filter to remove noise in the image;

[0067] Grayscale: converting a color image to a grayscale image;

[0068] Enhancing contrast: histogram equalization on grayscale image;

[0069] Wherein, the navigation positioning image after real-time image fusion processing is segmented by the optimal threshold value;

[0070] The optimal threshold value is determined by Otsu's algorithm, and the specific steps are as follows:

[0071] Calculate the gray histogram of the image: count the number of pixels at each gray level in the image;

[0072] Normalize each gray level: divide the number of pixels at each gray level by the total number of pixels to get the pixel proportion of each gray level;

[0073] Calculate the cumulative probability of each gray level: accumulate the pixel proportion of each gray level to get the cumulative probability;

[0074] Calculate the global average gray value: calculate the global average gray value by the cumulative probability;

[0075] Calculate the intra-class variance: calculate the intra-class variance of each gray level according to the global average gray value;

[0076] Calculate the inter-class variance: calculate the inter-class variance of each gray level by the cumulative probability and the intra-class variance;

[0077] Select the gray level corresponding to the maximum inter-class variance as the optimal threshold value;

[0078] Wherein, the feature extraction includes:

[0079] Use edge detection algorithm (such as Sobel, Canny, etc.) to extract image edge features;

[0080] Use gray level co-occurrence matrix (GLCM) to calculate the texture features of the image.

[0081] It can be understood that step one obtains the basic reference data, and step two obtains the real-time reference data, the difference is that the real-time reference data only includes the image part of the surgical wound part that can be obtained by the RGB lens, and the rest is supplemented by the basic reference data.

[0082] Step three, feature matching and comparison of the basic reference data and the real-time reference data, generating real-time navigation information according to the matching result, providing accurate operation guidance for doctors;

[0083] The feature matching and comparison of the basic reference data and the real-time reference data includes:

[0084] The basic reference data and the real-time reference data have characteristic parameters such as edge information and texture information;

[0085] It should be noted that: the algorithm involved in the characteristic matching of the basic reference data and the real-time reference data is:

[0086] The SIFT (Scale-Invariant Feature Transform) algorithm is a computer vision algorithm used to detect and describe local features in an image. It finds extreme points in spatial scales and extracts information such as position, scale, and rotation invariance. Finally, it generates a feature description that is invariant to image transformation, scaling, and rotation. The main steps of the SIFT algorithm are as follows:

[0087] Image registration: register visible light and infrared light images to ensure spatial consistency of different spectral images;

[0088] Scale space extreme detection: construct an image scale space pyramid through Gaussian filtering, and then find local extreme points as key points in different scale spaces. These key points are invariant in different scales and can be detected on targets of different sizes.

[0089] Key point positioning: based on the scale space extreme points, the position and scale of the key points are accurately positioned through interpolation of the scale space. In addition, to improve the stability of the key points, the gradient direction is further accurately calculated.

[0090] Direction assignment: to make the key points invariant to rotation, the gradient direction histogram of the pixels around the key points is calculated, and the main direction is selected as the direction of the key points. This makes the key points invariant to image rotation.

[0091] Feature description: after determining the position, scale, and direction of the key points, the gradient information of the surrounding area is extracted with the key point as the center, and it is converted into a feature vector with strong discrimination. These feature vectors can well describe the image information around the key points, thus realizing image matching and recognition.

[0092] After feature matching, the same feature part of the basic reference data and the real-time reference data is the real-time positioning point. Based on pixel-level fusion technology, the data of visible light and infrared light images are combined to enhance the details and contrast of the images, and the real-time surgical navigation information for the surgical site is generated in combination with the surgical plan.

[0093] Among them, the surgical target is located through the matching and comparison of edge information and texture information, and the real-time navigation information is generated in combination with the surgical plan.

[0094] The real-time navigation information is a virtual marker or contour displayed on the surgical area to indicate the correct operation position and path of the doctor.

[0095] Or the navigation information is the real-time tracking of the position and direction of the surgical target.

[0096] Step four, visualize the real-time navigation information, which is generated by digital conversion into graphics or text, to assist the doctor in surgical navigation.

[0097] The augmented reality technology is used to superimpose the fused image in the doctor's field of view to provide real-time surgical navigation information.

[0098] In an optional embodiment, during the visualization, human-computer interaction can be performed to provide an interactive interface, allowing the doctor to interact with the navigation system and adjust the display mode, zoom in or out, etc.

[0099] Step five, the real-time reference data updates with the doctor's head and line of sight position, and the real-time navigation information in the visualization is updated and adjusted.

[0100] In step five, the real-time reference data updates with the doctor's head and line of sight position based on head tracking and adjustment. The head-mounted MR device tracks the position of the doctor's head (IMU gyroscope on the head-mounted MR device and vslam algorithm to determine the 6dof coordinates of the head) and the line of sight direction (eye tracking on the head-mounted MR device).

[0101] The line of sight direction is tracked by the eye tracking camera to capture the doctor's line of sight direction in real time, using infrared cameras and sensors to detect the position and movement of the pupils to determine the position the doctor is observing.

[0102] The real-time image fusion processing navigation positioning image data corresponding to the doctor's observation position is obtained, and the operations of steps two, three and four are repeated to update the real-time reference data and obtain the real-time navigation information.

[0103] Adjust the content of the visualization according to the real-time navigation information to assist the surgery.

[0104] Set virtual markers and contours: based on the head position and line of sight direction information, superimpose virtual markers or contours on the surgical area to indicate the correct operation position and path of the doctor. These markers can be the contours of the target organ, the position of the surgical tool, etc.

[0105] Real-time image adjustment: dynamically adjust the focus or magnification of the displayed content according to the doctor's line of sight to ensure that the doctor can clearly observe the area of interest.

[0106] Tips and warnings: display prompt information or warning information related to the current operation according to the doctor's line of sight to help the doctor better understand the operation situation and take the correct action.

[0107] In one embodiment, by tracking the doctor's head in real time, mainly the tracking of the line of sight, due to the change of the line of sight, the real-time image fusion processing of the RGB camera collected image data changes, and the key features of the anatomical structure and the lesion area are identified in time to update the real-time reference data, and the pre-processed basic reference data is matched and compared again to generate real-time navigation information corresponding to the current line of sight, prevent the virtual image and the actual operation site from being incorrectly overlapped, and ensure the accurate overlap of the virtual image and the actual operation site, facilitate the indication of the operation and the implementation of the operation.

[0108] A MR surgery navigation system based on multi-camera acquisition, comprising:

[0109] The preoperative preparation module acquires the basic image positioning image data of the patient before surgery, and performs preprocessing, feature extraction and analysis to obtain basic reference data, and formulates a surgical plan;

[0110] The intraoperative processing module acquires real-time image fusion processing navigation positioning image data of the patient during surgery, and performs preprocessing, threshold segmentation and feature extraction to obtain real-time reference data;

[0111] The planning module performs feature matching and comparison of the basic reference data and the real-time reference data, and generates real-time navigation information according to the matching result;

[0112] The navigation module visually displays the real-time navigation information to assist the doctor's surgical navigation;

[0113] The real-time update module updates the real-time reference data with the doctor's head and line of sight position, and updates and adjusts the real-time navigation information in the visual display. In one embodiment, the workflow of the MR surgery navigation system based on multi-camera acquisition is as follows:

[0114] Data acquisition: multi-camera synchronously acquires images and depth information of the operation site.

[0115] Data processing: fuse multi-image data to generate high-precision surgical navigation images.

[0116] Real-time tracking: through infrared cameras and eye tracking lenses, real-time tracking of surgical tools and doctor's line of sight ensures real-time updating of navigation information.

[0117] Navigation Display: Display the processed images to the surgeon through the MR device, providing intuitive surgical navigation information.

[0118] Feedback Adjustment: Adjust the navigation information and display content in real-time according to the surgical progress and environmental changes, ensuring the precision of the surgery.

[0119] Among them, the image fusion processing further includes:

[0120] Data Preprocessing: Correct the color difference and geometric distortion of the camera, ensuring the alignment of the image data of multiple cameras.

[0121] Image Registration: Use feature matching algorithms (such as SIFT, SURF) to register visible light and infrared light images, ensuring the spatial consistency of different spectral images.

[0122] Image Fusion: Based on pixel-level fusion technology, combine the data of visible light and infrared light images to enhance the details and contrast of the images. Common methods include multi-scale fusion, wavelet transform, etc.

[0123] Enhanced Display: Use augmented reality technology to superimpose the fused image in the surgeon's field of view, providing real-time surgical navigation information.

[0124] Among them, the head-mounted MR device is equipped with multiple cameras and light sources: the multiple cameras include:

[0125] IR Infrared Camera: Used to capture infrared spectrum images, capable of working in low light or complete darkness. Mainly used for target recognition and tracking in surgical navigation systems, especially in situations where there is insufficient light in the operating field.

[0126] Fish-eye Camera: With a super-wide field of view, it can capture a larger range of images. Suitable for monitoring the entire operating room panorama, helping to provide comprehensive environmental perception.

[0127] TOF Lens (Time-of-Flight): Measures the time of flight of light to obtain depth information, generating high-precision 3D images. It can accurately capture the spatial position of surgical tools and human structures, providing accurate three-dimensional positioning support for surgical navigation.

[0128] RGB Lens: Used to capture high-resolution color images, providing clear and visible images of the surgical area. Combined with the data of other sensors, it can generate high-precision surgical navigation images.

[0129] Eye tracking: used to track the eye movement of the surgeon, which can adjust the display content in real time according to the surgeon's gaze direction. Improve the surgeon's attention to key areas during the operation and operation precision.

[0130] These cameras and lenses work together to provide accurate and real-time image and positioning information for surgical navigation through multi-channel data acquisition and fusion, improving the safety and efficiency of surgery.

[0131] It should be noted that: through image processing and analysis algorithms, the collected image data is processed to identify key features such as anatomical structures and lesion areas, and compared with the processed navigation positioning image data of the patient's medical images.

[0132] Among them, the algorithm processing is calculated by the embedded processing platform algorithm box, the head-mounted MR device is transmitted to the algorithm box through the type C interface, the algorithm processing is carried out, the data amount is reduced, the system MTP (Motion-To-Photon Latency) delay is reduced, and the response speed of real-time navigation information update is guaranteed.

[0133] It should be noted that: the head-mounted MR device has a built-in preset VPU that fuses multiple camera signals for image fusion preprocessing before transmitting them to the separate algorithm unit, reducing system latency and reducing bandwidth pressure on transmission links.

[0134] It should be understood that the above specific embodiments of the present application are only used for illustrative or explanatory purposes, and do not constitute a limitation on the present application. Therefore, any modifications, equivalent replacements, improvements, etc. made without departing from the spirit and scope of the present application shall be included within the protection scope of the present application. In addition, the appended claims of the present application are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or the equivalent forms of such scope and boundaries.

Claims

1. A method for MR surgical navigation based on multi-channel camera acquisition, characterized in that, Includes the following steps: Step 1: Obtain the patient's basic preoperative imaging localization data, perform preprocessing, feature extraction and analysis to obtain basic reference data, and formulate a surgical plan; Step 2: Obtain navigation and positioning data after real-time image fusion processing during the patient's surgery, and perform preprocessing, threshold segmentation and feature extraction to obtain real-time reference data; Step 3: Perform feature matching and comparison between the basic reference data and the real-time reference data, and generate real-time navigation information based on the matching results; Step 4: Visualize the real-time navigation information to assist the doctor in surgical navigation; Step 5: The real-time reference data is updated according to the doctor's head and line of sight position, and the real-time navigation information in the visualization is updated and adjusted.

2. The MR surgical navigation method based on multi-channel camera acquisition according to claim 1, characterized in that, In step one, the basic imaging localization image data comes from navigation localization imaging examinations such as MRI or CT scans after preoperative image fusion processing, to obtain the patient's anatomical structure data.

3. The MR surgical navigation method based on multi-channel camera acquisition according to claim 1, characterized in that, Step one, the preprocessing, feature extraction, and analysis of the basic image positioning data includes: The navigation and positioning data after the acquisition of spatial image fusion processing is preprocessed, including denoising, grayscale conversion, and contrast enhancement, to obtain the preprocessed base image; Extract feature parameters from the preprocessed base image, such as edge, texture, and color information; The preprocessed base image and extracted feature parameters are analyzed and processed to formulate a surgical plan and determine the surgical approach and operational objectives.

4. The MR surgical navigation method based on multi-channel camera acquisition according to claim 1, characterized in that, In step two, the navigation and positioning image data after real-time image fusion processing during the patient's surgery comes from the real-time capture and recording of the RGB lens of the head-mounted MR device; Among them, the navigation and positioning image data after real-time image fusion processing is preprocessed by denoising, grayscale conversion and contrast enhancement. Among them, the navigation and positioning image after real-time image fusion processing is segmented by using the optimal threshold; The optimal threshold is determined using Otsu's algorithm, and the specific steps are as follows: Calculate the grayscale histogram of an image: count the number of pixels at each grayscale level in the image; Normalize each gray level: Divide the number of pixels at each gray level by the total number of pixels to obtain the pixel percentage of each gray level; Calculate the cumulative probability for each gray level: sum the pixel proportions of each gray level to obtain the cumulative probability; Calculate the global average gray value: The global average gray value is calculated using cumulative probability. Calculate the intra-class variance: Based on the global average gray value, calculate the intra-class variance for each gray level; Calculate the inter-class variance: Calculate the inter-class variance for each gray level using the cumulative probability and the intra-class variance; Choose the gray level corresponding to the largest inter-class variance as the optimal threshold. Feature extraction includes: Use edge detection algorithms (such as Sobel, Canny, etc.) to extract image edge features; The texture features of an image are calculated using the Gray-Level Co-occurrence Matrix (GLCM).

5. The MR surgical navigation method based on multi-channel camera acquisition according to claim 1, characterized in that, Step three, the feature matching and comparison of the basic reference data and the real-time reference data, includes: The basic reference data and the real-time reference data have characteristic parameters such as edge information and texture information; The surgical target is located by matching and comparing edge information and texture information, and real-time navigation information is generated by combining the surgical plan. Among them, real-time navigation information is a virtual marker or outline superimposed on the surgical area to indicate the correct operating position and path for the doctor; Or the navigation information could be the real-time tracking of the surgical target's location and direction.

6. The MR surgical navigation method based on multi-channel camera acquisition according to claim 1, characterized in that, Step 4: Visualize the real-time navigation information to assist the doctor in surgical navigation; The real-time navigation information is digitized to generate graphics or text, which are then presented on the surgical navigation system for doctors to refer to in real time.

7. The MR surgical navigation method based on multi-channel camera acquisition according to claim 1, characterized in that, Step four also includes human-computer interaction, where the interactive interface can be adjusted to change the display mode, zoom in and out of the view, etc., as needed.

8. The MR surgical navigation method based on multi-channel camera acquisition according to claim 1, characterized in that, In step five, the real-time reference data is updated based on head tracking and adjustment as the doctor's head and gaze position changes. The head-mounted MR device tracks the position of the doctor's head in real time (the IMU gyroscope and vslam algorithm on the head-mounted MR device determine the 6DOF coordinates of the head) and the direction of gaze (eye tracking on the head-mounted MR device). Among them, the tracking of the direction of the gaze is achieved by capturing the doctor's gaze direction in real time through the eye tracker of the eye tracking lens, and by using infrared cameras and sensors to detect the position and movement of the pupil, thereby determining the position that the doctor is observing; Obtain navigation and positioning image data after real-time image fusion processing of the corresponding doctor's observation position, repeat steps two, three, and four, update the real-time reference data, and then obtain real-time navigation information; The visualization content is adjusted based on real-time navigation information to assist in the surgical procedure.

9. A multi-camera-based MR surgical navigation system, characterized in that, include: The preoperative preparation module acquires the patient's basic preoperative imaging localization image data, performs preprocessing, feature extraction and analysis to obtain basic reference data, and formulates a surgical plan; The intraoperative processing module acquires navigation and positioning image data after real-time image fusion processing during the patient's surgery, and performs preprocessing, threshold segmentation and feature extraction to obtain real-time reference data; The planning module performs feature matching and comparison between the basic reference data and the real-time reference data, and generates real-time navigation information based on the matching results. The navigation module visualizes the real-time navigation information to assist doctors in surgical navigation. The real-time update module updates the real-time reference data according to the doctor's head and line of sight position, thereby updating and adjusting the real-time navigation information in the visualization display.

10. A multi-camera-based MR surgical navigation system, characterized in that, This includes a head-mounted MR device, which is equipped with multiple cameras and light sources; The multi-camera system includes: an IR infrared camera, a Fisheye camera, a TOF lens, an RGB lens, and an Eye tracking lens.

Images