Trans-pedicle safe puncture AR navigation method and system

Abstract

The invention provides a transpedicular safe puncture AR navigation method and system. The system can be composed of a common notebook computer and a VR head mirror, is small in size and portable, does not occupy the space of an operating room, is small in navigation software memory, can be operated on any notebook computer, is good in software compatibility, and can be quickly registered and operated; the cost is low, the per-average cost is about 300 yuan to 500 yuan and is 10% of the per-average cost (3000 yuan to 5000 yuan) of the conventional navigation software, and the economic burden of a patient is greatly reduced; the operation is simple, and navigation configuration can be realized only by matching 3-4 point locations after positioning; operation disinfection is not needed, registration and navigation can be achieved only by scanning the two-dimensional code, the device decontamination process in the operation process is avoided, real-time navigation can be achieved in the operation, and therefore the percutaneous centrum puncture process is guided, and the good effect is achieved through clinical examination.

Description

Technical Field

[0001] The present invention relates to the field of medical technology, and in particular to an AR navigation method and system for safe transpedicular puncture. Background Art

[0002] With the development of science and technology and medical technology, surgery has entered the era of minimally invasive surgery and intelligent surgery. The application of orthopedic surgical robots is of great significance in improving surgical accuracy, optimizing surgical operations, and reducing surgical injuries. It is an important way to achieve minimally invasive and intelligent orthopedic surgery. The application of CAS in orthopedics is called Computer Assisted Orthopedic Surgery (CAOS). CAOS combines advanced medical imaging equipment such as Computer Tomography (CT), Magnetic Resonance Imaging (MRI), and Positron Emission Tomography (PET). The computer uses these medical image information to reconstruct a three-dimensional image and obtain the anatomical structure around the lesion, so that the doctor has a clear understanding of the surgical site and can plan the operation. Through the stereo positioning system, the relative position relationship between the lesion site and the robot and surgical instruments is determined, and navigation is provided for the doctor's operation in the virtual environment according to the preoperative plan. Finally, the robot performs surgical intervention. At present, CAOS is increasingly used in trauma orthopedics, spinal surgery, and artificial joint surgery.

[0003] Positioning and navigation are the core functions of existing orthopedic surgical robots: positioning refers to the use of spatial positioning algorithms to calculate the spatial coordinates of surgical targets. The accuracy of the positioning algorithm directly affects the accuracy and effectiveness of the surgery; navigation refers to the use of reasonable interactive methods to guide doctors to complete surgical operations. The convenience of the navigation mode directly affects the clinical adaptability and effectiveness of the system.

[0004] The mechanical positioning method is the earliest positioning method applied to computer-assisted surgery. In the positioning process, in order to ensure the flexibility of the robot, a multi-degree-of-freedom robotic arm (generally with more than five degrees of freedom) is usually selected. Various surgical instruments can be installed at the front end of the robotic arm to meet different clinical needs. In computer-assisted surgery, passive robotic arms were first used for positioning. The joint parameters of the passive robotic arm at different positions can be recorded by the encoders of each joint. The doctor can drive the robotic arm to move by holding the front end of the passive arm. When the target position is reached, the position of the surgical instrument is calculated based on the robotic arm model and the joint parameters of the robotic arm at this position. At present, the mechanical positioning process mostly uses active robotic arms, which can be programmed to control point positioning, path planning and trajectory planning in the workspace. When the robot reaches the target position, the joint can be locked, and it can also withstand a certain load, which effectively solves the jitter problem caused by the doctor holding the instrument during the operation.

[0005] Due to the characteristics of bone tissue itself: the relationship between bone structures is stable. This existence enables the orthopedic surgical robot system to use external landmarks to establish a mapping relationship between the surgical space and the image coordinates, thereby obtaining the corresponding relationship between the surgical instrument and the target position. With the increasing application of navigation technology in surgical operations, the scope of use has gradually expanded, but the wider application has made the need for updating navigation technology more and more urgent. The demand for AI intelligent navigation in orthopedic surgery has also gradually increased. Orthopedic surgery is highly flexible, and the complex and changeable recognition of bone and muscle tissue makes the intraoperative navigation operation process have a high repetition rate. This leads to a long intraoperative navigation operation time, and the mismatching of the target during the operation often occurs, which invisibly prolongs the operation time, increases the difficulty of the operation, and reduces the utilization rate of navigation.

[0006] The following are navigation solutions in the prior art:

[0007] 1. The SpineAssist system was developed and launched by Mazor Medical Technology Company. It is the first robot product that can be used for minimally invasive spinal surgery. The SpineAssist system is mainly used for translaminar facet screw fixation surgery and pedicle screw surgery, providing precise directional guidance for the process of manual screw implantation in spinal fusion surgery. Before the operation, the surgical planning is completed according to the CT image to determine the size and implantation path of the implant; during the operation, the doctor fixes the micro robot on the patient's spine with a spinal spinous process clamp, and then uses a C-arm to take intraoperative fluoroscopic images in the anteroposterior and lateral positions, and transmits the intraoperative fluoroscopic images to the working software of the SpineAssist system to complete the registration of the preoperative CT image and the intraoperative fluoroscopic image, and uses a micro parallel robot for navigation to establish a precise surgical instrument guidance channel. In the subsequent surgical engineering, the robot is completely locked and cannot move, and finally the doctor completes the surgical operation according to the established guidance channel. The SpineAssist system fixes the micro parallel robot directly on the patient's spine, simplifies the mapping process between the surgical space and the image, and improves the accuracy and reliability of the robot positioning and navigation system.

[0008] 2. The ROSA Spine system is a surgical robot system developed and produced by the French Medtech company. The system is mainly used for spinal and neurosurgery. In pedicle screw fixation, the ROSA Spine system completes preoperative planning through preoperative CT images, uses the photoelectric positioning system to determine the implantation path and position of the screws, and guides the doctor to perform the operation through the robotic arm. During the operation, the system tracks the movement caused by the patient's breathing in real time, avoiding the error introduced by the position change of the spine caused by the patient's breathing. Nicolas Lonjon and others have completed a clinical comparison of the ROSA Spine system and traditional surgery. The same group of doctors performed 20 pedicle screw implantation operations. The results showed that compared with traditional surgery, the surgery assisted by the ROSA Spine system has higher accuracy and shorter operation time, and also reduces the acquisition of intraoperative fluoroscopic images.

[0009] 3. The PinTrace system is a multi-purpose orthopedic surgery system developed by Medical Robotics in Sweden. The system uses a six-freedom robotic arm for surgical positioning, and through the combination with image processing methods, it realizes robotic surgical navigation based on intraoperative two-dimensional perspective images. During the operation, the PinTrace system needs to first collect two mutually perpendicular X-ray perspective images. The positioning tool installed at the end of the robot must be within the image acquisition range and ensure that it can be clearly displayed in the X-ray perspective image. The doctor establishes the coordinate system of the entire system by marking the fracture location and the positioning tool. This process does not require the implantation of markers in the patient, which can avoid damage to the patient.

[0010] The PinTrace system calculates the optimal position of the screws based on the fluoroscopic image and the fracture location, and displays the surgical path on the interface. The doctor can also adjust the surgical plan. When the doctor completes the surgical plan and confirms the surgical location, the robot can move to the target position, so that the positioning tool installed at the end of the robotic arm can be accurately placed at the surgical site. The doctor can perform corresponding operations according to the surgical location indicated by the robot. The PinTrace system is mainly used for robot-assisted hip fracture surgery. Clinical applications have shown that the PinTrace system can effectively improve the quality of surgery and reduce radiation accumulation during surgery.

[0011] 4. The CRIGOS system is an orthopedic surgical robot system based on fluoroscopic images jointly developed by Germany and France. The system uses two X-ray images for preoperative planning and positioning, and adopts a small parallel robot for surgical navigation. It can assist doctors in completing femoral neck hollow nail surgery, etc.

[0012] 5. The dual-plane orthopedic surgical robot system is a surgical navigation system based on fluoroscopic images, jointly developed by Beijing University of Aeronautics and Astronautics and Jishuitan Hospital. It is mainly used for fixation of leg fractures and can establish a precise implantation channel for the implantation of intramedullary nails. The main functions of the system are: solving the spatial coordinates of the surgical target position through the spatial coordinate mapping algorithm to complete the surgical positioning; guiding the doctor to perform the surgical operation at the target position according to the obtained target spatial position. The biggest advantage of the dual-plane orthopedic surgical robot is that the positioning process does not require CT images, which reduces the calculation and processing of image data.

[0013] 6. The Hybridot robot system is an active-passive hybrid orthopedic surgical robot system jointly developed by the Beijing University of Aeronautics and Astronautics and the Chinese University of Hong Kong. The system uses two perspective images for preoperative planning and surgical positioning. During use, the system can cooperate with photoelectric navigation and switch between active and passive motion modes. In active mode, the doctor can move the robot to the initial navigation position, and in passive mode, the robot can make fine adjustments to reach the final position.

[0014] 7. The BRIGIT robot system developed by Zimmer does not require image acquisition during the positioning process. The doctor collects human skeletal anatomical landmarks by holding the robot with force feedback, and makes angle corrections according to the surgical position to complete the positioning. Finally, the robot completes the bone cutting operation at the target position. The system uses a six-degree-of-freedom robotic arm, which is mainly used for high tibial osteotomy and can also be used for total knee replacement surgery. Since the system needs to be positioned according to human anatomical landmarks, the doctor needs to have rich surgical experience.

[0015] The positioning and navigation process of the above orthopedic surgical robot systems requires the implantation of titanium nails or markers in the patient's body before surgery. This method will cause damage to the patient on the one hand, and on the other hand, it also limits the application of surgical robot systems. To address this problem, an orthopedic surgical robot system has emerged that uses external landmarks and achieves positioning through intraoperative perspective images. At the same time, the positioning and navigation processes of the above orthopedic surgical robot systems are all based on image information, and image errors are easily introduced during the registration process, which affects the positioning accuracy.

[0016] Therefore, there is an urgent need for an AR navigation method and system for safe transpedicular puncture that can solve the above problems. Summary of the invention

[0017] The purpose of the present invention is to provide an AR navigation method and system for safe transpedicular puncture to solve the problems existing in the above-mentioned prior art.

[0018] To achieve the above object, the present invention provides the following solutions:

[0019] The present invention provides an AR navigation method for safe transpedicular puncture, comprising the following steps:

[0020] S1. Scan the QR code to register and log in to the navigation software;

[0021] S2. Import the patient case into the navigation software, analyze and process the patient case using the model in the navigation software, and export the navigation planning file;

[0022] S3. Use the VR head mirror to load the navigation planning file and perform AR navigation for safe transpedicular puncture.

[0023] Preferably, the model in the navigation software adopts a large medical multimodal model.

[0024] Preferably, the method for constructing a large multimodal medical model is:

[0025] Model pre-training;

[0026] Model fine-tuning;

[0027] The case data is used to train a large medical multimodal model.

[0028] Preferably, the model pre-training specifically includes: using transformer as the basic model, and using a learning mechanism combining semi-supervised learning and self-supervised learning to pre-train the labeled data and unlabeled data.

[0029] Preferably, the case data includes historical patient safety area annotation and depiction data, preoperative, intraoperative, and postoperative images, case records, physiological indicators, preliminary diagnosis, treatment plan, and follow-up.

[0030] Preferably, the medical multimodal large model is obtained by training with case data as follows:

[0031] Data preparation: Obtain and verify 4,000 actual case data;

[0032] Data labeling and cleaning: 1,000 cases out of 4,000 were selected as labeling data, ensuring that each case included an X-ray or CT image, as well as the corresponding plane or three-dimensional safety zone range labeling and text description;

[0033] Data format standardization: unify the size and resolution of images and pre-process the text;

[0034] Pre-trained multimodal model selection: Select a suitable multimodal pre-trained model to process and fuse image and text data;

[0035] Data preprocessing: normalize and enhance the image data, encode and vectorize the text;

[0036] Multimodal data fusion: The image and text input are passed through the visual and language encoders respectively to obtain the corresponding feature vectors and then fused;

[0037] Model training: Design a multi-task learning framework, combine image classification, region segmentation, and text matching tasks, and optimize through a joint loss function. After completing the above fully supervised fine-tuning, input the remaining 3,000 data into the model for self-supervised learning;

[0038] Multi-task learning: Through the multi-task learning framework, the training model can achieve good performance in image classification, region segmentation and text matching tasks;

[0039] Optimizer and hyperparameter tuning: Select the appropriate optimizer and learning rate, and perform hyperparameter tuning;

[0040] Model validation: Evaluate model performance using cross-validation and independent test sets;

[0041] Validation and test set evaluation: Regularly evaluate model performance on the validation set and adjust the training strategy based on the results.

[0042] Preferably, the multimodal pre-training model adopts CLIP or VLP.

[0043] Preferably, analyzing and processing the patient case using the model in the navigation software includes:

[0044] Add data: load CT images and import them into the software;

[0045] Segmentation usage: set thresholds to remove impurities and retain areas that need planning;

[0046] Model loading: place the model, adjust the position and angle, and quickly place it at a specified location;

[0047] Marking tools: provide point, line, surface and angle auxiliary tools;

[0048] Planning export: export the planning completion file.

[0049] Preferably, the navigation planning file is loaded using a VR head mirror, and the AR navigation for safe transpedicular puncture is specifically as follows:

[0050] Convert data: convert EasySpine export data format and import it into Unity;

[0051] Loading data: Read the JSON file in Unity to load the model into the correct spatial position;

[0052] Export deployment: export and deploy the environment;

[0053] QR code registration: real and virtual correspondence, click the four spatial position points corresponding to the same positions of the real model and the virtual model with the pointer, and design an algorithm for registration.

[0054] The present invention also provides an AR navigation system for safe transpedicular puncture, comprising a computer and a VR head mirror interconnected therewith, both of which are installed with the above-mentioned navigation software.

[0055] Compared with the prior art, the present invention has achieved the following beneficial technical effects:

[0056] The present invention provides an AR navigation method and system for safe transpedicular puncture. The system can be composed of an ordinary laptop computer and a VR head mirror, is small and portable, does not occupy the space of the operating room, and the navigation software has a small memory and can be operated on any laptop computer. The software has good compatibility and can be quickly registered and operated; the cost is low, with an average cost of about 300 yuan to 500 yuan per time, which is 10% of the average cost of conventional navigation software (3000 to 5000 yuan), greatly reducing the economic burden of patients; the operation is simple, and only 3-4 points need to be matched after positioning to achieve navigation configuration; no surgical disinfection is required, and registration and navigation can be achieved by scanning a QR code, thereby avoiding the equipment disinfection process during the operation, and real-time navigation can be performed during the operation to guide the percutaneous vertebral puncture process, which has good clinical results. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0058] Figure 1 A flow chart of an AR navigation method for safe transpedicular puncture provided by the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] The purpose of the present invention is to provide an AR navigation method and system for safe transpedicular puncture to solve the problems existing in the prior art.

[0061] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] Embodiment 1:

[0063] This embodiment provides an AR navigation method for safe transpedicular puncture, comprising the following steps:

[0064] S1. Scan the QR code to register and log in to the navigation software;

[0065] S2. Import the patient case into the navigation software, analyze and process the patient case using the model in the navigation software, and export the navigation planning file;

[0066] Among them, the model in the navigation software adopts a large medical multimodal model;

[0067] The method for constructing a large multimodal medical model is as follows:

[0068] The combination of semi-supervised learning and self-supervision can efficiently utilize labeled data and unlabeled data, while improving the generalization and adaptability of the model in multimodal tasks; this solution can make full use of existing labeled data and mine more potential features on unlabeled data, which is suitable for further improving the accuracy and robustness of bone cement injection safety area identification;

[0069] Using the transformer large model based on the modern attention mechanism can effectively focus on strongly correlated time series data and ignore weakly correlated sequences, which can ensure the capture of key information of the main features while avoiding overfitting;

[0070] Use case data to build a large medical multimodal model. Case data includes historical patient safety zone annotation and depiction data, preoperative, intraoperative and postoperative images, cases, physiological indicators, preliminary diagnosis, treatment plan, and follow-up;

[0071] Specifically include:

[0072] Data preparation: obtain and verify 4,000 actual case data, first of all to ensure the accuracy of images and texts; data annotation and cleaning: select 1,000 cases out of 4,000 as annotated data, ensuring that each case includes X-rays or CT images, as well as the corresponding plane or three-dimensional safety zone range annotations and text descriptions; data format standardization: unify the size and resolution of the image, and pre-process the text, such as word segmentation and removal of stop words; pre-trained multimodal model selection: select a suitable multimodal pre-trained model, such as CLIP or VLP, to process and fuse image and text data; data preprocessing: normalize and data enhance the image, and encode and vectorize the text; multimodal data fusion: pass the image and text input through the visual and language encoders respectively to obtain the corresponding feature vectors, and then perform Line fusion; Model training: Design a multi-task learning framework, combine image classification, region segmentation and text matching tasks, and optimize through a joint loss function; After completing the above fully supervised fine-tuning, input the remaining 3,000 data into the model for self-supervised learning to improve the generalization ability of the model while reducing training costs and improvements; Multi-task learning: Through the multi-task learning framework, the training model can achieve good performance in image classification, region segmentation and text matching tasks; Optimizer and hyperparameter tuning: Select a suitable optimizer and learning rate, and perform hyperparameter tuning to improve the convergence speed and performance of the model; Model validation: Use cross-validation and independent test sets to evaluate model performance to ensure its generalization ability on different data sets; Validation and test set evaluation: Regularly evaluate model performance on the validation set, and adjust the training strategy based on the results;

[0073] Analytical processing of patient cases using models in navigation software includes:

[0074] Add data: load CT images and import them into the software;

[0075] Segmentation usage: set thresholds to remove impurities and retain areas that need planning;

[0076] Model loading: place the model, adjust the position and angle, and quickly place it at a specified location;

[0077] Marking tools: provide point, line, surface and angle auxiliary tools;

[0078] Planning export: export planning completion file;

[0079] S3. Use the VR head mirror to load the navigation planning file and perform AR navigation for safe transpedicular puncture; specifically:

[0080] Convert data: convert EasySpine export data format and import it into Unity;

[0081] Loading data: Read the JSON file in Unity to load the model into the correct spatial position;

[0082] Export deployment: export and deploy the environment;

[0083] QR code registration: real and virtual correspondence, click the four spatial position points corresponding to the same positions of the real model and the virtual model with the pointer, and design an algorithm for registration.

[0084] This embodiment also provides an AR navigation system for safe transpedicular puncture, including a computer and a VR head mirror interconnected therewith, both of which are installed with the above-mentioned navigation software.

[0085] The present invention provides an AR navigation method and system for safe transpedicular puncture. The system can be composed of an ordinary laptop computer and a VR head mirror, is small and portable, does not occupy the space of the operating room, and the navigation software has a small memory and can be operated on any laptop computer. The software has good compatibility and can be quickly registered and operated; the cost is low, with an average cost of about 300 yuan to 500 yuan per time, which is 10% of the average cost of conventional navigation software (3000 to 5000 yuan), greatly reducing the economic burden of patients; the operation is simple, and only 3-4 points need to be matched after positioning to achieve navigation configuration; no surgical disinfection is required, and registration and navigation can be achieved by scanning a QR code, thereby avoiding the equipment disinfection process during the operation, and real-time navigation can be performed during the operation to guide the percutaneous vertebral puncture process, which has good clinical results.

[0086] The present invention uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only used to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. An AR navigation method for safe transpedicular puncture, characterized by: The following steps are involved: S1. Scan the QR code to register and log in to the navigation software; S2. Import the patient case into the navigation software, analyze and process the patient case using the model in the navigation software, and export the navigation planning file; S3. Use the VR head mirror to load the navigation planning file and perform AR navigation for safe transpedicular puncture.

2. The AR navigation method for safe transpedicular puncture according to claim 1, characterized in that: The model in the navigation software adopts a large medical multimodal model.

3. The AR navigation method for safe transpedicular puncture according to claim 2, characterized in that: The method for constructing a large multimodal medical model is as follows: Model pre-training; Model fine-tuning; The case data is used to train a large medical multimodal model.

4. The AR navigation method for safe transpedicular puncture according to claim 3, characterized in that: The specific model pre-training is as follows: using transformer as the basic model, and using a learning mechanism combining semi-supervised learning and self-supervised learning to pre-train labeled data and unlabeled data.

5. The AR navigation method for safe transpedicular puncture according to claim 3, characterized in that: Case data includes historical patient safety area annotation and depiction data, preoperative, intraoperative, and postoperative images, cases, physiological indicators, preliminary diagnosis, treatment plan, and follow-up.

6. The AR navigation method for safe transpedicular puncture according to claim 3, characterized in that: The medical multimodal large model trained with case data is as follows: Data preparation: Obtain and verify 4,000 actual case data; Data labeling and cleaning: 1,000 cases out of 4,000 were selected as labeling data, ensuring that each case included an X-ray or CT image, as well as the corresponding plane or three-dimensional safety zone range labeling and text description; Data format standardization: unify the size and resolution of images and pre-process the text; Pre-trained multimodal model selection: Select a suitable multimodal pre-trained model to process and fuse image and text data; Data preprocessing: normalize and enhance the image data, encode and vectorize the text; Multimodal data fusion: The image and text input are passed through the visual and language encoders respectively to obtain the corresponding feature vectors and then fused; Model training: Design a multi-task learning framework, combine image classification, region segmentation, and text matching tasks, and optimize through a joint loss function. After completing the above fully supervised fine-tuning, input the remaining 3,000 data into the model for self-supervised learning; Multi-task learning: Through the multi-task learning framework, the training model can achieve good performance in image classification, region segmentation and text matching tasks; Optimizer and hyperparameter tuning: Select the appropriate optimizer and learning rate, and perform hyperparameter tuning; Model validation: Evaluate model performance using cross-validation and independent test sets; Validation and test set evaluation: Regularly evaluate model performance on the validation set and adjust the training strategy based on the results.

7. The AR navigation method for safe transpedicular puncture according to claim 6, characterized in that: The multimodal pre-training model uses CLIP or VLP.

8. The AR navigation method for safe transpedicular puncture according to claim 1, characterized in that: Analytical processing of patient cases using models in navigation software includes: Add data: load CT images and import them into the software; Segmentation usage: set thresholds to remove impurities and retain areas that need planning; Model loading: place the model, adjust the position and angle, and quickly place it at a specified location; Marking tools: provide point, line, surface and angle auxiliary tools; Planning export: export the planning completion file.

9. The AR navigation method for safe transpedicular puncture according to claim 1, characterized in that: Using the VR head mirror to load the navigation planning file, the AR navigation for safe transpedicular puncture is as follows: Convert data: convert EasySpine export data format and import it into Unity; Loading data: Read the JSON file in Unity to load the model into the correct spatial position; Export deployment: export and deploy the environment; QR code registration: real and virtual correspondence, click the four spatial position points corresponding to the same positions of the real model and the virtual model with the pointer, and design an algorithm for registration.

10. An AR navigation system for safe transpedicular puncture, characterized by: It comprises a computer and a VR head mirror interconnected therewith, both of which are installed with the navigation software described in any one of claims 1-9.

Images