Real-time intraoperative navigation system and method based on optical and inertial navigation

Abstract

The invention relates to the technical field of medical surgical navigation, in particular to a real-time intraoperative navigation system and method based on optical and inertial navigation. The system comprises an optical data acquisition module which is used for acquiring images, identifying surgical instruments and anatomical structures of patients, and extracting corresponding feature points by adopting a mode of combining a feature point extraction algorithm based on deep learning with a traditional algorithm; calculating pose information by using a mode of fusing a pose resolving algorithm based on deep learning and a traditional algorithm; the inertial data acquisition module is used for integrating the data of each component of the IMU by using the IMU integrated component, establishing a state model and the observation module, and acquiring the attitude information of the surgical instrument; and the data fusion processing module is used for carrying out time synchronization and calibration of optical camera and IMU acquisition, and adopting different data fusion strategies according to the reliability and availability of optical data and inertial data. According to the technical scheme, the reliability of the intraoperative navigation system can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical surgical navigation, and particularly relates to a real-time intraoperative navigation system and method based on optical and inertial navigation. Background Art

[0002] In recent years, intraoperative navigation systems have played a key role in the process of surgical precision. Traditional intraoperative navigation systems mainly rely on a single positioning technology, such as optical navigation or inertial navigation, but these methods have certain limitations in practical applications.

[0003] The optical navigation system captures images of surgical instruments and the patient's anatomical structure, and uses image processing algorithms to identify and locate the position of the surgical instruments. This method can provide high-precision and high-resolution image information in the case of clear vision and good lighting conditions, thus accurately reflecting the details of surgical instruments and the patient's anatomical structure. However, the performance of the optical navigation system is easily affected by factors such as light changes, occlusions, and blood and body fluids generated during the operation, resulting in a decrease in image quality and thus affecting the accuracy of navigation.

[0004] The inertial navigation system uses an inertial measurement unit (IMU) to measure the attitude changes of surgical instruments, including information such as pitch angle, yaw angle, and roll angle. This method has advantages such as good dynamics and strong continuity, and can provide continuous attitude information when optical data is limited or unavailable. However, the inertial navigation system also has the problem of cumulative error, and the error will gradually increase after long-term operation, affecting the accuracy of navigation. Summary of the Invention

[0005] The purpose of the present invention is to propose a real-time intraoperative navigation system and method based on optical and inertial navigation, and this technical solution can improve the reliability of the intraoperative navigation system.

[0006] To achieve the above purpose, in the first aspect, the present invention provides a real-time intraoperative navigation system based on optical and inertial navigation, including: An optical data acquisition module, which is used to collect images, identify surgical instruments and the patient's anatomical structure, extract corresponding feature points by combining a feature point extraction algorithm based on deep learning with a traditional algorithm; and calculate pose information by fusing a pose solution algorithm based on deep learning with a traditional algorithm; An inertial data acquisition module, which is used to use an IMU integrated component to construct a state model and an observation module, fuse the data of each component of the IMU, and obtain the attitude information of the surgical instrument; The data fusion processing module is used to perform time synchronization and calibration for the acquisitions of the optical camera and the IMU. The data fusion algorithm adopts the Kalman filtering algorithm. The data fusion processing module is used to analyze the measurement data of the IMU and establish its noise model; use a large amount of IMU historical data to train the LSTM network; take the measurement data of the IMU and the corresponding noise model as inputs, and take the optimal parameters of the Kalman filter as outputs; the LSTM network learns the mapping relationship between the input data and the output parameters, and dynamically predicts the optimal parameters of the Kalman filter according to the current IMU measurement data; apply the Kalman filter parameters predicted by the LSTM network to the Kalman filtering algorithm to fuse the optical data and the inertial data; and adopt different data fusion strategies according to the reliability and availability of the optical data and the inertial data.

[0007] Beneficial effects of the basic solution: This technical solution fuses optical data and inertial data for real-time intraoperative navigation. The optical data can provide clear and accurate image information of the surgical instruments and the patient's anatomical structure, so as to accurately identify the positions of the surgical instruments and the details of the patient's body structure. The inertial data has good performance in terms of dynamics and continuity. Through the IMU integration component, the attitude changes of the surgical instruments can be tracked in real time. Even when the acquisition of optical data is restricted (such as being blocked, light changes, etc.), the inertial data can continuously provide information.

[0008] During fusion, time synchronization and calibration are performed for the acquisitions of the optical camera and the IMU to ensure the time consistency of different sensor data, and avoid navigation errors caused by time asynchronization. And according to the reliability and availability of the optical data and the inertial data, different data fusion strategies are adopted, which gives full play to the advantages of the two types of data, improves the calculation efficiency, and ensures the real-time performance of the system.

[0009] In terms of optical data acquisition, a method combining a feature point extraction algorithm based on deep learning and a traditional algorithm is adopted to extract feature points, and a method of fusing a pose calculation algorithm based on deep learning and a traditional algorithm is used to calculate the pose information, which improves the accuracy of the calculation results. At the same time, for complex images, the computational load of the traditional algorithm is very large, and the deep learning algorithm actually takes less time. Since the deep learning method has pre-learned a large number of complex images, it takes less time than the traditional algorithm without any "memory" when seeing complex images again, and can better ensure the real-time performance of the calculation.

[0010] Inertial data acquisition, on the one hand, can serve as a data source when optical data is inaccurate, and on the other hand, it can more accurately track the tiny movements of surgical instruments and complete surgical operations more precisely. By establishing a state model and a prediction model, based on physical laws and the measurement principles of IMUs, the data of multiple IMU components can be fused to accurately calculate the pose of surgical instruments in three-dimensional space, including information such as pitch angle, yaw angle, and roll angle, and accurately model the pose (such as angle, direction, etc.) of surgical instruments.

[0011] This technical solution combines the Kalman filter optimized by LSTM with different fusion strategies during data fusion, which can give full play to the advantages of optical and inertial data. The measurement environment of IMUs is complex and changeable, and the measurement characteristics of IMUs are different in different surgical scenarios. It is difficult for the traditional Kalman filter with fixed parameters to adapt. By learning the mapping relationship between IMU measurement data and the optimal parameters of the Kalman filter through the LSTM network, the optimal parameters can be dynamically predicted based on the current IMU measurement data, improving the accuracy and adaptability of data fusion. The training of the LSTM network enables it to adapt to various situations. When encountering a new surgical scenario or a change in the working state of the IMU, LSTM can quickly adjust the prediction results based on the learned knowledge and the current measurement data, enabling the system to flexibly adapt to changes and ensuring the stable performance of the navigation system.

[0012] This technical solution also establishes an IMU noise model and uses it as the input of the LSTM, enabling the LSTM to consider noise factors when predicting the Kalman filter parameters. There are problems such as drift and noise in IMU measurements. An accurate noise model can enable the LSTM to learn the influence law of noise on measurement data, predict more effective filtering parameters, reduce noise interference, and improve the accuracy and stability of measurement data.

[0013] This system combines the advantages of optical navigation and inertial navigation and can maintain stable navigation performance under different surgical environments and conditions. Optical navigation provides high-precision information in a clear field of view, while inertial navigation maintains continuity in complex or restricted environments. The two complement each other, enhancing the adaptability and robustness of the system.

[0014] In an implementable preferred solution, the optical data acquisition module is based on a multi-camera array, and the multi-camera array includes a binocular camera and a trinocular camera; it is configured with an adaptive optical acquisition algorithm for automatically adjusting the exposure parameters of the image acquisition component and dynamically adjusting the image resolution and frame rate according to surgical requirements; The feature points collected by the optical data acquisition module include the marking points on the surgical instruments, the anatomical feature points on the patient's body, and the artificially manufactured feature points; the artificially manufactured feature points include pasting marking stickers with special patterns around the patient's surgical site, and using combinations of marking points with different shapes, colors, and / or sizes to mark different surgical instruments.

[0015] In an implementable preferred solution, a method combining a feature point extraction algorithm based on deep learning with a traditional algorithm is used to extract the corresponding feature points, including the following: In a simple scenario, feature points are extracted through a traditional algorithm; in a complex scenario, the traditional algorithm is first used for preliminary screening of feature points, and then the deep learning algorithm is used for further extraction and classification of the screened feature points; The pose information is calculated by using a method that fuses a pose solution algorithm based on deep learning with a traditional algorithm, including the following: The traditional algorithm uses triangulation to preliminarily calculate the pose information of the surgical instrument and the patient, and then uses the result as the initial value of the ICP algorithm for iterative optimization; In a simple scenario, the result is quickly obtained through a traditional algorithm, and in a complex scenario, a traditional algorithm is used for a quick preliminary estimate, and then the deep learning algorithm is used for fine adjustment.

[0016] In an implementable preferred solution, the inertial data acquisition module is also used to compensate for the IMU temperature drift error in real time by using a mean filtering algorithm based on a sliding window. By integrating a temperature sensor inside the IMU, the working temperature of the IMU is monitored in real time; this algorithm processes the data collected by the IMU according to the set size of the sliding window. In each time window, the average value of the data is calculated, and this average value is used as the correction reference value. When new data enters the window, the old data moves out of the window, and the average value is recalculated; by continuously updating the average value, the zero bias error of the IMU is corrected in real time.

[0017] In an implementable preferred solution, a state model and an observation module are established to fuse the data of each component of the IMU, including the following: Construct a state module: Let the state vector , where are the four classifications of the quaternion, used to represent the pose, is the bias of the gyroscope; Based on the measured value of the gyroscope, the state transition equation is obtained:

[0018] Among them, is based on the state estimate at the previous moment The predicted state at the current moment is the state transition matrix is the control input matrix is the control input is the process noise, which follows a Gaussian distribution , is the process noise covariance matrix; The quaternion update formula is:

[0019] where is a matrix related to the angular velocity and the part related to quaternion update in the state transition matrix is obtained through discretization; Construct the observation model: The accelerometer measurement equation is as follows:

[0020] where is the measurement value of the accelerometer is the measurement matrix of the accelerometer is the measurement noise of the accelerometer, which follows a Gaussian distribution , is the measurement noise covariance matrix of the accelerometer; The magnetometer measurement equation

[0021] where is the measurement value of the magnetometer is the measurement matrix of the magnetometer is the measurement noise of the magnetometer, which follows a Gaussian distribution , is the measurement noise covariance matrix of the magnetometer; Prediction step: State prediction, predicting the state at the current moment according to the state transition equation:

[0022] Covariance prediction, predicting the covariance matrix of the state estimate:

[0023] where is the covariance matrix of the predicted state and the covariance matrix of the estimated state at the previous moment; Update step: Combining the measurement values of the accelerometer and magnetometer, calculate the Kalman gain :

[0024]

[0025]

[0026] Update the state estimate according to the measurement value:

[0027]

[0028] Update the covariance matrix of the state estimate:

[0029] Introduce an adaptively adjusted noise covariance matrix, and is an adaptive factor, and the parameters of the noise covariance matrix are dynamically adjusted according to the stability and reliability of the IMU measurement data.

[0030] In an implementable preferred solution, a data fusion processing module is used to perform time synchronization or calibration on the optical camera and IMU acquisitions, including the following: The FPGA generates a clock signal as the time reference for the entire system; when the optical camera and IMU acquire data, a timestamp is added to each data sample according to this clock signal; By sending a synchronization signal, record the time difference between the optical camera and the IMU when receiving the signal; by comparing the timestamp sequences of the optical data and the inertial data, calculate the average time deviation between the two. If the average time deviation exceeds the threshold, correct the timestamps of the optical data or the inertial data according to the deviation value.

[0031] In an implementable preferred solution, different data fusion strategies are adopted according to the reliability and availability of the optical data and the inertial data, including the following: When the lighting conditions are good and the optical feature points are clear, assign a high weight to the optical data; when the optical signal is partially blocked or interfered, reduce the weight of the optical data and increase the weight of the inertial data; When the optical signal is briefly lost, use the IMU data interpolation to predict the pose to avoid navigation interruption. By analyzing the measurement data of the IMU, establish a motion model of the surgical instrument; according to the motion model and the historical measurement data of the IMU, predict the pose of the surgical instrument; After the optical signal is restored, fuse the optical data with the predicted pose and recalibrate the navigation result.

[0032] In an implementable preferred solution, a navigation information generation module is further included, which is used to generate navigation information and includes a static navigation sub-module and a dynamic navigation sub-module; the static navigation sub-module is used to generate an initial navigation path according to preoperative planning information; the dynamic navigation sub-module is used to adjust the navigation path in real time according to the position and attitude changes of the surgical instrument in combination with the patient's anatomical structure.

[0033] In an implementable preferred solution, a navigation feedback module is further included. A vibration motor is integrated in the handle of the surgical instrument, and tactile feedback is realized by controlling the vibration intensity and frequency of the vibration motor; a vibration control algorithm is configured to dynamically adjust the vibration parameters of the vibration motor according to the deviation degree between the surgical instrument and the planned path.

[0034] In a second aspect, the present invention provides a real-time intraoperative navigation method based on optical and inertial navigation, which uses the above-mentioned real-time intraoperative navigation system based on optical and inertial navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic structural diagram of a real-time intraoperative navigation system based on optical and inertial navigation.

[0036] Figure 2 It is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0037] To make the technical solutions and their advantages of the present application clearer, the technical solutions of the present invention will be further described in detail below with reference to the drawings. It can be understood that the specific embodiments described herein are only partial embodiments of the present invention, which are only used to explain the present application and are not intended to limit the present application. It should be noted that the technical features or combinations of technical features described in the following embodiments should not be considered in isolation, and they can be combined with each other to achieve better technical effects. The same reference numerals in the drawings of the following embodiments represent the same features or components, and can be applied to different embodiments.

[0038] In addition, unless otherwise defined, the technical terms or scientific terms used in the description of the present invention should have the ordinary meanings understood by those of ordinary skill in the art to which the present invention belongs.

[0039] The present invention will be further described in detail below with reference to the drawings: Reference numerals: electronic device 500, processor 501, communication interface 502, memory 503, bus 504.

[0040] The embodiments of the present disclosure provide a real-time intraoperative navigation system based on optical and inertial navigation. Refer to Figure 1, including an optical data acquisition module, an inertial data acquisition module, a data fusion processing module, and a navigation information generation module.

[0041] The optical data acquisition module includes an image acquisition sub-module, a feature point extraction sub-module, and a pose solution operator module.

[0042] The image acquisition sub-module is based on a multi-camera array. The combination of a binocular camera and a trinocular camera is selected to construct the multi-camera array. The binocular camera has a stereoscopic vision function and can obtain the depth information of objects in the surgical scene through parallax calculation, which helps to more accurately locate surgical instruments and patient anatomical structures. The trinocular camera further expands the viewing angle range, can cover more angles of the surgical area, and reduces visual blind spots.

[0043] Supports high-resolution (≥1080p) and high-frame-rate (≥30Hz) dynamic capture, and configures an adaptive optical acquisition algorithm to adapt to the illumination changes in the operating room. In the high-resolution mode, it can clearly capture the tiny marker points on the surgical instruments and the subtle features on the surface of the patient's tissues, providing rich and accurate image details for subsequent feature point extraction and pose solution. At the same time, the high-frame-rate feature ensures that during the surgical process, even if the surgical instruments and the patient's body are in a dynamic movement state, a continuous image sequence can be captured in real time, so as to realize the real-time tracking of their movement states.

[0044] Configures an adaptive optical acquisition algorithm to monitor the brightness information of the image in real time and automatically adjust the exposure parameters of the image acquisition components (such as optical cameras), such as shutter speed, aperture size, and gain. When the illumination in the surgical area increases, the algorithm automatically reduces the exposure amount to avoid detail loss caused by over-bright images; conversely, when the illumination weakens, the algorithm increases the exposure amount to ensure that the image is clearly visible. For example, when using a strong light illumination device for local illumination during the surgical process, the adaptive optical acquisition algorithm can quickly respond to ensure that the images obtained by the camera are always in the best exposure state.

[0045] The adaptive optical acquisition algorithm is also used to dynamically adjust the image resolution and frame rate according to the specific requirements of the surgery. At the beginning stage of the surgery, when it is necessary to observe and locate the patient's overall anatomical structure, a lower resolution (such as 1080p) is selected to reduce the data volume and computational load and improve the system response speed. When entering the fine operation stage, such as blood vessel suture and nerve repair, the image resolution is increased to 4K or even higher to obtain richer detailed information to assist the doctor in precise operation. The image frame rate is always maintained at a level of ≥30Hz to ensure that the movement states of the surgical instruments and the patient can be tracked in real time. In some surgeries with extremely high real-time requirements, such as cardiac surgery, due to the beating of the heart causing the surgical area to be in rapid dynamic change, the frame rate is increased to 60Hz or higher to ensure that the instantaneous position changes of the surgical instruments and the heart tissue can be accurately captured, providing accurate data support for navigation.

[0046] The image acquisition sub-module can also select a high-definition electronic endoscope or an intraoperative ultrasound device according to actual needs.

[0047] The electronic endoscope has high-resolution imaging capabilities and can clearly present the fine structures of deep tissues inside the patient. The optical lens design of the endoscope can adapt to the observation requirements of different surgical sites. During the surgery, the endoscope enters the patient's body together with the surgical instruments and real-time collects the image information of the surgical area. To ensure the image quality, the endoscope is equipped with a dedicated lighting system that can provide uniform and bright illumination to avoid image blurring caused by insufficient light. The endoscope image is registered and fused with the surgical scene image obtained by the multi-eye camera. Using the feature point matching algorithm, the corresponding feature points in the endoscope image and the surgical scene image are found, and the transformation matrix between the two is calculated to achieve the spatial alignment of the images.

[0048] The intraoperative ultrasound device is equipped with different types of ultrasound probes, and a suitable probe is selected according to the surgical site and requirements. For example, a linear array probe is used for superficial tissue imaging, and a convex array probe is used for deep tissue imaging. During the surgery, the ultrasound probe is placed near the patient's surgical site, and by emitting and receiving ultrasonic waves, the ultrasound images of deep tissues and organs inside the patient are collected in real time. By establishing the spatial mapping relationship between the ultrasound image and the optical image, the information in the ultrasound image (such as the position of deep organs, the position of lesions, etc.) is superimposed on the display interface of the optical navigation, and the changes in deep tissues are observed with the help of the intraoperative ultrasound image to improve the accuracy and safety of the surgery.

[0049] The feature point extraction submodule is based on the active marker recognition unit. In addition to selecting the marker points on the surgical instruments and the anatomical feature points on the patient's body, it also introduces artificially created feature points. For example, marker stickers with special patterns are pasted around the patient's surgical site. The pattern design of these stickers has unique geometric shapes and texture features, which are easy to identify and track in the image. At the same time, the marker points on the surgical instruments are optimized and designed, and a combination of marker points of different shapes, colors and / or sizes is used to increase the recognition of the marker points. Specifically, active light-emitting marker points are integrated on the surgical instruments, and infrared LEDs are preferably used as the marking light source. Infrared light has good penetrability. In low-light environments, such as when blood or tissue blocks visible light during surgery, infrared light can still be effectively transmitted, thereby significantly improving the recognition accuracy of feature points.

[0050] Each active light-emitting marker has a unique coding identification. During the image recognition process, different surgical instruments and markers can be distinguished more accurately to avoid confusion. For example, by encoding the flashing frequency, duty cycle and other parameters of the infrared LED, the system can quickly identify and track specific surgical instruments.

[0051] The feature extraction submodule extracts feature points by combining a deep learning-based feature point extraction algorithm with a traditional algorithm.

[0052] The traditional feature extraction algorithm is improved by using Harris or SIFT feature extraction algorithm. According to the characteristics of surgical images, the scale space construction parameters and key point screening threshold in the algorithm are adjusted to improve the algorithm's operating efficiency and feature point extraction accuracy in surgical images. By reducing the number of scale space layers, the algorithm can be accelerated while ensuring the quality of feature point extraction; at the same time, the key point screening threshold is increased to remove some unstable feature points and improve the quality of feature points.

[0053] For deep learning, you can choose a feature point extraction model based on Mask R-CNN, and use a large amount of labeled surgical image data to train the model so that it can learn the feature representation of feature points in different surgical scenarios. This enables the feature point extraction model to accurately identify instances of surgical instruments and patient anatomical structures in the image, and extract the corresponding feature points at the same time.

[0054] Combine the feature point extraction algorithm based on deep learning with traditional algorithms. In relatively simple scenarios (such as the initial stage of surgery), the image quality is good, and traditional algorithms can quickly and accurately extract feature points. In relatively complex scenarios (such as during surgery, with lighting changes, or blood obscuring surgical instruments / patient tissues), it is difficult for traditional algorithms to accurately identify feature points or the calculation speed cannot meet the requirements of the real-time intraoperative navigation system. Deep learning algorithms can learn complex feature representations from a large amount of image data, have strong robustness to lighting changes, occlusion, image deformation, etc., and can accurately identify feature points in complex environments, and can be flexibly adjusted and optimized according to different surgical needs and scenarios. By adjusting the structure and parameters of the model, or using different training data, the feature point extraction performance of the model in specific surgical scenarios can be improved. By combining the two, in relatively complex scenarios, first use traditional algorithms for preliminary feature point screening, and then use deep learning algorithms to further accurately extract and classify the screened feature points, which can give full play to the powerful feature learning ability of deep learning algorithms and the stability of traditional algorithms in simple scenarios, improve the accuracy and robustness of feature point extraction, and also ensure the calculation efficiency, thus improving the real-time performance of the entire system.

[0055] The pose solver module adopts a method of fusing the pose calculation algorithm based on deep learning with traditional algorithms, and uses a Transformer-based pose calculation model. Transformer has powerful global feature capture ability. Through a large amount of surgical image data, the Transformer model is trained to enable it to learn the complex mapping relationship between feature points and poses.

[0056] Traditional algorithms use triangulation to initially calculate the pose information of surgical instruments and patients, and then use this result as the initial value of the ICP (Iterative Closest Point) algorithm for iterative optimization. In this way, both the fast calculation characteristics of triangulation and the high-precision optimization ability of the ICP algorithm can be utilized to improve the accuracy and stability of pose calculation.

[0057] In actual optical navigation applications, various uncertain factors will be faced, such as lighting changes, occlusion, image noise, etc. Traditional algorithms have poor robustness in these complex situations, while deep learning algorithms have strong robustness. Although deep learning algorithms have high accuracy and robustness, they consume a large amount of computing resources. Therefore, for relatively simple scenarios, directly calculating through traditional algorithms is relatively fast, and a preliminary pose calculation result can be obtained in a short time. For relatively complex scenarios, traditional algorithms are used for a quick preliminary estimate, and then deep learning algorithms are used for fine adjustment, which can improve the real-time performance of the system while ensuring the calculation accuracy.

[0058] The inertial data acquisition module includes an IMU (Inertial Measurement Unit) measurement sub-module, a data processing sub-module, and an attitude solution operator module.

[0059] The IMU measurement sub-module is based on a micro IMU integrated component. The micro IMU integrated component preferably embeds a 9-axis IMU (accelerometer + gyroscope + magnetometer) into the handle of the surgical instrument. The accelerometer is used to measure the acceleration of the surgical instrument in three axes (x, y, z axes), and can sense the changes in the motion states such as acceleration, deceleration, and vibration of the surgical instrument in real time. The gyroscope accurately measures the angular velocity of the surgical instrument in three axes, obtains its rotational motion information, and can monitor the attitude adjustment and rotational operation of the surgical instrument. The magnetometer can measure the intensity of the earth's magnetic field in different directions, and assist in determining the orientation of the surgical instrument. Especially when the optical navigation signal is severely interfered, the orientation information provided by the magnetometer can provide a key supplement for the navigation system.

[0060] The IMU measurement sub-module adjusts the data acquisition frequency of the IMU according to the surgical information. For some surgical operations that require quick response and high-precision tracking, such as micro-operations in neurosurgery, the data acquisition frequency is set to 1000Hz or even higher. This can obtain a large amount of measurement data in a short time and capture the minute motion changes of the surgical instrument in real time. For some relatively slow surgical operations, such as organ exploration in abdominal surgery, the acquisition frequency can be appropriately reduced to 200 - 500Hz to reduce the data volume and processing burden, while still meeting the basic requirements of navigation.

[0061] The IMU measurement sub-module preferably transmits the raw data through low-latency Bluetooth 5.0. Bluetooth 5.0 has the characteristics of high speed, low power consumption, and low latency. During the surgical process, it can ensure that the raw data collected by the IMU is quickly and stably transmitted to the data acquisition card and subsequent processing devices, avoiding the lag of navigation information caused by data transmission delay, and ensuring the real-time performance of the navigation system. For example, during delicate surgical operations, the minute motion changes of the surgical instrument can be transmitted in a timely manner through Bluetooth 5.0, enabling the navigation system to respond quickly.

[0062] Since the IMU will generate temperature drift error due to temperature changes during operation, which affects the measurement accuracy. In this embodiment, the data processing module adopts a technical solution for real-time compensation of the IMU temperature drift error, and integrates a temperature sensor inside the IMU to monitor the working temperature of the IMU in real time.

[0063] An adaptive zero-bias correction algorithm is adopted, specifically the mean filtering algorithm based on a sliding window. This algorithm processes the data collected by the IMU according to the set size of the sliding window. Within each time window, the average value of the data is calculated and used as the correction reference value. When new data enters the window, the old data moves out of the window, and the average value is recalculated. By continuously updating the average value, the zero-bias error of the IMU is corrected in real time. For example, during a surgical procedure, as time goes by, the temperature of the IMU gradually increases. The mean filtering algorithm based on the sliding window can adjust the zero-bias correction value in real time according to the temperature change to ensure the accuracy of the IMU measurement data.

[0064] The attitude resolver module uses a data fusion algorithm to fuse the data of the accelerometer, gyroscope, and magnetometer of the IMU to obtain more accurate attitude information. The Kalman filtering algorithm is preferably used as the data fusion algorithm, and the measurement data of the IMU is used as the observation value. The Kalman filtering algorithm is used to estimate the attitude of the surgical instrument in real time.

[0065] The attitude resolver module establishes a state model and an observation model. The state model describes the dynamic change process of the attitude of the surgical instrument, and the observation model describes the relationship between the IMU measurement data and the attitude state. At each sampling moment, the state at the current moment is predicted according to the state estimate value at the previous moment and the dynamic model of the system, and then the predicted state is updated according to the IMU measurement data at the current moment to obtain a more accurate attitude estimate value.

[0066] Specifically, a state module is constructed: Assume the state vector , where are the four components of the quaternion, used to represent the pose, is the bias of the gyroscope.

[0067] Based on the measured value of the gyroscope, the state transition equation is obtained:

[0068] where, is the state at the current moment predicted based on the state estimate at the previous moment, is the state transition matrix, is the control input matrix, is the control input, is the process noise, which follows a Gaussian distribution , is the process noise covariance matrix.

[0069] The quaternion update formula is:

[0070] Among them, is a matrix related to the angular velocity and the part related to the quaternion update in the state transition matrix can be obtained through discretization.

[0071] Construct the observation model: The accelerometer measurement equation is as follows:

[0072] Among them, is the measurement value of the accelerometer, is the measurement matrix of the accelerometer, is the measurement noise of the accelerometer, which follows a Gaussian distribution , is the measurement noise covariance matrix of the accelerometer.

[0073] The magnetometer measurement equation,

[0074] Among them, is the measurement value of the magnetometer, is the measurement matrix of the magnetometer, is the measurement noise of the magnetometer, which follows a Gaussian distribution , is the measurement noise covariance matrix of the magnetometer.

[0075] Prediction step: Perform state prediction based on the measurement value of the gyroscope, and predict the state at the current moment according to the state transition equation:

[0076] Covariance prediction, predict the covariance matrix of the state estimate:

[0077] Among them, is the covariance matrix of the predicted state and the covariance matrix of the estimated state at the previous moment.

[0078] Update step: Combine the measurement values of the accelerometer and the magnetometer to calculate the Kalman gain :

[0079]

[0080]

[0081] Update the state estimate according to the measurement values:

[0082]

[0083] Update the covariance matrix of the state estimate:

[0084] In the Kalman filter algorithm of this embodiment, an adaptively adjusted noise covariance matrix is introduced. and are adaptive factors, which dynamically adjust the parameters of the noise covariance matrix according to the stability and reliability of the IMU measurement data, enabling the filtering algorithm to better adapt to different surgical scenarios and IMU working states, and improving the accuracy and robustness of data fusion.

[0085] The attitude solution operator module sets the attitude update frequency according to the real-time requirements of surgical operations and the data acquisition frequency of the IMU. In most surgical scenarios, the attitude update frequency is set to be the same as or slightly lower than the data acquisition frequency of the IMU to ensure that the attitude changes of surgical instruments can be reflected in a timely manner. For example, when the data acquisition frequency of the IMU is 1000Hz, the attitude update frequency is set to 500 - 1000Hz. In some surgical operations with extremely high real-time requirements, such as cardiac interventional surgery, the attitude update frequency can be increased to more than 1000Hz to meet the rapid response requirements of the surgery. At the same time, to ensure the accuracy and stability of attitude updates, when updating the attitude each time, the new measurement data is checked for validity and filtered to remove the influence of abnormal data.

[0086] The data fusion processing module includes a time synchronization sub-module and a data fusion sub-module.

[0087] The time synchronization sub-module uses an FPGA (Field Programmable Gate Array) to achieve microsecond-level synchronization of optical and inertial data. The FPGA has the ability of high-speed and parallel processing, and can accurately add timestamps to the data collected by the optical camera and the IMU.

[0088] Specifically, the FPGA generates a high-precision clock signal as the time reference for the entire system. When the optical camera and the IMU collect data, timestamps are added to each data sample according to this clock signal. Specifically, when the optical camera captures a frame of image, the FPGA immediately records the clock value at that moment and uses it as the timestamp of this frame of image; similarly, when the IMU collects a set of measurement data, corresponding timestamps are also added according to the clock signal. In this way, it is ensured that the optical data and the inertial data are highly consistent in time, eliminating the time sequence deviation of multi-sensors.

[0089] The time synchronization sub-module will also adopt a time calibration algorithm to further improve the accuracy of time synchronization. During the system initialization phase, time synchronization calibration is performed on the optical camera and the IMU. By sending a synchronization signal, the time difference between the optical camera and the IMU receiving the signal is recorded, and the subsequent acquired data is time-calibrated according to this time difference.

[0090] During the surgical process, the time synchronization status of the optical camera and the IMU is monitored in real time. When it is found that the time deviation exceeds a certain threshold, the time calibration algorithm is started for adjustment. Specifically, by comparing the timestamp sequences of the optical data and the inertial data, the average time deviation between the two is calculated. If the average time deviation exceeds the threshold (e.g., 10 microseconds), the timestamps of the optical data or the inertial data are corrected according to the deviation value to ensure that the two are synchronized in time.

[0091] The data fusion sub-module is used to analyze the measurement data of the IMU and establish its noise model; Gaussian white noise describes the random error in the IMU measurement data, and random walk reflects the slow change of the IMU zero bias over time. A large amount of IMU historical data is used to train the LSTM network. During the training process, the measurement data of the IMU and the corresponding noise model are used as inputs, and the optimal parameters of the Kalman filter are used as outputs. The LSTM network can dynamically predict the optimal parameters of the Kalman filter according to the current IMU measurement data by learning the mapping relationship between the input data and the output parameters. The Kalman filter parameters predicted by the LSTM network are applied to the Kalman filter algorithm to fuse the optical data and the inertial data. In this way, the Kalman filter algorithm can adaptively adjust the filter parameters, better adapt to different surgical scenarios and IMU working states, improve the accuracy and robustness of data fusion, and better track the movement trajectory of the surgical instrument.

[0092] The data fusion sub-module adopts different data fusion strategies according to the reliability and availability of the optical data and the inertial data. When both the optical data and the inertial data are normal and reliable, a weighted fusion method is adopted. Different weights are assigned to the two according to the measurement accuracy and stability of the optical data and the inertial data. Specifically, when the lighting conditions are good and the optical feature points are clear, the reliability of the optical data is high and a higher weight is given; while when the optical signal is blocked or interfered to a certain extent, the weight of the optical data is appropriately reduced and the weight of the inertial data is increased.

[0093] When the optical signal is temporarily lost, the IMU data interpolation is used to predict the pose to avoid navigation interruption. By analyzing the measurement data of the IMU, a motion model of the surgical instrument is established. According to the motion model and the historical measurement data of the IMU, the pose of the surgical instrument is predicted. After the optical signal is restored, the optical data is fused with the predicted pose to recalibrate the navigation result. For example, when the optical camera is blocked by the surgical instrument or the patient's body, the acceleration and angular velocity data measured by the IMU during the occlusion are used to predict the displacement and rotation of the surgical instrument through integral operation to maintain the continuity of navigation.

[0094] A navigation information generation module is used to generate navigation information, including a static navigation sub-module and a dynamic navigation sub-module.

[0095] The static navigation sub-module is used to generate an initial navigation path according to the preoperative planning information. Through a path planning algorithm (such as the navigation information generation module for generating navigation information), the shortest path or the optimal path to the surgical target position is searched to generate an initial navigation path.

[0096] The dynamic navigation sub-module is used to combine the position and attitude changes of the surgical instrument with the patient's anatomical structure. When there are obstacles, the surgical instrument deviates from the predetermined path, or the patient's anatomical structure changes, the navigation path is adjusted in real time to ensure that the surgical instrument can reach the surgical target position safely and accurately.

[0097] Embodiment 2 The distinguishing technical feature of this embodiment from the above embodiment is that it further includes a navigation feedback module. A vibration motor is integrated in the surgical instrument handle, and tactile feedback is achieved by controlling the vibration intensity and frequency of the vibration motor. The vibration motor adopts a small and efficient design to ensure that it will not affect the operating performance of the surgical instrument.

[0098] A vibration control algorithm is configured to dynamically adjust the vibration parameters of the vibration motor according to the deviation degree of the surgical instrument from the planned path. When the surgical instrument deviates slightly from the planned path, the vibration motor generates a slight low-frequency vibration as a slight warning signal; when the deviation degree gradually increases, the vibration intensity and frequency are gradually increased to remind the doctor to adjust the position of the surgical instrument in time with a stronger vibration. For example, when the distance of the surgical instrument deviating from the planned path is less than 1 mm, the vibration motor vibrates at a frequency of 10 Hz and a maximum vibration intensity of 10%; when the deviation distance reaches 2 - 3 mm, the vibration frequency is increased to 30 Hz and the vibration intensity is increased to 30%; when the deviation distance exceeds 3 mm, the vibration frequency is increased to 50 Hz and the vibration intensity reaches 50%, which can attract the doctor's high attention.

[0099] Embodiment 3 The technical features differentiating this embodiment from the above embodiments are that it further includes an augmented reality module. Before the operation, the augmented reality module uses medical image processing technology to perform three-dimensional reconstruction on the patient's CT / MRI images to obtain a three-dimensional model of the patient's internal body structure. Then, the navigation path is determined according to the surgical plan, and the navigation path information is fused with the three-dimensional model.

[0100] During the operation, the system obtains the position and attitude information of the surgical instrument in real time, and projects the virtual model of the surgical instrument, the fused three-dimensional model, and the navigation path onto the field of view of the AR device (such as Hololens). Through the AR device, the doctor can intuitively see the position and movement direction of the surgical instrument in the patient's body, as well as the relative position relationship with the surrounding tissues and organs. For example, during a brain operation, through the AR device, the doctor can see the real-time position of the surgical instrument in the three-dimensional model of the brain and its movement trajectory along the navigation path. At the same time, the distance between the surgical instrument and important structures such as surrounding nerves and blood vessels can be clearly observed, providing accurate visual guidance for the surgical operation.

[0101] The position and attitude of the AR device are monitored and calibrated in real time through an optical camera and an IMU to ensure the accurate alignment of the virtual model and the navigation path with the actual surgical scene. At the same time, according to the movement of the surgical instrument and the movement of the patient's body, the AR display content is updated in real time to ensure that the doctor can always obtain accurate navigation information.

[0102] The augmented reality module is also used to fuse the real-time surgical images collected by the optical camera with other modality images (such as ultrasound images, fluorescence images), and align the different modality images in space so as to observe multiple pieces of information of the surgical area simultaneously. For example, during a liver operation, by fusing the real-time liver surface image collected by the optical camera with the ultrasound image, the doctor can, while seeing the surface condition of the liver, understand the blood vessels and lesion conditions inside the liver, providing more comprehensive information support for the surgical operation.

[0103] The embodiment of the present disclosure also provides a real-time intraoperative navigation method based on optical and inertial navigation, which uses the above real-time intraoperative navigation system based on optical and inertial navigation.

[0104] The embodiment of the present disclosure also provides a storage medium, in which a computer program is stored. When the computer program is executed by a processor, it can implement all the steps of the above real-time intraoperative navigation method based on optical and inertial navigation.

[0105] Those of ordinary skill in the art can understand that all or part of the processes in implementing a real-time intraoperative navigation method based on optical and inertial navigation can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of various embodiments of a real-time intraoperative navigation method based on optical and inertial navigation. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in this application can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0106] An embodiment of this application also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of the above-mentioned real-time intraoperative navigation method based on optical and inertial navigation. In the embodiment of this application, the processor is the control center of the computer method, which can be the processor of a physical machine or the processor of a virtual machine.

[0107] Referring to Figure 2 , the electronic device 500 includes: at least one processor 501, at least one communication interface 502, at least one memory 503, and at least one bus 504. Among them, the bus 504 is used to implement connection communication between these components, the communication interface 502 is used to communicate signaling or data with other node devices, and the memory 503 stores machine-readable instructions executable by the processor 501. When the electronic device 500 runs, communication occurs between the processor 501 and the memory 503 through the bus 504. When the machine-readable instructions are called by the processor 501, they execute the steps of the above-mentioned real-time intraoperative navigation method based on optical and inertial navigation.

[0108] The above content is only an embodiment of the present invention. Specific structures and common knowledge such as characteristics that are well-known in the art are not described in detail herein. Those of ordinary skill in the art know all the general technical knowledge in the technical field to which the invention pertains before the filing date or the priority date, can learn all the existing technologies in this field, and have the ability to apply conventional experimental means before this date. Those of ordinary skill in the art can, under the inspiration given in this application, combine their own abilities to improve and implement this solution. Some typical well-known structures or well-known methods should not become an obstacle for those of ordinary skill in the art to implement this application. It should be noted that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can still be made, and these should also be regarded as the protection scope of the present invention, and these will not affect the implementation effect of the present invention and the practicality of the patent. The protection scope required by this application should be based on the content of its claims, and the specific implementation manners and the like recorded in the specification can be used to interpret the content of the claims.

Claims

1. A real-time intraoperative navigation system based on optical and inertial navigation, characterized in that: include: The optical data acquisition module is used to collect images, identify surgical instruments and patient anatomical structures, extract corresponding feature points by combining a deep learning-based feature point extraction algorithm with a traditional algorithm, and calculate pose information by combining a deep learning-based pose solution algorithm with a traditional algorithm. Inertial data acquisition module, used to build state model and observation module by using IMU integrated components, fuse the data of IMU components, and obtain the posture information of surgical instruments; The data fusion processing module is used for time synchronization and calibration of optical camera and IMU acquisition. The data fusion algorithm adopts Kalman filter algorithm. The data fusion processing module is used to analyze the measurement data of IMU and establish its noise model; a large amount of IMU historical data is used to train the LSTM network; the IMU measurement data and the corresponding noise model are used as input, and the optimal parameters of Kalman filter are used as output; the LSTM network dynamically predicts the optimal parameters of Kalman filter according to the current IMU measurement data by learning the mapping relationship between input data and output parameters; the Kalman filter parameters predicted by the LSTM network are applied to the Kalman filter algorithm to fuse the optical data and inertial data; and different data fusion strategies are adopted according to the reliability and availability of optical data and inertial data.

2. The real-time intraoperative navigation system based on optical and inertial navigation according to claim 1, characterized in that: The optical data acquisition module is based on a multi-eye camera array, which includes a binocular camera and a trinocular camera; and is equipped with an adaptive optical acquisition algorithm for automatically adjusting the exposure parameters of the image acquisition component and dynamically adjusting the image resolution and frame rate according to surgical requirements; The feature points collected by the optical data acquisition module include marking points on surgical instruments, anatomical feature points on the patient's body and artificially created feature points; the artificially created feature points include marking stickers with special patterns pasted around the patient's surgical site, and marking different surgical instruments with a combination of marking points of different shapes, colors and / or sizes.

3. The real-time intraoperative navigation system based on optical and inertial navigation according to claim 1, characterized in that: The corresponding feature points are extracted by combining the feature point extraction algorithm based on deep learning with the traditional algorithm, including the following: In simple scenarios, feature points are extracted using traditional algorithms; in complex scenarios, traditional algorithms are used to perform preliminary feature point screening, and then deep learning algorithms are used to further extract and classify the screened feature points. The pose information is calculated by combining the pose solving algorithm based on deep learning with the traditional algorithm, including the following: The traditional algorithm uses triangulation to preliminarily calculate the posture information of surgical instruments and patients, and then uses the result as the initial value of the ICP algorithm for iterative optimization; Simple scenarios can quickly obtain results using traditional algorithms, while complex scenarios use traditional algorithms for quick preliminary estimates and then use deep learning algorithms for fine-tuning.

4. The real-time intraoperative navigation system based on optical and inertial navigation according to claim 1, characterized in that: The inertial data acquisition module is also used to compensate for the IMU temperature drift error in real time using a sliding window-based mean filter algorithm. By integrating a temperature sensor inside the IMU, the IMU's operating temperature is monitored in real time. The algorithm processes the data collected by the IMU according to the set sliding window size. In each time window, the average value of the data is calculated and used as a correction reference value. When new data enters the window, the old data is moved out of the window and the average value is recalculated; by continuously updating the average value, the zero bias error of the IMU is corrected in real time.

5. The real-time intraoperative navigation system based on optical and inertial navigation according to claim 1, characterized in that: Establish a state model and observation module to fuse the data of each IMU component, including the following: Build the status module: Let the state vector ,in There are four categories of quaternions, used to represent postures. is the deviation of the gyroscope; Based on gyroscope measurements , and the state transfer equation is obtained: in, It is based on the state estimate of the previous moment The predicted current state, is the state transition matrix, is the control input matrix, is the control input, is process noise, which follows a Gaussian distribution , is the process noise covariance matrix; The quaternion update formula is: in, is a function of the angular velocity The related matrix is ​​discretized to obtain the state transfer matrix The part related to quaternion update; Constructing the observation model: The accelerometer measurement equation is as follows: in, is the accelerometer measurement, is the measurement matrix of the accelerometer, is the measurement noise of the accelerometer, which follows a Gaussian distribution , is the measurement noise covariance matrix of the accelerometer; Magnetometer measurement equation, in, is the measurement value of the magnetometer, is the measurement matrix of the magnetometer, is the measurement noise of the magnetometer, which follows a Gaussian distribution , is the measurement noise covariance matrix of the magnetometer; Prediction steps: State prediction, predicting the current state based on the state transition equation: Covariance prediction, predicting the covariance matrix of the state estimate: in, is the covariance matrix of the predicted state, is the covariance matrix of the estimated state at the previous moment; Update steps: Combine the measurements from the accelerometer and magnetometer to calculate the Kalman gain : Update the state estimate based on the measurements: Update the covariance matrix of the state estimate: Introducing the adaptively adjusted noise covariance matrix, and is an adaptive factor that dynamically adjusts the parameters of the noise covariance matrix according to the stability and reliability of the IMU measurement data.

6. The real-time intraoperative navigation system based on optical and inertial navigation according to claim 1, characterized in that: The data fusion processing module is used for time synchronization or calibration of optical camera and IMU acquisition, including the following: The FPGA generates a clock signal as the time reference for the entire system. When the optical camera and IMU collect data, they add a timestamp to each data sample based on this clock signal. By sending a synchronization signal, the time difference between the optical camera and the IMU receiving the signal is recorded; by comparing the timestamp sequences of the optical data and the inertial data, the average time deviation between the two is calculated. If the average time deviation exceeds the threshold, the timestamp of the optical data or the inertial data is corrected according to the deviation value.

7. The real-time intraoperative navigation system based on optical and inertial navigation according to claim 1, characterized in that: Depending on the reliability and availability of optical and inertial data, different data fusion strategies are adopted, including the following: When the lighting conditions are good and the optical feature points are clear, the optical data is given a high weight; when the optical signal is blocked or interfered to a certain extent, the weight of the optical data is reduced and the weight of the inertial data is increased; When the optical signal is temporarily lost, the IMU data is used to interpolate and predict the position and posture to avoid navigation interruption. The motion model of the surgical instrument is established by analyzing the IMU measurement data. The position and posture of the surgical instrument are predicted based on the motion model and the historical measurement data of the IMU. After the optical signal is restored, the optical data is fused with the predicted pose and the navigation result is recalibrated.

8. The real-time intraoperative navigation system based on optical and inertial navigation according to claim 1, characterized in that: It also includes a navigation information generation module for generating navigation information, including a static navigation submodule and a dynamic navigation submodule; The static navigation submodule is used to generate the initial navigation path based on the preoperative planning information; The dynamic navigation submodule is used to adjust the navigation path in real time according to the position and posture changes of the surgical instruments combined with the patient's anatomical structure.

9. The real-time intraoperative navigation system based on optical and inertial navigation according to claim 1, characterized in that: It also includes a navigation feedback module, which integrates a vibration motor in the handle of the surgical instrument to achieve tactile feedback by controlling the vibration intensity and frequency of the vibration motor; and configures a vibration control algorithm to dynamically adjust the vibration parameters of the vibration motor according to the degree of deviation between the surgical instrument and the planned path.

10. A real-time intraoperative navigation method based on optical and inertial navigation, characterized in that: A real-time intraoperative navigation system based on optical and inertial navigation as described in any one of claims 1 to 9 is used.

Images