Intraoperative tracking method and system for surgical instruments

Abstract

This invention provides a method and system for intraoperative tracking of surgical instruments, relating to the field of surgical instrument calibration and tracking technology. The method includes: collecting multi-source data during surgery and performing preoperative calibration; establishing initial transformation relationships between various coordinate systems based on the preoperative calibration results; filtering visual data and kinematic data respectively using an extended Kalman filter based on the initial transformation relationships to obtain instrument pose estimation results; correcting the instrument pose estimation results using a fuzzy logic algorithm, combined with the statistical characteristics of visual measurement residuals and kinematic residuals, to obtain a fused pose result; updating the three-dimensional position and orientation of the surgical instrument in real time based on the fused pose information; mapping the updated three-dimensional position and orientation to an intraoperative digital model; and tracking the surgical instrument in real time based on the updated three-dimensional position and orientation, combined with historical tracking trajectories, until the end of the surgery, and outputting the three-dimensional position and orientation of the surgical instrument.

Description

Technical Field

[0001] This invention relates to the field of surgical instrument calibration and tracking technology, and in particular to a method and system for intraoperative tracking of surgical instruments. Background Technology

[0002] With the rapid development of minimally invasive surgery, robot-assisted surgery and image-guided technology, higher requirements have been placed on the real-time high-precision tracking of surgical instruments in the operating room. However, the complex surgical scenarios (including blood, smoke, obstruction, metal reflection, etc.) and the rapid mutual movement between instruments and human tissues make it difficult for a single sensing channel to stably and reliably provide continuous high-precision pose information.

[0003] Intraoperative tracking of surgical instruments is a key means to achieve precise navigation and intelligent assisted surgery. Current mainstream solutions include vision-based tracking, inertial tracking based on inertial measurement units (IMUs), and kinematic calculation based on the robotic arm / instrument itself. However, these single-channel technologies each have their own advantages and disadvantages, and pure kinematic calculation is limited by model errors and assembly inaccuracies.

[0004] With the development of algorithms such as multi-sensor fusion, filtering estimation, and fuzzy control, researchers have begun to explore the joint use of visual, inertial, and kinematic data to improve robustness and accuracy. However, in real-world surgical environments, latency, noise characteristics, and sudden anomalies (such as short-term occlusion, sensor misalignment, or sudden collisions) between data sources can still lead to decreased or unstable tracking accuracy, resulting in tracking interruptions or further accuracy reductions. Furthermore, the reliability of each data source cannot be adaptively adjusted according to the dynamic intraoperative environment, making it difficult to cope with sudden interference and complex motion scenarios. Overall robustness and real-time performance still need improvement. Summary of the Invention

[0005] To address the technical challenges of real-world surgical environments where latency, noise characteristics, and sudden anomalies (such as brief occlusion, sensor malfunction, or sudden collisions) between data sources still lead to decreased or unstable tracking accuracy, resulting in tracking interruptions or further accuracy degradation. Furthermore, these systems struggle to adaptively adjust the reliability of data sources based on the dynamic intraoperative environment, cope with sudden interference and complex motion scenarios, and ultimately require improvement in overall robustness and real-time performance. This invention provides an intraoperative tracking method and system for surgical instruments. The technical solution is as follows:

[0006] First aspect:

[0007] An intraoperative tracking method for surgical instruments provided in this embodiment of the invention includes:

[0008] S1: Collect multi-source data during the operation and perform preoperative calibration of the surgical instrument tracking system;

[0009] S2: Based on the preoperative calibration results, establish multiple initial transformation relationships between the camera coordinate system, the inertial measurement unit coordinate system, the robot coordinate system, and the surgical instrument coordinate system;

[0010] S3: Based on the initial transformation relationships, the visual data and kinematic data in the intraoperative multi-source data are filtered by the extended Kalman filter to obtain the instrument pose estimation result;

[0011] S4: By using fuzzy logic algorithm and combining the statistical characteristics of visual measurement residuals and kinematic residuals, the device pose estimation results are corrected to obtain the fused pose results;

[0012] S5: Based on the fused pose results, the three-dimensional position and orientation of the surgical instruments are updated in real time;

[0013] S6: Map the updated 3D position and orientation to the intraoperative digital model for 3D visualization;

[0014] S7: Based on the updated 3D position and orientation and 3D visualization results, combined with historical tracking trajectories, the surgical instruments are tracked in real time to obtain the real-time pose results of the surgical instruments.

[0015] S8: Determine whether the surgery has ended; if yes, output the real-time pose of the surgical instruments; otherwise, return to step S3 and continue tracking until the surgery ends.

[0016] The second aspect:

[0017] An intraoperative tracking system for surgical instruments provided in this embodiment of the invention includes:

[0018] processor;

[0019] The memory stores computer-readable instructions, which, when executed by a processor, implement an intraoperative tracking method for surgical instruments as described in the first aspect.

[0020] Third aspect:

[0021] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an intraoperative tracking method for surgical instruments as described in the first aspect.

[0022] The beneficial effects of the technical solutions provided by the embodiments of the present invention include at least the following:

[0023] In this embodiment of the invention, by introducing the fusion processing of multi-source data such as vision, inertial measurement unit (IMU) data, and robot kinematics data during surgery, and combining extended Kalman filtering and fuzzy logic adaptive weighting algorithms, the reliability of each channel can be dynamically adjusted even when the accuracy of different sensors fluctuates or partially fails, achieving high-precision and continuous tracking of surgical instrument pose. Through residual statistics and fuzzy inference mechanisms, the influence of visual occlusion, illumination changes, and IMU drift on the results is effectively suppressed, significantly improving the stability and robustness of tracking. Simultaneously, by combining real-time visualization and force feedback mapping, surgeons can intuitively perceive the spatial position and force state of the instruments in the surgical field, improving operational safety and surgical precision. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating an intraoperative tracking method for surgical instruments provided in an embodiment of the present invention.

[0026] Figure 2 This is a schematic diagram of the structure of an intraoperative tracking system for surgical instruments provided in an embodiment of the present invention. Detailed Implementation

[0027] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0028] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0029] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0030] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0031] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0032] Reference manual attached Figure 1 The diagram shows a flowchart of an intraoperative tracking method for surgical instruments provided in an embodiment of the present invention.

[0033] This invention provides an intraoperative tracking method for surgical instruments. This method can be implemented using an intraoperative tracking device for surgical instruments, which can be a terminal or a server. The processing flow of the intraoperative tracking method for surgical instruments may include the following steps:

[0034] S1: Collect multi-source data during the operation and perform preoperative calibration of the surgical instrument tracking system.

[0035] Intraoperative multi-source data refers to data collected synchronously by multiple sensors during the operation, including visual information acquired by the stereo camera, acceleration and angular velocity data output by the inertial measurement unit (IMU), and kinematic parameters collected by the robotic system.

[0036] The surgical instrument tracking system is a comprehensive tracking platform consisting of a vision module, an inertial measurement module, a robot control module, and a fusion algorithm module, used to realize the real-time calculation and updating of instrument pose.

[0037] Preoperative calibration refers to establishing precise spatial transformation relationships between the camera coordinate system, inertial measurement unit coordinate system, robot coordinate system, and surgical instrument coordinate system using a specific calibration plate or reference coordinate system before surgery.

[0038] It should be noted that through multi-source data acquisition and preoperative calibration, the data from various sensors can be correlated and fused within a unified coordinate system, eliminating sensor installation errors and measurement deviations. This provides a high-precision foundation for subsequent pose estimation and data fusion. It significantly improves the overall spatial consistency and tracking accuracy of the system, making the positioning results of intraoperative instruments more stable and reliable, thus laying a solid foundation for real-time navigation and control in complex surgical scenarios.

[0039] In one possible implementation, the intraoperative multi-source data includes visual data acquired by a stereo camera, acceleration and angular velocity data output by an inertial measurement unit, and kinematic data acquired by a robot.

[0040] In one possible implementation, the preoperative calibration in S1 specifically includes: calibrating the intrinsic and extrinsic parameters of the camera manually or semi-automatically, calibrating the base coordinates of the robotic arm and the camera coordinate system, and calibrating the relationship between the instrument end coordinates and the surgical reference point.

[0041] Specifically, surgical reference points are spatial reference points in the intraoperative navigation system used to define the location and orientation of the surgical operation area.

[0042] It should be noted that by performing precise calculations on the relationships between multiple coordinate systems manually or semi-automatically, the camera, robotic arm, and surgical instruments achieve coordinate unification under the same spatial reference system. This high-precision calibration can significantly reduce spatial errors and installation deviations between systems, ensuring the accuracy of subsequent pose estimation and fusion calculations, thereby improving the overall accuracy and stability of intraoperative instrument tracking and providing a reliable foundation for safe and controllable surgical navigation.

[0043] S2: Based on the preoperative calibration results, establish multiple initial transformation relationships between the camera coordinate system, the inertial measurement unit coordinate system, the robot coordinate system, and the surgical instrument coordinate system.

[0044] The camera coordinate system is a coordinate system established with the camera's optical center as the origin and the optical axis as the principal axis. It is used to describe the position and orientation of the target in the visual data.

[0045] The inertial measurement unit coordinate system is a reference coordinate system with the center of the IMU sensor as the origin, used to output acceleration and angular velocity information.

[0046] The robot coordinate system is a coordinate system established with the robot base or joint reference point as the origin, and it serves as the spatial reference for calculating the kinematics of the robotic arm.

[0047] Among them, the surgical instrument coordinate system is a local coordinate system fixed on the surgical instrument, used to accurately describe the position and posture of the instrument end.

[0048] The initial transformation relationship refers to the spatial mapping relationship between these different coordinate systems, which is usually represented in the form of rotation matrix and translation vector, and is used to achieve spatial alignment of multi-source data.

[0049] It should be noted that by establishing the initial transformation relationship between the coordinate systems of the camera, IMU, robot, and surgical instruments, the data collected by each sensor can be fused and calculated under the same spatial reference, ensuring spatial consistency of information transmission. This step effectively eliminates coordinate deviations caused by different installation positions and orientations in multi-sensor systems, significantly improving the accuracy and stability of subsequent pose estimation and data fusion, laying a crucial foundation for achieving high-precision, real-time intraoperative tracking.

[0050] In one possible implementation, S2 specifically involves: calculating the pose transformation matrices between the camera, the inertial measurement unit, and the robot and surgical instruments, i.e., the initial transformation relationships, based on the preoperative calibration results.

[0051] S3: Based on the initial transformation relationships, the visual data and kinematic data in the intraoperative multi-source data are filtered by the extended Kalman filter to obtain the instrument pose estimation result.

[0052] Among them, the Extended Kalman Filter (EKF) is a recursive algorithm suitable for state estimation of nonlinear systems. It combines the process model and the observation model through a prediction-update mechanism to make the optimal estimate of the dynamic state in a noisy environment.

[0053] Visual data consists of image information acquired by stereo cameras or other optical sensors, used to extract the three-dimensional position and orientation of the device. Kinematic data consists of parameters such as joint angles, pose, or velocity provided by sensors within the robot system or device, reflecting the device's motion state.

[0054] Among them, the instrument pose estimation result refers to the estimated position and orientation of the surgical instrument in three-dimensional space obtained after fusing multi-source data.

[0055] It should be noted that by dynamically fusing visual and kinematic data using the extended Kalman filter algorithm, measurement errors from a single data source can be effectively suppressed under conditions of noise interference and nonlinear motion. Through a combination of prediction and correction, the system can maintain the continuity and stability of pose estimation even in the event of transient visual failures or robotic arm sensing errors. This method significantly improves the accuracy and robustness of surgical instrument tracking, providing reliable basic data support for subsequent fuzzy fusion correction and real-time visualization.

[0056] In one possible implementation, S3 specifically includes:

[0057] S301: Acquire the attitude information of the surgical instruments, including position information, quaternion attitude, linear velocity, and angular velocity.

[0058] In this context, attitude information refers to the motion state of surgical instruments in three-dimensional space, including position, orientation, linear velocity, and angular velocity. Quaternion attitude is a rotational representation method that avoids Euler angle singularities and is commonly used in three-dimensional attitude calculations.

[0059] S302: Establish a nonlinear fusion model based on linear velocity and angular velocity.

[0060] Among them, the nonlinear fusion model is used to describe the dynamic changes of the instrument over continuous time, and can simultaneously consider nonlinear factors such as displacement and rotation.

[0061] S303: Based on a nonlinear fusion model and combined with attitude information, determine the state transition function:

[0062]

[0063] in, l ( k )express k The position vector of the surgical instruments at time t. q ( k )express k The quaternion of the posture of surgical instruments at any given moment. express k The real part of the attitude quaternion at time. v ( k )express k The linear velocity vector at time t. express k Angular velocity vector at time t, f ( ) represents the state transition function. l ( k -1) indicates k Position vector of the surgical instruments at time -1 Indicates the time step. v ( k -1) indicates k The linear velocity vector at time -1 q ( k -1) indicates k The quaternion of the surgical instrument posture at time -1 express k The angular velocity vector at time -1 express k The real part of the attitude quaternion at time -1. express k Attitude information at time -1.

[0064] S304: Calculate the prediction covariance matrix based on the state transition function and the process noise covariance matrix:

[0065]

[0066]

[0067] in, express k The prediction error covariance matrix at time 1. F k express k The state transition Jacobian matrix at time step 1. express kThe posterior error covariance matrix at time -1 express k The process noise covariance matrix at time step 1. T Indicates matrix transpose. This represents the state transition function with respect to the state variables. x The partial derivatives, F k11 , F k12 , F k21 as well as F k22 Both represent block Jacobian matrices.

[0068] S305: Based on the predicted covariance matrix, perform EKF updates on both the visual and kinematic data to obtain the device pose estimation results:

[0069]

[0070] in, r k express k The residual at time step is the difference between the actual measurement and the predicted measurement. The residual calculated from visual data is called the visual residual, and the residual calculated from kinematic data is called the kinematic residual. z k express k Actual measurement of time, Represents the observation model function, S k express k The innovation covariance matrix at any given moment H k express k The Jacobian matrix of the observation at time t, R k express k The measurement noise covariance matrix at time step. K k express k The Kalman gain matrix at time 10:00. The matrix representing the inverse of the innovation covariance matrix. Indicates the posterior state. Indicates the prior state. express k The posterior error covariance matrix at time t. I Represents the identity matrix.

[0071] It should be noted that using the Extended Kalman Filter (EKF) algorithm to jointly estimate visual and kinematic data enables dynamic optimization of surgical instrument pose under nonlinear motion and noise interference environments. By establishing a nonlinear state transition model and a covariance propagation mechanism, the system can effectively predict the motion state at the next moment and make corrections based on real-time measurements, thus maintaining continuous and stable tracking results even in the event of temporary visual failure or increased sensor noise. This method not only improves the accuracy and robustness of pose estimation but also enhances the interpretability and real-time performance of the model through block Jacobian and residual analysis mechanisms, providing highly reliable input data for subsequent fuzzy weighted fusion and 3D visualization.

[0072] S4: By using fuzzy logic algorithm and combining the statistical characteristics of visual measurement residuals and kinematic residuals, the device pose estimation results are corrected to obtain the fused pose results.

[0073] Among them, fuzzy logic algorithm is a reasoning method based on fuzzy set theory. It achieves adaptive adjustment of system output by transforming uncertain or fuzzy information into computable membership values.

[0074] Among them, visual measurement residual is the difference between the actual measured value and the predicted value in the visual channel, which is used to reflect the reliability and accuracy of visual data.

[0075] Among them, kinematic residuals are the errors between the predicted pose calculated by the kinematic model of the robot or machine and the actual measured pose, and are used to evaluate the reliability of kinematic data.

[0076] Among them, statistical characteristics usually refer to the mean, variance, or standard deviation of the residuals, which are used to reflect the distribution pattern and stability of the error.

[0077] The fused pose result is the final optimized pose estimate obtained after fuzzy logic weight correction, which combines the advantages of multi-channel data.

[0078] It should be noted that the reliability of each data source is dynamically evaluated using residual statistical characteristics, and the fusion ratio is automatically adjusted according to environmental changes (such as occlusion, lighting fluctuations, or robotic arm tremors), thereby significantly improving the robustness and stability of the system. Through fuzzy inference and defuzzification processes, the system can achieve optimal weighting when multi-source information is inconsistent, reducing the impact of single sensor anomalies on overall pose estimation, making the spatial positioning of surgical instruments more accurate, smooth, and reliable, and ensuring real-time tracking performance in complex intraoperative environments.

[0079] In one possible implementation, S4 specifically includes:

[0080] S401: Obtain the residuals from the device pose estimation results, where the residuals include visual residuals and kinematic residuals.

[0081] S402: Calculate the mean and standard deviation of the residuals.

[0082] S403: Calculate the visual fit index and the kinematic fit index based on the mean and standard deviation, respectively:

[0083]

[0084] in, DoM This represents the matching degree metric. Represents the actual residual covariance matrix. r i Indicates the first i The residual of the second time, i = k - N +1,…., k , k Indicates the time index. N Indicates the length of the sliding window.

[0085] Among them, the matching degree index is an evaluation index calculated based on the statistical characteristics of the residuals, which is used to measure the consistency between the measurement data and the prediction model.

[0086] Specifically, the visual matching index is calculated based on the visual residual, and the kinematic matching index is calculated based on the kinematic residual.

[0087] S404: Convert the visual matching degree index and the kinematic matching degree index into membership degrees belonging to multiple fuzzy sets.

[0088] Fuzzy sets are sets that divide continuous variables into different fuzzy levels, such as zero (Z), small (S), medium (M), large (L), and very large (VL), used to represent the degree of uncertainty.

[0089] S405: Based on the membership degree, determine the relative weights of the visual channel and the relative weights of the kinematic channel by querying the fuzzy rule table.

[0090] Membership degree represents the degree to which a value belongs to a certain fuzzy set, and its value ranges from 0 to 1. The fuzzy rule table is an inference rule based on experience or training, used to determine the output result based on the input membership degree.

[0091] S406: Deblur the relative weights of the visual channel and the relative weights of the kinematic channel, and then normalize the deblurred relative weights of the visual channel and the relative weights of the kinematic channel.

[0092]

[0093] in, weightk This represents the relative weights of the kinematic channels after normalization. weight v This represents the relative weights of the visual channels after normalization. w k This represents the relative weights of the kinematic channels after deblurring. w v This represents the relative weights of the visual channels after deblurring.

[0094] Defuzzification is the process of converting the result of fuzzy reasoning into a computable, precise numerical value.

[0095] S407: The device pose estimation results are weighted and fused based on the normalized relative weights of the visual and kinematic channels to obtain the fused pose result.

[0096]

[0097] in, This indicates the result of the pose fusion. Indicates the posterior state of kinematic data. This represents the posterior state of visual data.

[0098] It should be noted that the reliability of the visual and kinematic channels is first dynamically assessed using the mean and standard deviation of the residuals. Then, through fuzzification, fuzzy inference, and defuzzification processes, intelligent adjustment of the weights of different channels is achieved, thus avoiding the insufficient adaptability of fixed weights in complex surgical scenarios. This mechanism enables the system to autonomously increase the weights of reliable channels and weaken the influence of abnormal channels under conditions such as visual occlusion, changes in lighting, or robotic arm tremors, significantly improving the robustness and accuracy of pose estimation. Simultaneously, through sliding window statistics and weighted fusion, the algorithm achieves temporal smoothing and dynamic compensation, making the tracking of surgical instruments more stable, continuous, and reliable, providing reliable data support for intraoperative navigation and control.

[0099] S5: Based on the fused pose results, the three-dimensional position and orientation of surgical instruments are updated in real time.

[0100] The three-dimensional position refers to the specific coordinate position of the surgical instrument in space, which is usually described in the form of three axes: X, Y, and Z.

[0101] In this context, posture refers to the direction and angle of rotation of surgical instruments in space, usually described by quaternions or Euler angles, to reflect the orientation and rotational state of the instruments.

[0102] It should be noted that by updating the fused pose information in real time, the system can dynamically reflect the spatial motion of surgical instruments during intraoperative procedures, ensuring that the positioning and display results are consistent with the actual operation. This high-frequency real-time refresh mechanism not only significantly improves the response speed and accuracy of the tracking system but also effectively reduces the accumulation of errors caused by data latency or pose drift. Simultaneously, through continuous pose updates, the system can provide accurate input information for subsequent 3D visualization, force feedback control, and safety monitoring, thereby enhancing the accuracy and safety of surgical navigation and ensuring the stability and reliability of the surgeon's operations in complex surgical environments.

[0103] S6: Map the updated 3D position and orientation to the intraoperative digital model for 3D visualization.

[0104] Intraoperative digital model refers to a virtual simulation environment of the surgical scene established through three-dimensional modeling technology, which usually includes the patient's anatomical structure, surgical instrument model and reference coordinate system of key surgical areas.

[0105] Among them, the three-dimensional visualization display presents the updated pose data to the doctor in the form of three-dimensional graphics, realizing the dynamic visualization of the movement, direction and spatial relationship of instruments during the operation.

[0106] It should be noted that by mapping real-time updated pose information onto a digital model, the movement of surgical instruments can be intuitively displayed in a three-dimensional visualization, greatly improving the comprehensibility of intraoperative information and the precision of operation. Surgeons can grasp the spatial position and movement trend of instruments in the surgical field in real time through a visual interface without relying on simple image or text data, thereby improving the safety of surgical procedures and spatial awareness.

[0107] In one possible implementation, S6 specifically includes:

[0108] S601: Transform the updated 3D position and orientation to the world coordinate system.

[0109] The world coordinate system is the system's global reference coordinate system, used to uniformly represent the position and orientation of all objects in the surgical scene, enabling different data sources to perform calculations and displays under the same spatial reference.

[0110] S602: Based on timestamps, synchronize the updated pose information and torque data to generate synchronized pose data and corresponding torque data in the world coordinate system.

[0111] Synchronized pose data is a dataset obtained by synchronizing pose information with corresponding torque data over time. Torque data refers to the force and torque information generated when surgical instruments come into contact with tissues or equipment, and is used to reflect the interaction state and operational intensity.

[0112] S603: Create translation and rotation matrices based on synchronized pose and torque data.

[0113] Among them, the translation matrix and the rotation matrix represent the position translation and attitude rotation of the instrument in space, respectively, and are the basic mathematical representations for constructing three-dimensional transformation relationships.

[0114] S604: Combine the translation matrix and the rotation matrix to obtain the transformation matrix of the intraoperative digital model.

[0115] The transformation matrix is ​​a matrix that combines translation and rotation information, used to accurately describe the overall motion of the model in three-dimensional space.

[0116] S605: Calculate the resultant force of torque data.

[0117] S606: Based on the safety force threshold, the resultant force is mapped to color coding, and the force feedback attribute is output in a visual form.

[0118] It should be noted that those skilled in the art can set the size of the safety force threshold according to actual needs, and this invention does not limit it.

[0119] The color coding includes green, yellow, and red. Green represents safety, yellow represents warning, and red represents exceeding limits.

[0120] Among them, force feedback attributes are intuitive tactile or visual information transmitted to the operator through color and physical quantities.

[0121] S607: Uses the transformation matrix to drive the spatial transformation of the 3D model of the surgical instrument, and updates the synchronous pose data and torque data based on the visualized force feedback properties.

[0122] S608: Packages the updated synchronized pose and torque data and sends them to the client via the WebSocket protocol for 3D visualization.

[0123] WebSocket is a network protocol that supports bidirectional real-time communication and is used for high-speed transmission of pose and visualization data between servers and clients.

[0124] It should be noted that by transforming the 3D pose to a unified world coordinate system and establishing a transformation matrix based on torque data, the system can not only accurately reproduce the dynamic posture of surgical instruments in 3D space but also reflect the force changes during the operation in real time. A color-coding mechanism based on safety force thresholds allows surgeons to intuitively perceive the force state of surgical instruments, thereby avoiding tissue damage caused by excessive thrust or torque. Simultaneously, real-time data transmission via the WebSocket protocol seamlessly displays the 3D model and force feedback information on the client, enhancing intraoperative interactivity and information transparency. Overall, this solution effectively improves the visualization, real-time performance, and safety control capabilities of the surgical navigation system, providing high-precision, low-latency dynamic feedback support for intelligent assisted surgery.

[0125] S7: Based on the updated 3D position and orientation and 3D visualization results, combined with historical tracking trajectories, the surgical instruments are tracked in real time to obtain the real-time pose results of the surgical instruments.

[0126] Historical tracking trajectories are sequences of instrument poses continuously recorded by the system during surgery, used to describe the temporal evolution path of instrument movement. Real-time tracking refers to the system dynamically calculating the current and future positions and orientations of surgical instruments using prediction and update mechanisms while continuously receiving new data, making the tracking process continuous, smooth, and with low latency.

[0127] It is worth noting that by combining the latest pose data with historical trajectory information, the system can achieve continuous, stable, and highly robust real-time tracking in a 3D visualization environment. The historical trajectory not only provides motion trends and inertial information but also compensates for positional errors through predictive algorithms in the event of short-term sensor data loss or occlusion, ensuring the continuity of instrument movement. Simultaneously, combined with the 3D visualization results, it enables intuitive display and dynamic monitoring of the instrument's motion status, allowing surgeons to grasp the instrument's position, orientation, and movement trends in real time within complex surgical fields. This step significantly improves the system's response speed and anti-interference capabilities in dynamic scenarios, providing a solid data foundation for subsequent state prediction, noise adaptive adjustment, and safety control, thereby enhancing the accuracy, safety, and stability of surgical navigation and operation.

[0128] In one possible implementation, S7 specifically includes:

[0129] S701: Based on the 3D visualization results, construct a buffer for historical trajectory pose sequences.

[0130] The historical trajectory pose sequence buffer is a cache structure used to store continuous pose data of surgical instruments within a certain time range, which can provide motion trend information.

[0131] S702: Based on the historical trajectory pose sequence buffer, it predicts the pose of surgical instruments through a kinematic model and outputs the pose prediction.

[0132] Among them, the kinematic model is a mathematical model that describes the motion law of surgical instruments and is used to predict future poses based on historical motion information.

[0133] S703: Based on pose prediction and state transition function, combined with motion smoothness index, the process noise covariance matrix and visual search region are dynamically processed respectively.

[0134] Among them, the motion smoothness index is an indicator that reflects the continuity and stability of the movement of surgical instruments, and is usually calculated from the characteristics of displacement or velocity changes.

[0135] The process noise covariance matrix represents the degree of noise caused by uncertainty during system state changes. The visual search region is the image area used in visual algorithms to detect and track targets.

[0136] The processing includes adjusting the process noise covariance matrix and narrowing the visual search area.

[0137] S704: Based on the dynamic processing results, calculate the residual between the predicted pose and the actual measured pose.

[0138] S705: Weighted fusion of residuals to obtain the real-time pose of surgical instruments.

[0139] It should be noted that by constructing a trajectory buffer and combining it with a kinematic model, the system can learn the movement trend of the instrument and perform forward prediction when visual information is briefly lost or interfered with by noise, maintaining the stability of pose estimation. By dynamically adjusting the process noise covariance matrix and visual search area using a motion smoothness index, filtering parameters can be adaptively adjusted according to the instrument's motion state, thus achieving an intelligent response that is "stable during fast movements and precise during slow movements." Finally, residual weighted fusion further improves the accuracy and smoothness of the predicted pose. This mechanism significantly enhances the system's real-time performance, robustness, and environmental adaptability, enabling surgical instruments to maintain high-precision continuous tracking even under complex conditions such as occlusion, vibration, or rapid movement, providing solid technical support for safe and efficient surgical navigation.

[0140] In one possible implementation, S703 specifically includes:

[0141] S7031: Calculate the displacement difference between consecutive frames based on the historical position vector of the surgical instruments.

[0142]

[0143] in, D (i ) indicates the surgical instrument in the first i -1 time and the i The displacement between time points, i.e., the displacement difference. l ( i -1) indicates i The position vector of the surgical instruments at time -1 W Indicates the length of the sliding window.

[0144] S7032: Calculate the mean displacement of the displacement difference within the sliding window:

[0145]

[0146] in, This represents the mean displacement of the displacement difference.

[0147] S7033: Determine the standard deviation of displacement difference based on the mean displacement.

[0148] S7034: Calculate the motion smoothness index based on the displacement standard deviation:

[0149]

[0150] MSI stands for Motion Smoothness Index. The standard deviation of displacement represents the difference in displacement.

[0151] S7035: Based on pose prediction and state transition function, combined with motion smoothness index, the process noise covariance matrix and visual search region are dynamically processed respectively.

[0152] It should be noted that by calculating the mean and standard deviation of the displacement difference, the system can accurately capture the stability and trend of instrument movement. When the movement is stable (higher MSI), the system automatically reduces process noise and narrows the visual search area to improve estimation accuracy; while during rapid or unstable movement (lower MSI), the system increases the noise covariance and expands the search range to enhance robustness and tracking continuity. This adaptive mechanism allows the system to flexibly adjust parameters at different stages of surgical procedures, balancing response speed and accuracy, significantly improving dynamic tracking capabilities and system stability in complex environments, and providing efficient and reliable technical support for intraoperative intelligent navigation.

[0153] S8: Determine if the surgery is complete. If yes, output the real-time pose of the surgical instruments. Otherwise, return to step S3 and continue tracking until the surgery is complete.

[0154] It should be noted that by establishing a judgment and feedback mechanism, closed-loop control and adaptive looping of the system are achieved throughout the entire surgical process. This step not only ensures that the system can continuously estimate and update its pose, but also automatically terminates data processing and outputs the final results after the surgery, thereby improving the system's operational efficiency and safety. By outputting the final 3D pose information, doctors or the system can perform postoperative review, trajectory analysis, and accuracy verification of the surgical process, providing a reliable basis for clinical records and algorithm optimization. Simultaneously, this loop judgment mechanism prevents performance degradation due to system malfunctions or data accumulation, ensuring the orderly, stable, and efficient execution of the tracking process throughout the surgery, thereby enhancing the practicality and clinical controllability of the intelligent navigation system.

[0155] The beneficial effects of the technical solutions provided by the embodiments of the present invention include at least the following:

[0156] In this embodiment of the invention, by introducing the fusion processing of multi-source data such as vision, inertial measurement unit (IMU) data, and robot kinematics data during surgery, and combining extended Kalman filtering and fuzzy logic adaptive weighting algorithms, the reliability of each channel can be dynamically adjusted even when the accuracy of different sensors fluctuates or partially fails, achieving high-precision and continuous tracking of surgical instrument pose. Through residual statistics and fuzzy inference mechanisms, the influence of visual occlusion, illumination changes, and IMU drift on the results is effectively suppressed, significantly improving the stability and robustness of tracking. Simultaneously, by combining real-time visualization and force feedback mapping, surgeons can intuitively perceive the spatial position and force state of the instruments in the surgical field, improving operational safety and surgical precision.

[0157] Reference manual attached Figure 2 The diagram shows a schematic of the structure of an intraoperative tracking system for a surgical instrument provided by the present invention.

[0158] The present invention also provides an intraoperative tracking system 20 for surgical instruments, applied to the above-mentioned intraoperative tracking method for surgical instruments, comprising:

[0159] Processor 201.

[0160] The memory 202 stores computer-readable instructions, which, when executed by the processor 201, implement the intraoperative tracking method of surgical instruments as described in the method embodiment.

[0161] The intraoperative tracking system 20 for surgical instruments provided by the present invention can perform the above-described intraoperative tracking method for surgical instruments and achieve the same or similar technical effects. To avoid repetition, the present invention will not elaborate further.

[0162] The beneficial effects of the technical solutions provided by the embodiments of the present invention include at least the following:

[0163] In this embodiment of the invention, by introducing the fusion processing of multi-source data such as vision, inertial measurement unit (IMU) data, and robot kinematics data during surgery, and combining extended Kalman filtering and fuzzy logic adaptive weighting algorithms, the reliability of each channel can be dynamically adjusted even when the accuracy of different sensors fluctuates or partially fails, achieving high-precision and continuous tracking of surgical instrument pose. Through residual statistics and fuzzy inference mechanisms, the influence of visual occlusion, illumination changes, and IMU drift on the results is effectively suppressed, significantly improving the stability and robustness of tracking. Simultaneously, by combining real-time visualization and force feedback mapping, surgeons can intuitively perceive the spatial position and force state of the instruments in the surgical field, improving operational safety and surgical precision.

[0164] It should be understood that the processor in the embodiments of the present invention can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0165] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0166] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0167] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0168] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0169] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0170] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0171] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0172] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0173] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0174] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0175] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0176] This invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the intraoperative tracking method for surgical instruments as described in the method embodiment.

[0177] The present invention provides a computer-readable storage medium that can implement the steps and effects of the intraoperative tracking method of the surgical instruments in the above-described method embodiments. To avoid repetition, the present invention will not elaborate further.

[0178] The beneficial effects of the technical solutions provided by the embodiments of the present invention include at least the following:

[0179] In this embodiment of the invention, by introducing the fusion processing of multi-source data such as vision, inertial measurement unit (IMU) data, and robot kinematics data during surgery, and combining extended Kalman filtering and fuzzy logic adaptive weighting algorithms, the reliability of each channel can be dynamically adjusted even when the accuracy of different sensors fluctuates or partially fails, achieving high-precision and continuous tracking of surgical instrument pose. Through residual statistics and fuzzy inference mechanisms, the influence of visual occlusion, illumination changes, and IMU drift on the results is effectively suppressed, significantly improving the stability and robustness of tracking. Simultaneously, by combining real-time visualization and force feedback mapping, surgeons can intuitively perceive the spatial position and force state of the instruments in the surgical field, improving operational safety and surgical precision.

[0180] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0181] The following points need to be explained:

[0182] (1) The accompanying drawings of the embodiments of the present invention only involve the structures involved in the embodiments of the present invention. Other structures can refer to the general design.

[0183] (2) For clarity, the thickness of layers or regions is enlarged or reduced in the drawings used to describe embodiments of the invention, i.e., these drawings are not drawn to scale. It is understood that when an element such as a layer, film, region or substrate is referred to as being “above” or “below” another element, the element may be “directly” located “above” or “below” the other element or there may be intermediate elements.

[0184] (3) Where there is no conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other to obtain new embodiments.

[0185] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An intraoperative tracking system for surgical instruments, characterized in that, The operation steps corresponding to the system include: S1: Collect intraoperative multi-source data and perform preoperative calibration of the surgical instrument tracking system, wherein the intraoperative multi-source data includes visual data and kinematic data; S2: Based on the preoperative calibration results, establish multiple initial transformation relationships between the camera coordinate system, the inertial measurement unit coordinate system, the robot coordinate system, and the surgical instrument coordinate system; S3: Based on the initial transformation relationships, the visual data and the kinematic data are filtered by an extended Kalman filter to obtain the device pose estimation result; S4: By using a fuzzy logic algorithm and combining the statistical characteristics of visual measurement residuals and kinematic residuals, the device pose estimation results are corrected to obtain a fused pose result; S5: Based on the fused pose results, update the three-dimensional position and orientation of the surgical instrument in real time; S6: Map the updated 3D position and orientation to the intraoperative digital model for 3D visualization; S7: Based on the updated 3D position and orientation and 3D visualization results, combined with historical tracking trajectories, the surgical instrument is tracked in real time to obtain the real-time pose result of the surgical instrument; S8: Determine whether the surgery is over; if so, output the real-time pose result of the surgical instruments; otherwise, return to step S3 and continue tracking until the surgery is over.

2. The intraoperative tracking system for surgical instruments according to claim 1, characterized in that, The intraoperative multi-source data includes visual data acquired by the stereo camera, acceleration and angular velocity data output by the inertial measurement unit, and kinematic data acquired by the robot.

3. The intraoperative tracking system for surgical instruments according to claim 1, characterized in that, The preoperative calibration in S1 specifically includes: calibrating the camera's intrinsic and extrinsic parameters manually or semi-automatically, calibrating the robotic arm's base coordinates and the camera's coordinate system, and calibrating the relationship between the instrument's end-effector coordinates and the surgical reference point.

4. The intraoperative tracking system for surgical instruments according to claim 1, characterized in that, Specifically, S2 involves calculating the pose transformation matrices between the camera, the inertial measurement unit, the robot, and the surgical instruments, i.e., the initial transformation relationships, based on the preoperative calibration results.

5. The intraoperative tracking system for surgical instruments according to claim 1, characterized in that, S3 specifically includes: S301: Obtain the posture information of the surgical instrument, wherein the posture information includes position information, quaternion posture, linear velocity, and angular velocity; S302: Establish a nonlinear fusion model based on the linear velocity and the angular velocity; S303: Based on the nonlinear fusion model and the attitude information, determine the state transition function; S304: Calculate the prediction covariance matrix based on the state transition function and the process noise covariance matrix; S305: Based on the predicted covariance matrix, perform EKF updates on the visual data and the kinematic data respectively to obtain the device pose estimation result.

6. The intraoperative tracking system for surgical instruments according to claim 5, characterized in that, S4 specifically includes: S401: Obtain the residual from the device pose estimation result, wherein the residual includes visual residual and kinematic residual; S402: Calculate the mean and standard deviation of the residuals; S403: Calculate the visual matching index and the kinematic matching index based on the mean and the standard deviation, respectively; S404: Convert the visual matching degree index and the kinematic matching degree index into membership degrees belonging to multiple fuzzy sets; S405: Based on the membership degree, determine the relative weights of the visual channel and the relative weights of the kinematic channel by querying the fuzzy rule table; S406: Deblur the relative weights of the visual channel and the relative weights of the kinematic channel, and normalize the deblurred relative weights of the visual channel and the relative weights of the kinematic channel. S407: The device pose estimation results are weighted and fused according to the normalized relative weights of the visual channel and the relative weights of the kinematic channel to obtain the fused pose result.

7. The intraoperative tracking system for surgical instruments according to claim 1, characterized in that, S6 specifically includes: S601: Transform the updated 3D position and orientation to the world coordinate system; S602: Based on the timestamp, the updated pose information and torque data are synchronized to generate synchronized pose data and corresponding torque data in the world coordinate system; S603: Create a translation matrix and a rotation matrix based on the synchronized pose data and the torque data; S604: Combine the translation matrix and the rotation matrix to obtain the transformation matrix of the intraoperative digital model; S605: Calculate the resultant force of the torque data; S606: Based on the safety force threshold, the resultant force is mapped to color coding, and a visualized force feedback attribute is output; S607: Use the transformation matrix to drive the spatial transformation of the three-dimensional model of the surgical instrument, and update the synchronous pose data and the torque data according to the visualized force feedback attributes; S608: Packages the updated synchronized pose and torque data and sends them to the client via the WebSocket protocol for 3D visualization.

8. The intraoperative tracking system for surgical instruments according to claim 1, characterized in that, Specifically, S7 includes: S701: Construct a buffer for historical trajectory pose sequences based on the 3D visualization results; S702: Based on the historical trajectory pose sequence buffer, the pose of the surgical instrument is predicted using a kinematic model, and the pose prediction is output. S703: Based on the pose prediction and state transition function, and combined with the motion smoothness index, the process noise covariance matrix and the visual search region are dynamically processed respectively. S704: Based on the dynamic processing results, calculate the residual between the predicted pose and the actual measured pose; S705: Perform weighted fusion on the residuals to obtain the real-time pose result of the surgical instrument.

9. The intraoperative tracking system for surgical instruments according to claim 8, characterized in that, The S703 specifically includes: S7031: Calculate the displacement difference between consecutive frames based on the historical position vector of the surgical instruments; S7032: Calculate the mean displacement of the displacement difference within the sliding window; S7033: Determine the standard deviation of the displacement difference based on the mean displacement; S7034: Calculate the motion smoothness index based on the displacement standard deviation; S7035: Based on the pose prediction and state transition function, and combined with the motion smoothness index, the process noise covariance matrix and the visual search region are dynamically processed respectively.

10. An intraoperative tracking system for surgical instruments, characterized in that, The system also includes: processor; The memory stores computer-readable instructions, which, when executed by the processor, implement the operation steps corresponding to the intraoperative tracking system of the surgical instruments as described in any one of claims 1 to 9.

Images