Surgical navigation system based on real-time visual analysis of the surgical field and fusion of inertial navigation with artificial intelligence

Abstract

This invention provides a surgical navigation system based on artificial intelligence, integrating real-time surgical field visual analysis and inertial navigation, belonging to the field of computer-aided surgery technology. This artificial intelligence-based real-time surgical field visual analysis and inertial navigation fusion surgical navigation system includes a surgical field acquisition module, comprising an image acquisition unit, an illumination adjustment unit, and a data transmission unit, configured to acquire real-time video stream data of the surgical area and generate a raw surgical field video stream; and an inertial sensing module, comprising an inertial measurement unit, an acquisition unit, a time synchronization unit, and a data transmission unit. By fusing high-definition endoscopic visual data and miniature inertial sensor information in real time, a lightweight convolutional neural network is used to achieve surgical instrument instance segmentation and anatomical landmark detection, significantly improving feature extraction efficiency and reducing computational latency. Combined with sliding window filtering and pre-integration algorithms to process inertial data, noise interference is effectively suppressed and high-precision relative pose changes are calculated.

Description

Technical Field

[0001] This invention relates to the field of computer-assisted surgery technology, specifically to a surgical navigation system based on the fusion of real-time surgical field visual analysis and inertial navigation using artificial intelligence. Background Technology

[0002] Computer-assisted surgery refers to systems that utilize computer technology, medical imaging technology, spatial positioning technology, and virtual reality technology to provide surgeons with real-time guidance, planning, and decision support during surgery. Its core objective is to improve the precision, safety, and efficiency of surgery, reducing over-reliance on the surgeon's personal experience and hand-eye coordination. Among these, a surgical navigation system based on artificial intelligence and real-time surgical field visual analysis combined with inertial navigation is an intelligent navigation platform specifically designed for surgical procedures. This system integrates high-definition endoscopic visual data with data from miniature inertial sensors, using artificial intelligence algorithms for real-time fusion processing. It aims to provide surgeons with high-precision, low-latency spatial positioning and navigation information for surgical instruments. Its applications include: achieving precise real-time tracking of the three-dimensional spatial position of surgical instruments within the patient's body; predicting and compensating for dynamic deformation of soft tissues during surgery to ensure accurate registration between preoperative planning maps and actual intraoperative anatomical locations; planning safe movement paths for instruments and warning of potential collision risks; and ultimately, using augmented reality technology to visually overlay key navigation information onto the real surgical field, assisting surgeons in performing complex surgical procedures more safely and efficiently.

[0003] Current computer-assisted surgical systems rely excessively on optical positioning technology, making them susceptible to interference from blood, smoke, or instrument obstruction in the surgical field. This can lead to lost tracking signals or positioning jumps, forcing surgeons to frequently perform manual recalibration. Preoperative static anatomical atlases cannot adapt to real-time soft tissue deformation during surgery, causing deviations between navigation information and actual anatomical positions. These errors are exacerbated, especially during organ repositioning or traction manipulations, requiring surgeons to rely on mental compensation based on experience. Path planning, based on static anatomical models, lacks responsiveness to dynamic tissue movements during surgery, and the planned path may momentarily fail or lead to dangerous areas. Augmented reality navigation employs fixed transparency and overlay strategies, which can easily lead to information overload during critical operational phases, obscuring the view of important anatomical structures. System parameters are fixed to preoperative training data, making it impossible to adjust feature extraction models and sensor parameters in real-time based on patient specificity or environmental changes during surgery. This can result in recognition failures under conditions of tissue variation or sudden changes in lighting. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a surgical navigation system that integrates real-time surgical field visual analysis and inertial navigation based on artificial intelligence. This system solves the problems of existing computer-assisted surgical systems that rely excessively on optical positioning technology, are easily affected by blood, smoke, or instruments obstructing the surgical field, leading to loss of tracking signals or positioning jumps, forcing doctors to frequently manually recalibrate, and the inability of preoperative static anatomical atlases to adapt to real-time soft tissue deformation during surgery, causing deviations between navigation information and actual anatomical positions. These errors are exacerbated, especially during organ displacement or traction operations, requiring doctors to rely on their experience for intracranial compensation.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a surgical navigation system based on the fusion of real-time surgical field visual analysis and inertial navigation using artificial intelligence, comprising the following modules: The surgical field acquisition module, including an image acquisition unit, an illumination adjustment unit, and a data transmission unit, is configured to acquire real-time video stream data of the surgical area and generate the original surgical field video stream. The inertial sensing module, including an inertial measurement unit, an acquisition unit, a time synchronization unit, and a data transmission unit, is configured to acquire raw acceleration and angular velocity data from the IMU integrated on the surgical instrument and generate raw IMU timing data. The visual feature extraction module, including an instrument segmentation unit, a landmark detection unit, and a feature encoding unit, is configured to process the original surgical field video stream, perform surgical instrument instance segmentation and key anatomical landmark detection, and output preliminary estimates of their 2D positions and poses to generate visual feature information; wherein, the instrument instance segmentation is implemented through a lightweight convolutional neural network, and the segmentation function is expressed as: in, To segment the mask, For the current frame image, For neural network model parameters; The inertial data processing module, including a data filtering unit, a pre-integration calculation unit, and an error estimation unit, is configured to perform sliding window filtering and pre-integration processing on the raw IMU time-series data, calculate the relative pose change (displacement increment, rotation increment) over a short period of time, and generate IMU pre-integration results; wherein, the pre-integration is based on the IMU kinematic model: in, These are the velocity and position increments, respectively. For rotation matrix, For acceleration; The multimodal fusion localization module, including a confidence assessment unit, a feature fusion unit, and a pose regression unit, is configured to fuse the visual feature information and the IMU pre-integration results. It dynamically assesses the confidence of each modality through an attention-based multimodal fusion network and performs fusion at the feature level. It then outputs a high-precision, low-latency six-degree-of-freedom pose estimate through an SE(3) isovariant regression network to generate the device pose. The attention mechanism is expressed as: in, For modal weights, These are visual and IMU features, respectively. Learnable weights; The dynamic atlas registration module, including a deformation field prediction unit, an atlas deformation unit, and a registration optimization unit, is configured to use the preoperatively constructed patient digital anatomical atlas and the real-time acquired instrument pose as sparse reference points. It utilizes a graph neural network to predict the soft tissue deformation field of the current surgical area and uses this deformation field to perform non-rigid deformation on the preoperative atlas to achieve matching with the actual anatomical position during surgery, generating a dynamic anatomical atlas. The deformation field prediction is expressed as follows: in, For deformation field, This is an anatomical atlas diagram. This serves as a reference point for the instrument's pose. The path planning and early warning module includes a path calculation unit, a safety space modeling unit, and a risk warning unit. It is configured to calculate the safe movement path of the instrument to the target based on the dynamic anatomical atlas and real-time instrument pose by rapidly exploring a random tree optimization algorithm, and to establish a dynamic safety space model to calculate the distance information between the instrument and the surrounding structures in real time. When the distance is lower than the safety threshold, an early warning is triggered, and the optimal path suggestion and collision warning signal are generated. The augmented reality navigation module includes a virtual-real registration unit, an information rendering unit, and a display management unit. It is configured to overlay navigation information onto the real surgical field display device based on the instrument pose, dynamic anatomical atlas, optimal path suggestion, and collision warning signal, using deep learning-based perspective rendering technology. It also dynamically manages the display of AR information according to the surgical stage to generate an immersive AR navigation interface. The intraoperative self-learning module, including a self-supervised training unit, a feature library update unit, and an IMU correction unit, is configured to train a feature extraction network to adapt to changes based on the original surgical field video stream and instrument pose, using continuous inter-frame consistency and pose information as supervision signals, and generating an adaptive feature library and IMU correction parameters through an online self-supervised learning framework.

[0006] Preferably, the surgical field acquisition module includes: Video acquisition submodule: Based on the surgical area, it uses a high-definition endoscope or camera to acquire raw image sequences in real time and generate raw surgical field image sequences; Lighting adjustment submodule: Based on ambient lighting conditions, it uses automatic exposure and white balance control algorithms to optimize image quality and generate lighting optimization parameters; Data transmission submodule: Based on the original surgical field image sequence and illumination optimization parameters, it uses a high-speed data compression and transmission protocol to transmit the data and generate a preprocessed surgical field data stream.

[0007] Preferably, the inertial sensing module includes: IMU data acquisition submodule: Uses an inertial measurement unit to acquire raw acceleration and angular velocity data of surgical instruments and generate raw IMU sensor data; Time synchronization submodule: Based on the original IMU sensor data, it uses hardware timestamps and a synchronization protocol to perform time synchronization and generate time-synchronized IMU data; Data transmission submodule: Based on the time-synchronized IMU data, it uses a high-speed, low-latency transmission interface for transmission to generate a pre-processed IMU data stream.

[0008] Preferably, the visual feature extraction module includes: Instrument segmentation submodule: Based on the preprocessed surgical field data stream, a lightweight convolutional neural network segmentation algorithm is used to identify the instrument region and generate an instrument segmentation mask; Landmark Detection Submodule: Based on the preprocessed surgical field data stream and instrument segmentation mask, a key point detection neural network algorithm is used to locate anatomical landmarks and generate two-dimensional coordinates of the anatomical landmarks; Feature encoding submodule: Based on the two-dimensional coordinates of the instrument segmentation mask and anatomical landmarks, a feature encoding network is used to extract the initial pose estimate and generate visual feature information.

[0009] Preferably, the inertial data processing module includes: Data filtering submodule: Based on the preprocessed IMU data stream, a sliding window filtering algorithm is used to remove noise and generate filtered IMU data; Pre-integration calculation submodule: Based on the filtered IMU data, the IMU pre-integration algorithm is used to calculate the relative pose change and generate the IMU pre-integration result; Error estimation submodule: Based on the IMU pre-integration results, the confidence level is estimated using the error state modeling method, and the IMU pre-integration confidence level is generated.

[0010] Preferably, the multimodal fusion positioning module includes: Confidence assessment submodule: Based on the visual feature information and IMU pre-integration results, a multimodal attention mechanism algorithm is used to evaluate modal weights and generate modal confidence weights; Feature fusion submodule: Based on the visual feature information, IMU pre-integration results and modal confidence weights, a feature-level fusion network algorithm is used to fuse them and generate a fused feature vector; Pose regression submodule: Based on the fused feature vector, the pose is calculated using the SE(3) equal-variable pose regression network algorithm to generate a high-precision instrument pose.

[0011] Preferably, the dynamic map registration module includes: Deformation field prediction submodule: Based on the patient's digital anatomical atlas and the high-precision instrument pose, a graph neural network deformation prediction algorithm is used to infer the deformation field and generate the soft tissue deformation field; Atlas Deformation Submodule: Based on the patient's digital anatomical atlas and soft tissue deformation field, a non-rigid deformation interpolation algorithm is used to perform geometric deformation and generate a deformed anatomical atlas. Registration optimization submodule: Based on the deformed anatomical atlas and high-precision instrument pose, an iterative optimization algorithm is used to optimize the registration accuracy and generate a dynamic anatomical atlas.

[0012] Preferably, the path planning early warning module includes: Path calculation submodule: Based on the dynamic anatomical atlas and high-precision instrument pose, a fast exploratory random tree optimization algorithm is used to plan a safe trajectory and generate a safe movement path for the instrument; Safety space modeling submodule: Based on the dynamic anatomical atlas and high-precision instrument pose, a distance field calculation algorithm is used to construct the safety space and generate a dynamic safety space model; Risk warning submodule: Based on the dynamic safety space model, high-precision instrument pose and safe movement path of the instrument, a collision detection algorithm is used to monitor risks and generate optimal path suggestions and collision warning signals.

[0013] Preferably, the augmented reality navigation module includes: Virtual-real registration submodule: Based on the high-precision instrument pose, dynamic anatomical atlas, optimal path suggestion, collision warning signal and preprocessed surgical field data stream, the virtual coordinates are calculated using a 3D to 2D projection transformation algorithm to generate registered virtual information coordinates; Information rendering submodule: Based on the registered virtual information coordinates, virtual elements are generated using deep learning-based perspective rendering technology to generate augmented reality rendering elements; Display Management Submodule: Based on the augmented reality rendering elements, it uses a context-aware information management strategy to control the display output and generate an immersive AR navigation interface.

[0014] Preferably, the intraoperative self-learning module includes: Self-supervised training submodule: Based on the preprocessed surgical field data stream and high-precision instrument pose, an online self-supervised learning algorithm is used to update network parameters and generate adaptive network weight update amounts; Feature library update submodule: Based on the adaptive network weight update amount, it integrates new features using an incremental feature library update strategy to generate an updated adaptive feature library; IMU calibration submodule: Based on the updated adaptive feature library and preprocessed IMU data stream, it calculates the calibration value using the online sensor error estimation method and generates online IMU calibration parameters.

[0015] This invention provides a surgical navigation system based on artificial intelligence, integrating real-time surgical field visual analysis with inertial navigation. It offers the following advantages: This invention integrates high-definition endoscopic visual data with information from miniature inertial sensors in real time. A lightweight convolutional neural network is used to segment surgical instruments and detect anatomical landmarks, significantly improving feature extraction efficiency and reducing computational latency. Sliding window filtering and pre-integration algorithms are combined to process inertial data, effectively suppressing noise interference and calculating high-precision relative pose changes. A multimodal fusion network based on an attention mechanism dynamically evaluates the reliability weights of the visual and inertial modalities, achieving deep coupling at the feature level. Finally, a SE(3) isovariant regression network outputs a six-degree-of-freedom pose, eliminating the localization issues of a single modality in occluded or motion-blurred scenarios. The system employs graph neural networks to predict soft tissue deformation fields, uses intraoperative instrument poses as sparse reference points for non-rigid registration of preoperative maps, and compensates for anatomical displacement caused by tissue deformation in real time. It uses a dynamic safe space model combined with rapid exploration of random tree path planning to generate obstacle avoidance trajectories and monitor collision risks in real time. Based on perspective rendering technology, it aligns virtual navigation elements with the real surgical field geometry, dynamically manages information overlay through a context-aware strategy, and uses continuous inter-frame consistency and pose information as self-supervised signals to optimize feature extraction network parameters online and correct inertial sensor errors, adapting to individual patient differences and changes in the intraoperative environment. Attached Figure Description

[0016] Figure 1 This is a system block diagram of the present invention; Figure 2 This is a schematic diagram of the surgical field acquisition module of the present invention; Figure 3 This is a schematic diagram of the inertial sensing module of the present invention; Figure 4 This is a schematic diagram of the visual feature extraction module of the present invention; Figure 5 This is a schematic diagram of the inertial data processing module of the present invention; Figure 6 This is a schematic diagram of the multimodal fusion positioning module of the present invention; Figure 7 This is a schematic diagram of the dynamic map registration module of the present invention; Figure 8 This is a schematic diagram of the path planning early warning module of the present invention; Figure 9 This is a schematic diagram of the augmented reality navigation module of the present invention; Figure 10 This is a schematic diagram of the intraoperative self-learning module of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example: like Figure 1-10 As shown, this embodiment of the invention provides a surgical navigation system based on artificial intelligence-driven real-time surgical field visual analysis and inertial navigation fusion, comprising the following modules: The surgical field acquisition module, including an image acquisition unit, an illumination adjustment unit, and a data transmission unit, is configured to acquire real-time video stream data of the surgical area and generate the original surgical field video stream. The inertial sensing module, including an inertial measurement unit, an acquisition unit, a time synchronization unit, and a data transmission unit, is configured to acquire raw acceleration and angular velocity data from the IMU integrated on the surgical instrument and generate raw IMU timing data. The visual feature extraction module, including an instrument segmentation unit, a landmark detection unit, and a feature encoding unit, is configured to process the original surgical field video stream, perform surgical instrument instance segmentation and key anatomical landmark detection, and output preliminary estimates of their 2D positions and poses to generate visual feature information; wherein, the instrument instance segmentation is implemented through a lightweight convolutional neural network, and the segmentation function is expressed as: in, To segment the mask, For the current frame image, For neural network model parameters; The inertial data processing module, including a data filtering unit, a pre-integration calculation unit, and an error estimation unit, is configured to perform sliding window filtering and pre-integration processing on the raw IMU time-series data, calculate the relative pose change (displacement increment, rotation increment) over a short period of time, and generate IMU pre-integration results; wherein, the pre-integration is based on the IMU kinematic model: in, These are the velocity and position increments, respectively. For rotation matrix, For acceleration; The multimodal fusion localization module, including a confidence assessment unit, a feature fusion unit, and a pose regression unit, is configured to fuse the visual feature information and the IMU pre-integration results. It dynamically assesses the confidence of each modality through an attention-based multimodal fusion network and performs fusion at the feature level. It then outputs a high-precision, low-latency six-degree-of-freedom pose estimate through an SE(3) isovariant regression network to generate the device pose. The attention mechanism is expressed as: in, For modal weights, These are visual and IMU features, respectively. Learnable weights; The dynamic atlas registration module, including a deformation field prediction unit, an atlas deformation unit, and a registration optimization unit, is configured to use the preoperatively constructed patient digital anatomical atlas and the real-time acquired instrument pose as sparse reference points. It utilizes a graph neural network to predict the soft tissue deformation field of the current surgical area and uses this deformation field to perform non-rigid deformation on the preoperative atlas to achieve matching with the actual anatomical position during surgery, generating a dynamic anatomical atlas. The deformation field prediction is expressed as follows: in, For deformation field, This is an anatomical atlas diagram. This serves as a reference point for the instrument's pose. The path planning and early warning module includes a path calculation unit, a safety space modeling unit, and a risk warning unit. It is configured to calculate the safe movement path of the instrument to the target based on the dynamic anatomical atlas and real-time instrument pose by rapidly exploring a random tree optimization algorithm, and to establish a dynamic safety space model to calculate the distance information between the instrument and the surrounding structures in real time. When the distance is lower than the safety threshold, an early warning is triggered, and the optimal path suggestion and collision warning signal are generated. The augmented reality navigation module includes a virtual-real registration unit, an information rendering unit, and a display management unit. It is configured to overlay navigation information onto the real surgical field display device based on the instrument pose, dynamic anatomical atlas, optimal path suggestion, and collision warning signal, using deep learning-based perspective rendering technology. It also dynamically manages the display of AR information according to the surgical stage to generate an immersive AR navigation interface. The intraoperative self-learning module, including a self-supervised training unit, a feature library update unit, and an IMU correction unit, is configured to train a feature extraction network to adapt to changes based on the original surgical field video stream and instrument pose, using continuous inter-frame consistency and pose information as supervision signals, and generating an adaptive feature library and IMU correction parameters through an online self-supervised learning framework.

[0019] The surgical field acquisition module includes: Video acquisition submodule: Based on the surgical area, it uses a high-definition endoscope or camera to acquire raw image sequences in real time and generate raw surgical field image sequences; The video acquisition submodule is based on the surgical area and uses a high-definition endoscope or camera to acquire raw image sequences in real time. The endoscope captures RGB images with a resolution of 1920×1080 pixels at a rate of 30 frames per second. For example, in laparoscopic cholecystectomy, when the lens is 15 cm away from the surgical field, it acquires images of tissue surface texture and instrument movement. Each frame of the image is converted into a raw digital signal by photoelectric conversion through a CMOS sensor, and then quantized to 8-bit depth by an analog-to-digital converter to generate a continuous image sequence containing timestamps.

[0020] Lighting adjustment submodule: Based on ambient lighting conditions, it uses automatic exposure and white balance control algorithms to optimize image quality and generate lighting optimization parameters; The lighting adjustment submodule optimizes image quality based on ambient lighting conditions using an automatic exposure and white balance control algorithm. The algorithm first calculates the global average brightness value of the current frame image. (Range 0-255), if If the value is below 50, it is considered underexposed. The formula increases the exposure time; for example, with an initial brightness of 30, the exposure time is increased from 10 milliseconds to 23 milliseconds. Simultaneously, the color temperature deviation is calculated based on the CIE XYZ color space. When the white balance coefficient R / G < 0.9, it is adjusted accordingly. Adjusting the red channel gain, the final output includes illumination control commands containing exposure time parameters and a white balance matrix.

[0021] Data transmission submodule: Based on the original surgical field image sequence and illumination optimization parameters, it uses a high-speed data compression and transmission protocol to transmit the data and generate a preprocessed surgical field data stream.

[0022] The data transmission submodule is based on the original surgical field image sequence and illumination optimization parameters. It uses a high-speed data compression and transmission protocol to transmit the data. The single frame of uncompressed image (1920×1080×3 bytes ≈ 6MB) is compressed using H.264 encoding. When the quantization parameter QP=28 is set, the compression ratio can reach 10:1, generating a 600KB frame data packet. At the same time, the exposure time and white balance coefficient are encapsulated into 32 bytes of metadata and transmitted via UDP protocol at a bandwidth of 100Mbps. The data packet sequence number is set to be bound to the timestamp to ensure that the transmission delay is less than 10 milliseconds, generating a standardized data stream containing compressed images and adjustment parameters.

[0023] The inertial sensing module includes: IMU data acquisition submodule: Uses an inertial measurement unit to acquire raw acceleration and angular velocity data of surgical instruments and generate raw IMU sensor data; The IMU data acquisition submodule employs a six-axis MEMS inertial measurement unit with an accelerometer range covering ±8g and a gyroscope range of ±2000 dps. Embedded in the surgical instrument handle, the sensor captures raw analog signals of triaxial linear acceleration and triaxial angular velocity at a sampling frequency of 1000Hz during tissue cutting operations. These signals are then converted into digital values ​​by an onboard 24-bit ADC, and the accelerometer output value is displayed. (Corresponding voltage value 1.8V), gyroscope output value (Corresponding voltage value 2.1V), each frame of data includes a timestamp and acceleration components. and angular velocity components It is encapsulated as a 32-byte data packet.

[0024] Time synchronization submodule: Based on the original IMU sensor data, it uses hardware timestamps and a synchronization protocol to perform time synchronization and generate time-synchronized IMU data; The time synchronization submodule is based on an FPGA hardware clock source (precision). Add a 64-bit nanosecond-level timestamp to the raw IMU sensor data when a master-slave clock deviation exceeding [a certain threshold] is detected. At the same time, the IEEE 1588v2 precise time protocol is used for clock correction, and the clock offset is calculated. 2. For example, if the master device's timestamp is 1620000000ns and the slave device's is 1620000100ns, the slave device's time will be corrected to 1620000050ns. After synchronization, the inter-frame time jitter will be controlled within... Inside.

[0025] Data transmission submodule: Based on the time-synchronized IMU data, it uses a high-speed, low-latency transmission interface for transmission to generate a pre-processed IMU data stream.

[0026] The data transmission submodule uses a differential coding compression algorithm to construct a data block from 10 consecutive frames of time-synchronized IMU data and calculates the changes between adjacent frames. ,when Run-length encoding is enabled, increasing the compression rate to 70%. Data is transmitted via Gigabit Ethernet interface using UDP protocol. A packet priority flag is set (surgical instrument motion data priority = 7). When the network latency exceeds 20ms, a redundant packet dropping mechanism is triggered, generating a data stream with continuous timestamps.

[0027] The visual feature extraction module includes: Instrument segmentation submodule: Based on the preprocessed surgical field data stream, a lightweight convolutional neural network segmentation algorithm is used to identify the instrument region and generate an instrument segmentation mask; The instrument segmentation submodule loads a pre-trained lightweight convolutional neural network model, which contains 12 convolutional layers and 3 deconvolutional layers. The input is a 512×512 pixel RGB image slice. The feature map is extracted through a 3×3 convolutional kernel. In the laparoscopic cholecystectomy scenario, the model performs pixel-level classification of the electrosurgical instrument region. Pixels with a probability value greater than 0.85 in the output probability map are identified as instrument regions. After morphological closure operation to fill the holes, a binary mask is generated. The white area in the mask corresponds to the set of instrument pixel coordinates.

[0028] Landmark Detection Submodule: Based on the preprocessed surgical field data stream and instrument segmentation mask, a key point detection neural network algorithm is used to locate anatomical landmarks and generate two-dimensional coordinates of the anatomical landmarks; The landmark detection submodule receives the current frame image and instrument segmentation mask from the preprocessed surgical field data stream. Based on the mask bounding box, it expands outward by 20% to form a region of interest. Within this region, the HRNet network is used to predict the anatomical landmark heatmap. The peak coordinates of the heatmap are the landmark locations. For example, in liver surgery, when detecting the junction of the cystic duct and the common hepatic duct, if the maximum response value of the heatmap exceeds 0.75 and the second highest peak ratio is less than 0.3, the coordinates are determined to be valid detection points, and the [x,y] coordinate pairs in the pixel coordinate system are output.

[0029] Feature encoding submodule: Based on the two-dimensional coordinates of the instrument segmentation mask and anatomical landmarks, a feature encoding network is used to extract the initial pose estimate and generate visual feature information.

[0030] The feature encoding submodule inputs the two-dimensional coordinates of the instrument segmentation mask and anatomical landmarks into the spatial transformation network, first calculating the coordinates of the center point of the minimum bounding rectangle of the mask. Establish a local coordinate system around this point and convert the coordinates of the marker point into relative vectors. The instrument's main axis orientation angle is regressed through a two-layer fully connected network. By combining the mask area S (pixel count) to estimate the instrument depth scaling factor, a model containing the location is generated. An attitude parameter vector consisting of orientation angle θ and scale factor s.

[0031] The inertial data processing module includes: Data filtering submodule: Based on the preprocessed IMU data stream, a sliding window filtering algorithm is used to remove noise and generate filtered IMU data; The data filtering submodule receives continuous acceleration and angular velocity sequences from the preprocessed IMU data stream, sets the sliding window length to 10 frames, sorts the data within the window in chronological order, and calculates the median acceleration within the window. When the current frame acceleration value and When the difference exceeds 0.2g, use replace A first-order low-pass filter is applied to the gyroscope data, with a cutoff frequency set to 5Hz and a filter coefficient α=0.2. The calculation... For example, if the initial angular velocity is 200 dps, the output after filtering is 185 dps.

[0032] Pre-integration calculation submodule: Based on the filtered IMU data, the IMU pre-integration algorithm is used to calculate the relative pose change and generate the IMU pre-integration result; The pre-integration calculation submodule calculates the data based on the filtered IMU data within the time interval. Integrating the internal acceleration and angular velocity, we establish the rotation matrix increment. ,in The skew-symmetric matrix is ​​the angular velocity vector, and the position increment is... Speed ​​increment Taking the movement of the instrument from point A to point B as an example, Second, Time calculation radian, hour rice.

[0033] Error estimation submodule: Based on the IMU pre-integration results, the confidence level is estimated using the error state modeling method, and the IMU pre-integration confidence level is generated.

[0034] The error estimation submodule calculates the covariance matrix Σ of the pre-integration result and sets the acceleration noise variance. Angular velocity noise variance Covariance elements ,when A value less than 0.0001 is considered a high confidence level, for example... seconds The corresponding confidence level is 0.95.

[0035] The multimodal fusion positioning module includes: Confidence assessment submodule: Based on the visual feature information and IMU pre-integration results, a multimodal attention mechanism algorithm is used to evaluate modal weights and generate modal confidence weights; The confidence assessment submodule receives the initial estimation parameters of the instrument's attitude from the visual feature information and the displacement increment of the IMU pre-integration result, and calculates the visual attitude angle. With IMU attitude angle The difference ,when When the degree is less than 5 degrees, the confidence weight of the visual modality is set to 0.7; otherwise, it is reduced to 0.3. Simultaneously, the IMU pre-integration confidence value is detected. If this value is greater than 0.8, the weight of the inertial modality is set to 0.6, using a weighted formula. Calculate the fusion weight coefficients, where ,For example When the IMU confidence level is 0.9, the output visual weight is 0.65 and the inertial weight is 0.35.

[0036] Feature fusion submodule: Based on the visual feature information, IMU pre-integration results and modal confidence weights, a feature-level fusion network algorithm is used to fuse them and generate a fused feature vector; The feature fusion submodule combines visual feature vectors (Dimension 128) and IMU pre-integral feature vector (Dimension 64) concatenated into a 192-dimensional vector, and then weighted according to modal confidence. Multiply by the weighting coefficient α, for Multiply by a coefficient β, project the weighted features onto a 256-dimensional fusion space through a fully connected layer, and calculate the fusion vector. The weight matrix With dimensions of 256×192, in an electrosurgical translation operation scenario, when α=0.6... The proportion of features has increased to 60%.

[0037] Pose regression submodule: Based on the fused feature vector, the pose is calculated using the SE(3) equal-variable pose regression network algorithm to generate a high-precision instrument pose.

[0038] The pose regression submodule inputs the 256-dimensional fused feature vector into the SE(3) equivariant network, which contains four equivariant convolutional layers. The output of each layer satisfies rotation and translation equivariance. The pose parameters of the 6 degrees of freedom are regressed through the Lie algebra space mapping layer. The rotation component is represented by quaternions and converted to Euler angles through orthogonalization. For example, the channel with the largest input feature activation value corresponds to an output Δz = 0.2 mm. The final output is the three-dimensional coordinates and attitude angle of the instrument tip in the global coordinate system.

[0039] The dynamic map registration module includes: Deformation field prediction submodule: Based on the patient's digital anatomical atlas and the high-precision instrument pose, a graph neural network deformation prediction algorithm is used to infer the deformation field and generate the soft tissue deformation field; The deformation field prediction submodule receives a triangular mesh model from the patient's digital anatomical atlas and uses the high-precision instrument pose as the input node features of the graph neural network. In a liver surgery scenario, when the instrument tip contacts the surface of the right lobe of the liver, a radius region is established with the contact point as the center. The Euclidean distance d from each mesh vertex within the region to the instrument point is calculated, and distance weights are set. Where σ is the region radius parameter, displacement information is propagated through a three-layer graph convolutional network, and the displacement vector of each vertex is output. This generates soft tissue deformation field data.

[0040] Atlas Deformation Submodule: Based on the patient's digital anatomical atlas and soft tissue deformation field, a non-rigid deformation interpolation algorithm is used to perform geometric deformation and generate a deformed anatomical atlas. The atlas deformation submodule loads the preoperatively constructed 3D liver model and applies the vertex displacement vectors from the deformation field data to the corresponding mesh vertices. For internal vertices not directly subjected to force, a thin-plate spline interpolation algorithm is used to calculate the displacement, with the interpolation function based on the control point set. (The point of action of the instrument) and its displacement vector Calculate the displacement of any vertex v. , where φ is the radial basis function, and the displacement of 0.5 mm is obtained by interpolation at the internal vertex 30 mm away from the instrument point, generating the anatomical model with updated geometry.

[0041] Registration optimization submodule: Based on the deformed anatomical atlas and high-precision instrument pose, an iterative optimization algorithm is used to optimize the registration accuracy and generate a dynamic anatomical atlas.

[0042] The registration optimization submodule calculates the three-dimensional distance between the vascular branch points in the deformed anatomical atlas and the real-time instrument tip. When the distance deviation exceeds a threshold, the deformation field parameters are updated using the gradient descent method, and a learning rate is set. iteration step size The loss function For example, when the initial deviation is 2 mm, it is reduced to 0.3 mm after 5 iterations, and the output dynamic anatomical atlas with a registration error of less than 0.5 mm is obtained.

[0043] The path planning early warning module includes: Path calculation submodule: Based on the dynamic anatomical atlas and high-precision instrument pose, a fast exploratory random tree optimization algorithm is used to plan a safe trajectory and generate a safe movement path for the instrument; The path calculation submodule receives the voxelized mesh model of the dynamic anatomical atlas and the current pose coordinates of the instrument. Starting from the instrument tip and ending at the target position, it generates random sampling points by evenly scattering points in three-dimensional space. It filters reachable nodes through collision detection and gradually expands to quickly explore the random tree structure. When a new node is less than a threshold away from the target point, it backtracks to generate a path. In the liver surgery scenario, it plans the instrument's travel route to avoid the branches of the portal vein. The spacing between path points is set to a multiple of the instrument diameter. Finally, it outputs a safe trajectory sequence composed of a series of three-dimensional coordinate points.

[0044] Safety space modeling submodule: Based on the dynamic anatomical atlas and high-precision instrument pose, a distance field calculation algorithm is used to construct the safety space and generate a dynamic safety space model; The safety space modeling submodule calculates the minimum distance from the instrument tip to the surface of each anatomical structure based on triangular patch data from the anatomical atlas. It uses ray projection to detect the nearest tissue surface along the instrument axis and sets a safety distance threshold as a multiple of the instrument radius. When the detected distance is less than the threshold, it is marked as a danger zone. In cholecystectomy, the distance value is calculated in real time when the instrument approaches the common bile duct. If the distance value is less than the safety threshold, a spherical danger zone model is constructed, and a dynamically updated safety space voxel map is output.

[0045] Risk warning submodule: Based on the dynamic safety space model, high-precision instrument pose and safe movement path of the instrument, a collision detection algorithm is used to monitor risks and generate optimal path suggestions and collision warning signals.

[0046] The risk warning submodule calculates the distance difference Δd between the current position of the instrument and the safe space model in real time. When Δd is less than the warning threshold, a yellow warning signal is triggered; when it is less than the collision threshold, a red warning signal is triggered. At the same time, it detects the deviation angle θ between the actual movement path of the instrument and the planned path. If θ is greater than the allowable angle, a path correction vector is generated. In neurosurgery, when the instrument deviates from the planned path and approaches the motor functional area, a corrected path suggestion containing an obstacle avoidance direction arrow and an audible and visual alarm signal are output.

[0047] The augmented reality navigation module includes: Virtual-real registration submodule: Based on the high-precision instrument pose, dynamic anatomical atlas, optimal path suggestion, collision warning signal and preprocessed surgical field data stream, the virtual coordinates are calculated using a 3D to 2D projection transformation algorithm to generate registered virtual information coordinates; The virtual-real registration submodule receives the 3D coordinates of the high-precision instrument pose and the vascular structure model from the dynamic anatomical atlas. It establishes a transformation matrix from the 3D model coordinate system to the 2D endoscopic image coordinate system. The matrix parameters are obtained from the endoscopic optical parameter calibration. In the laparoscopic surgical scenario, the coordinates [X,Y,Z] of the instrument tip are projected and transformed to calculate its coordinates [u,v] on the image plane, using the following formula: , ,in , For focal length parameters, Using the main point coordinates, for example, when the instrument is at a spatial position of (10,20,100) millimeters, it is projected onto the image coordinates of (320,240) pixels, and the two-dimensional projection coordinate data of all virtual elements is output.

[0048] Information rendering submodule: Based on the registered virtual information coordinates, virtual elements are generated using deep learning-based perspective rendering technology to generate augmented reality rendering elements; The information rendering submodule, based on the two-dimensional position data in the registered virtual information coordinates, uses a convolutional neural network to generate a semi-transparent rendering layer with a perspective effect for virtual elements such as the instrument movement path line and the highlighted box of the danger zone. The network input is the surgical field image and the coordinates of the virtual elements. The U-Net structure predicts the transparency α and color increment Δc of each pixel, and the transparency is calculated using the Sigmoid function. Where d is the distance from the pixel to the boundary of the virtual element, and k is the attenuation coefficient. When rendering the instrument path line, k=0.1 is set so that the transparency of the path line drops to 0.3 at a distance of 10 pixels from the boundary, generating an enhanced image layer that blends with the light and shadow of the real tissue.

[0049] Display Management Submodule: Based on the augmented reality rendering elements, it uses a context-aware information management strategy to control the display output and generate an immersive AR navigation interface.

[0050] The display management submodule dynamically controls the display priority of augmented reality rendering elements based on surgical stage state variables (input by the surgeon's interface). It sets the display of instrument paths and danger zones during the anatomical separation stage, hides the path and only displays vascular structures during the hemostasis stage, and displays suture point markers during the suturing stage. At the same time, it automatically adjusts the transparency of information according to the speed of instrument movement. When the instrument speed is greater than a threshold, the transparency of non-critical information is reduced. For example, when the speed is greater than 5 mm / s, the transparency of anatomical annotations is set to 0.2. The output is an AR overlay image stream optimized according to the surgical context.

[0051] The intraoperative self-learning module includes: Self-supervised training submodule: Based on the preprocessed surgical field data stream and high-precision instrument pose, an online self-supervised learning algorithm is used to update network parameters and generate adaptive network weight update amounts; The self-supervised training submodule receives preprocessed surgical field data streams and corresponding high-precision instrument pose data from consecutive frames. It calculates the Intersection over Union (IoU) of the instrument segmentation masks between adjacent frames. When the IoU value is greater than a set threshold, the pose change is used as a supervision signal to construct a loss function. The loss function is in the form of... ,in For the current frame pose, The IMU pre-integrated displacement is used to update the visual feature extraction network weights using stochastic gradient descent. The learning rate is set to an exponential decay form of the initial value. For example, when the initial learning rate is 0.001, it decays to 0.0005 every 1000 frames. The network weight update parameter file is then output.

[0052] Feature library update submodule: Based on the adaptive network weight update amount, it integrates new features using an incremental feature library update strategy to generate an updated adaptive feature library; The feature library update submodule loads the existing 128-dimensional feature vector set from the feature library, arranges the new feature vectors generated by self-supervised training in chronological order, calculates the cosine similarity between the new feature and the features in the library, and adds the new feature as an independent vector to the feature library when the maximum similarity is below a threshold; when the similarity is above the threshold, it is added to the feature library according to a weighted formula. Update the old feature vectors, where α is the retention coefficient. For example, when the similarity is 0.8, α is 0.7. At the same time, delete old features in the feature library that have exceeded the time window, and output the updated feature vector set.

[0053] IMU calibration submodule: Based on the updated adaptive feature library and preprocessed IMU data stream, it calculates the calibration value using the online sensor error estimation method and generates online IMU calibration parameters.

[0054] The IMU calibration submodule estimates the accelerometer bias in the preprocessed IMU data stream based on the device motion pattern features in the updated adaptive feature library. It then uses the least squares method to fit the acceleration measurements during the static interval (when the device is stationary). The bias calculation model is as follows: Where g is the gravity vector, and the kinematic constraint equations are also applied. Calculate the dynamic compensation value by sampling 10 sets of data during the instrument's circular motion period, taking the mean value, and outputting a correction parameter matrix containing triaxial zero bias compensation values.

[0055] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A surgical navigation system based on artificial intelligence-driven real-time surgical field visual analysis and inertial navigation fusion, characterized in that, Includes the following modules: The surgical field acquisition module includes an image acquisition unit, an illumination adjustment unit, and a data transmission unit, configured to acquire real-time video stream data of the surgical area and generate the original surgical field video stream; The inertial sensing module, including an inertial measurement unit, an acquisition unit, a time synchronization unit, and a data transmission unit, is configured to acquire raw acceleration and angular velocity data from the IMU integrated on the surgical instrument and generate raw IMU timing data. The visual feature extraction module includes an instrument segmentation unit, a landmark detection unit, and a feature encoding unit. It is configured to process the original surgical field video stream, perform surgical instrument instance segmentation and key anatomical landmark detection, and output preliminary estimates of their 2D positions and poses to generate visual feature information. The instrument instance segmentation is implemented using a lightweight convolutional neural network, and the segmentation function is expressed as: in, To segment the mask, For the current frame image, For neural network model parameters; The inertial data processing module, including a data filtering unit, a pre-integration calculation unit, and an error estimation unit, is configured to perform sliding window filtering and pre-integration processing on the raw IMU time-series data, calculate the relative pose change (displacement increment, rotation increment) over a short period of time, and generate IMU pre-integration results; wherein, the pre-integration is based on the IMU kinematic model: in, These are the velocity and position increments, respectively. For rotation matrix, For acceleration; The multimodal fusion localization module includes a confidence assessment unit, a feature fusion unit, and a pose regression unit. It is configured to fuse the visual feature information and the IMU pre-integration results, dynamically assess the confidence of each modality through an attention-based multimodal fusion network, fuse them at the feature level, and output a high-precision, low-latency six-degree-of-freedom pose estimate through an SE(3) variable regression network to generate the device pose. The attention mechanism is expressed as: in, For modal weights, These are visual and IMU features, respectively. Learnable weights; The dynamic atlas registration module includes a deformation field prediction unit, an atlas deformation unit, and a registration optimization unit. It is configured to use a graph neural network to predict the soft tissue deformation field of the current surgical area based on the preoperatively constructed patient digital anatomical atlas and the real-time acquired instrument pose as sparse reference points. This module then uses a graph neural network to predict the soft tissue deformation field of the current surgical area and applies this deformation field to the preoperative atlas using non-rigid deformation to achieve matching with the actual anatomical position during surgery, generating a dynamic anatomical atlas. The deformation field prediction is expressed as follows: in, For deformation field, This is an anatomical atlas diagram. This serves as a reference point for the instrument's pose. The path planning and early warning module includes a path calculation unit, a safety space modeling unit, and a risk warning unit. It is configured to calculate the safe movement path of the instrument to the target based on the dynamic anatomical atlas and real-time instrument pose by rapidly exploring a random tree optimization algorithm, and to establish a dynamic safety space model to calculate the distance information between the instrument and the surrounding structures in real time. When the distance is lower than the safety threshold, an early warning is triggered, and the optimal path suggestion and collision warning signal are generated. The augmented reality navigation module includes a virtual-real registration unit, an information rendering unit, and a display management unit. It is configured to overlay navigation information onto the real surgical field display device based on the instrument pose, dynamic anatomical atlas, optimal path suggestion, and collision warning signal, using deep learning-based perspective rendering technology. It also dynamically manages the display of AR information according to the surgical stage to generate an immersive AR navigation interface. The intraoperative self-learning module includes a self-supervised training unit, a feature library update unit, and an IMU correction unit. It is configured to train a feature extraction network to adapt to changes based on the original surgical field video stream and instrument pose, using continuous inter-frame consistency and pose information as supervision signals, and generating an adaptive feature library and IMU correction parameters through an online self-supervised learning framework.

2. The surgical navigation system based on artificial intelligence and fusion of real-time surgical field visual analysis and inertial navigation as described in claim 1, characterized in that: The surgical field acquisition module includes: Video acquisition submodule: Based on the surgical area, it uses a high-definition endoscope or camera to acquire raw image sequences in real time and generate raw surgical field image sequences; Lighting adjustment submodule: Based on ambient lighting conditions, it uses automatic exposure and white balance control algorithms to optimize image quality and generate lighting optimization parameters; Data transmission submodule: Based on the original surgical field image sequence and illumination optimization parameters, it uses a high-speed data compression and transmission protocol to transmit the data and generate a preprocessed surgical field data stream.

3. The surgical navigation system based on artificial intelligence and fusion of real-time surgical field visual analysis and inertial navigation as described in claim 1, characterized in that: The inertial sensing module includes: IMU data acquisition submodule: Uses an inertial measurement unit to acquire raw acceleration and angular velocity data of surgical instruments and generate raw IMU sensor data; Time synchronization submodule: Based on the original IMU sensor data, it uses hardware timestamps and a synchronization protocol to perform time synchronization and generate time-synchronized IMU data; Data transmission submodule: Based on the time-synchronized IMU data, it uses a high-speed, low-latency transmission interface for transmission to generate a pre-processed IMU data stream.

4. The surgical navigation system based on artificial intelligence and fusion of real-time surgical field visual analysis and inertial navigation as described in claim 1, characterized in that: The visual feature extraction module includes: Instrument segmentation submodule: Based on the preprocessed surgical field data stream, a lightweight convolutional neural network segmentation algorithm is used to identify the instrument region and generate an instrument segmentation mask; Landmark Detection Submodule: Based on the preprocessed surgical field data stream and instrument segmentation mask, a key point detection neural network algorithm is used to locate anatomical landmarks and generate two-dimensional coordinates of the anatomical landmarks; Feature encoding submodule: Based on the two-dimensional coordinates of the instrument segmentation mask and anatomical landmarks, a feature encoding network is used to extract the initial pose estimate and generate visual feature information.

5. The surgical navigation system based on artificial intelligence and fusion of real-time surgical field visual analysis and inertial navigation according to claim 1, characterized in that: The inertial data processing module includes: Data filtering submodule: Based on the preprocessed IMU data stream, a sliding window filtering algorithm is used to remove noise and generate filtered IMU data; Pre-integration calculation submodule: Based on the filtered IMU data, the IMU pre-integration algorithm is used to calculate the relative pose change and generate the IMU pre-integration result; Error estimation submodule: Based on the IMU pre-integration results, the confidence level is estimated using the error state modeling method, and the IMU pre-integration confidence level is generated.

6. The surgical navigation system based on artificial intelligence and fusion of real-time surgical field visual analysis and inertial navigation according to claim 1, characterized in that: The multimodal fusion positioning module includes: Confidence assessment submodule: Based on the visual feature information and IMU pre-integration results, a multimodal attention mechanism algorithm is used to evaluate modal weights and generate modal confidence weights; Feature fusion submodule: Based on the visual feature information, IMU pre-integration results and modal confidence weights, a feature-level fusion network algorithm is used to fuse them and generate a fused feature vector; Pose regression submodule: Based on the fused feature vector, the pose is calculated using the SE(3) variable pose regression network algorithm to generate a high-precision instrument pose.

7. The surgical navigation system based on artificial intelligence and fusion of real-time surgical field visual analysis and inertial navigation according to claim 1, characterized in that: The dynamic map registration module includes: Deformation field prediction submodule: Based on the patient's digital anatomical atlas and the high-precision instrument pose, a graph neural network deformation prediction algorithm is used to infer the deformation field and generate the soft tissue deformation field; Atlas Deformation Submodule: Based on the patient's digital anatomical atlas and soft tissue deformation field, a non-rigid deformation interpolation algorithm is used to perform geometric deformation and generate a deformed anatomical atlas. Registration optimization submodule: Based on the deformed anatomical atlas and high-precision instrument pose, an iterative optimization algorithm is used to optimize the registration accuracy and generate a dynamic anatomical atlas.

8. The surgical navigation system based on artificial intelligence and fusion of real-time surgical field visual analysis and inertial navigation according to claim 1, characterized in that: The path planning early warning module includes: Path calculation submodule: Based on the dynamic anatomical atlas and high-precision instrument pose, a fast exploratory random tree optimization algorithm is used to plan a safe trajectory and generate a safe movement path for the instrument; Safety space modeling submodule: Based on the dynamic anatomical atlas and high-precision instrument pose, a distance field calculation algorithm is used to construct the safety space and generate a dynamic safety space model; Risk warning submodule: Based on the dynamic safety space model, high-precision instrument pose and safe movement path of the instrument, a collision detection algorithm is used to monitor risks and generate optimal path suggestions and collision warning signals.

9. The surgical navigation system based on artificial intelligence and fusion of real-time surgical field visual analysis and inertial navigation according to claim 1, characterized in that: The augmented reality navigation module includes: Virtual-real registration submodule: Based on the high-precision instrument pose, dynamic anatomical atlas, optimal path suggestion, collision warning signal and preprocessed surgical field data stream, the virtual coordinates are calculated using a 3D to 2D projection transformation algorithm to generate registered virtual information coordinates; Information rendering submodule: Based on the registered virtual information coordinates, virtual elements are generated using deep learning-based perspective rendering technology to generate augmented reality rendering elements; Display Management Submodule: Based on the augmented reality rendering elements, it uses a context-aware information management strategy to control the display output and generate an immersive AR navigation interface.

10. The surgical navigation system based on artificial intelligence and fusion of real-time surgical field visual analysis and inertial navigation according to claim 1, characterized in that: The intraoperative self-learning module includes: Self-supervised training submodule: Based on the preprocessed surgical field data stream and high-precision instrument pose, an online self-supervised learning algorithm is used to update network parameters and generate adaptive network weight update amounts; Feature library update submodule: Based on the adaptive network weight update amount, it integrates new features using an incremental feature library update strategy to generate an updated adaptive feature library; IMU calibration submodule: Based on the updated adaptive feature library and preprocessed IMU data stream, it calculates the calibration value using the online sensor error estimation method and generates online IMU calibration parameters.

Images