Visual and tactile intelligent operation navigation system

Abstract

The application discloses a visual and sensible intelligent operation navigation system, which comprises an auxiliary diagnosis module, a multi-scale lesion information presentation module and an intraoperative force protection module; the auxiliary diagnosis module realizes automatic segmentation and automatic reconstruction of a lesion, the automatic segmentation realizes fine delineation of a lesion area, the automatic reconstruction of the lesion realizes automatic three-dimensional reconstruction of the lesion, and the auxiliary diagnosis module assists a doctor in identifying and judging lesion information; the multi-scale lesion information presentation module realizes two-dimensional, three-dimensional and mixed reality display of lesion information; and the intraoperative force protection module realizes real-time force feedback and accidental touch early warning during operation, and guides operation. The application realizes AI-assisted diagnosis in the preoperative diagnosis stage, and realizes operation planning, so as to guide the operation of the doctor, reduce operation errors and reduce the disability rate caused by operation.

Description

Technical Field

[0001] This invention relates to the field of surgical navigation technology, and in particular to a visual and sensory intelligent surgical navigation system. Background Technology

[0002] In traditional surgery, lesion localization relies on preoperative imaging, surgeon's visual perception, and experience, making precise location of the surgical area impossible and lacking reliable intraoperative guidance. Therefore, traditional surgery inevitably results in a high rate of damage to adjacent tissues and blood vessels, which can lead to disability or even death in severe cases. Surgical navigation systems, however, are an effective measure to reduce disability rates; their use can reduce disability rates by up to 50%.

[0003] However, existing surgical navigation solutions suffer from a lack of intelligence, intuitiveness, and flexibility. In the preoperative diagnosis stage, they rely heavily on the doctor's experience and manual operation, resulting in strong subjective judgment and insufficient intelligence. In the preoperative planning stage, lesion information is mostly presented as two-dimensional medical images, which is not intuitive enough. In the intraoperative operation stage, they rely on the doctor's eyesight and touch, without feedback on force and position, making them inflexible. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a visual and perceptible intelligent surgical navigation system that enables AI-assisted diagnosis in the preoperative diagnosis stage, three-dimensional medical image reconstruction in the surgical planning stage, and force feedback during surgery, thereby guiding the surgeon's operation, reducing surgical errors, and lowering the surgical disability rate.

[0005] To address the aforementioned technical problems, this invention provides a visual and perceptible intelligent surgical navigation system, comprising: an auxiliary diagnostic module, a multi-scale lesion information presentation module, and an intraoperative force protection module; the auxiliary diagnostic module enables automatic segmentation and reconstruction of lesions, with automatic segmentation achieving refined delineation of lesion areas and automatic lesion reconstruction achieving automatic three-dimensional reconstruction of lesions, assisting doctors in identifying and judging lesion information; the multi-scale lesion information presentation module enables two-dimensional, three-dimensional, and mixed reality display of lesion information; and the intraoperative force protection module enables real-time force feedback and accidental touch warnings during surgery, guiding surgical operations.

[0006] Preferably, the auxiliary diagnostic module achieves automatic lesion segmentation, specifically including the following steps:

[0007] Step 1: Set up the network, which consists of two cascaded U-Net networks. Each U-Net includes a four-layer downsampled encoder and a four-layer upsampled decoder.

[0008] Step 2: The ultrasound image with a size of 256×256 is fed into the first U-Net network to obtain coarse segmentation results, which include the probability of each pixel being predicted to be a category, with a size of 256×256×C, where C is the number of categories;

[0009] Step 3: An adaptive uncertainty feature fusion module based on channel attention mechanism is used to adaptively fuse uncertainty features, ultrasound image features and structural features obtained from coarse segmentation, and then feeds them as input into the second U-Net of the cascaded network.

[0010] Step 4: In the encoding layer of the second U-Net, an uncertainty-based attention submodule (UAM) is introduced. The feature weights are adjusted through cross-attention calculation and self-attention calculation to extract key features of the lesion region. The UAM is added before each convolutional layer of the second U-Net encoder. The feature extraction of the second U-Net is adjusted by uncertainty features and attention submodule. The network can adjust the coarse segmentation results and re-decode to obtain fine segmentation results.

[0011] Preferably, an iterative edge correction submodule based on uncertainty is set up. In the lesion boundary region, the weights of the confidence features around the features with high uncertainty are increased to correct the fuzzy boundary prediction errors caused by low confidence features. Through iterative methods, the predicted boundary is continuously corrected. While reducing the boundary prediction errors caused by low confidence features, the confidence of the correct boundary prediction is improved, and the fuzzy boundary is accurately segmented. Finally, the edge-corrected result is used as the final prediction output of the entire network.

[0012] Preferably, the auxiliary diagnostic module realizes automatic reconstruction of lesions, and the network structure includes an image feature extraction branch and an optical flow feature extraction branch.

[0013] Preferably, the input to the image feature extraction branch is two 256×256 ultrasound images, and the input to the optical flow feature extraction branch is obtained by feeding the ultrasound images into the FlowNetCSS module.

[0014] Preferably, the multi-scale lesion information presentation module realizes two-dimensional, three-dimensional and mixed reality display of lesion information. The two-dimensional display of lesion information is realized in the form of a medical image "plane", the three-dimensional display is presented intuitively in the form of a medical model "stereo", and the mixed reality display uses mixed reality (MR) technology, virtual reality (VR) technology and augmented reality (AR) technology to realize the fusion of virtual images and real environment, and the coexistence of digital and physical objects in virtual and real environments.

[0015] Preferably, during the surgical planning stage, the user needs to wear mixed reality glasses to achieve mixed reality display of lesion information. The user can interact with the lesion information by touching it, observe the details of the lesion information, conduct surgical rehearsals, and refine surgical operations, thereby planning the surgery in a more intuitive way. At the same time, different 3D anatomical structures of lesions can be imported into the mixed reality display device to realize the planning and rehearsals of different surgeries.

[0016] Preferably, the intraoperative force protection module includes an intraoperative force feedback auxiliary unit and a magnetically positioned surgical instrument posture tracking unit. The force feedback auxiliary unit realizes real-time force feedback and accidental touch warning during the operation, guiding the surgeon's operation. The magnetically positioned surgical instrument posture tracking unit realizes the posture relationship between the surgical instrument and the lesion.

[0017] Preferably, the intraoperative accidental touch warning unit includes an intraoperative force simulation module, an accidental touch detection module, and an interactive deformation calculation module. The user performs corresponding operations on the surgical instruments through the interactive interface. The interactive interface feeds back the simulated force between the surgical instruments and the organ tissue to the accidental touch detection module, which judges and warns of accidental touch. If it is judged to be a virtual collision, an accidental touch warning is issued to the user. If it is a real touch, the processed feedback force is then input to the interactive deformation module. After processing, the module presents the deformation information generated by the lesion in three dimensions. At the same time, the force magnitude and direction of the corresponding deformation displacement information are fed back to the user in real time by the interactive interface.

[0018] Preferably, the surgical instrument pose tracking unit based on magnetic positioning uses magnetic positioning technology to achieve real-time tracking of surgical instruments. The surgical instruments are embedded with magnetic positioning devices. In the magnetic field generated by the magnetic field generator, the movement of the surgical instruments causes changes in the magnetic field, ultimately achieving position tracking.

[0019] The beneficial effects of this invention are as follows: For the "brain"—in the preoperative diagnosis stage, the system utilizes an automatic segmentation algorithm to achieve precise identification of lesion areas, assisting doctors in diagnosis; for the "eye"—in the surgical planning stage, the system utilizes two-dimensional, three-dimensional, and VR+AR formats to intuitively present lesion information, assisting doctors in specifying surgical plans; for the "hand"—in the intraoperative operation stage, the system uses magnetic positioning technology and force feedback technology to achieve surgical instrument pose tracking and intraoperative accidental touch feedback, guiding doctors in surgical operations. This invention achieves the coordination of the doctor's "eye," "brain," and "hand," realizing "intelligent" preoperative diagnosis, "intuitive" surgical planning, and "flexible" intraoperative operation. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the image segmentation algorithm of the present invention.

[0021] Figure 2 This is a schematic diagram of the three-dimensional reconstruction algorithm of the present invention.

[0022] Figure 3 This is a schematic diagram illustrating the presentation of multi-scale lesion information according to the present invention.

[0023] Figure 4 This is a schematic diagram of the intraoperative accidental touch warning auxiliary unit of the present invention. Detailed Implementation

[0024] A visual and perceptible intelligent surgical navigation system includes: an auxiliary diagnostic module, a multi-scale lesion information presentation module, and an intraoperative force protection module. The auxiliary diagnostic module enables automatic segmentation and reconstruction of lesions. Automatic segmentation enables fine delineation of lesion areas, and automatic lesion reconstruction enables automatic three-dimensional reconstruction of lesions, assisting doctors in identifying and judging lesion information. The multi-scale lesion information presentation module enables two-dimensional, three-dimensional, and mixed reality display of lesion information. The intraoperative force protection module enables real-time force feedback and accidental touch warnings during surgery, guiding surgical operations.

[0025] Automatic Segmentation: In image segmentation tasks, high-pixel regions predicted by the network often overlap with complex lesion regions that are difficult to segment, as well as blurred or even disappearing boundary regions in ultrasound images. Therefore, enhancing the confidence of the neural network prediction results can improve the segmentation performance of complex lesion regions and blurred boundary regions in ultrasound images. This system proposes a segmentation method based on enhancing the uncertainty features of network segmentation prediction. By introducing uncertainty features, the network's segmentation effect on noisy lesion regions and blurred boundary regions is improved, thereby enhancing the network's segmentation performance on ultrasound images. On the one hand, through a cascaded network structure, we can calculate the predicted uncertainty features from the segmentation results of the first network and add them as features to the second network, thereby improving the segmentation results of high-uncertainty regions (i.e., complex lesion regions) in the previous coarse segmentation results. On the other hand, based on the uncertainty features of the final prediction results, we introduce an iterative edge correction module, which iteratively adjusts the weights of edge features according to the predicted uncertainty, improving the network's performance in segmenting boundary regions.

[0026] Network framework such as Figure 1As shown, the network consists of two cascaded U-Net networks. Each U-Net comprises a four-layer downsampling encoder and a four-layer upsampling decoder. Feeding a 256×256 ultrasound image into the first U-Net yields a coarse segmentation result, containing the probability of each pixel being predicted to belong to a specific category, with dimensions of 256×256×C, where C is the number of categories. Based on this probability map, we can calculate the entropy value of each pixel prediction, reflecting the uncertainty of the prediction. Then, through an adaptive uncertainty feature fusion module based on channel attention, we adaptively fuse the uncertainty features, ultrasound image features, and structural features obtained from the coarse segmentation, and feed this as input into the second U-Net of the cascaded network. Simultaneously, in the encoding layer of the second U-Net, we introduce an uncertainty-based attention submodule (UAM), which adjusts feature weights through cross-attention and self-attention calculations to extract key features of the lesion region. Since the encoding layer of the U-Net contains four downsampling operations, to fully utilize information at different scales, we add the UAM before each convolutional layer of the second U-Net encoder. By adjusting the feature extraction of the second U-Net through uncertainty features and the attention submodule, the network can adjust the coarse segmentation results and re-decode to obtain the fine segmentation results.

[0027] To further refine the ambiguous boundaries of lesion regions, we perform edge correction on the precise segmentation results. We propose an iterative edge correction submodule based on uncertainty. In the lesion boundary region, we increase the weights of surrounding confidence features with high uncertainty to correct prediction errors caused by low-confidence features. Through iteration, we continuously refine the predicted boundaries, reducing prediction errors caused by low-confidence features while increasing the confidence of correct boundary predictions, thus achieving accurate segmentation of ambiguous boundaries. Finally, the edge-corrected result is used as the final prediction output of the entire network.

[0028] Automatic Reconstruction: The automatic reconstruction algorithm used in this system is a deep learning-based network model. This model can learn the feature information in ultrasound images, predict the spatial pose information between adjacent images, and incorporate modal features such as optical flow and inertial measurement units to improve the network's estimation accuracy of spatial pose.

[0029] The structure of the network is as follows Figure 2 As shown, it can be mainly divided into image feature extraction branch and optical flow feature extraction branch. The input of the image feature extraction branch is two frames of ultrasound images with a size of 256×256, while the input of the optical flow feature extraction branch is obtained by feeding the ultrasound images into the FlowNetCSS module.

[0030] The multi-scale lesion information presentation module enables two-dimensional, three-dimensional, and mixed reality display of lesion information. The two-dimensional display of lesion information is realized in the form of a medical image in a "plane" form, the three-dimensional display is presented intuitively in the form of a medical model in a "three-dimensional" form, and the mixed reality display uses VR+AR to realize the fusion of virtual images and real environment. The coexistence of digital and physical objects in virtual and real environments allows doctors to interact directly with lesions in the real environment.

[0031] like Figure 3 As shown, during the surgical planning stage, with the help of this module, doctors only need to wear mixed reality glasses to achieve mixed reality display of lesion information. Doctors can interact with the lesion information by touching it, observe the details of the lesion information, conduct surgical rehearsals, and refine surgical operations, thereby planning the surgery in a more intuitive way. At the same time, importing different 3D anatomical structures of lesions into the mixed reality display device can realize the planning and rehearsals of different surgeries.

[0032] The intraoperative force protection module includes an intraoperative accidental touch warning and auxiliary unit and a magnetically positioned surgical instrument posture tracking unit. The accidental touch warning and auxiliary unit realizes real-time force feedback and accidental touch warning during the operation to guide the surgeon's operation. The magnetically positioned surgical instrument posture tracking unit realizes the posture relationship between the surgical instrument and the lesion.

[0033] Intraoperative accidental touch warning unit: This unit primarily alerts the operator when they are about to accidentally touch an organ or tissue, preventing harm to the patient due to surgical errors. For example... Figure 4 As shown, this unit adopts a modular design, with main modules including an intraoperative force simulation module, a mis-touch detection module, and an interactive deformation calculation module. The user operates the surgical instruments via an interactive interface. The interface feeds back the simulated forces between the surgical instruments and the organ tissue to the mis-touch detection module, which judges and warns of mis-touches. If a virtual collision is detected, a mis-touch warning is issued to the user; if a real touch is detected, the processed feedback force is input to the interactive deformation module. This module processes the force and presents the deformation information generated by the lesion in three dimensions. Simultaneously, the magnitude and direction of the force of the corresponding deformation displacement information are fed back to the user in real time via the interactive interface. The force simulation module simulates various forces, such as elasticity and friction, generated by the interaction between the virtual surgical instruments and the 3D model of the lesion. We designed a force feedback device based on OpenHaptics that serves as the interactive interface, enabling it to realistically reflect the output of the force simulation module to the user, thus achieving real-time force feedback during intraoperative operations. We also designed an interactive deformation module that can simulate the deformation information caused by collisions.

[0034] Magnetic positioning-based surgical instrument pose tracking unit: Real-time tracking of surgical instruments is achieved using magnetic positioning technology. The surgical instruments are embedded with magnetic positioning devices. In the magnetic field generated by the magnetic field generator, the movement of the surgical instruments causes changes in the magnetic field, ultimately achieving position tracking.

Claims

1. A visual and sensory intelligent surgical navigation system, characterized in that, include: The module includes an auxiliary diagnostic module, a multi-scale lesion information presentation module, and an intraoperative force protection module. The auxiliary diagnostic module enables automatic segmentation and reconstruction of lesions. Automatic segmentation enables fine delineation of lesion areas, and automatic lesion reconstruction enables automatic three-dimensional reconstruction of lesions, assisting doctors in identifying and judging lesion information. The multi-scale lesion information presentation module enables two-dimensional, three-dimensional, and mixed reality display of lesion information. The intraoperative force protection module enables real-time force feedback and accidental touch warning during surgery, guiding surgical operations. The intraoperative force protection module includes an intraoperative accidental touch warning assistance unit and a magnetically positioned surgical instrument pose tracking unit. The accidental touch warning assistance unit provides real-time force feedback and accidental touch warnings during surgery to guide the surgeon's operation. The magnetically positioned surgical instrument pose tracking unit presents the pose relationship between the surgical instrument and the lesion. The intraoperative accidental touch warning unit includes an intraoperative force simulation module, an accidental touch detection module, and an interactive deformation calculation module. The user performs corresponding operations on the surgical instrument through the interactive interface. The interactive interface feeds back the simulated force between the surgical instrument and the organ tissue to the accidental touch detection module, which judges and warns of accidental touch. If it is judged to be a virtual collision, an accidental touch warning is issued to the user. If it is a real touch, the processed feedback force is then input to the interactive deformation module. After processing, this module presents the deformation information generated by the lesion in three dimensions. At the same time, the force magnitude and direction of the corresponding deformation displacement information are fed back to the user in real time through the interactive interface. The auxiliary diagnostic module enables automatic segmentation of lesions, specifically including the following steps: Step 1: Set up the network, which consists of two cascaded U-Net networks. Each U-Net includes a four-layer downsampled encoder and a four-layer upsampled decoder. Step 2: The ultrasound image with a size of 256×256 is fed into the first U-Net network to obtain coarse segmentation results, which include the probability of each pixel being predicted to be a category, with a size of 256×256×C, where C is the number of categories; Step 3: An adaptive uncertainty feature fusion module based on channel attention mechanism is used to adaptively fuse uncertainty features, ultrasound image features and structural features obtained from coarse segmentation, and then feeds them as input into the second U-Net of the cascaded network. Step 4: In the encoding layer of the second U-Net, an uncertainty-based attention submodule (UAM) is introduced. The feature weights are adjusted through cross-attention calculation and self-attention calculation to extract key features of the lesion region. The UAM is added before each convolutional layer of the second U-Net encoder. The feature extraction of the second U-Net is adjusted by uncertainty features and attention submodule. The network can adjust the coarse segmentation results and re-decode to obtain fine segmentation results.

2. The visual and sensory intelligent surgical navigation system as described in claim 1, characterized in that, An iterative edge correction submodule based on uncertainty is set up. In the lesion boundary region, the weights of the confidence features around the features with high uncertainty are increased to correct the fuzzy boundary prediction errors caused by low confidence features. Through iterative methods, the predicted boundary is continuously corrected. While reducing the boundary prediction errors caused by low confidence features, the confidence of the correct boundary prediction is improved, and the fuzzy boundary is accurately segmented. Finally, the edge correction result is used as the final prediction output of the entire network.

3. The visual and sensory intelligent surgical navigation system as described in claim 1, characterized in that, The auxiliary diagnostic module enables automatic reconstruction of lesions, and its network structure includes an image feature extraction branch and an optical flow feature extraction branch.

4. The visual and sensory intelligent surgical navigation system as described in claim 3, characterized in that, The input to the image feature extraction branch is two 256×256 ultrasound images, while the input to the optical flow feature extraction branch is obtained by feeding the ultrasound images into the FlowNetCSS module.

5. The visual and sensory intelligent surgical navigation system as described in claim 1, characterized in that, The multi-scale lesion information presentation module realizes two-dimensional, three-dimensional and mixed reality display of lesion information. The two-dimensional display of lesion information is realized in the form of medical image "plane", the three-dimensional display is presented intuitively in the form of medical model "stereo", and the mixed reality display uses mixed reality MR technology, virtual reality VR technology and augmented reality AR technology to realize the fusion of virtual images and real environment, and the coexistence of digital and physical objects in virtual and real environments.

6. The visual and sensory intelligent surgical navigation system as described in claim 5, characterized in that, During the surgical planning stage, users need to wear mixed reality glasses to display lesion information. Users can interact with the lesion information by touching it, observe the details of the lesion information, conduct surgical rehearsals, and refine surgical operations. This allows for more intuitive surgical planning. At the same time, different 3D anatomical structures of lesions can be imported into the mixed reality display device to realize the planning and rehearsals of different surgeries.

7. The visual and sensory intelligent surgical navigation system as described in claim 1, characterized in that, The magnetic positioning-based surgical instrument pose tracking unit uses magnetic positioning technology to achieve real-time tracking of surgical instruments. The surgical instruments are embedded with magnetic positioning devices. In the magnetic field generated by the magnetic field generator, the movement of the surgical instruments causes changes in the magnetic field, ultimately achieving position tracking.

Images