Thoracoscope minimally invasive cardiac surgery navigation system based on multi-modal image fusion

Abstract

The invention discloses a thoracoscope minimally invasive cardiac surgery navigation system based on multi-modal image fusion, and the system comprises the steps: constructing a 5D heart digital twin containing an anatomical structure, a soft tissue boundary, metabolic activity and electrophysiological information through fusing preoperative CT, MRI, PET and intraoperative ultrasonic images; real-time dynamic registration is realized by using an electrocardio gating driven biomechanical model; superposing the multi-color coding navigation information to the visual field of the thoracoscope through a wavelength selection type spectroscope; a dynamic risk scoring model is constructed based on the distance, the speed and the tissue vulnerability, and three-level grading intervention is triggered; a miniature force feedback actuator is integrated to realize closed-loop control; and pre-operation virtual drilling and post-operation federal learning optimization are supported. The system remarkably improves the surgical precision and safety, and is suitable for high-difficulty heart minimally invasive surgeries such as mitral valve repair and atrial fibrillation ablation.

Description

Technical Field

[0001] This invention belongs to the field of surgical navigation and computer-aided surgical technology, specifically relating to a thoracoscopic minimally invasive cardiac surgery navigation system based on multimodal image fusion. Background Technology

[0002] Thoracoscopic minimally invasive cardiac surgery, as a cutting-edge technology in cardiovascular surgery, has become an important means of treating diseases such as valvular heart disease, congenital heart disease, and arrhythmias due to its advantages of minimal trauma and rapid recovery. However, the current technology system still has three major bottlenecks, which seriously restrict the accuracy and safety of the surgery. First, current surgical navigation systems generally use a single imaging modality (such as CT or ultrasound), resulting in incomplete display of anatomical structures. Although CT can clearly show bony structures and calcified plaques, it has insufficient resolution for soft tissue boundaries; MRI can capture the trajectory of myocardial motion, but lacks metabolic function information; although PET can display the active area of ​​myocardium, its spatial positioning accuracy is limited. Current technologies only achieve simple image superposition and have not established a feature-level fusion mechanism of anatomical-functional-metabolic information. Second, dynamic organ registration is difficult: the heart continuously beats during surgery and is affected by respiration, with displacement reaching 10-15 mm. Traditional rigid or affine registration cannot compensate for non-rigid deformation, resulting in serious misalignment between the virtual model and the real organ, and navigation failure. Third, the human-computer interaction and safety mechanisms are weak: the surgeon needs to frequently switch his gaze between the monitor and the surgical field; safety warnings rely only on fixed distance thresholds, ignoring dynamic factors such as instrument speed and tissue fragility; the actuator control lacks a force feedback mechanism, resulting in a high rate of misoperation. Summary of the Invention

[0003] This invention aims to provide a high-precision, high-safety, and adaptively evolving navigation system for thoracoscopic minimally invasive cardiac surgery, solving the problems of fragmented multimodal information, inaccurate dynamic registration, fragmented human-computer interaction, and insufficient safety protection in existing technologies. It achieves millimeter-level precise navigation and proactive risk intervention during surgery, facilitating medical surgical navigation.

[0004] The present invention achieves its objective by employing the following technical solution: This invention provides a navigation system for thoracoscopic minimally invasive cardiac surgery based on multimodal image fusion, comprising a multimodal image fusion unit, including: a preoperative fusion subunit configured to perform feature-level fusion of preoperative CT images, MRI images, and PET images to generate a 5D digital twin of the heart containing anatomical structures, soft tissue boundaries, metabolic activity, and electrophysiological characteristics; and an intraoperative correction subunit configured to receive intraoperative ultrasound images / optical coherence tomography data, extract local anatomical boundaries or microstructural features, and perform online non-rigid correction on the 5D digital twin of the heart based on the extracted features to compensate for intraoperative anatomical deformation; The dynamic registration unit couples ECG signals, respiratory waveforms and blood pressure signals, and compensates for non-rigid displacements caused by cardiac pulsation and respiratory movements in real time based on a patient-specific biomechanical deformation model. The augmented reality display unit overlays the navigation information of the 5D cardiac digital twin onto the real-time field of view of the thoracoscope using a multi-color encoding method through a wavelength-selective beam splitter. The safety early warning unit calculates a dynamic risk score and triggers a graded intervention mechanism based on the distance and speed between surgical instruments and risky structures, as well as the tissue vulnerability index. The actuator control unit integrates a 6DoF electromagnetic sensor and a micro force feedback module to adjust the operating freedom and tactile feedback intensity of the surgical instruments based on the dynamic risk score.

[0005] As a further limitation of this technical solution, the preoperative fusion subunit adopts the following network structure: The CT encoder uses a three-level cascaded 3D U-Net to extract high-resolution anatomical features from CT images; An MRI encoder, combining the cine sequence and T1 mapping included in MRI images, calculates the myocardial motion vector field using optical flow. A PET encoder employing a 3D Swin Transformer captures the global spatial dependence of metabolic activity from PET images; The cross-attention fusion module uses extracted CT features as queries and MRI and PET features as key-value pairs to generate aligned multimodal fusion features. The spatiotemporal decoder models along 20 cardiac phases using a bidirectional LSTM and a temporal convolutional network, outputting a 5D digital twin of the heart that includes anatomical labels, motion vectors, metabolic intensity, electrophysiological properties, and tissue vulnerability.

[0006] As a further limitation of this technical solution, the wavelength-selective beam splitter has reflection bands of 515±10nm and 590±10nm, respectively, and transmits the remaining visible light bands.

[0007] As a further limitation of this technical solution, the safety early warning unit is specifically configured to identify thin-walled areas based on the patient's preoperative MRI images and low-metabolic areas based on PET images, so as to generate a patient-specific tissue vulnerability map, and introduce it as a spatial weighting factor into the dynamic risk scoring model.

[0008] As a further limitation of this technical solution, the dynamic risk scoring model is expressed as: ; in, For dynamic risk scoring, The current position of the instrument tip calculated from preoperative images. The organization's vulnerability index value at that location, , , The weights for distance, speed, and tissue fragility are respectively adjusted automatically based on the type of surgery. The distance from the tip of the instrument to the nearest risk structure. This is a correction value. The speed at which the equipment is propelled.

[0009] As a further limitation of this technical solution, the system also includes a preoperative planning module that supports virtual surgical simulation and path optimization based on a 5D heart model.

[0010] As a further limitation of this technical solution, the system supports real-time updating and dynamic registration of multimodal images to adapt to changes in anatomical structures during surgery.

[0011] As a further limitation of this technical solution, the augmented reality display unit supports dual-mode output of the surgeon's eyepiece and an external display to meet the needs of different surgical scenarios.

[0012] As a further limitation of this technical solution, the safety warning unit is specifically configured such that when the dynamic risk score exceeds a first threshold, the actuator control unit activates a tactile vibration prompt; when the dynamic risk score exceeds a second threshold, the operating resistance is increased; and when the dynamic risk score exceeds a third threshold, the instrument's propulsion degrees of freedom are physically restricted.

[0013] Compared with the prior art, the advantages and positive effects of the present invention are: 1. By using a multimodal image fusion unit to perform feature-level fusion of CT, MRI and PET images, a 5D digital twin of the heart containing anatomical structures, soft tissue boundaries and metabolic activity is generated. The twin is then dynamically registered to achieve zero-displacement alignment with the real heart during surgery, thereby improving the anatomical recognition under thoracoscopic vision.

[0014] 2. The safety warning unit calculates the Euclidean distance between the surgical instrument and the risk structure in real time. Combined with the force feedback and tactile vibration prompts from the actuator control unit, it restricts the instrument's degree of freedom of movement when a preset threshold is reached, thereby reducing the probability of accidental injury.

[0015] 3. The preoperative planning module performs virtual surgical drills and path optimization based on a 5D cardiac model. Postoperative results are written back to the multimodal image fusion unit and risk scoring model to achieve self-updating of algorithm parameters, shorten the surgical learning cycle and standardize the operation process. Attached Figure Description

[0016] Figure 1This is a system flowchart of the present invention. Detailed Implementation

[0017] The following detailed description of a specific embodiment of the present invention is provided in conjunction with the accompanying drawings. However, it should be understood that the scope of protection of the present invention is not limited to the specific embodiment.

[0018] In one specific embodiment of the present invention: like Figure 1 As shown, the thoracoscopic minimally invasive cardiac surgery navigation system based on multimodal image fusion provided by the present invention includes a multimodal image fusion unit, comprising: a preoperative fusion subunit configured to perform feature-level fusion of preoperative CT images, MRI images, and PET images to generate a 5D digital twin of the heart containing anatomical structures, soft tissue boundaries, metabolic activity, and electrophysiological characteristics; and an intraoperative correction subunit configured to receive intraoperative ultrasound images / optical coherence tomography data, extract local anatomical boundaries or microstructural features, and perform online non-rigid correction of the 5D digital twin of the heart based on the extracted features to compensate for intraoperative anatomical deformation. The dynamic registration unit couples ECG signals, respiratory waveforms and blood pressure signals, and compensates for non-rigid displacements caused by cardiac pulsation and respiratory movements in real time based on a patient-specific biomechanical deformation model. The augmented reality display unit overlays the navigation information of the 5D cardiac digital twin onto the real-time field of view of the thoracoscope using a multi-color encoding method through a wavelength-selective beam splitter. The safety early warning unit calculates a dynamic risk score and triggers a graded intervention mechanism based on the distance and speed between surgical instruments and risky structures, as well as the tissue vulnerability index. The actuator control unit integrates a 6DoF electromagnetic sensor and a micro force feedback module to adjust the operating freedom and tactile feedback intensity of the surgical instruments based on the dynamic risk score.

[0019] Preoperative CT images can reveal anatomical structures, MRI images can reveal soft tissue boundaries, myocardial strain, and fiber orientation, and positron emission tomography (PET) is a functional imaging technique that can demonstrate metabolic activity. This invention overcomes the limitations of traditional systems that only use a single modality (such as CT images), pixel-level or decision-level fusion. Through a preoperative fusion subunit within a multimodal image fusion unit, it performs feature-level fusion of preoperative CT, MRI, and PET images to generate a 5D digital twin of the heart containing both temporal and spatial dimensions. Furthermore, since images acquired under preoperative static conditions cannot reflect anatomical deformations caused by intraoperative changes in body position, pericardial traction, and hemodynamics, this invention also utilizes an intraoperative correction subunit to receive real-time intraoperative ultrasound images and perform online registration and correction with the preoperative 5D digital twin of the heart, further incorporating the myocardial electrical conduction velocity field and the valvular regurgitation eddy current field. The revised 5D cardiac digital twin not only outputs anatomy / metabolism, but also electrophysiological maps (for atrial fibrillation ablation) and hemodynamic parameters (such as wall stress and shear force).

[0020] Among them, "5D heart digital twin" refers to a voxel-level heart model that includes three-dimensional spatial coordinates, time dimension (t, corresponding to the phase of the cardiac cycle), and functional dimension (f, including metabolic activity, electrical conduction velocity, and tissue fragility).

[0021] The risk structures include, but are not limited to, the main coronary artery, the atrioventricular conduction bundle, the aortic valve annulus, and thin-walled regions (thickness <2 mm) identified in preoperative imaging.

[0022] The color coding method includes rendering blood vessels in green and displaying warning areas in orange-red. A wavelength-selective beam splitter is used for reflecting specific colors of light. This invention, specifically for thoracoscopic AR intraocular navigation scenarios, configures it to reflect specific wavelengths, such as only the green and orange-red bands (515nm green is brightest, 590nm orange is more prominent), and matches the wavelengths sensitive to the human eye with clinical semantics, seamlessly integrating virtual and real-world vision. The augmented reality display unit in this system eliminates the traditional "looking at the screen - looking at the surgical field" visual switching, improving operational smoothness and focus.

[0023] Among them, the actuator control unit refers to the integrated sensing-drive module inside the thoracoscopic surgical instrument.

[0024] In practice, the system's workflow is as follows: acquiring preoperative CT, MRI, and PET images; generating a 5D digital twin of the heart through feature-level fusion; dynamically registering the digital twin with the real heart during surgery based on electrocardiogram signals; overlaying the registered navigation information onto the thoracoscopic field of view in a color-coded form; monitoring the distance and motion status of surgical instruments and risk structures in real time, and calculating a dynamic risk score; and controlling the operating freedom of surgical instruments and providing force feedback according to the dynamic risk score.

[0025] The thoracoscopic minimally invasive cardiac surgery navigation system provided by this invention uses a digital twin heart, AR intraocular navigation, and force feedback control as its core architecture: a multimodal image fusion unit synthesizes anatomical, boundary, and metabolic information from preoperative CT, MRI, and PET images into a 5D heart model; a dynamic registration unit performs frame-by-frame correction of the biomechanical model based on ECG gating and respiratory signals, ensuring real-time alignment of the model coordinates with the real heart. An augmented reality display unit overlays 515nm and 590nm band images onto the thoracoscopic field of view using an optical synthesizer, outputting vascular rendering and warning colors. A safety warning unit emits vibration and voice signals when the distance between the instrument and a risk structure reaches a threshold, and controls the actuator control unit to reduce the instrument advancement speed. Through these modules, the system achieves millimeter-level positioning under thoracoscopic guidance, reducing complications and standardizing operating procedures. This system is applicable to mitral valve repair, atrial fibrillation ablation, or congenital heart disease closure procedures.

[0026] In traditional systems, even with AR display, the lack of real-time dynamic registration results in a 5 to 8 mm displacement between the virtual model and the real heart, causing navigation failure. However, this invention uses an ECG-gated biomechanical deformation model to control the registration error to within 0.5 mm, making AR navigation truly clinically usable.

[0027] As one embodiment, the preoperative fusion subunit adopts the following network structure: The CT encoder uses a three-level cascaded 3D U-Net to extract high-resolution anatomical features from CT images; An MRI encoder, combining the cine sequence and T1 mapping included in MRI images, calculates the myocardial motion vector field using optical flow. A PET encoder employing a 3D Swin Transformer captures the global spatial dependence of metabolic activity from PET images; The cross-attention fusion module uses extracted CT features as queries and MRI and PET features as key-value pairs to generate aligned multimodal fusion features. The spatiotemporal decoder models along 20 cardiac phases using a bidirectional LSTM and a temporal convolutional network, outputting a 5D digital twin of the heart that includes anatomical labels, motion vectors, metabolic intensity, electrophysiological properties, and tissue vulnerability.

[0028] The CT encoder branch takes CT images as input and employs a three-layer cascaded 3D U-Net with 2×2×2 downsampling and 32, 64, and 128 channels respectively. It progressively extracts ventricular contours, valve geometry, and coronary artery orientation, preserving edge gradients through skip connections. The MRI encoder branch takes Cine MRI (20 phases) + T1 mapping as input. This branch combines Motion-CNN and optical flow estimation, using FlowNet 2.0 to extract the myocardial motion vector field, with T1 values ​​as additional channel input. The PET encoder branch takes FDG-PET SUV images as input. This branch uses a Swin Transformer with a 7×7×7 window size, a shift step of 3, and an embedding dimension of 96 projected to 128. The metabolic intensity map is windowed and then self-attention is calculated through the shifted window to obtain global functional dependencies. It is important to note that all input images are pre-registered to the same space (based on the left ventricular long axis), and the voxel size is uniformly 1.0 mm. 3 .

[0029] Among them, the cine sequence is a dynamic cine imaging that covers multiple cardiac cycle phases and is used to extract myocardial motion information; T1 mapping is an imaging technique that quantitatively measures the longitudinal relaxation time of myocardium pixel by pixel. It directly measures the longitudinal relaxation time (T1 value, unit ms) of each pixel of myocardial tissue through special pulse sequences (such as MOLLI, ShMOLLI, SAPPHIRE). This invention is used to quantify the degree of myocardial fibrosis.

[0030] It is worth noting that traditional concatenation or weighted averaging cannot model intermodal dependencies. This invention employs a learnable cross-attention fusion module. The fusion process involves projecting trimodal features onto a unified query (Q), key (K), and value (V) space, calculating attention weights, and generating fused features. The advantage of this fusion method lies in the strong anatomical prior provided by CT, guiding the network to focus on functional regions aligned with structures in MRI / PET (such as metabolic hotspots at the valve annulus).

[0031] Electrophysiological characteristics (such as conduction velocity and refractory period) cannot be directly obtained from images, but can be introduced through indirect surrogate indicators. Examples of indirect surrogate indicators include: T1 mapping values, which indicate the degree of myocardial fibrosis and correspond to conduction delay; wall thickness variability (MRI), which indicates abnormal local strain and corresponds to re-entrant matrix; and low-metabolic areas in PET, which indicate electroquiescent areas. During the training phase, the system receives preoperative electrophysiological mapping (if available) as a weak supervisory signal, and guides feature learning through multi-task loss.

[0032] In the spatiotemporal decoder, 20 cardiac phases are treated as sequences and input into a bidirectional temporal convolutional network (TCN) + LSTM, outputting a 5D digital twin of the heart, where each voxel contains anatomical labels, motion vectors, metabolic intensity, electrical conduction velocity, and vulnerability index.

[0033] Training is performed using a hybrid loss function to ensure multi-objective collaborative optimization. ; in, This represents Dice loss, used to constrain anatomical margins; This represents the mean squared error loss, used to constrain the regression of the motion field; To predict the temporal-spatial correlation coefficient with real PET SUVs, and to constrain metabolic time series; For deformation field smoothing loss; , , , These are the weighting coefficients for each item.

[0034] The intraoperative correction subunit is specifically configured as follows: receiving transesophageal ultrasound or intracardiac ultrasound images and registering them to the navigation coordinate system based on the probe pose; calling the corresponding lightweight neural network to segment local anatomical structures or detect key points in real time according to the current surgical stage; converting the segmentation boundaries or key points into a sparse observation point cloud with normals and confidence; constructing an energy function with biomechanical stiffness as the regularization term within the corresponding anatomical region of the 5D cardiac digital twin, and solving the local non-rigid displacement field through nonlinear optimization; updating only the model vertices constrained by observation and their associated functional attributes to achieve millisecond-level online correction.

[0035] It should be noted that, because real-time ultrasound imaging (ICE) or optical coherence tomography (OCT) data are intraoperative local imaging modalities with limited fields of view and high noise, they are not suitable for direct input into the same depth network for end-to-end training as CT / MRI / PET images. This invention employs a "prior model + online correction" strategy: preoperative fusion provides a globally high-confidence model, while the intraoperative modality is only used for dynamic updates of locally reliable regions, thus balancing accuracy and robustness.

[0036] In the first stage (preoperative), the U-Net+Transformer+LSTM network architecture is used to perform end-to-end feature-level fusion of CT, MRI and PET images to construct a global 5D digital twin of the heart (high-precision prior). In the second stage (intraoperative), ICE / OCT is used to perform online registration and local correction of the output results of the first stage to obtain a dynamically updated 5D digital twin model of the heart.

[0037] ICE fusion: Lightweight UNet is used to segment the left atrial / left ventricular boundaries in the ICE; the segmentation results are used as observations, and local deformation adjustments are made to the surface of the preoperative model through Kalman filtering or non-rigid ICP (Iterative Closest Point); updates are only applied to the area covered by the ICE field of view (such as near the mitral valve annulus) to avoid global perturbations.

[0038] OCT fusion: Utilizes high-resolution OCT to identify leaflet edges or calcifications; overlays "microstructure maps" at corresponding locations on the digital twin; used to enhance AR rendering details (e.g., displaying calcified areas as blinking white dots).

[0039] The augmented reality display unit includes a micro-projection module, a wavelength-selective beam splitter, and a refractive adjustment lens group. The micro-projection module generates a multi-color encoded navigation image with emission wavelengths limited to 515±10 nm and 590±10 nm. Specifically, 515 nm is near the peak of the human eye's spectral luminous efficiency V(λ) (555 nm), and green has high contrast and is less tiring against a blood-red background. At 590 nm, orange-yellow has the highest Weber contrast in a dark red surgical field, making it suitable as a warning color. The navigation image is rendered in real-time by a GPU. The rendering engine automatically allocates color channels based on the voxel tags of the 5D digital twin and controls brightness and flicker frequency (e.g., 2Hz pulsation in high-risk areas). The wavelength-selective beam splitter has a reflectivity of no less than 85% for the aforementioned wavelengths and a transmittance of no less than 95% for the remaining visible light bands, ensuring that the navigation information and the thoracoscopic surgical field image are superimposed collinearly on the human eye's observation plane, achieving unobstructed, high-contrast augmented reality display.

[0040] For example, the augmented reality display unit includes the following components, arranged in the order of the optical path: a micro-projection module: an OLED or LCoS microdisplay (1920×1080 resolution, 120 Hz refresh rate) for generating navigation images; a narrowband light source / filter system: emitting only light of specific wavelengths (e.g., 515 nm green light, 590 nm orange-red light); a wavelength-selective beam splitter: placed in the thoracoscope eyepiece optical path to reflect navigation light and transmit surgical field light; an eye box and refractive adjustment lens group: adapting to different surgeons' visual acuity (-5D ~ +3D); and a thoracoscope imaging optical path: the white light reflected image from the patient's body is transmitted to the beam splitter via the objective lens and relay lens. Preferably, the wavelength-selective beam splitter reflects wavelengths of 515±10nm and 590±10nm, respectively, and transmits the remaining visible light wavelengths.

[0041] In one embodiment, the dynamic configuration unit receives an ECG R wave as the electrocardiogram signal, a respiratory waveform acquired by a chest impedance or abdominal pressure sensor, and an arterial pressure waveform as the blood pressure signal, used to estimate the ventricular filling state. The biomechanical deformation model is preferably a finite element model based on Mooney-Rivlin hyperelastic material, but this invention does not limit it to this.

[0042] As one embodiment, the dynamic registration unit includes: The electrocardiogram (ECG) signal acquisition module is used to acquire the R-wave trigger signal; The patient-specific finite element model construction module reconstructs a tetrahedral mesh of the left ventricle based on preoperative CT images. The material properties are modeled using the Mooney-Rivlin hyperelasticity model, where C... 10 =8.5 kPa, C 01 =2.1 kPa, where C 10 C reflects the material's resistance to the first strain invariant I1 (i.e., the overall degree of tensile stress). 01 C reflects the material's sensitivity to the second strain invariant I2 (related to shear and shape distortion). 10 C 01 All were determined by preoperative MRI T1 mapping; The real-time pose fusion module receives 6DoF electromagnetic sensor data and thoracoscopic visual SLAM point cloud, and solves the node displacement vector u(t) by extended Kalman filtering; The non-rigid transformation module updates the surface vertex V0 of the 5D heart digital twin according to V(t)=V0+u(t), achieving frame-by-frame alignment with the real heart.

[0043] As one embodiment, the augmented reality display unit supports semantic and context-aware AR rendering, and adopts different rendering strategies for different structure types, as shown in Table 1 below: Table 1 In one embodiment, the safety warning unit is specifically configured to identify thin-walled regions based on the patient's preoperative MRI images, identify low-metabolic regions based on PET images, generate a patient-specific tissue vulnerability map, and introduce it as a spatial weighting factor into a dynamic risk scoring model.

[0044] In practice, firstly, based on preoperative MRI images, thin-walled regions with a wall thickness of less than 2.0 mm are identified by quantitative analysis of myocardial wall thickness; then, based on preoperative PET images, low-metabolic regions are identified by a threshold where the standardized uptake value (SUV) is lower than 60% of the average SUV of the myocardium; then, the thin-walled regions and low-metabolic regions are fused according to weighted logic or rules to generate a three-dimensional tissue vulnerability map, with the values ​​of its elements ranging from 0 to 1; (4) During the operation, the tissue vulnerability index value corresponding to the current position of the instrument is queried in real time and introduced as a spatial weight factor into the dynamic risk scoring model to realize patient-specific risk warning.

[0045] The dynamic risk scoring model is expressed as follows: ; in, For dynamic risk scoring, The current position of the instrument tip calculated from preoperative images. The organization's vulnerability index value ranges from 0 to 1. , , The weights for distance, speed, and tissue fragility are automatically adjusted based on the type of surgery (calibrated by clinical experts). The distance from the tip of the instrument to the hazardous structure. =0.1 is a correction value, the purpose of which is to prevent the denominator from being 0. The speed at which the equipment is propelled.

[0046] In one embodiment, the safety warning unit is specifically configured to activate tactile vibration alert when the dynamic risk score exceeds a first threshold; increase operating resistance when the dynamic risk score exceeds a second threshold; and physically restrict the instrument's propulsion degrees of freedom when the dynamic risk score exceeds a third threshold.

[0047] An optional implementation: When the dynamic risk score R < 0.8, the risk level is determined to be safe (green) and no prompt is required; when 0.8 < R < 1.5, the risk level is determined to be early warning (yellow), the AR flashes and is accompanied by a 200 Hz vibration, and at the same time the resistance is doubled; when R > 1.5, the risk level is determined to be dangerous (red), the z-axis propulsion is cut off, the AR is full-screen red-framed and a voice prompt "High risk! Stop!" is given.

[0048] The actuator control unit integrates a 6DoF electromagnetic sensor with a sampling frequency of 200 Hz, a positioning accuracy of 0.3 mm, and supports force feedback and tactile vibration prompts.

[0049] As one of the embodiments, the actuator control unit sets a 6DoF electromagnetic microcoil at each of the distal and proximal ends of the thoracoscope instrument rod to form a rigid reference pair; two sets of six-dimensional pose data are synchronously collected by a 200 Hz radio frequency field, and the rod deflection is calculated in real time through vector difference and the terminal drift is compensated with an accuracy of 0.3 mm. The sensor data is fed forward by FPGA hardware with a delay < 5 ms, directly driving the micro linear motor and the magnetorheological damper: when the distance between the forceps tip and the mitral annulus is less than the preset threshold, the damper instantaneously increases the torsional resistance; if further advancement continues, an 8 kHz linear resonant motor outputs a narrowband vibration prompt. The force feedback level is mapped one-to-one with the AR color scale, the blue area maintains the basic resistance, the yellow area has a sharp increase in resistance, and the red area cuts off the propulsion freedom. The sensing-driving control module has an outer diameter ≤ 5 mm and can be integrated into a standard 5 mm thoracoscope instrument without an additional channel.

[0050] As one of the embodiments, the system further includes a preoperative planning module that supports virtual surgical rehearsal and path optimization based on a 5D heart model.

[0051] The preoperative planning module constructs an interactive virtual model based on a 5D heart digital twin, realizes the layer-by-layer display and hiding of myocardial, rib and soft tissue structures and 360° perspective switching through an AR helmet and a handle input device, and synchronously plays the valve opening and closing, coronary artery deformation and blood flow time series state during the cardiac cycle; the system calls an individualized biomechanical deformation model to calculate the surgical field exposure degree at different incision coordinates and outputs the optimal approach coordinate set in real time. After the operator selects any point on the valve annulus, the module immediately generates the stitch distance, the needle entry angle and the tension curve, and drives the force feedback handle to provide the corresponding resistance. After completing the virtual suture, the valve area, the stress concentration value and the residual regurgitation volume are automatically measured. If the evaluation result does not reach the preset threshold, it is allowed to retreat and readjust the path, and iterate until the conditions are met. The pose, mechanics and evaluation data generated during the rehearsal process are recorded as a surgical fingerprint, and the intraoperative navigation unit directly calls this fingerprint to set an individualized safety boundary, making the preoperative planning and intraoperative execution share the same coordinate system and mechanical standard.

[0052] As one embodiment, the system supports real-time updates and dynamic registration of multimodal images to adapt to changes in anatomical structures during surgery.

[0053] As one embodiment, the augmented reality display unit supports dual-mode output of the surgeon's eyepiece and an external display to meet the needs of different surgical scenarios.

[0054] The system can optimize the multimodal fusion algorithm and risk scoring model based on surgical outcome data.

[0055] Through the above technical analysis, this invention achieves breakthrough innovations in key aspects such as image fusion, deformation compensation, human-computer interaction, and safety protection, providing a fully intelligent solution for thoracoscopic minimally invasive cardiac surgery.

[0056] Throughout this specification, the terms "an embodiment" or "an embodiment" mean that a specific feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the disclosed subject matter. Therefore, the phrases "in one embodiment" or "in an embodiment" appearing throughout this specification do not necessarily refer to the same embodiment. Furthermore, any suitable manner may be adopted to incorporate a specific feature, structure, or characteristic in one or more embodiments. It should be understood that this specification is not intended to limit the invention. Rather, exemplary embodiments are intended to cover alternatives, modifications, and equivalents that are included within the spirit and scope of the invention as defined by the appended claims. Furthermore, numerous specific details are set forth in the detailed description of exemplary embodiments to provide a comprehensive understanding of the claimed invention. However, those skilled in the art will understand that various embodiments may also be practiced without these specific details.

[0057] Although features and elements of these exemplary embodiments have been described in particular combination in the embodiments, each feature and element may be used alone without the other features and elements of the embodiments, or in combination with or without the other features and elements disclosed herein.

[0058] This written description uses examples, including the best mode, to disclose the invention and enables any person skilled in the art to practice the invention, including making and utilizing any apparatus or system and performing any combined methods. The patentable scope of the invention is defined by the claims and may include other examples as would be apparent to those skilled in the art. Such other examples are considered to be included within the scope of the claims if they have structural elements that are not different from the verbal language of the claims, or if they include structural elements equivalent to those described in the verbal language of the claims.

Claims

1. A navigation system for thoracoscopic minimally invasive cardiac surgery based on multimodal image fusion, characterized in that, include: The multimodal image fusion unit includes: a preoperative fusion subunit, configured to perform feature-level fusion of preoperative CT images, MRI images, and PET images to generate a 5D cardiac digital twin containing anatomical structures, soft tissue boundaries, metabolic activity, and electrophysiological characteristics; and an intraoperative correction subunit, configured to receive intraoperative ultrasound images / optical coherence tomography data, extract local anatomical boundaries or microstructural features, and perform online non-rigid correction of the 5D cardiac digital twin based on the extracted features to compensate for intraoperative anatomical deformation. The dynamic registration unit couples ECG signals, respiratory waveforms and blood pressure signals, and compensates for non-rigid displacements caused by cardiac pulsation and respiratory movements in real time based on a patient-specific biomechanical deformation model. The augmented reality display unit overlays the navigation information of the 5D cardiac digital twin onto the real-time field of view of the thoracoscope using a multi-color encoding method through a wavelength-selective beam splitter. The safety early warning unit calculates a dynamic risk score and triggers a graded intervention mechanism based on the distance and speed between surgical instruments and risky structures, as well as the tissue vulnerability index. The actuator control unit integrates a 6DoF electromagnetic sensor and a micro force feedback module to adjust the operating freedom and tactile feedback intensity of the surgical instruments based on the dynamic risk score.

2. The thoracoscopic minimally invasive cardiac surgery navigation system based on multimodal image fusion according to claim 1, characterized in that: The preoperative fusion subunit adopts the following network structure: The CT encoder uses a three-level cascaded 3D U-Net to extract high-resolution anatomical features from CT images; An MRI encoder, combining the cine sequence and T1 mapping included in MRI images, calculates the myocardial motion vector field using optical flow. A PET encoder employing a 3D Swin Transformer captures the global spatial dependence of metabolic activity from PET images; The cross-attention fusion module uses extracted CT features as queries and MRI and PET features as key-value pairs to generate aligned multimodal fusion features. The spatiotemporal decoder models along 20 cardiac phases using a bidirectional LSTM and a temporal convolutional network, outputting a 5D digital twin of the heart that includes anatomical labels, motion vectors, metabolic intensity, electrophysiological properties, and tissue vulnerability.

3. The thoracoscopic minimally invasive cardiac surgery navigation system based on multimodal image fusion according to claim 1, characterized in that: The wavelength-selective beam splitter has reflection bands of 515±10nm and 590±10nm, respectively, and transmits the remaining visible light bands.

4. The thoracoscopic minimally invasive cardiac surgery navigation system based on multimodal image fusion according to claim 1, characterized in that: The safety early warning unit is specifically configured to identify thin-walled areas based on the patient's preoperative MRI images and low-metabolic areas based on PET images, in order to generate a patient-specific tissue vulnerability map, which is then introduced as a spatial weighting factor into the dynamic risk scoring model.

5. The thoracoscopic minimally invasive cardiac surgery navigation system based on multimodal image fusion according to claim 4, characterized in that: The dynamic risk scoring model is expressed as follows: ； in, For dynamic risk scoring, The current position of the instrument tip calculated from preoperative images. The organization's vulnerability index value at that location, , , The weights for distance, speed, and tissue fragility are respectively adjusted automatically based on the type of surgery. The distance from the tip of the instrument to the nearest risk structure. This is a correction value. The speed at which the equipment is propelled.

6. The thoracoscopic minimally invasive cardiac surgery navigation system based on multimodal image fusion according to any one of claims 1-5, characterized in that: The system also includes a preoperative planning module that supports virtual surgical simulations and pathway optimization based on a 5D heart model.

7. The thoracoscopic minimally invasive cardiac surgery navigation system based on multimodal image fusion according to claim 3, characterized in that: The augmented reality display unit supports dual-mode output of the surgeon's eyepiece and an external display, meeting the needs of different surgical scenarios.

8. The thoracoscopic minimally invasive cardiac surgery navigation system based on multimodal image fusion according to claim 5, characterized in that: The safety warning unit is specifically configured such that when the dynamic risk score exceeds the first threshold, the actuator control unit activates a tactile vibration prompt. When the dynamic risk score exceeds the second threshold, the operating resistance is increased; when the dynamic risk score exceeds the third threshold, the propulsion degree of the device is physically restricted.

Images