Lung nodule surgery simulation system, method, device and medium based on digital twinning

Abstract

The application discloses a lung nodule surgery simulation system and method based on digital twinning, a device and a medium, relates to the technical field of medical digital twinning and surgery simulation, and comprises a depth camera, a dynamic twin containing a surgery gesture dynamic model and a digital lung dynamic model, a display feedback device and a control center, the depth camera is used for acquiring operation data of a first target object; the surgery gesture dynamic model is used for receiving the operation data and mapping the operation data into movement information of a virtual surgical instrument in real time; the control center is used for aligning the surgery gesture dynamic model with the digital lung dynamic model and processing, so that multi-modal interaction instructions are generated according to the interaction state between the movement information and the digital lung dynamic model; and the display feedback device is used for displaying a holographic operation scene and feeding back information according to the multi-modal interaction instructions. The system can generate multi-modal feedback according to the real-time interaction state of the instrument and the lung tissue.

Description

Technical Field

[0001] This invention relates to the field of medical digital twin and surgical simulation technology, and particularly to a surgical simulation system, method, device and medium for lung nodules based on digital twins. Background Technology

[0002] Digital twin and natural interaction technologies are increasingly being used in medical surgical simulation. In lung nodule surgical simulation, existing technologies mainly focus on lung nodule image segmentation, static 3D reconstruction, and human-computer interaction based on markers or wearable devices.

[0003] (1) In terms of lung nodule modeling, patent CN109350265A proposed a puncture positioning method based on marker-assisted mixed reality technology, which achieves millimeter-level spatial positioning through bone-like material markers. Although this method can provide static spatial coordinate alignment, its drawback is that it relies on physical markers, which need to be placed and marked on the human chest, making the operation cumbersome and invasive; the model is static and cannot simulate the dynamic changes of lung nodules with respiratory movements, let alone simulate changes in blood supply and tissue mechanical response.

[0004] (2) In terms of human-computer interaction, patent CN106406544A uses electromyography (EMG) sensors and inertial navigation sensors to achieve gesture recognition and control. However, this method relies on wearable sensors, which limits the doctor's operational flexibility and freedom of aseptic operation. Patent CN114995657A proposes a multimodal fusion natural interaction method, but it is mainly aimed at general robot interaction scenarios and has not been adapted to the high precision and low latency requirements of surgical operations. The accuracy of the interaction feedback is difficult to meet the needs of medical simulation.

[0005] (3) Regarding dynamic modeling of digital twins, patent CN120409049 B proposed a real-time linkage method for digital twin models driven by cyber-physical fusion, but it failed to solve complex physiological dynamics problems in the medical field, such as the movement of lung nodules with respiration, changes in intravascular hemodynamics, and viscoelastic deformation of tissues when squeezed by instruments. These physiological dynamic characteristics are crucial in surgical simulation, but existing medical digital twin models generally lack integrated simulation of such dynamic processes.

[0006] In summary, how to establish a two-way dynamic linkage mechanism between operation and digital lung physiological response, so that the system can generate multimodal feedback based on the real-time interaction between the device and lung tissue, is a technical problem to be solved in this field. Summary of the Invention

[0007] In view of this, the purpose of this invention is to provide a surgical simulation system, method, device, and medium for lung nodules based on digital twins, establishing a bidirectional dynamic linkage mechanism between operation and digital lung physiological response, enabling the system to generate multimodal feedback based on the real-time interaction between the instrument and lung tissue. The specific solution is as follows: In a first aspect, this application discloses a lung nodule surgical simulation system based on digital twins, including a depth camera, a dynamic twin containing a dynamic model of surgical gestures and a dynamic model of the digital lung, a display feedback device, and a control center, wherein... The depth camera is used to acquire operational data of the first target object; The surgical gesture dynamic model is used to receive the operation data and map the operation data into the motion information of virtual surgical instruments in real time; The control center is used to align the surgical gesture dynamic model with the digital lung dynamic model to generate multimodal interaction commands based on the interaction state between the motion information and the digital lung dynamic model. The display feedback device is used to display the holographic operation scene and provide information feedback according to the multimodal interaction instructions.

[0008] Optionally, the digital lung dynamic model integrates any two or more of the following: respiratory motion simulation unit, hemodynamic simulation unit, and tissue mechanics simulation unit.

[0009] Optionally, the digital lung dynamic model includes: The respiratory motion simulation unit is used to construct the motion field of the lung anatomy based on multi-respiratory cycle CT and MR images of a second target object and through a spatiotemporal attention mechanism to determine the current respiratory motion phase. The hemodynamic simulation unit is used to construct a blood supply calculation model for calculating blood flow velocity and pressure distribution based on the blood vessel segmentation results and hemodynamic data of the second target object. The tissue mechanics simulation unit is used to invert the viscoelastic parameters of each region of lung tissue based on the radiomics features and clinical mechanics data of the second target object, and to establish a finite element calculation model for calculating tissue deformation and pressure distribution; wherein, the radiomics features are image features extracted from the CT images and the MR images.

[0010] Optionally, the control center includes: A spatial alignment unit is used to construct a right-handed Cartesian coordinate system with the geometric center of the surgical simulation training cabin as the origin, and to align the three-dimensional coordinates of the surgical gesture dynamic model with the three-dimensional coordinates of the digital lung dynamic model in the right-handed Cartesian coordinate system through a coordinate transformation matrix. The collision detection unit is used to detect the motion information of the virtual surgical instrument and the current state of the digital lung dynamic model in real time, and to record the pressure value of the contact point when the current state is detected to be in contact. The deformation and blood supply calculation unit is used to, when the current state is detected to be a contact state, call the current respiratory motion phase in the respiratory motion simulation unit, the viscoelastic parameters of the tissue mechanics simulation unit and the finite element calculation model and the blood supply calculation model of the hemodynamics simulation unit, and calculate the current tissue deformation, current stress distribution and current blood supply change in real time based on the pressure value of the contact point. The instruction generation unit is used to generate multimodal interactive instructions based on the current tissue deformation and current blood supply changes.

[0011] Optionally, the deformation and blood supply calculation unit includes: The vascular topology reconstruction subunit is used to adjust the vascular tree structure through node deletion and edge reconnection algorithms when a simulated vascular ligation or severance operation is detected, and to update the current blood flow velocity and current pressure distribution in the hemodynamic parameters in real time.

[0012] Optionally, the surgical gesture dynamic model includes: A key point detection unit is used to extract the three-dimensional spatial coordinates of key points of the hand from the operation data; wherein, the key points of the hand are the position points of the hand joints and fingertips; The action recognition unit is used to input the three-dimensional spatial coordinates of continuous frames into a preset long short-term memory network to recognize virtual surgical actions; The motion mapping unit is used to map the virtual surgical actions into motion information of virtual surgical instruments using a forward kinematics algorithm.

[0013] Optionally, the multimodal interaction instructions include any one or more of visual highlighting instructions, tactile vibration instructions, or audio prompt instructions.

[0014] Secondly, this application discloses a method for simulating lung nodule surgery based on digital twins, including: Obtain the operation data of the first target object; The system receives the operation data and maps it into motion information of virtual surgical instruments in real time using a surgical gesture dynamic model. The surgical gesture dynamic model is aligned with the digital lung dynamic model to generate multimodal interaction commands based on the interaction state between the motion information and the digital lung dynamic model; the digital lung dynamic model integrates any two or more of the following: respiratory motion simulation unit, hemodynamic simulation unit, and tissue mechanics simulation unit. Display the holographic operation scene and provide information feedback according to the multimodal interaction instructions.

[0015] Thirdly, this application discloses an electronic device, including: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the steps of the aforementioned disclosed digital twin-based surgical simulation method for lung nodules.

[0016] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed digital twin-based lung nodule surgical simulation method.

[0017] As can be seen, this application discloses a lung nodule surgical simulation system based on digital twins, including a depth camera, a dynamic twin containing a surgical gesture dynamic model and a digital lung dynamic model, a display feedback device, and a control center. The depth camera is used to acquire operation data of a first target object; the surgical gesture dynamic model is used to receive the operation data and map it in real time to the motion information of virtual surgical instruments; the control center is used to align the surgical gesture dynamic model with the digital lung dynamic model to generate multimodal interaction commands based on the interaction state between the motion information and the digital lung dynamic model; the display feedback device is used to display a holographic operation scene and provide information feedback according to the multimodal interaction commands. Thus, by using a depth camera to collect bare-hand operation data without markings, mapping it in real time to virtual instrument motion information via the surgical gesture dynamic model, aligning the gesture model with the digital lung dynamic model integrating respiratory, blood supply, and biomechanical characteristics, generating multimodal commands based on the real-time interaction state between the instruments and lung tissue, and displaying a holographic scene and providing feedback according to the commands, the system achieves this. This solves the problems of traditional methods that rely on physical markers / wearable devices, lack dynamic physiological linkage, and have one-way operation feedback without a closed loop. It enables markerless bare-hand operation, two-way dynamic coupling of physiological reality, and closed-loop multimodal feedback, thereby improving the realism of surgical simulation. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the structure of a lung nodule surgical simulation system based on digital twin disclosed in this application; Figure 2 This application discloses a flowchart of a surgical simulation method for lung nodules based on digital twins. Figure 3 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0021] Digital twin and natural interaction technologies are increasingly being used in the field of medical surgical simulation. In lung nodule surgical simulation, existing technologies mainly focus on lung nodule image segmentation, static 3D reconstruction, and human-computer interaction based on markers or wearable devices.

[0022] Regarding the localization and modeling of pulmonary nodules, patent CN109350265A proposes a puncture localization technique using marker-assisted mixed reality technology. While this method can control errors to the millimeter level, it relies on physical markers made of bone-like materials, which need to be placed and marked on the chest, making the operation cumbersome and invasive. Furthermore, this patent only focuses on static spatial localization and lacks modeling and simulation of the physiological dynamics of pulmonary nodules, such as respiratory movements and changes in blood supply.

[0023] In terms of lung nodule detection and segmentation, patent CN110310281A uses Mask-RCNN deep learning technology to realize lung nodule detection and segmentation in a virtual medical environment. However, this method is mainly for offline processing of static CT (Computed Tomography) images. The model cannot be continuously optimized with data accumulation, lacks evolution capabilities, and does not involve dynamic interaction with doctors in real time.

[0024] Regarding natural human-computer interaction, patent CN114995657A proposes a multimodal fusion natural interaction method for intelligent robots. Although it can combine multimodal features such as audio and vision for intent understanding, this method is not adapted to high-precision, low-latency medical operation scenarios, and the accuracy and real-time performance of its interactive feedback cannot meet the needs of surgical simulation. Patent CN106406544A (China Ordnance Industry Computer Application Technology Research Institute) achieves gesture recognition and control through electromyography sensors and inertial navigation sensors, but it relies on wearable devices, which limits the flexibility and naturalness of the operation.

[0025] In the area of ​​dynamic modeling of digital twins, patent CN120409049 B proposes a real-time linkage method for digital twin models driven by cyber-physical fusion; patent CN202410345650.3 relates to the construction of a digital twin platform for BIM model simulation; and patent CN202411864769.8 focuses on the dynamic configuration and online updating of business models. These patents achieve dynamic simulation in their respective fields, but their technical approaches all originate from the industrial or architectural sectors. They fail to effectively solve complex physiological dynamics problems in the medical field (such as the movement of lung nodules with respiration and dynamic changes in blood supply), nor do they cover the dynamic modeling and recognition of doctors' surgical gestures.

[0026] In summary, existing technologies suffer from the following shortcomings: Static models and lack of dynamic physiological features: Most existing digital models of lung nodules are static or simply dynamic, lacking detailed simulations of key physiological processes such as respiratory movements and hemodynamics, leading to a disconnect between the simulated environment and the real surgical scenario. Unnatural interaction and reliance on specific devices: Interaction methods largely rely on physical markers or wearable sensors, failing to achieve unmarked, bare-handed, high-precision natural interaction, thus limiting the doctor's immersion and operational freedom.

[0027] To this end, the present invention provides a surgical simulation scheme for lung nodules based on digital twins, establishing a two-way dynamic linkage mechanism between operation and digital lung physiological response, enabling the system to generate multimodal feedback based on the real-time interaction status between the instrument and lung tissue.

[0028] like Figure 1 As shown, the present invention provides a lung nodule surgical simulation system based on digital twin, including a depth camera 11, a dynamic twin 12 comprising a surgical gesture dynamic model 121 and a digital lung dynamic model 122, a display feedback device 13, and a control center 14, wherein, The depth camera 11 is used to acquire operation data of the first target object; The surgical gesture dynamic model 121 is used to receive the operation data and map the operation data into the motion information of virtual surgical instruments in real time. The control center 14 is used to align the surgical gesture dynamic model 121 with the digital lung dynamic model 122 to generate multimodal interaction commands based on the interaction state between the motion information and the digital lung dynamic model 122. The display feedback device 13 is used to display the holographic operation scene and provide information feedback according to the multimodal interaction instructions.

[0029] It is understood that the depth cameras consist of four high-frame-rate depth cameras, deployed at 0°, 45°, 90°, and 135° on the ceiling of the simulated operating room, with a frame rate of no less than 120fps and a depth accuracy better than 0.5mm. The four cameras simultaneously acquire depth images and RGB (Red, Green, Blue) images of the first target object during bare-hand manipulation, covering eight surgical actions such as grasping, separation, and suturing, providing multi-view operational data for gesture recognition and motion mapping.

[0030] The surgical gesture dynamic model 121 includes a key point detection unit, used to extract the three-dimensional spatial coordinates of hand key points from the operation data; wherein, the hand key points are the position points of hand joints and fingertips; it can be understood that after receiving depth image and RGB image data, the key point detection unit runs the MediaPipe hand key point detection algorithm to extract the three-dimensional coordinates of 21 hand key points frame by frame in each image. The hand key points cover all major joints and fingertips of the hand, including the wrist joint, each metacarpophalangeal joint, interphalangeal joint, and fingertip point. For the same key point observed simultaneously by multiple cameras in the same frame, the system uses the three-dimensional coordinates calculated by multiple cameras as input, and solves for the optimal world coordinates using the least squares method, so that the sum of the squared residuals between the optimal coordinates and the coordinates calculated by each camera observation is minimized, thereby effectively reducing the occlusion error or depth noise that may be generated by a single viewpoint. In this way, the key point detection unit outputs the accurate three-dimensional spatial coordinates of 21 key points of the first target object's hand in each frame image, providing high-fidelity temporal data for action recognition.

[0031] The surgical gesture dynamic model 121 includes: an action recognition unit, used to input the three-dimensional spatial coordinates of consecutive frames into a preset long short-term memory network to recognize virtual surgical actions; that is, the action recognition unit receives the three-dimensional coordinates of 21 key points output by the key point detection unit for each frame, and concatenates the coordinates of several consecutive frames (e.g., 30 frames, corresponding to approximately 0.25 seconds) in chronological order into a high-dimensional temporal feature vector, which contains information such as the motion trajectory, speed, and acceleration of each key point of the hand changing over time. Then, the feature vector is input into a pre-trained long short-term memory (LSTM) network model, which can effectively capture long-term dependencies and dynamic patterns in hand movements. The LSTM model outputs the recognition probability of 8 types of surgical actions, including but not limited to core operations in lung nodule surgery such as grasping, separation, suturing, electrocoagulation, cutting, traction, rotation, and release. The system takes the category with the highest probability as the recognition result at the current moment, thereby determining the type of surgical action being performed by the first target object in real time. In this way, the system can understand the operational intent of the first target object and upgrade the original coordinate data into semantic surgical action labels.

[0032] The surgical gesture dynamic model 121 includes a motion mapping unit, used to map the virtual surgical actions into motion information of virtual surgical instruments using a forward kinematics algorithm. Understandably, the motion mapping unit receives the surgical action recognition results output by the action recognition unit, and combines this with the real-time hand keypoint coordinates output by the keypoint detection unit, using a forward kinematics algorithm to construct a motion model of the virtual surgical instruments. This algorithm establishes a kinematic chain relationship between the joint positions of the first target object's hand and the end effector of the virtual instrument (e.g., the tip of a surgical forceps, an electrosurgical head, etc.), and calculates the corresponding position, posture, and opening / closing angle of the virtual instrument in three-dimensional space based on the hand's translation, rotation, opening / closing movements.

[0033] To eliminate motion unsmoothness caused by hand tremors or image noise, the motion mapping unit integrates a Kalman filter algorithm to smooth the virtual instrument motion trajectory in real time between consecutive frames. The Kalman filter recursively predicts the optimal state of the current frame based on the optimal estimate from the previous frame and the observation from the current frame, thus significantly reducing jagged transitions in the trajectory while maintaining response speed. The motion mapping unit outputs stable, smooth, and accurate virtual surgical instrument motion information, including the three-dimensional coordinates of the instrument's end effector, Euler angles (or quaternions), and jaw opening and closing parameters, achieving a 1:1 precise mapping between real bare-hand movements and virtual instrument motion. The primary target user does not need to wear any sensors or markers; they can manipulate instruments in the virtual surgical environment solely through natural hand movements, achieving a highly immersive operating experience.

[0034] In this way, through the collaborative work of the three units mentioned above, the surgical gesture dynamic model gradually transforms the raw image data collected by the depth camera into virtual instrument motion information that can be used to drive digital lung interaction.

[0035] The digital lung dynamic model 122 integrates any two or more of the following: respiratory motion simulation unit, hemodynamic simulation unit, and tissue mechanics simulation unit. It is understood that the digital lung dynamic model 122 is another component of the dynamic twin, used to simulate the physiological dynamic response of the lungs of the second target object during surgery. This model integrates at least two or more of the physiological dynamic characteristics from respiratory motion simulation, hemodynamic simulation, and tissue mechanics simulation. Through the coupling of multiple physiological dynamic characteristics, the digital lung dynamic model can realistically reproduce the displacement and deformation of lung tissue during the respiratory cycle, the changes in blood flow velocity and pressure distribution within the vascular system, and the viscoelastic stress and strain response generated in the tissue upon instrument contact, thereby providing a biorealistic interactive environment for surgical simulation.

[0036] In this embodiment of the invention, the digital lung dynamic model integrates all three physiological dynamic characteristics mentioned above, namely, it includes a respiratory motion simulation unit, a hemodynamic simulation unit, and a tissue mechanics simulation unit, in order to approximate the physiological and mechanical behavior of the real human lung to the greatest extent. The three units are described in detail below.

[0037] Specifically, the digital lung dynamic model 122 includes a respiratory motion simulation unit, used to construct a motion field of the lung anatomy based on multi-respiratory cycle CT and MR images of a second target object and through a spatiotemporal attention mechanism to determine the current respiratory motion phase. It can be understood that the respiratory motion simulation unit is used to construct a continuous nonlinear motion field of the lung anatomy based on multi-respiratory cycle CT and MR (Magnetic Resonance) images of the second target object and through a spatiotemporal attention mechanism to determine the current respiratory motion phase. First, time-series CT images (4D-CT) and time-series MR images (4D-MR) of multiple complete respiratory cycles of the patient are acquired. CT images provide high-resolution lung tissue density information, and MR images provide excellent soft tissue contrast. These are aligned by time points using a registration algorithm based on mutual information to form a fused 4D image sequence. Further, based on the CT images, a dynamic global thresholding algorithm is used to lock the entire lung region, excluding background elements such as the chest wall and bones. The local area was further divided into 3×3×3 voxel blocks. The mean and standard deviation of the CT value (HU) were calculated for each region. The segmentation threshold for lung tissue was adaptively adjusted to mean ± standard deviation, and the threshold for lung nodules was adjusted from mean + standard deviation to mean + 3 × standard deviation, achieving fine segmentation of lung parenchyma and nodules. Based on the periodic change curve of lung volume over time, a respiratory peak detection algorithm was used to align the inspiratory and expiratory peaks of multiple respiratory cycles. The Lucas-Kanade optical flow method was used to extract the displacement vectors of each voxel in the lung from the 4D images, and the lung was divided into high-ventilation, intermediate-ventilation, and low-ventilation zones using the K-means clustering algorithm to reflect the differences in respiratory motion among different lobes and segments. An improved spatiotemporal attention U-Net (Convolutional Networks for Biomedical Image Segmentation) model was constructed. Based on the classic U-Net encoder-decoder structure, this model adds a spatiotemporal dual-branch attention module and a lung ventilation functional partitioning mask layer. The input is 4D-CT / MR fusion data from multiple respiratory cycles (size [T, H, W, D], where T is the number of phases, and H, W, and D are the spatial dimensions). The output is the displacement vector (i.e., motion field) of each voxel in the next time step. The temporal attention branch consists of three temporal convolutional layers with kernel sizes of 3×1×1, 5×1×1, and 3×1×1 (convolution only along the time dimension), each followed by batch normalization and ReLU activation. This branch extracts temporal features from the input 4D data and outputs a temporal attention weight map of the same size as the input. Voxels with larger values ​​in the weight map indicate that the motion pattern at that location is more representative during the respiratory cycle, resulting in a higher response during subsequent encoding.Through this branch, the network can automatically learn the different features of the inspiratory phase, expiratory phase, and transition phase, avoiding the tedious manual annotation of respiratory phases. The temporal attention branch extracts the temporal motion features of 4D images with multiple respiratory cycles through temporal convolution, capturing the dynamic changes of lung tissue with respiration. The spatial attention branch averages the input 4D data over time to obtain a 3D spatial feature map. Then, two convolutional layers (1×1×1 kernels, channel count halved and then restored) generate a spatial attention weight map, which is normalized to [0,1] using the Sigmoid function. The weight map is then element-wise multiplied with the original spatial feature map, and the result is added back to the original feature map via residual connections, ultimately outputting a spatial feature map that enhances key anatomical locations such as the lung apex, hilum, and subpleural region. This branch makes the network pay more attention to areas with large motion amplitude and frequent surgical manipulations (such as blood vessels near the lung hilum and subpleural nodules). Therefore, the spatial attention branch enhances the spatial features of different anatomical locations such as the lung apex, hilum, and subpleural region through spatial convolution (3×3×3) and attention weight allocation. The lung ventilation zoning mask layer pre-extracts the patient's high, intermediate, and low ventilation functional zones from lung ventilation-perfusion imaging (e.g., MRI or nuclear medicine ventilation scans) and generates corresponding masks. In the final layer of the network encoder, the mask image is downsampled to the same resolution as the feature image and then concatenated with the feature image as input to the decoder. Through this mask layer, the network can identify lung tissue pixel features in different ventilation functional zones and assign differentiated motion weights to different regions: high ventilation zones allow for larger displacement changes, while low ventilation zones restrict displacement changes. This design overcomes the limitations of traditional U-Net, which can only extract single spatial features and cannot adapt to the functional-anatomical-motor correlation of lung tissue. The lung ventilation functional zone mask layer enables the model to accurately identify pixel features in high, medium, and low ventilation regions, achieving a two-dimensional fusion of physiological functional features and spatiotemporal anatomical features. Finally, the model outputs regionalized, personalized, continuous dynamic motion fields for different anatomical structures such as the tracheal tree, vascular tree, lung lobes, and lung segments. Based on these motion fields and combined with respiratory wave curves, a nonlinear interpolation method (binding the interpolation step size to the respiratory phase) is used to calculate the respiratory motion phase (e.g., end of inspiration, end of expiration, mid-inspiration, etc.) in real time, providing a time reference for other units when calculating tissue deformation and blood supply changes.

[0038] The digital lung dynamic model 122 includes a hemodynamic simulation unit, used to construct a blood supply calculation model for calculating blood flow velocity and pressure distribution based on the vascular segmentation results and hemodynamic data of the second target object. Specifically, the hemodynamic simulation unit constructs a blood supply calculation model for calculating blood flow velocity and pressure distribution based on the vascular segmentation results and hemodynamic data of the second target object. Based on the time-series CT or MR vascular imaging data of the second target object, an improved U-Net++ vascular segmentation algorithm (adding residual links and attention gating to standard U-Net++) is used to automatically segment the pulmonary arteries and pulmonary veins. Topological parameters such as vessel diameter, branch angle, and tortuosity are extracted from the segmentation results to construct a complete pulmonary vascular tree structure (nodes represent vascular bifurcation points, and edges represent vascular segments). The acquired hemodynamic data of the second target object (including blood flow velocity measured by Doppler ultrasound and flow rate data measured by magnetic resonance imaging) are fused, and the vascular topological parameters are used as dynamic boundary conditions to construct a hemodynamic model using the improved Navier-Stokes equations. This model can calculate the blood flow velocity vector field and pressure distribution field at every location within the vascular tree. Due to the complex non-Newtonian fluid properties and pulsating flow characteristics of the pulmonary vascular system, the improved Navier-Stokes equations introduce dynamic viscosity correction terms related to vessel diameter and blood flow velocity to improve computational accuracy. This blood supply calculation model supports real-time parameter updates after changes in vascular topology. When simulating surgical procedures (such as ligating or cutting a vascular branch), the system can quickly reconstruct the hemodynamic equations based on the new topology (deleting corresponding nodes and edges and reconnecting edges), recalculating the blood flow velocity and pressure distribution in the distal region, and realistically reflecting the impact of surgery on lung tissue blood supply.

[0039] The digital lung dynamic model 122 includes: a tissue mechanics simulation unit, used to invert the viscoelastic parameters of each region of lung tissue based on the radiomics features and clinical mechanics data of the second target object, and to establish a finite element calculation model for calculating tissue deformation and pressure distribution; wherein, the radiomics features are image features extracted from the CT images and the MR images. It can be understood that the tissue mechanics simulation unit is used to invert the viscoelastic parameters of each region of lung tissue based on the radiomics features and clinical mechanics data of the second target object, and to establish a finite element calculation model for calculating tissue deformation and pressure distribution. First, from the aforementioned registered 4D-CT and 4D-MR images, based on the segmentation results of lung tissue, pulmonary nodules, and pulmonary vessels, three types of radiomics features are extracted: ① first-order statistical features (grayscale mean, variance, skewness, kurtosis, etc.); ② morphological features (volume, surface area, sphericity, compactness, etc.); ③ texture features (contrast, correlation, energy, homogeneity, etc. of the grayscale co-occurrence matrix). The above features are concatenated in time sequence to form a high-dimensional radiomics feature vector. Clinical biomechanical experimental data of the second target object are collected, such as stress-strain curves of lung tissue obtained through indentation tests or uniaxial tensile tests. Using radiomics features as input and biomechanical experimental data as labels, a gradient boosting regression model is trained. This model learns the mapping relationship between imaging features and tissue biomechanical properties, thereby predicting personalized viscoelastic material parameters for each voxel or region of the digital lung, including elastic modulus, Poisson's ratio, relaxation time, and viscosity coefficient. Based on these viscoelastic parameters, the digital lung is discretized into a mesh model composed of a large number of tetrahedral or hexahedral elements using an explicit finite element method. Each element is assigned a corresponding viscoelastic constitutive equation (e.g., Maxwell model or Kelvin-Voigt model) to describe the stress-strain time response of the tissue under external force. This finite element calculation model can receive external input boundary conditions such as contact pressure, contact position, and application time, and calculate the displacement (i.e., deformation) of each node within the tissue and the stress distribution of each element in real time. The network employs a mean squared error loss function (the difference between predicted displacement and actual optical flow displacement), uses the Adam optimizer, has an initial learning rate of 0.001, and is trained for 100 epochs. The training data consists of 4D-CT / MR fusion data from 50 patients, each containing 10 respiratory phases. After training, the network can quickly infer a continuous motion field from any temporal input, and then generate displacement vectors for any respiratory phase through nonlinear interpolation, thereby achieving continuous, nonlinear, and regional dynamic motion simulation of the digital lung.

[0040] In this way, the calculations of the tissue mechanics simulation unit are not performed independently, but rather within the current respiratory phase provided by the respiratory motion simulation unit, taking into account the pre-stretching state of the lung tissue caused by respiration. At the same time, changes in blood supply (such as changes in vascular pressure) also affect the tissue filling degree, thereby altering its mechanical response. The coupling of the three units is synchronized through a unified physical space coordinate system and time step, ensuring the physiological consistency of the calculation results.

[0041] Control Center 14 is the core of the system's computation and control, connected to the depth camera, dynamic twin (including the surgical gesture dynamic model and the digital lung dynamic model), and display feedback device. The control center is responsible for aligning the surgical gesture dynamic model and the digital lung dynamic model in a unified spatial coordinate system, detecting the interaction status between the virtual surgical instruments and the digital lung in real time, calling various physiological simulation units in the digital lung dynamic model to calculate tissue deformation, stress distribution, and blood supply changes, and generating multimodal interaction commands based on the calculation results to drive the display feedback device to provide visual, tactile, and audio feedback to the doctor. The control center specifically includes a spatial alignment unit, a collision detection unit, a deformation and blood supply calculation unit, and a command generation unit. The deformation and blood supply calculation unit also includes a vascular topology reconstruction subunit. The specific units are as follows: The control center 14 includes a spatial alignment unit, used to construct a right-handed Cartesian coordinate system with the geometric center of the surgical simulation training cabin as the origin, and to align the three-dimensional coordinates of the surgical gesture dynamic model with the three-dimensional coordinates of the digital lung dynamic model in the right-handed Cartesian coordinate system through a coordinate transformation matrix. In essence, the spatial alignment unit first establishes a right-handed Cartesian coordinate system (X-axis to the right, Y-axis up, Z-axis forward) with the geometric center of the surgical simulation training cabin as the origin. This coordinate system serves as the reference space for the entire system. For the surgical gesture dynamic model, the spatial alignment unit acquires the intrinsic parameter matrix (including focal lengths fx, fy and optical center coordinates (cx, cy)) and extrinsic parameter matrix (rotation matrix R, translation vector T) of the depth camera. Based on these parameters, the three-dimensional coordinates of the hand key points obtained through the surgical gesture dynamic model (located in the camera coordinate system) are transformed to the cabin's physical space coordinate system. The transformation formula is: P_world = R·P_camera + T. Since multiple depth cameras acquire data from different angles, the spatial alignment unit uses Kalman filtering to dynamically correct the transformation matrix, suppressing coordinate errors caused by camera shake or temperature drift. For the digital lung dynamic model, the spatial alignment unit reads the origin information from the DICOM file of the patient's CT / MR images and transforms the voxel coordinates in the image coordinate system to the cockpit physical space coordinate system. To improve alignment accuracy, the system also uses manually marked anatomical feature points such as the lung apex, carina, and diaphragm top as matching anchor points to finely adjust the transformation results. Similarly, Kalman filtering is used to suppress noise in the transformation matrix, ensuring accurate registration between the medical image space and the physical space. After the above processing, the virtual surgical instrument motion information output by the surgical gesture dynamic model and the lung anatomical structure position information output by the digital lung dynamic model are both in the same right-handed Cartesian coordinate system, laying a spatial foundation for subsequent collision detection and deformation calculation.

[0042] The control center 14 includes a collision detection unit, used to detect the motion information of the virtual surgical instruments and the current state of the digital lung dynamic model in real time, and to record the pressure value of the contact point when the current state is detected to be a contact state. It is understood that the collision detection unit uses a voxel-based collision detection algorithm. First, the lung tissue, lung nodules, blood vessels, and other structures described by the digital lung dynamic model are discretized into a three-dimensional voxel mesh (voxel side length can be set to 0.5mm to 1mm). The virtual surgical instruments (surgical forceps, electrosurgical head) are represented by their geometric bounding boxes or point clouds. Within each time step (e.g., 10ms), the collision detection unit determines whether there is an intersection between the bounding box or point cloud of the virtual surgical instruments and the voxel mesh of the digital lung, based on the unified coordinate system provided by the spatial alignment unit. If there is an intersection, it is determined to be a contact state. When contact is detected, the collision detection unit further obtains the viscoelastic parameters (elastic modulus, hardness, etc.) of the contact point area from the tissue mechanics simulation unit of the digital lung dynamic model, and combines this with the instrument movement speed, angle, and other information output by the surgical gesture dynamic model to estimate the instantaneous pressure value of the contact point. The pressure value is recorded and transmitted to the deformation and blood supply calculation unit. For multi-point contact scenarios, the collision detection unit generates a list of contact points, each containing attributes such as coordinates, pressure value, and direction of application. It should be noted that the collision detection unit is only responsible for detecting contact and recording pressure values; it does not handle deformation calculations. Deformation calculations are performed by the subsequent deformation and blood supply calculation unit.

[0043] The control center 14 includes a deformation and blood supply calculation unit, which, when the current state is detected to be a contact state, calls the current respiratory motion phase in the respiratory motion simulation unit, the viscoelastic parameters of the tissue mechanics simulation unit, the finite element calculation model, and the blood supply calculation model of the hemodynamics simulation unit, and calculates the current tissue deformation, current stress distribution, and current blood supply change in real time based on the pressure value of the contact point. It can be understood that when the collision detection unit determines that the state is a contact state, the deformation and blood supply calculation unit calls the respiratory motion simulation unit, tissue mechanics simulation unit, and hemodynamics simulation unit in the digital lung dynamic model, and calculates the current tissue deformation, current stress distribution, and current blood supply change in real time in combination with the pressure value of the contact point.

[0044] Specifically, the respiratory motion simulation unit is invoked, and the deformation and blood supply calculation unit first obtains the respiratory motion phase (such as the end of inspiration, the end of expiration, or an intermediate phase) from the respiratory motion simulation unit. Since lung tissue has different initial deformation states at different respiratory phases (e.g., lung tissue is stretched during inspiration and retracts during expiration), this phase information will be used as the initial boundary condition for tissue mechanics calculation.

[0045] Furthermore, the tissue mechanics simulation unit is invoked, and the deformation and blood supply calculation unit obtains the following information from it: viscoelastic parameters (elastic modulus, Poisson's ratio, relaxation time, viscosity coefficient, etc.) of the region where the contact point is located, and the corresponding finite element calculation model (i.e., a mesh model composed of tetrahedral or hexahedral elements and the viscoelastic constitutive equation). Then, using the contact point pressure value provided by the collision detection unit as the external force input, the Newton-Raphson iterative method is used to solve the nonlinear finite element equation, obtaining the displacement (i.e., deformation) of each node within the tissue and the stress distribution of each element. During the calculation process, the deformation and blood supply calculation unit also considers the initial strain field caused by respiratory motion, making the calculation results more consistent with physiological reality.

[0046] Secondly, the hemodynamics simulation unit is invoked, and the deformation and blood supply calculation unit obtains the blood supply calculation model (including the vascular tree topology, blood flow velocity, and pressure distribution of each vascular segment) from the hemodynamics simulation unit. When tissue deformation causes changes in vascular geometry (e.g., vascular compression and narrowing), the deformation and blood supply calculation unit feeds back the updated vascular diameter and tortuosity to the hemodynamics simulation unit, recalculating local blood flow velocity and pressure. Furthermore, when simulated vascular ligation or transection is detected, the deformation and blood supply calculation unit activates its internal vascular topology reconstruction subunit.

[0047] Based on the above calls and calculations, the deformation and blood supply calculation unit outputs three key quantities at each time step: current tissue deformation (represented in the form of displacement field or deformation mesh), current stress distribution (represented in the form of stress cloud map or stress value of each element), and current blood supply changes (including changes in blood flow velocity, changes in pressure distribution, and possible ischemic area identification). These outputs are transmitted to the instruction generation unit in real time and are also sent back to the display feedback device for visual presentation (e.g., displaying deformation effects, stress color mapping, blood flow direction arrows, etc.).

[0048] The control center 14 includes an instruction generation unit, used to generate multimodal interactive instructions based on the current tissue deformation and current blood supply changes. The multimodal interactive instructions include any one or more of visual highlighting instructions, tactile vibration instructions, or audio prompts. It is understood that the instruction generation unit converts the received tissue deformation data into visual instructions; for example, areas with large deformation are represented by a red heatmap, and areas with small deformation are represented by blue; areas where stress exceeds a tissue strength threshold are highlighted with flashing light to alert the first target object to avoid excessive force. For changes in blood supply, the instruction generation unit generates the following visual instructions: the area distal to the ligated or severed blood vessel changes color to dark purple (representing ischemia), areas with reduced blood flow velocity are represented by thin arrows, and areas with blocked blood flow are marked with a cross. Simultaneously, the instruction generation unit generates tactile instructions based on the pressure value at the contact point and the tissue viscoelastic parameters: when the pressure exceeds a preset safety threshold, the instruction generation unit sends a vibration signal to the force feedback device in the display feedback device, with the frequency and amplitude proportional to the degree of pressure exceeding the limit; when simulating cutting hard tissue (e.g., calcified nodules), high-frequency damped vibration is sent. In addition, the instruction generation unit also generates audio instructions. When the virtual instrument approaches areas with dense blood vessels or dangerous areas (large blood vessels, nerves), it emits a low-frequency warning sound; when accidental damage (cutting a blood vessel) occurs, it emits a rapid alarm sound accompanied by a voice prompt "Caution: Blood vessel damage." All multimodal interactive instructions use structured data formats (JSON or Protobuf) and are transmitted in real time to the display feedback device via a high-speed data interface (USB 3.0 or Ethernet). The instruction update frequency is no less than 60Hz to ensure the continuity of the doctor's operation and the immediacy of the feedback.

[0049] In this way, the control center provides the other two units with the current respiratory motion phase and the position and velocity of each anatomical structure at the current moment through the respiratory motion simulation unit; the hemodynamic simulation unit provides the intravascular pressure (as the internal boundary condition for tissue stress) to the tissue mechanics simulation unit; and the tissue mechanics simulation unit feeds back the geometric dimensions of the blood vessels after deformation (affecting blood flow resistance) to the hemodynamic simulation unit. Through this collaboration, the digital lung dynamic model can generate physiologically consistent lung tissue deformation, changes in vascular blood supply, and stress distribution in real time when the surgeon applies operations using virtual surgical instruments, and output these dynamic responses to the control center for generating multimodal interactive commands.

[0050] Furthermore, the deformation and blood supply calculation unit includes a vascular topology reconstruction subunit, used to adjust the vascular tree structure through node deletion and edge reconnection algorithms when a simulated vascular ligation or transection operation is detected, and to update the current blood flow velocity and current pressure distribution in hemodynamic parameters in real time. It is understood that the vascular topology reconstruction subunit is triggered when the collision detection unit detects an interaction between a virtual surgical instrument and a vascular segment, and the surgical gesture dynamic model identifies the action as "ligation" or "transection". This subunit first obtains the current vascular tree structure (nodes represent branching points, edges represent vascular segments, and edge attributes include diameter, length, and resistance coefficient) from the hemodynamic simulation unit. For ligation operations, the subunit finds the edge corresponding to the target vascular segment, marks it as "occluded," and inserts a node with infinite resistance at the midpoint of the edge, equivalent to complete blood flow blockage. Then, through a graph-based node deletion and edge reconnection algorithm, all unnecessary branches downstream of the occluded vascular segment are removed, and the blood supply source in the downstream region is redirected to other unoccluded collateral vessels (if they exist). Reconstruction latency is controlled within 20ms to ensure real-time performance. For transection operations, the subunit first performs a occlusion process similar to ligation, then physically deletes the edges corresponding to the severed vessel segment and its downstream subtrees. For the two severed ends, the subunit adds boundary nodes at the ends and sets the pressure to zero or a constant value (simulating open vessel bleeding). Simultaneously, a connectivity check is performed on the remaining vessel tree to ensure there are no isolated nodes. After completing the topology reconstruction, the vessel topology reconstruction subunit returns the new vessel tree structure (including node and edge updates) to the hemodynamic simulation unit. The hemodynamic simulation unit immediately resolves the Navier-Stokes equations based on the new topology, calculating the current blood flow velocity and current pressure distribution for each vessel segment. These updated hemodynamic parameters are used by the deformation and blood supply calculation unit to output the current blood supply changes, and finally presented to the physician through the instruction generation unit (e.g., a darker color in the distal area of ​​ligation indicates ischemia, and a spurting effect is displayed at the transection site).

[0051] Through the above mechanism, the deformation and blood supply calculation unit and its vascular topology reconstruction subunit can realistically simulate the dynamic changes in blood supply during pulmonary vascular surgery, enhancing the physiological realism and training value of the surgical simulation.

[0052] It is important to note that after the surgical simulation training is completed, the system automatically extracts the time-series data and model response data throughout the entire operation, runs a multi-dimensional operation evaluation algorithm to generate a quantitative training report. The evaluation feedback process includes the following steps: Operation Time Scoring: The system extracts the system timestamp from the start of the surgical simulation to the end of the operation and calculates the total operation time. The industry average operation time for different surgical procedures is used as the benchmark of 100 points (e.g., average time for wedge resection of the lung is 20 minutes, average time for lobectomy is 60 minutes). 5 points are deducted for every 5% overtime, up to a maximum deduction of 0. No points are awarded if the operation is completed ahead of schedule (maximum score 100). For example, if a doctor completes a wedge resection of the lung in 25 minutes, exceeding the time limit by 25%, 5 x 5 = 25 points are deducted, resulting in a score of 75.

[0053] Tissue vascular damage severity scoring: Tissue damage scoring: An improved region growing algorithm is used, with contact points recorded by collision detection units as seed points, and a threshold of 0.5 mm for 3D region growing to calculate the damaged tissue volume. A base score of 50 is used, with 1 point deducted for every 10 mm³ of damage (adjustable), down to a minimum of 0 points. Vascular damage scoring: From the vascular segmentation mask, vascular segments overlapping with the damaged tissue mask are selected. A skeleton extraction algorithm is used to extract the centerline of the damaged vessel and calculate its length. A base score of 50 is used, with 5 points deducted for every 1 mm of damage length. The minimum score from both tests is taken as the score for this item (maximum 100). For example, a 50 mm³ damage volume deducts 5 points, and a 3 mm damaged vessel length deducts 15 points, resulting in a score of 85 points for this item.

[0054] Operational standardization scoring, trajectory similarity: Standard surgical action trajectories of three senior thoracic surgeons were collected to generate standard trajectory templates. A dynamic time warping algorithm was used to calculate the distance between the surgeon's actual trajectory and the standard template, scoring based on similarity: ≥80% similarity = 50 points, ≤60% similarity = 0 points, with intermediate linear interpolation. Suture consistency: The coordinates of the suture needle landing points were extracted from the operation time sequence data, and the standard deviation of the distance between adjacent landing points was calculated. Standard deviation ≤1mm = 50 points, standard deviation ≥2mm = 0 points, with intermediate linear interpolation. The two scores were added together to obtain the total score for this section (out of 100).

[0055] Surgical completion score, nodule residue: The initial nodule volume is calculated based on a voxel counting algorithm. The residual nodule area is segmented post-operatively, and the percentage of residual volume is calculated. Residual volume ≤ 5% receives 80 points; residual volume ≥ 20% receives 0 points, with intermediate values ​​calculated using linear interpolation. Blood flow occlusion: Blood flow data distal to the target vessel after ligation is extracted using a hemodynamic simulation unit, and blood flow velocity and occlusion rate are calculated. Distal blood flow velocity ≤ 1 cm / s and occlusion rate ≥ 95% receive 20 points; distal blood flow velocity ≥ 5 cm / s and occlusion rate ≤ 85% receive 0 points, with intermediate values ​​calculated using linear interpolation. The two scores are added together to obtain the total score for this section (out of 100).

[0056] The overall scoring and report generation process uses the analytic hierarchy process (AHP) to determine the weights of the four scoring components. Example weights: operation time 20%, damage severity 30%, standardization 30%, completion rate 20%. A weighted total score is calculated, with a maximum of 100 points. ≥80 points is excellent, 60-79 points is satisfactory, and ≤60 points indicates room for improvement.

[0057] The system automatically generates training reports, including scores for each indicator and comparison curves with industry standard values; specific optimization suggestions, such as "operation speed is too fast, it is recommended to reduce the suturing speed to 3mm / s" and "the length of accidentally damaged blood vessels exceeds the standard, it is recommended to strengthen training in identifying dangerous areas," etc. The report can be printed out as a paper copy via a connected printer or saved as a PDF. This assessment and feedback mechanism helps doctors quantitatively understand their own skill level and conduct targeted training.

[0058] As can be seen, this application discloses a lung nodule surgical simulation system based on digital twins, including a depth camera, a dynamic twin containing a surgical gesture dynamic model and a digital lung dynamic model, a display feedback device, and a control center. The depth camera is used to acquire operation data of a first target object; the surgical gesture dynamic model is used to receive the operation data and map it in real time to the motion information of virtual surgical instruments; the control center is used to align the surgical gesture dynamic model with the digital lung dynamic model to generate multimodal interaction commands based on the interaction state between the motion information and the digital lung dynamic model; the display feedback device is used to display a holographic operation scene and provide information feedback according to the multimodal interaction commands. Thus, by using a depth camera to collect bare-hand operation data without markings, mapping it in real time to virtual instrument motion information via the surgical gesture dynamic model, aligning the gesture model with the digital lung dynamic model integrating respiratory, blood supply, and biomechanical characteristics, generating multimodal commands based on the real-time interaction state between the instruments and lung tissue, and displaying a holographic scene and providing feedback according to the commands, the system achieves this. This solves the problems of traditional methods that rely on physical markers / wearable devices, lack dynamic physiological linkage, and have one-way operation feedback without a closed loop. It enables markerless bare-hand operation, two-way dynamic coupling of physiological reality, and closed-loop multimodal feedback, thereby improving the realism of surgical simulation.

[0059] In this embodiment, the system has a built-in surgical scenario configuration function on the computer, which allows the first target object (such as a training teacher or doctor) to customize lung nodules and surgery-related parameters according to teaching or training needs, so as to generate diverse simulated training scenarios. Specifically, the first target object can configure the following parameters through a graphical interface: Nodule parameters include nodule size (e.g., diameter 5mm, 10mm, 20mm, etc.), location (e.g., lobe, segment, subpleural, etc.), and benign / malignant type (benign nodule, early adenocarcinoma, metastatic tumor, etc.); Virtual surgical methods include common lung nodule procedures such as wedge resection, segmentectomy, lobectomy, radiofrequency ablation, and microwave ablation; Physiological baseline of the second target subject: different breathing modes (normal breathing, shallow and rapid breathing) and different hemodynamic states (normal blood pressure, hypertension, etc.) can be selected.

[0060] Based on the above configuration parameters, the system automatically calls upon the corresponding geometric, kinematic, mechanical, and blood supply parameters from the digital lung dynamic model to generate a patient digital lung instance that meets the set conditions. For example, if the user sets a 10mm ground-glass nodule located in the right upper lobe and intends to perform a segmentectomy, the system will adjust the CT value and texture features of the nodule area and assign an appropriate range of motion to the area in the respiratory motion simulation unit. Simultaneously, it will pre-set the vascular tree branches of the corresponding lung segment for surgical operations. After configuration, the system presents the corresponding simulation scenario through a display feedback device, allowing doctors to begin training. This function enables the same system to simulate multiple clinical scenarios, enriching the training content.

[0061] like Figure 2 As shown, the present invention also discloses a surgical simulation method for lung nodules based on digital twins, comprising: Step S11: Obtain the operation data of the first target object; Step S12: Receive the operation data and map the operation data into the motion information of virtual surgical instruments in real time through the surgical gesture dynamic model; Step S13: Align the surgical gesture dynamic model with the digital lung dynamic model to generate multimodal interaction commands based on the interaction state between the motion information and the digital lung dynamic model; the digital lung dynamic model integrates any two or more of the following: respiratory motion simulation unit, hemodynamic simulation unit, and tissue mechanics simulation unit. Step S14: Display the holographic operation scene and provide information feedback according to the multimodal interaction instructions.

[0062] For more detailed processing procedures in steps S11 to S14, please refer to the aforementioned disclosed embodiments, which will not be repeated here.

[0063] Therefore, by constructing a digital lung model that simulates breathing, blood supply, and tissue characteristics, and establishing a two-way dynamic simulation model of surgical instrument movement and respiratory movement, hemodynamics, and tissue mechanics, we can immerse ourselves in the real response of the lung during actual surgical operations, thereby achieving preoperative planning optimization and surgical simulation training.

[0064] Furthermore, embodiments of this application also disclose an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0065] Figure 3 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the digital twin-based lung nodule surgical simulation method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0066] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0067] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0068] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0069] The operating system 221 manages and controls the various hardware devices and computer programs 222 on the electronic device 20 to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. It can be Windows Server, Netware, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing the digital twin-based lung nodule surgical simulation method disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the electronic device from external devices, as well as data collected by its own input / output interface 25.

[0070] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed method for simulating lung nodule surgery based on digital twins. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0071] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0072] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly in hardware, software modules executed by a processor, or a combination of both. The software module may be located in random access memory (RAM), memory, read-only memory (ROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, CD-ROMs (Compact Disc-Read Only Memory), or any other form of storage medium known in the art.

[0073] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0074] The solution provided by the present invention has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A surgical simulation system for lung nodules based on digital twins, characterized in that, It includes a depth camera, a dynamic twin containing dynamic models of surgical gestures and digital lungs, display feedback devices, and a control center. The depth camera is used to acquire operational data of the first target object; The surgical gesture dynamic model is used to receive the operation data and map the operation data into the motion information of virtual surgical instruments in real time; The control center is used to align the surgical gesture dynamic model with the digital lung dynamic model to generate multimodal interaction commands based on the interaction state between the motion information and the digital lung dynamic model. The display feedback device is used to display the holographic operation scene and provide information feedback according to the multimodal interaction instructions.

2. The lung nodule surgical simulation system based on digital twin according to claim 1, characterized in that, The digital lung dynamic model integrates any two or more of the following: respiratory motion simulation unit, hemodynamic simulation unit, and tissue mechanics simulation unit.

3. The lung nodule surgical simulation system based on digital twin according to claim 2, characterized in that, The digital lung dynamic model includes: The respiratory motion simulation unit is used to construct the motion field of the lung anatomy based on multi-respiratory cycle CT and MR images of a second target object and through a spatiotemporal attention mechanism to determine the current respiratory motion phase. The hemodynamic simulation unit is used to construct a blood supply calculation model for calculating blood flow velocity and pressure distribution based on the blood vessel segmentation results and hemodynamic data of the second target object. The tissue mechanics simulation unit is used to invert the viscoelastic parameters of each region of lung tissue based on the radiomics features and clinical mechanics data of the second target object, and to establish a finite element calculation model for calculating tissue deformation and pressure distribution; wherein, the radiomics features are image features extracted from the CT images and the MR images.

4. The lung nodule surgical simulation system based on digital twin according to claim 3, characterized in that, The control center includes: A spatial alignment unit is used to construct a right-handed Cartesian coordinate system with the geometric center of the surgical simulation training cabin as the origin, and to align the three-dimensional coordinates of the surgical gesture dynamic model with the three-dimensional coordinates of the digital lung dynamic model in the right-handed Cartesian coordinate system through a coordinate transformation matrix. The collision detection unit is used to detect the motion information of the virtual surgical instrument and the current state of the digital lung dynamic model in real time, and to record the pressure value of the contact point when the current state is detected to be in contact. The deformation and blood supply calculation unit is used to, when the current state is detected to be a contact state, call the current respiratory motion phase in the respiratory motion simulation unit, the viscoelastic parameters of the tissue mechanics simulation unit and the finite element calculation model, the blood supply calculation model of the hemodynamics simulation unit, and calculate the current tissue deformation, current stress distribution and current blood supply change in real time based on the pressure value of the contact point. The instruction generation unit is used to generate multimodal interactive instructions based on the current tissue deformation and current blood supply changes.

5. The lung nodule surgical simulation system based on digital twin according to claim 4, characterized in that, The deformation and blood supply calculation unit includes: The vascular topology reconstruction subunit is used to adjust the vascular tree structure through node deletion and edge reconnection algorithms when a simulated vascular ligation or severance operation is detected, and to update the current blood flow velocity and current pressure distribution in the hemodynamic parameters in real time.

6. The lung nodule surgical simulation system based on digital twin according to claim 1, characterized in that, The surgical gesture dynamic model includes: A key point detection unit is used to extract the three-dimensional spatial coordinates of key points of the hand from the operation data; wherein, the key points of the hand are the position points of the hand joints and fingertips; The action recognition unit is used to input the three-dimensional spatial coordinates of continuous frames into a preset long short-term memory network to recognize virtual surgical actions; The motion mapping unit is used to map the virtual surgical actions into motion information of virtual surgical instruments using a forward kinematics algorithm.

7. The lung nodule surgical simulation system based on digital twin according to claim 1, characterized in that, The multimodal interaction commands include any one or more of the following: visual highlight commands, tactile vibration commands, or audio prompt commands.

8. A method for simulating lung nodule surgery based on digital twins, characterized in that, include: Obtain the operation data of the first target object; The system receives the operation data and maps it into motion information of virtual surgical instruments in real time using a surgical gesture dynamic model. The surgical gesture dynamic model is aligned with the digital lung dynamic model to generate multimodal interaction commands based on the interaction state between the motion information and the digital lung dynamic model; the digital lung dynamic model integrates any two or more of the following: respiratory motion simulation unit, hemodynamic simulation unit, and tissue mechanics simulation unit. Display the holographic operation scene and provide information feedback according to the multimodal interaction instructions.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the steps of the digital twin-based lung nodule surgical simulation method as described in claim 8.

10. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the digital twin-based lung nodule surgical simulation method as described in claim 8.

Images