Thoracentesis robot and storage medium

Abstract

This invention provides a thoracentesis robot and its storage medium, belonging to the field of medical robot technology. It includes: a multimodal perception module that acquires images, force feedback, and respiratory information from an external respiratory monitoring device at the end of the robotic arm; a spatial calibration module that establishes a unified spatial benchmark; a feature extraction and fusion module that acquires a CT 3D reconstruction model as prior knowledge and dynamically fuses visual, tactile, and respiratory features through a dual attention mechanism to generate an enhanced core feature vector; a digital twin module that constructs a real-time digital twin to achieve virtual-real synchronization, respiratory motion prediction, and puncture path deduction, outputting predicted displacement and risk correction factors; a decision control module that generates action commands based on a reinforcement learning framework; and an execution and display module that drives the robotic arm and generates a virtual-real fused guidance view. This invention proactively incorporates respiratory information into the perception dimension and improves the robot's control accuracy in dynamic environments through collaborative decision-making using digital twins and reinforcement learning.

Description

Technical Field

[0001] This invention relates to the field of medical robot technology, and in particular to a thoracentesis robot and a storage medium. Background Technology

[0002] Thoracentesis is a minimally invasive diagnostic and treatment technique that involves inserting a needle through the chest wall into the pleural cavity or lung tissue to drain effusion, perform biopsies, or inject medication. Traditional thoracentesis relies heavily on the physician's tactile sense and experience; however, the accuracy and safety of the puncture are difficult to guarantee due to the dynamic displacement of the target site caused by respiratory movements. In recent years, with the development of robotics and artificial intelligence, thoracentesis robots have been extensively studied.

[0003] Existing thoracentesis robots typically navigate using optical positioning and plan the puncture path using preoperative CT images, with a robotic arm performing the puncture procedure. However, such robots still face the following technical challenges in practical applications: First, the system lacks the ability to process respiratory motion. Existing systems typically treat respiratory motion as an interference factor and suppress it through filtering or gating techniques, failing to incorporate respiratory information as an active sensing dimension into robot control. Because respiratory motion causes periodic displacement of the chest wall and internal organs (with an amplitude of up to 20-30 mm), existing robots struggle to achieve accurate target tracking in dynamic environments.

[0004] Second, the fusion capability of multimodal perception information is limited. Although existing robots can simultaneously acquire information from multiple sources such as visual images and force feedback signals, they mostly adopt simple feature splicing or fixed weight fusion strategies, which cannot dynamically adjust the contribution weight of each modality according to the real-time state. This results in insufficient feature representation capability of the robot when there are complex tissue interfaces or sudden changes in respiratory phase.

[0005] Third, the integration of digital twin technology into robots is relatively shallow. In existing technologies, digital twins are mainly used for preoperative planning, offline simulation, or surgical teaching, failing to drive robot control in real time. Robots lacking virtual-real synchronous mapping and respiratory motion prediction capabilities cannot utilize digital twins to provide dynamic correction information for online control.

[0006] Fourth, reinforcement learning decision modules do not fully utilize modal correlation information. Some studies have attempted to apply reinforcement learning to the control of puncture robots, but general reinforcement learning algorithms (such as DDPG, SAC, TD3, etc.) have low sample efficiency when processing multimodal high-dimensional state spaces, and their decision modules do not have specific experience sampling and feature enhancement mechanisms designed for the correlation between vision, touch, and respiration. As a result, the control accuracy and safety of the robot in dynamic environments such as respiratory movements cannot meet clinical requirements.

[0007] In summary, there is an urgent need for a thoracentesis robot that can achieve deep fusion of visual, tactile, and respiratory modalities, possess real-time digital twin synchronization and prediction capabilities during surgery, and fully utilize modal correlation information in the decision-making module, thereby improving the accuracy and safety of puncture control in dynamic environments. Summary of the Invention

[0008] This invention provides a thoracentesis robot and a storage medium, aiming to solve the technical problem of low puncture control accuracy of existing thoracentesis robots in dynamic environments.

[0009] To achieve the aforementioned objectives, the first aspect of this invention provides a thoracentesis robot, comprising: The multimodal perception module is used to collect multimodal raw data in real time. The multimodal raw data includes image information of the location of the robotic arm end effector, force feedback information, and respiratory information output by an external respiratory monitoring device. The spatial calibration module is used to establish a unified spatial reference between the world coordinate system, the robot arm base coordinate system, the optical positioning instrument coordinate system, and the chest phantom coordinate system, and to obtain the calibrated spatial coordinate parameters. The feature extraction and fusion module is used to acquire a three-dimensional reconstructed anatomical model of CT image data as prior knowledge. Based on the multimodal raw data and spatial coordinate parameters, it performs structured feature extraction on the image information, force feedback information, and respiratory information, and dynamically fuses them through a dual attention mechanism to generate an enhanced core feature vector. The digital twin module is used to construct a real-time digital twin based on the enhanced core feature vector, realize the virtual-real synchronous mapping between the physical robotic arm and the virtual model, output the predicted displacement at future moments through the breathing time-series prediction model, and deduce multiple puncture paths in the virtual environment and output the puncture path damage risk and resistance mutation probability as risk correction factors. The decision control module is used to generate action instructions based on the reinforcement learning framework according to the enhanced core feature vector, predicted displacement and risk correction factor. The execution and display module is used to drive the robotic arm to perform corresponding actions based on the action instructions. At the same time, according to the spatial coordinate parameters, it fuses the preoperative CT three-dimensional anatomical model with the real-time acquired images to generate a virtual-real fusion guide view and displays it in real time. It also feeds back the real-time pose and status information of the robotic arm to the digital twin module to update the digital twin.

[0010] Furthermore, the multimodal sensing module includes: A visual perception unit is used to acquire the image information, which includes a two-dimensional image and depth information. A tactile sensing unit is used to collect the force feedback information, which includes force feedback and tissue resistance information. A respiratory sensing unit is used to collect the respiratory information, which includes respiratory phase and displacement.

[0011] Furthermore, the spatial calibration module includes: The coordinate acquisition unit is used to acquire multi-point corresponding data in the world coordinate system, the robot arm base coordinate system, the optical positioning instrument coordinate system, and the chest phantom coordinate system. The transformation matrix calculation unit is used to solve the rigid body transformation matrix between different coordinate systems using the singular value decomposition method, and to establish a unified coordinate transformation relationship.

[0012] Furthermore, the feature extraction and fusion module includes: The structured representation unit is used to extract features and mathematically model the image information, force feedback information, and respiratory information, and to construct a core feature state space composed of visual geometric features, tactile mechanical features, respiratory physiological features, and multimodal coupling features. The prior knowledge fusion unit is used to acquire CT image data and perform three-dimensional reconstruction. Through the real-time tracking of optical positioning markers by the optical positioning system, the reconstructed three-dimensional anatomical model is accurately registered with the current body position. Dual attention units, including cross-modal attention subunits and spatial attention subunits; The cross-modal attention subunit is used to learn the interdependencies among the three modal features: image information, force feedback information, and breathing information. The enhanced image features, force feedback features, and breathing features are obtained through the following methods: For the target modality and the source modality, they are mapped to the same space using a learnable query mapping matrix and a key mapping matrix, respectively. The dot product of the two is calculated and divided by a scaling factor, then normalized using Softmax to obtain the attention weight. This weight is then used to perform a weighted summation on the value mapping matrix output of the source modality to obtain the enhanced cross-modal features of the target modality. Specifically, the above operations are performed on the image, force feedback, and breathing modalities as the target modalities to obtain the enhanced image features, enhanced force feedback features, and enhanced breathing features. The spatial attention subunit is used to combine the position encoding of the robotic arm end coordinates and the target coordinates, and to concatenate the cross-modal enhanced image features with the position encoding. After processing by a three-dimensional convolutional layer, a spatial attention weight map is generated by the Sigmoid activation function. The weight map is then multiplied element-wise with the cross-modal enhanced image features to obtain the spatially weighted image features. The dual attention unit fuses the spatially weighted image features, the cross-modal enhanced force feedback features, and the cross-modal enhanced breathing features to obtain multimodal fusion features; The feature extraction and fusion module also concatenates the multimodal fusion features with the prior knowledge features output by the prior knowledge fusion unit to obtain the enhanced core feature vector.

[0013] Furthermore, the digital twin module includes: The virtual-real synchronization unit is used to map the enhanced core feature vector to the virtual space in real time, and drive the digital twin model containing the robotic arm, puncture needle and organ model reconstructed based on the CT image data to synchronize with the physical process. The breathing prediction unit is used to build a breathing time-series prediction model in a digital twin. It takes historical breathing phase sequences and corresponding displacements as input and outputs the predicted displacements at future moments as additional state features. The path deduction unit is used to deduce multiple potential optimal puncture paths in a digital twin environment, assess the damage risk and resistance mutation probability of the puncture path in real time, and input the deduction results as a risk correction factor into the reinforcement learning reward function.

[0014] Furthermore, the reinforcement learning framework is a maximum entropy reinforcement learning framework, whose optimization objective is to maximize the weighted sum of expected cumulative reward and policy entropy, where the discount factor is used to balance current and future rewards, the entropy regularization coefficient is used to control the degree of exploration, and the policy entropy is defined as the negative expected value of the log probability of the action.

[0015] Furthermore, the decision control module also includes a policy network construction unit, which is used for: The state features consisting of the enhanced core feature vector, predicted displacement, and risk correction factor are subjected to a first-level linear transformation, activation function processing, and normalization to obtain the first hidden layer features; then, the first hidden layer features are subjected to a second-level linear transformation and activation function normalization to obtain the second hidden layer features. The self-attention operation is performed on the second hidden layer features. Specifically, the second hidden layer features are weighted by learnable gating coefficients for image, force feedback and breathing modality, and then self-attention is calculated for each. The weighted features are then added back to the original features to obtain the attention-enhanced features. The attention-enhanced features are sequentially input into two fully connected layers, which output the mean and log-standard deviation of the actions, respectively. The log-standard deviation of the actions is used to construct a Gaussian distribution to sample the actual actions.

[0016] Furthermore, the decision control module also includes a weighted experience sampling unit, which is used to assign higher sampling priority to experience samples with high modal correlation. The sampling priority is calculated as follows: the absolute value of the temporal difference error is used as the base term, and a modal weighting term is added. The modal weighting term is the absolute value of the temporal difference error multiplied by the modal weighting coefficient, then multiplied by the maximum value between the visual-breathing correlation and the tactile-visual correlation, and then superimposed with a preset positive number to prevent division by zero, so as to obtain the final sampling priority.

[0017] Furthermore, the execution and display module includes: The communication unit is used to establish a real-time two-way communication link between the digital twin and the physical robotic arm, forming a closed-loop interaction; The robotic arm drive unit is used to send motion commands to the robotic arm controller, and drive the robotic arm to move along the planned path through inverse kinematics calculation; The virtual-real fusion display unit is used to spatially register the three-dimensional reconstructed anatomical model with optical positioning markers, generate a three-dimensional virtual guidance layer from the predicted displacement and path deduction results output by the digital twin, and perform image fusion and overlay with the real-time acquired image to generate a virtual-real fusion guidance view. The puncture completion judgment unit is used to output a puncture completion signal when the end of the robotic arm reaches the target point and meets the path accuracy requirements.

[0018] A second aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the functions of the thoracentesis robot described in any of the preceding claims.

[0019] Compared with the prior art, the thoracentesis robot provided by the present invention has the following advantages: (1) This invention uses a multimodal perception module to collect image information, force feedback information and breathing information output by an external breathing monitoring device in real time at the location of the end effector of the robotic arm. This upgrades breathing motion from a traditional passive interference to an active perception dimension, providing a complete original data foundation for subsequent feature fusion and dynamic compensation.

[0020] (2) This invention establishes a unified spatial reference between the world coordinate system, the robotic arm base coordinate system, the optical positioning instrument coordinate system and the chest phantom coordinate system through a spatial calibration module, which ensures the accurate alignment of multi-source information under a unified coordinate system and lays a spatial consistency foundation for subsequent feature extraction and robotic arm control.

[0021] (3) The present invention obtains the three-dimensional reconstructed anatomical model of CT image data as prior knowledge through the feature extraction and fusion module. Based on the multimodal raw data and spatial coordinate parameters, it performs structured feature extraction on image information, force feedback information and respiratory information, and adopts a dual attention mechanism for dynamic fusion. This effectively solves the problem of insufficient feature representation ability of traditional simple splicing or fixed weight fusion methods in dynamic environments, and generates an enhanced core feature vector with rich information and strong adaptability.

[0022] (4) This invention constructs a real-time digital twin based on the enhanced core feature vector through the digital twin module, realizes the virtual-real synchronous mapping between the physical robotic arm and the virtual model, outputs the predicted displacement at future moments through the breathing time-series prediction model, and deduces multiple puncture paths in the virtual environment and outputs the puncture path damage risk and resistance mutation probability as risk correction factors, thus extending the digital twin from offline simulation to real-time decision core, significantly improving the system's adaptability and safety to dynamic environments such as breathing movements.

[0023] (5) The present invention generates action commands based on the reinforcement learning framework by the decision control module according to the enhanced core feature vector, predicted displacement and risk correction factor. It can learn the optimal control strategy autonomously in the dynamic environment without precise modeling, which solves the problem that traditional control methods rely on precise models and are difficult to adapt to respiratory movements.

[0024] (6) The present invention drives the robotic arm to perform corresponding actions based on the action instructions through the execution and display module. At the same time, it fuses the preoperative CT three-dimensional anatomical model with the real-time acquired image according to the spatial coordinate parameters to generate a virtual-real fusion guide view and displays it in real time. The real-time pose and status information of the robotic arm is fed back to the digital twin module to update the digital twin, forming a complete "perception-fusion-decision-execution-feedback" closed-loop control system, which significantly improves the control accuracy and safety of the thoracentesis robot in dynamic environments. Attached Figure Description

[0025] Figure 1 A schematic block diagram of the structure of a thoracentesis robot according to an embodiment of the invention; Figure 2 A schematic block diagram of the structure of a multimodal sensing module according to an embodiment of the invention; Figure 3 A schematic block diagram of the structure of a spatial calibration module according to an embodiment of the invention; Figure 4 This is a schematic block diagram of the feature extraction and fusion module according to an embodiment of the invention; Figure 5 A schematic block diagram of the structure of a digital twin module according to an embodiment of the invention; Figure 6This is a schematic block diagram of the structure of a decision control module according to an embodiment of the invention; Figure 7 This is a schematic block diagram of the execution and display module according to an embodiment of the invention; the realization of the purpose, functional features and advantages of the invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0027] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any modules and all combinations of one or more associated listed items.

[0028] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0029] Reference Figure 1 This application provides a thoracentesis robot, comprising: The multimodal sensing module 10 is used to collect multimodal raw data in real time. The multimodal raw data includes image information of the location of the robotic arm end effector, force feedback information, and respiratory information output by an external respiratory monitoring device.

[0030] Image information: refers to two-dimensional color images and their corresponding depth images acquired by vision sensors (such as RGB-D cameras and binocular cameras), used to identify the visual appearance and image coordinates of target areas, robotic arm ends and puncture needles (their absolute spatial coordinates are determined by the spatial calibration module in combination with the optical positioning instrument).

[0031] Force feedback information: refers to the contact force, torque, and resistance information generated by the tissue during puncture, collected by force / torque sensors, which is used to determine the tissue type and detect abnormal resistance.

[0032] Respiratory information: refers to the phase signals (inspiratory phase / expiratory phase) and body surface displacement related to respiratory movements output by external respiratory monitoring devices (such as laser displacement sensors, pressure bands, infrared markers).

[0033] For example, during the factory testing phase, a chest phantom (bionic dummy) is placed on a testing platform. An Intel RealSense D435i RGB-D camera (visual perception) and an ATI Mini40 six-dimensional force sensor (tactile perception) are installed near the end effector of the robotic arm. Optical markers are affixed to the surface of the phantom, and a laser rangefinder (Keyence LK-G500) is used to collect respiratory displacement signals from the phantom's chest wall. A respiratory simulator is set to drive the phantom to perform sinusoidal breathing movements at a frequency of 12 times per minute. The multimodal perception module simultaneously acquires image frames, force / torque data, and respiratory displacement data at a frequency of 100Hz, forming a synchronous multimodal raw data stream. Through simultaneous acquisition of three modalities, rich environmental information is provided for subsequent fusion. In particular, by making respiratory information an independent perception dimension, the system can actively perceive dynamic environmental changes, laying the foundation for respiratory prediction and dynamic compensation.

[0034] The spatial calibration module 20 is used to establish a unified spatial reference between the world coordinate system, the robotic arm base coordinate system, the optical positioning instrument coordinate system, and the chest phantom coordinate system, and to obtain the calibrated spatial coordinate parameters.

[0035] World coordinate system: An absolute reference coordinate system, usually with a fixed point at the test site (such as the reference frame of an optical positioning instrument) as the origin.

[0036] Robotic arm base coordinate system: fixed to the robotic arm base, with the origin at the center of the base.

[0037] The coordinate system of the optical positioning instrument is fixed to the optical positioning instrument (such as NDI Polaris Vega), with its origin at its optical center.

[0038] Chest phantom coordinate system: fixed to the chest phantom, the origin can be set at a certain marked point on the phantom surface.

[0039] For example, the optical positioning probe sequentially collects the coordinates of at least four non-coplanar marker points in the world coordinate system, at least three non-collinear marker points on the robotic arm base, the corner points of the optical positioning probe's calibration plate, and at least four marker points on the surface of the chest phantom. The singular value decomposition (SVD) method is used to solve for the rigid body transformation matrix (rotation matrix and translation vector) between each pair of coordinate systems, such as the homogeneous transformation matrix for transforming points in the phantom coordinate system to the robotic arm's base coordinate system. The calculated transformation matrix parameters are stored in the system parameter file for subsequent coordinate transformations. A unified spatial reference allows for precise coordinate transformations between vision, the robotic arm, optical positioning, and the phantom, providing spatial consistency for subsequent feature extraction, path planning, and robotic arm control.

[0040] The feature extraction and fusion module 30 is used to acquire a three-dimensional reconstructed anatomical model of CT image data as prior knowledge. Based on the multimodal raw data and spatial coordinate parameters, it performs structured feature extraction on the image information, force feedback information, and respiratory information, and dynamically fuses them through a dual attention mechanism to generate an enhanced core feature vector.

[0041] Structured feature extraction: The original image, force signal, and respiratory signal are transformed into feature vectors with clear physical meaning, such as edges, corners, and depth gradients in the image; mean, variance, and impact peak in the force signal; and phase value and displacement amplitude in the respiratory signal.

[0042] Dual attention mechanisms include cross-modal attention (learning dependencies between different modalities) and spatial attention (focusing on key regions in an image).

[0043] Enhanced core feature vector: A high-dimensional feature vector obtained after dual attention fusion, which integrates trimodal information, and serves as the input for subsequent digital twin and decision modules.

[0044] For example, for a frame of RGB-D image, a pre-trained convolutional neural network (such as ResNet-18) is used to extract visual geometric feature vectors; for the force / torque data at the current moment, tactile mechanical features (including triaxial force, triaxial torque, and drag rate of change) are extracted; for the breathing phase and displacement, respiratory physiological features (including the sine and cosine values ​​of the current phase, displacement, and first derivative) are constructed. These three features are then concatenated and input into a dual attention network: first, the cross-modal attention subunit calculates the attention weights among vision, touch, and breathing to obtain enhanced modal features; then, the spatial attention subunit combines the positional encoding of the robotic arm's end-effector coordinates and the preset target coordinates to spatially weight the visual features; finally, the spatially weighted visual features are fused with the cross-modal enhanced tactile and respiratory features to obtain the enhanced core feature vector. This dual attention mechanism enables the system to dynamically focus on the modality and spatial region most relevant to the current task. For example, when the needle tip approaches the target, the spatial attention weights are concentrated around the needle tip, and the cross-modal attention enhances force feedback and respiratory information, thereby improving the discriminative power of the feature representation.

[0045] The digital twin module 40 is used to construct a real-time digital twin based on the enhanced core feature vector, realize the virtual-real synchronous mapping between the physical robotic arm and the virtual model, output the predicted displacement at future moments through the breathing time-series prediction model, and deduce multiple puncture paths in the virtual environment and output the puncture path damage risk and resistance mutation probability as risk correction factors.

[0046] Digital twin: A digital model constructed in virtual space and synchronized in real time with the physical robotic arm and testing environment, including the robotic arm kinematic model, the puncture needle model, and the organ model based on CT reconstruction.

[0047] Respiratory timing prediction model: A predictor built on a recurrent neural network (RNN) or a temporal convolutional network (TCN), which takes historical respiratory phases and displacement sequences as input and outputs the displacement prediction for future time moments.

[0048] Risk correction factor: After simulating multiple possible paths in a virtual environment, the path deduction unit evaluates the collision probability and resistance mutation probability of each path, and the combined correction value is used to adjust the reward function of reinforcement learning.

[0049] For example, a digital twin environment can be built in Unity3D or ROS Gazebo, importing a robotic arm URDF model, a puncture needle CAD model, and a chest organ mesh model reconstructed from CT data segmentation. The virtual-real synchronization unit receives the robotic arm joint angles and end-effector pose from the enhanced core feature vector at a frequency of 50Hz and drives the virtual model to follow the movement. The breathing prediction unit uses a two-layer LSTM network, taking into input the breathing phase (sine-coded) and displacement every 0.1 seconds over the past 2 seconds, and outputting the predicted displacement every 0.1 seconds over the next 0.5 seconds. The path deduction unit rapidly samples 100 potential paths in the virtual environment using Monte Carlo methods, calculates the collision probability of each path with the ribs and the steepness of resistance changes when passing through different tissues (such as muscles and pleura), and takes the maximum value as the risk correction factor (denoted as ρ, ranging from [0, 1]). The digital twin module not only achieves virtual-real synchronization but also provides forward-looking respiratory motion prediction and path risk assessment, enabling the decision-making module to avoid risks in advance and significantly improving safety in dynamic environments.

[0050] The decision control module 50 is used to generate action instructions based on the enhanced core feature vector, predicted displacement, and risk correction factor, using a reinforcement learning framework.

[0051] Reinforcement learning framework: a machine learning paradigm that learns optimal policies through trial and error by interacting with the environment. This module employs a reinforcement learning framework, using the core feature vectors, predicted displacement, and risk correction factors as states to output motion commands for the robotic arm.

[0052] Action commands: typically the three-dimensional position increments (Δx, Δy, Δz) and attitude increments (Δρ, Δθ, Δφ) of the robotic arm end effector in Cartesian space, totaling 6 continuous quantities.

[0053] For example, the decision control module 50 runs on an Ubuntu system and is implemented using the PyTorch framework. Its inputs are an enhanced core feature vector (256-dimensional), a predicted displacement (1-dimensional), and a risk correction factor (1-dimensional), which are concatenated into a state vector (258-dimensional). The policy network (Actor) receives this 258-dimensional state vector and outputs action commands (6-dimensional continuous values). This policy network is deployed and trained using a reinforcement learning framework. The specific framework type (maximum entropy reinforcement learning), its optimization objective, and network structure details are given in Examples 6, 7, and 8, respectively. During training, the reward function is designed as follows: +10 for reaching the target point, -10 for collision penalty, -0.01 for path length penalty, and -5 × risk correction factor penalty for sudden resistance change. The reinforcement learning framework enables the robot to autonomously learn the optimal control strategy through interaction with the environment, adapting to dynamic changes such as breathing movements without the need for precise modeling.

[0054] The execution and display module 60 is used to drive the robotic arm to perform corresponding actions based on the action instructions. At the same time, according to the spatial coordinate parameters, it fuses the preoperative CT three-dimensional anatomical model with the real-time acquired images to generate a virtual-real fusion guide view and displays it in real time. It also feeds back the real-time pose and status information of the robotic arm to the digital twin module 40 to update the digital twin.

[0055] Virtual-Real Fusion Guided View: Virtual CT 3D anatomical models, predicted paths, safety warnings, etc. are superimposed on the real camera screen in a semi-transparent manner to provide operators with augmented reality navigation.

[0056] Inverse kinematics calculation: A numerical method for calculating the angles of each joint of the robotic arm based on the desired end-effector pose (such as the Jacobi pseudo-inverse method).

[0057] For example, the execution and display module communicates with the robotic arm controller via an EtherCAT bus. The motion commands (end-effector pose increments) output by the decision module are first converted into joint angle commands through inverse kinematics calculation and sent to the servo driver. Simultaneously, the data is displayed on the monitor in a virtual-real fusion manner: the left window shows the real camera view overlaid with a semi-transparent CT reconstruction model and a virtual puncture path; the right window displays a digital twin 3D view. The robotic arm's real-time pose (feedback every 10ms) is sent back to the digital twin module to update the virtual model's position. The operator can intuitively see the relationship between the puncture needle and the virtual anatomical structure, increasing their sense of safety and operational confidence; closed-loop feedback ensures the consistency between the digital twin and the physical entity.

[0058] This embodiment constructs a complete closed-loop control system through the collaborative work of six modules: multimodal perception, spatial calibration, feature fusion, digital twin, reinforcement learning decision-making, and execution display. It elevates respiratory information to an active perception dimension, effectively compensating for target displacement caused by respiratory movements when combined with a respiratory prediction model; a dual attention mechanism achieves deep fusion of visual, tactile, and respiratory features, enhancing feature representation capabilities; the digital twin module is extended from offline simulation to a real-time decision-making core, providing path deduction and risk correction; and the reinforcement learning framework enables the system to autonomously learn and adapt to dynamic environments without requiring precise modeling.

[0059] Reference Figure 2 In one embodiment, the multimodal sensing module 10 includes: The visual perception unit 11 is used to acquire the image information, which includes a two-dimensional image and depth information.

[0060] This unit specifically employs an RGB-D camera (such as an Intel RealSense D435i) to output 640×480 resolution color and depth images at a frame rate of 30fps. For example, in dummy testing, the RGB-D camera is fixed to the flange at the end of a robotic arm and moves with the arm. The camera acquires color and depth images of the puncture area in real time and transmits them to an industrial computer via USB 3.0. The color images are used to identify marker points on the phantom surface, and the depth images are used to calculate the 3D distance between the needle tip and the target point. This unit provides rich geometric and texture information, laying the foundation for spatial localization and path planning.

[0061] The tactile sensing unit 12 is used to collect the force feedback information, which includes force feedback and tissue resistance information.

[0062] This unit uses a six-dimensional force / torque sensor (e.g., ATI Mini40) with a range of 40N / 2Nm and a resolution of 0.01N. Tissue resistance information is extracted through the first derivative (stiffness) and second derivative (abrupt change) of the force signal. For example, the sensor is mounted between the end effector of the robotic arm and the needle holder. When the robotic arm pushes the needle to contact the phantom surface, the sensor acquires triaxial force and triaxial torque in real time. As the needle tip passes through simulated tissues of different densities (such as sponge or silicone), the resistance information changes, and the system records the peak resistance and rate of change. This unit enables the system to sense differences in tissue stiffness and stop promptly when encountering abnormally high resistance, avoiding damage to the dummy or instruments.

[0063] The breathing sensing unit 13 is used to collect the breathing information, which includes the breathing phase and displacement.

[0064] This unit employs a laser displacement sensor (e.g., Keyence LK-G500) or infrared optical markers in conjunction with an optical positioning device. The respiratory phase is defined by the extreme points of the displacement signal (the highest point at the end of expiration, and the lowest point at the end of inspiration). For example, a lightweight reflective marker is attached to the chest wall of the phantom, and the laser sensor is aligned with this marker at a sampling frequency of 100Hz. The real-time acquired displacement signal is filtered, and a peak detection algorithm is used to calculate the respiratory phase (a continuous value between 0 and 2π) and displacement amplitude in real time. Simultaneously, the displacement amount is recorded as the state of respiratory motion. This unit's independent and high-precision respiratory sensing provides a reliable input for respiratory prediction and dynamic compensation.

[0065] In this embodiment, the type and output of each sensing unit are clearly defined, enabling those skilled in the art to select appropriate sensors according to actual needs, thus ensuring the richness and accuracy of the raw data.

[0066] Reference Figure 3 The spatial calibration module 20 includes: The coordinate acquisition unit 21 is used to acquire multi-point corresponding data in the world coordinate system, the robot arm base coordinate system, the optical positioning instrument coordinate system, and the chest phantom coordinate system.

[0067] This unit uses the NDI Polaris Vega optical positioning system. First, a reference frame with four marker points is placed at the origin of the world coordinate system, and its coordinates are recorded. Second, a calibration plate with at least three non-collinear marker points is fixed to the robotic arm base. The optical positioning system probe sequentially touches these marker points, and the coordinates in the optical positioning system's coordinate system are recorded. Simultaneously, the theoretical coordinates in the robotic arm's base coordinate system are read from the robotic arm controller (because the calibration plate is fixed relative to the base). Then, a calibrated calibration tool is placed within the optical positioning system's field of view, and the optical positioning system's own coordinate system is recorded. Finally, at least four marker points are attached to the surface of the chest phantom, and their coordinates in the optical positioning system's coordinate system are collected using a probe. Simultaneously, the design coordinates in the phantom's own coordinate system are recorded. Collecting a sufficient number of corresponding point pairs provides the data foundation for subsequent calculation of the transformation matrix.

[0068] The transformation matrix calculation unit 22 is used to solve the rigid body transformation matrix between different coordinate systems by using the singular value decomposition method, and to establish a unified coordinate transformation relationship.

[0069] For two coordinate systems A and B, there are n pairs of corresponding points. and First, calculate the centroids of each component:

[0070] Then calculate the covariance matrix:

[0071] Perform singular value decomposition on H:

[0072] The rotation matrix is ​​then:

[0073] The translation vector is:

[0074] If det(R) < 0, then correct the last column of R (multiply by -1). Solve the transformation matrices sequentially for world → robotic arm base, optical positioning device → robotic arm base, optical positioning device → phantom, etc., and finally obtain the transformation relationship of any point in different coordinate systems.

[0075] in, : The three-dimensional coordinate vector (column vector) of the i-th corresponding point in coordinate systems A and B.

[0076] : The centroid of the corresponding point.

[0077] H: 3×3 covariance matrix.

[0078] U, V: Orthogonal matrices, Σ: Diagonal singular value matrix.

[0079] R: 3×3 rotation matrix, t: 3×1 translation vector.

[0080] This unit obtains the optimal rigid body transformation in the least squares sense through the SVD method, with calibration accuracy reaching the sub-millimeter level, providing a high-precision benchmark for subsequent coordinate transformations.

[0081] In this embodiment, the transformation relationship between the four coordinate systems is precisely known, which lays a solid foundation for subsequent feature extraction (which requires transforming visual features to the robotic arm coordinate system) and robotic arm control.

[0082] Reference Figure 4 In one embodiment, the feature extraction and fusion module 30 includes: The structured representation unit 31 is used to extract features and perform mathematical modeling on the image information, force feedback information, and respiratory information, and to construct a core feature state space composed of visual geometric features, tactile mechanical features, respiratory physiological features, and multimodal coupling features.

[0083] Visual geometric features: For each frame of RGB image, ResNet-18 pre-trained on ImageNet is used, and its global average pooling layer output is taken as a 512-dimensional feature vector, which is then reduced to 128-dimensional through a fully connected layer. For depth images, geometric features such as point cloud normal vectors and curvature near the puncture needle are calculated and fused with RGB features.

[0084] Tactile mechanical characteristics: For a six-dimensional force signal, the mean, variance, maximum peak value, first derivative, and second derivative over the past 0.1 seconds are calculated to form 30-dimensional characteristics.

[0085] Respiratory physiological characteristics: respiratory phase Encoded as sin cos Add the current displacement y and its first derivative This forms a 4-dimensional feature, which is then expanded to 16 dimensions through a linear layer.

[0086] Multimodal coupling features: The three types of features (128-dimensional visual geometric features, 30-dimensional tactile mechanical features, and 16-dimensional respiratory physiological features) are simply spliced ​​together to obtain 174-dimensional intermediate features, which are then compressed into 128-dimensional coupling features by an autoencoder.

[0087] The prior knowledge fusion unit 32 is used to acquire CT image data and perform three-dimensional reconstruction. Through the real-time tracking of optical positioning markers by the optical positioning system, the reconstructed three-dimensional anatomical model is accurately registered with the current body position, and 128-dimensional anatomical structural features (including spatial position codes of ribs, pleura, lung regions and relative position codes of target points) are extracted as prior knowledge features.

[0088] For example, a chest CT sequence (DICOM format) is acquired, and a triangular mesh model of the ribs, lungs, and pleura is reconstructed using the MarchingCubes algorithm. At least four non-collinear optical positioning markers are attached to the phantom surface, and the coordinates of these markers in the CT image are recorded (either by scanning the markers into the CT scan or by measuring them in the phantom coordinate system using a probe). During testing, the optical positioning system tracks the spatial coordinates of these markers in real time, calculates the transformation matrix between the current phantom pose and the pose during CT reconstruction using singular value decomposition, and transforms the CT model to the current pose, achieving dynamic registration.

[0089] This unit transforms the original signal into physically meaningful features, reducing the input dimension of the subsequent attention module, while preserving the correlation information between modalities, and introducing anatomical constraints through prior knowledge.

[0090] The dual attention unit 33 includes a cross-modal attention subunit and a spatial attention subunit. The cross-modal attention subunit is used to learn the interdependencies among the three modal features: image information, force feedback information, and breathing information. It obtains the cross-modal enhanced image features, force feedback features, and breathing features in the following ways: For the target modality and the source modality, they are mapped to the same space using a learnable query mapping matrix and a key mapping matrix, respectively. The dot product of the two is calculated and divided by a scaling factor, then normalized using Softmax to obtain the attention weight. This weight is then used to perform a weighted summation on the value mapping matrix output of the source modality to obtain the cross-modal enhanced features of the target modality. Specifically, the above operations are performed on the image, force feedback, and breathing modalities as the target modalities to obtain the cross-modal enhanced features. The system comprises enhanced image features, cross-modal enhanced force feedback features, and cross-modal enhanced breathing features. The spatial attention subunit combines the position encoding of the robotic arm end-effector coordinates and the target coordinates, performs channel concatenation of the cross-modal enhanced image features with the position encoding, processes them through a 3D convolutional layer, generates a spatial attention weight map through a Sigmoid activation function, and multiplies the weight map element-wise with the cross-modal enhanced image features to obtain spatially weighted image features. The dual attention unit fuses the spatially weighted image features (128-dimensional), the cross-modal enhanced force feedback features (30-dimensional), and the cross-modal enhanced breathing features (16-dimensional), and compresses them through a fully connected layer to obtain a 128-dimensional multimodal fusion feature.

[0091] The feature extraction and fusion module 30 concatenates the 128-dimensional multimodal fusion features mentioned above with the 128-dimensional prior knowledge features output by the prior knowledge fusion unit 32 to form a 256-dimensional enhanced core feature vector, which serves as the input for the subsequent digital twin module and decision control module.

[0092] The specific formula for the cross-modal attention subunit is: Let the visual feature vector be The tactile feature vector is The respiratory feature vector is For the target mode m and the source mode n, the first step is to use a learnable query matrix. Bond matrix Map the features to the same space, then calculate the attention weights:

[0093] Where d k Here, is the scaling factor (e.g., 64), and softmax is the Softmax normalization function. Then, the value matrix of the source modes... The outputs are weighted and summed to obtain the cross-modal enhancement features of the target modality:

[0094] Perform the above operation on the visual, tactile, and respiratory modalities as target modalities m respectively, and obtain the following results. , , .

[0095] The specific formula for the spatial attention subunit is as follows: Let the coordinates of the robotic arm's end effector be... The target coordinates are After performing sinusoidal position encoding on the two, we obtain... Image features enhanced across modalities The spatial attention weight map is obtained by concatenating the location encoding along the channel dimension and inputting it into a 3×3 convolutional layer, followed by a sigmoid activation function.

[0096] Where σ is the Sigmoid function, [ ; [] indicates channel splicing. Then, this weight map is combined with... Element-wise multiplication yields spatially weighted image features:

[0097] Final Fusion: By fusing spatially weighted image features, cross-modal enhanced tactile features, and breathing features (e.g., by stitching or weighted summation), multimodal fusion features are obtained.

[0098] In the formulas of this embodiment: f v f t f r Original visual, tactile, and respiratory feature vectors.

[0099] d v d t d r Dimensions of each feature.

[0100] d k Attention scaling factor.

[0101] , , Learnable parameter matrix.

[0102] α m，n Attention weights from source mode n to target mode m.

[0103] : Cross-modal enhancement features of target mode m.

[0104] p e p g : Coordinates of the robotic arm's end effector and the target point.

[0105] pos( ): Sine position encoding function.

[0106] M: Spatial attention weight map.

[0107] ⊙: Element-by-element multiplication.

[0108] Cross-modal attention dynamically allocates modal weights, enabling the system to automatically increase the weight of the respiratory modality when there are abrupt changes in the respiratory phase and increase the weight of the tactile modality when the needle tip contacts the tissue; spatial attention focuses on the area near the needle tip and target, reducing interference from irrelevant background. The two work together to improve the discriminativeness of feature representation.

[0109] This embodiment discloses a structured representation, prior knowledge fusion, and dual attention mechanism. Through these techniques, the system can efficiently extract key information from multimodal raw data and adaptively fuse it, providing high-quality state representation for subsequent digital twin and decision-making modules.

[0110] Reference Figure 5 The digital twin module 40 includes: The virtual-real synchronization unit 41 is used to map the enhanced core feature vector to the virtual space in real time, driving the digital twin model containing the robotic arm, puncture needle and organ model reconstructed based on the CT image data to synchronize with the physical process.

[0111] For example, import the URDF model of the robotic arm, the STL model of the puncture needle, and the CT-reconstructed organ model into Unity3D. Communicate with the ROS node of the physical robotic arm via ROS#, subscribe to the "joint_states" topic to obtain real-time joint angles, and update the joint angles of the virtual model; simultaneously, publish the end-effector pose of the virtual model for display. The synchronization frequency is 100Hz, and the latency is less than 10ms. Operators can observe the virtual model from any perspective, obtaining visual feedback consistent with the physical process; it also provides the same virtual environment for path deduction.

[0112] The breathing prediction unit 42 is used to build a breathing time-series prediction model in the digital twin. It takes the historical breathing phase sequence and corresponding displacement as input and outputs the predicted displacement (1-dimensional scalar) at a future time (e.g., 0.1 seconds later) as an additional state feature.

[0113] For example, construct a single-layer LSTM network with an input window length of 20 (i.e., the past 2 seconds, sampling rate of 10Hz), and the feature at each time step is [sin cos The output window length is 1 (0.1 seconds into the future), resulting in a 1-dimensional predicted displacement. During training, supervised learning is performed using collected breathing data (10,000 cycles), with the loss function being mean squared error. During testing, the predicted displacement for the next 0.1 seconds is output in real-time, denoted as y[i, y](3D). By anticipating breathing movements, the robotic arm can compensate in advance, reducing tracking errors.

[0114] The path deduction unit 43 is used to deduce multiple potential optimal puncture paths in a digital twin environment, evaluate the damage risk and resistance mutation probability of the puncture path in real time, and input the deduction results as a risk correction factor into the reinforcement learning reward function.

[0115] In a digital twin environment, given the current needle tip and target locations, the RRT* (Fast Exploratory Random Tree Star) algorithm is used to quickly generate 100 candidate paths. For each path, the minimum distance to the rib model (puncture path damage risk) and the integral change in resistance along the path through different tissues are calculated (resistance mutation probability). The maximum value of the puncture path damage risk and the maximum value of the resistance mutation probability among all paths are taken and weighted to obtain the risk correction factor.

[0116] Where ρ∈[0,1], a larger value indicates a higher risk. This factor is multiplied into the penalty term in the reward function of reinforcement learning.

[0117] Where ρ: risk correction factor, scalar, range [0, 1].

[0118] collision max : The maximum collision probability among all candidate paths.

[0119] resistance max : The probability of the maximum resistance mutation among all candidate paths.

[0120] By identifying high-risk paths in advance through offline simulations and imposing penalties during decision-making, the policy network is guided to choose safer paths.

[0121] This embodiment elevates digital twins from passive display to active prediction and inference tools. Breathing prediction and path inference provide reinforcement learning with forward-looking state characteristics and risk corrections, significantly improving control safety in dynamic environments.

[0122] In one embodiment, the reinforcement learning framework is a maximum entropy reinforcement learning framework, whose optimization objective is to maximize the weighted sum of expected cumulative reward and policy entropy, wherein the discount factor is used to balance current and future rewards, the entropy regularization coefficient is used to control the degree of exploration, and the policy entropy is defined as the negative expected value of the log probability of the action.

[0123] In this embodiment, the reinforcement learning framework used by the decision control module is specifically the maximum entropy reinforcement learning framework, and its optimization objective is:

[0124] in: s t The state at time t is the combination of enhanced core feature vector, predicted displacement, and risk correction factor.

[0125] a t The action at time t is the pose increment of the robotic arm's end effector (6-dimensional).

[0126] γ∈[0,1): Discount factor, set to 0.99, used to balance current and future rewards.

[0127] r(s t a t ): The instant reward function is defined during factory testing based on factors such as the distance between the needle tip and the target point, and whether a collision occurs.

[0128] α>0: Entropy regularization coefficient, set to 0.2, used to control the degree of exploration.

[0129] H(π(·∣st Policy entropy, defined as follows: It measures the randomness of the distribution of actions.

[0130] For example, during the training phase, the policy network parameters are updated using the optimization objective described above. At each time step, the TD error is calculated, and the critic and policy networks are updated. Initially, entropy is high, and actions are random; as training progresses, entropy gradually decreases, but a certain degree of exploratory nature is always maintained. The maximum entropy framework allows the algorithm to achieve a balance between exploration and exploitation, avoiding getting trapped in local optima, and is particularly suitable for high-dimensional continuous action spaces.

[0131] This embodiment clarifies the specific type (maximum entropy) of the reinforcement learning framework upon which the decision control module is based and its optimization objective. This objective function forms the basis for subsequent updates to the policy network and the commentator network.

[0132] Reference Figure 6 In one embodiment, the decision control module 50 further includes a policy network construction unit 51, which is used for: The state features consisting of the enhanced core feature vector, predicted displacement, and risk correction factor are subjected to a first-level linear transformation, activation function processing, and normalization to obtain the first hidden layer features; then, the first hidden layer features are subjected to a second-level linear transformation and activation function normalization to obtain the second hidden layer features. The self-attention operation is performed on the second hidden layer features. Specifically, the second hidden layer features are weighted by learnable gating coefficients for image, force feedback and breathing modality, and then self-attention is calculated for each. The weighted features are then added back to the original features to obtain the attention-enhanced features. The attention-enhanced features are sequentially input into two fully connected layers, which output the mean and log-standard deviation of the actions, respectively. The log-standard deviation of the actions is used to construct a Gaussian distribution to sample the actual actions.

[0133] In this embodiment, the input state is: to enhance the core feature vector. Predicted displacement (1-dimensional) and risk correction factor ρ∈R (1-dimensional) are concatenated to form a state vector. .

[0134] First layer:

[0135] in , ReLU is the linear rectified activation function, and LayerNorm is the layer normalization function.

[0136] Second layer:

[0137] in , .

[0138] Cross-modal self-attention operation: Split h2 into three sub-vectors according to modality: , , Calculate self-attention (scaled dot product self-attention) for each subvector:

[0139] Where β img ,β force ,β resp The learnable scalar gating coefficients are initially set to 0.5; SelfAttn is defined as:

[0140] Here, X is a subvector reshaped into a sequence (length 1, dimension equal to the subvector dimension), W Q W K W V For a learnable matrix, d k This is the scaling factor.

[0141] Output the mean and logarithmic standard deviation of the actions:

[0142]

[0143] in , Final action That is, sampling from a Gaussian distribution.

[0144] In the formulas of this embodiment: s: State vector.

[0145] W1, b1, W2, b2: Weight matrices and biases for the first and second layers.

[0146] h1, h2: Features of the first and second hidden layers.

[0147] β img ,β force ,β resp : Gating coefficients for each mode.

[0148] μ, logσ: mean and logarithmic standard deviation of the action.

[0149] W μ W μ1 b μ1 b μ : Parameters of the mean output layer.

[0150] W σ W σ1 b σ1 b σ : Logarithmic standard deviation output layer parameters.

[0151] Introducing cross-modal self-attention into the policy network enables the network to dynamically adjust the degree of attention given to different modal information. For example, the weight of the breathing modality can be increased when the breathing phase changes drastically, thereby improving the adaptability of action decision-making.

[0152] This embodiment details the specific structure of the policy network, including the number of layers, activation functions, normalization methods, and cross-modal self-attention enhancement. These designs improve the efficiency of the policy network in utilizing multimodal information.

[0153] In one embodiment, the decision control module 50 further includes a weighted experience sampling unit 52, which is used to assign higher sampling priority to experience samples with high modal correlation. The sampling priority is calculated as follows: the absolute value of the temporal difference error is used as the base term, and a modal weighting term is added. The modal weighting term is the absolute value of the temporal difference error multiplied by the modal weighting coefficient and then multiplied by the maximum value between the visual-breathing correlation and the tactile-visual correlation, and then a preset positive number to prevent division by zero is added to obtain the final sampling priority.

[0154] In this embodiment, each experience e t (s t a t r t s t+1 Sampling priority p t Calculate using the following formula:

[0155] in: Preset positive number (e.g., 10) 6 This prevents the priority from being zero.

[0156] δ t Timing difference error, i.e. The absolute value of.

[0157] λ modal : Modal weighting coefficient, set to 0.5, controls the strength of modal correlation terms.

[0158] R vr Visual-respiratory correlation is calculated as the absolute value of the cosine similarity between visual features and respiratory features in the current experience.

[0159] R tv: Tactile-visual correlation, calculated as the absolute value of the cosine similarity between the force feedback feature and the visual feature.

[0160] For example, 10,000 experiences are stored in the experience replay buffer. For each newly generated experience, its |δ is calculated. t | Simultaneously, image features and respiratory feature vectors are extracted from this experience, and the correlation coefficient R is calculated. vr =∣corr(f v f r ) |; Similar to calculating R tv =∣corr(f t f v Take the maximum of the two values. If the breathing movements in this experience are intense and highly correlated with image motion (e.g., large fluctuations in the phantom), then R... vr When the value is close to 1, the priority increases significantly. During sampling, weighted sampling is performed according to the priority probability distribution, and experiences with higher priority have a greater probability of being selected.

[0161] In the formulas of this embodiment: p t : Sampling priority of the t-th experience.

[0162] : Preset positive number to prevent division by zero.

[0163] δ t Timing difference error.

[0164] λ modal Modal weighting coefficients.

[0165] R vr Visual-respiratory correlation (absolute value).

[0166] R tv : Tactile-visual correlation (absolute value).

[0167] Traditional priority experience playback relies solely on TD error, easily overlooking modal correlation information. This embodiment adds modal correlation terms, enabling the system to focus more on experiences with tightly coupled multimodal information that are crucial for learning breathing compensation and force perception, thereby accelerating learning convergence and improving final performance.

[0168] This embodiment provides a novel weighted experience sampling mechanism that prioritizes training with high modal correlation, effectively improving the sample efficiency of reinforcement learning, and is particularly suitable for multimodal continuous control tasks.

[0169] Reference Figure 7 The execution and display module 60 includes: Communication unit 61 is used to establish a real-time bidirectional communication link between the digital twin and the physical robotic arm, forming a closed-loop interaction.

[0170] For example, ROS (Robot Operating System) is used as the communication middleware. The digital twin module runs on a PC and publishes the topics " / virtual / joint_states" and " / virtual / end_effector_pose". The physical robot arm controller subscribes to the topic " / cmd_joint_angles" and publishes the topic " / real / joint_states". The communication unit establishes a ROS node, subscribes to both topics, forwards virtual commands to the physical controller, and feeds back the physical state to the virtual model, forming a two-way closed loop with a communication cycle of 10ms. This ensures the real-time performance and reliability of virtual-physical synchronization, providing a foundation for closed-loop control.

[0171] The robotic arm drive unit 62 is used to send motion commands to the robotic arm controller and drive the robotic arm to move along the planned path through inverse kinematics calculation.

[0172] The robotic arm is a UR10e collaborative robot, and the controller runs inside the robot control cabinet. Motion commands are end-effector pose increments (6D). The robotic arm drive unit first adds the increment to the current end-effector pose to obtain the desired pose, then calls an inverse kinematics library (such as TraciK) to calculate each joint angle, and finally sends the joint angles to the controller via Modbus / TCP. The control cycle is 20ms. High-level motion commands are translated into low-level motor control to achieve smooth motion.

[0173] The virtual-real fusion display unit 63 is used to spatially register the three-dimensional reconstructed anatomical model with optical positioning markers, generate a three-dimensional virtual guide layer from the predicted displacement and path deduction results output by the digital twin, and perform image fusion and overlay with the real-time acquired image to generate a virtual-real fusion guide view.

[0174] For example, the display interface is developed using OpenCV and OpenGL. First, the coordinates of marker points on the phantom surface are acquired in real time using an optical locator, transforming the CT-reconstructed model to its current pose. Then, the predicted displacement (the trajectory 0.2 seconds ahead) output by the digital twin and the optimal path (a 3D curve) from the path deduction unit are rendered as a semi-transparent green curve. Finally, using a real camera view as the background, the virtual model and guide lines are drawn on the screen in a semi-transparent overlay, generating an augmented reality view. Operators can observe this on a monitor, providing intuitive navigation information and enhancing safety and operational precision.

[0175] The puncture completion judgment unit 64 is used to output a puncture completion signal when the end of the robotic arm reaches the target point and meets the path accuracy requirements.

[0176] For example, the target point is defined as a spherical region with a radius of 0.5 mm. The Euclidean distance between the robotic arm's end effector and the target point is calculated in real time. When the distance is less than 0.5 mm and remains within tolerance for 10 consecutive control cycles (0.2 seconds), while simultaneously meeting path accuracy requirements (the maximum deviation between the actual path and the planned path is less than 0.5 mm), a puncture completion signal (high level) is output, the system automatically stops moving, and an alarm sound is emitted. This automated judgment requires no manual intervention, improving the consistency and reliability of the test.

[0177] This embodiment encompasses communication, driving, display, and completion judgment, forming a complete execution loop. The virtual-real fusion guided view significantly enhances the intuitiveness of human-computer interaction.

[0178] One embodiment of this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the functions of the thoracentesis robot described in any of the preceding claims. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0179] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in this application and in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0180] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0181] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A thoracentesis robot, characterized in that, include: The multimodal perception module is used to collect multimodal raw data in real time. The multimodal raw data includes image information of the location of the robotic arm end effector, force feedback information, and respiratory information output by an external respiratory monitoring device. The spatial calibration module is used to establish a unified spatial reference between the world coordinate system, the robot arm base coordinate system, the optical positioning instrument coordinate system, and the chest phantom coordinate system, and to obtain the calibrated spatial coordinate parameters. The feature extraction and fusion module is used to acquire a three-dimensional reconstructed anatomical model of CT image data as prior knowledge. Based on the multimodal raw data and spatial coordinate parameters, it performs structured feature extraction on the image information, force feedback information, and respiratory information, and dynamically fuses them through a dual attention mechanism to generate an enhanced core feature vector. The digital twin module is used to construct a real-time digital twin based on the enhanced core feature vector, realize the virtual-real synchronous mapping between the physical robotic arm and the virtual model, output the predicted displacement at future moments through the breathing time-series prediction model, and deduce multiple puncture paths in the virtual environment and output the puncture path damage risk and resistance mutation probability as risk correction factors. The decision control module is used to generate action instructions based on the enhanced core feature vector, predicted displacement, and risk correction factor, using a reinforcement learning framework. The execution and display module is used to drive the robotic arm to perform corresponding actions based on the action instructions. At the same time, according to the spatial coordinate parameters, it fuses the preoperative CT three-dimensional anatomical model with the real-time acquired images to generate a virtual-real fusion guide view and displays it in real time. It also feeds back the real-time pose and status information of the robotic arm to the digital twin module to update the digital twin.

2. The thoracentesis robot according to claim 1, characterized in that, The multimodal sensing module includes: A visual perception unit is used to acquire the image information, which includes a two-dimensional image and depth information. A tactile sensing unit is used to collect the force feedback information, which includes force feedback and tissue resistance information. A respiratory sensing unit is used to collect the respiratory information, which includes respiratory phase and displacement.

3. The thoracentesis robot according to claim 1, characterized in that, The spatial calibration module includes: The coordinate acquisition unit is used to acquire multi-point corresponding data in the world coordinate system, the robot arm base coordinate system, the optical positioning instrument coordinate system, and the chest phantom coordinate system. The transformation matrix calculation unit is used to solve the rigid body transformation matrix between different coordinate systems using the singular value decomposition method, and to establish a unified coordinate transformation relationship.

4. The thoracentesis robot according to claim 1, characterized in that, The feature extraction and fusion module includes: The structured representation unit is used to extract features and mathematically model the image information, force feedback information, and respiratory information, and to construct a core feature state space composed of visual geometric features, tactile mechanical features, respiratory physiological features, and multimodal coupling features. The prior knowledge fusion unit is used to acquire CT image data and perform three-dimensional reconstruction. Through the real-time tracking of optical positioning markers by the optical positioning system, the reconstructed three-dimensional anatomical model is accurately registered with the current body position. Dual attention units, including cross-modal attention subunits and spatial attention subunits; The cross-modal attention subunit is used to learn the interdependencies among the three modal features: image information, force feedback information, and breathing information. The enhanced image features, force feedback features, and breathing features are obtained through the following methods: For the target modality and the source modality, they are mapped to the same space using a learnable query mapping matrix and a key mapping matrix, respectively. The dot product of the two is calculated and divided by a scaling factor, then normalized using Softmax to obtain the attention weight. This weight is then used to perform a weighted summation on the value mapping matrix output of the source modality to obtain the enhanced cross-modal features of the target modality. Specifically, the above operations are performed on the image, force feedback, and breathing modalities as the target modalities to obtain the enhanced image features, enhanced force feedback features, and enhanced breathing features. The spatial attention subunit is used to combine the position encoding of the robotic arm end coordinates and the target coordinates, and to concatenate the cross-modal enhanced image features with the position encoding. After processing by a three-dimensional convolutional layer, a spatial attention weight map is generated by the Sigmoid activation function. The weight map is then multiplied element-wise with the cross-modal enhanced image features to obtain the spatially weighted image features. The dual attention unit fuses the spatially weighted image features, the cross-modal enhanced force feedback features, and the cross-modal enhanced breathing features to obtain multimodal fusion features; The feature extraction and fusion module also concatenates the multimodal fusion features with the prior knowledge features output by the prior knowledge fusion unit to obtain the enhanced core feature vector.

5. The thoracentesis robot according to claim 1, characterized in that, The digital twin module includes: The virtual-real synchronization unit is used to map the enhanced core feature vector to the virtual space in real time, and drive the digital twin model containing the robotic arm, puncture needle and organ model reconstructed based on the CT image data to synchronize with the physical process. The breathing prediction unit is used to build a breathing time-series prediction model in a digital twin. It takes historical breathing phase sequences and corresponding displacements as input and outputs the predicted displacements at future moments as additional state features. The path deduction unit is used to deduce multiple potential optimal puncture paths in a digital twin environment, assess the damage risk and resistance mutation probability of the puncture path in real time, and input the deduction results as a risk correction factor into the reinforcement learning reward function.

6. The thoracentesis robot according to claim 1, characterized in that, The reinforcement learning framework is a maximum entropy reinforcement learning framework, whose optimization objective is to maximize the weighted sum of expected cumulative reward and policy entropy. The discount factor is used to balance current and future rewards, the entropy regularization coefficient is used to control the degree of exploration, and the policy entropy is defined as the negative expected value of the log probability of the action.

7. The thoracentesis robot according to claim 6, characterized in that, The decision control module further includes a policy network construction unit, which is used for: The state features consisting of the enhanced core feature vector, predicted displacement, and risk correction factor are subjected to a first-level linear transformation, activation function processing, and normalization to obtain the first hidden layer features; then, the first hidden layer features are subjected to a second-level linear transformation and activation function normalization to obtain the second hidden layer features. The self-attention operation is performed on the second hidden layer features. Specifically, the second hidden layer features are weighted by learnable gating coefficients for image, force feedback and breathing modality, and then self-attention is calculated for each. The weighted features are then added back to the original features to obtain the attention-enhanced features. The attention-enhanced features are sequentially input into two fully connected layers, which output the mean and log-standard deviation of the actions, respectively. The log-standard deviation of the actions is used to construct a Gaussian distribution to sample the actual actions.

8. The thoracentesis robot according to claim 6, characterized in that, The decision control module also includes a weighted experience sampling unit, which is used to assign higher sampling priority to experience samples with high modal correlation. The sampling priority is calculated as follows: the absolute value of the time-series difference error is used as the base term, and a modal weighting term is added. The modal weighting term is the absolute value of the time-series difference error multiplied by the modal weighting coefficient, then multiplied by the maximum value between the visual-breathing correlation and the tactile-visual correlation, and then superimposed with a preset positive number to prevent division by zero, so as to obtain the final sampling priority.

9. The thoracentesis robot according to claim 1, characterized in that, The execution and display module includes: The communication unit is used to establish a real-time two-way communication link between the digital twin and the physical robotic arm, forming a closed-loop interaction; The robotic arm drive unit is used to send motion commands to the robotic arm controller, and drive the robotic arm to move along the planned path through inverse kinematics calculation; The virtual-real fusion display unit is used to spatially register the three-dimensional reconstructed anatomical model with optical positioning markers, generate a three-dimensional virtual guidance layer from the predicted displacement and path deduction results output by the digital twin, and perform image fusion and overlay with the real-time acquired image to generate a virtual-real fusion guidance view. The puncture completion judgment unit is used to output a puncture completion signal when the end of the robotic arm reaches the target point and meets the path accuracy requirements.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program performs the functions of the thoracentesis robot as described in any one of claims 1 to 9.

Images