Deep reinforcement learning robot-assisted ultrasonic scanning system and method based on security constraints

Abstract

The invention discloses a deep reinforcement learning robot-assisted ultrasonic scanning system and method based on security constraints. The invention discloses a deep reinforcement learning mechanical arm ultrasonic scanning method and system based on security constraints. A scanning system comprising a mechanical arm, an ultrasonic probe, a joint torque and visual sensor and a control unit is constructed. The environment state space covers the joint angle, the tail end pose, the contact force, the scanning path and the safety state variable of the mechanical arm, and the action space is a tail end linear speed instruction of the mechanical arm. A deep reinforcement learning model is built based on a near-end strategy optimization (PPO) algorithm, and a reward function including image quality, task completion reward and multiple penalty terms is designed. A control barrier function (CBFs) is used for implementing safety constraint on motion of the mechanical arm, and the safety constraint is fused into deep reinforcement learning training through a penalty function method. A model is trained in a simulation environment simulating an actual scanning scene and then deployed to an actual system, and a mechanical arm is controlled to scan and monitor safety constraints according to a real-time environment state. The mechanical arm can efficiently execute the ultrasonic scanning task under the condition that the safety constraint is met, and a reliable scheme is provided for ultrasonic scanning automation. Meanwhile, the system is provided with a data acquisition module, a state construction module, a reward calculation module, a security constraint module, a control module and the like, and the model performance can be improved through simulation training.

Description

Technical Field

[0001] This invention belongs to the field of medical robotics and intelligent control technology, specifically a safety-constrained deep reinforcement learning method and system for robotic arm ultrasound scanning. This technology integrates robotic arm control, ultrasound imaging, force sensing, visual sensing, the Proximal Policy Optimization (PPO) algorithm, and control barrier functions (CBFs). It aims to enable autonomous, efficient, and safe operation of the robotic arm during ultrasound scanning tasks. It has applications in a variety of medical scenarios, including medical diagnosis, intraoperative navigation, and telemedicine. Background Art

[0002] In the field of medical ultrasound scanning, traditional manual ultrasound scanning relies on the operator's experience and technique, resulting in poor repeatability, low scanning efficiency, high labor costs, and unstable scan quality due to operator fatigue. While robotic-assisted ultrasound scanning technology can improve scanning standardization, it faces numerous challenges in practical application.

[0003] On the one hand, if the robotic arm lacks an effective safety control mechanism during its interaction with the human body, it may easily cause harm to the patient due to collision or excessive contact force. For example, when the patient's position changes slightly or there is an error in the robotic arm's motion model, the contact force between the ultrasound probe and the human tissue may exceed the safe range, which not only affects the image quality but also poses a safety hazard.

[0004] On the other hand, existing robotic arm ultrasound scanning control methods mostly use preset fixed paths or simple feedback control, which are difficult to adapt to complex and changing clinical scenarios, such as the anatomical differences between different patients and the patient's autonomous movement during scanning. This leads to low scanning efficiency and cannot meet the clinical demand for high-quality, personalized ultrasound scanning.

[0005] While deep reinforcement learning has demonstrated powerful autonomous learning capabilities in robotic control, its direct application to robotic ultrasound scanning can lead to policy instability and safety boundary violations during training and actual operation due to the lack of effective safety constraints. While control barrier functions (CBFs) can provide system safety, integrating them with deep reinforcement learning algorithms to achieve a coordinated optimization of safety and efficiency remains a pressing technical challenge. Therefore, a robotic ultrasound scanning method that balances safety and efficiency is urgently needed to meet the growing demands of the medical field. Summary of the Invention

[0006] The purpose of the present invention is to provide a deep reinforcement learning robot-assisted ultrasound scanning system and method based on safety constraints to achieve autonomous, efficient and safe operation of the robotic arm in ultrasound scanning tasks.

[0007] To achieve the above object, the technical solution of the present invention is as follows:

[0008] A safety-constrained deep reinforcement learning robot-assisted ultrasound scanning system and method includes the following steps:

[0009] Step 1: Use the MUJOCO simulation platform to build a simulation model of the Panda robotic arm using XML format files. Based on the actual geometric dimensions, joint parameters, and kinematic characteristics of the Panda robotic arm, define the physical properties of each link, such as shape, mass, and inertia matrix, as well as the type, range of motion, and dynamic parameters of the joints.

[0010] Define the link by label, <joint>Tags define joints and set corresponding attribute values. At the same time, a virtual ultrasound probe model is added to the end of the robotic arm to simulate the movement and interaction of the probe during the actual scanning process.

[0011] Step 1.1: In the simulation environment, add a virtual human tissue model based on the actual clinical scanning scenario. Different parts of human tissue can be simulated by setting different geometric shapes and physical properties (such as Young's modulus, density, stiffness, and damping).

[0012] Step 1.2: Using the rendering function of the simulation platform, set up a virtual camera in the simulation environment, define its parameters such as position, posture, viewing angle, and resolution, so that they are consistent with the parameters of the real visual sensor, generate an image of the virtual scene through the virtual camera, and process the image;

[0013] Step 2: Construct an environment state space containing multi-dimensional information, as follows:

[0014] The robot's own state information: collects the robot's joint angles, which directly determine the robot's configuration; obtains the end position and posture, which are used to determine the specific position of the ultrasound probe in Cartesian space;

[0015] Interaction status information: The torque of each joint of the robot arm is measured by the torque sensor, and the relationship between the end contact force and the torque of each joint is obtained. Calculate the contact force between the ultrasound probe and the human body, where is the pseudo-inverse of the Jacobian matrix of the robotic arm. This contact force directly affects the ultrasound image quality and patient comfort;

[0016] Task-related information: Incorporate scanning path information to clarify the robot's scanning target area and execution order;

[0017] Safety state information: Define safety state variables based on control barrier functions (CBFs). Use the safety function values ​​calculated by CBFs as part of the state variables to quantify the distance between the system and the safety boundary.

[0018] Step 3: Define the motion space of the manipulator as the linear velocity and angular velocity vectors of the end effector in Cartesian space.

[0019] This vector contains three linear velocity components (v x ,v y ,v z ) and three angular velocity components (ω x ,ω y ,ω z ), which correspond to translation and rotation in three-dimensional space respectively. By planning the motion directly in the Cartesian coordinate system, the probe posture adjustment can be realized more intuitively, and it is convenient to combine visual feedback and safety constraints for trajectory planning;

[0020] In step 4, a deep reinforcement learning model is introduced based on the proximal policy optimization (PPO) algorithm. The model architecture consists of a policy network and a value network.

[0021] The policy network takes the current environment state as input and outputs an action probability distribution through forward propagation, which defines the probability of choosing different actions in a given state. The value network also receives the same environment state input and outputs a scalar value used to estimate the expected reward or value in that state.

[0022] Step 5: Use control barrier functions (CBFs) to constrain the motion of the manipulator, define a safety set, and satisfy the CBFs inequality through control input;

[0023] In step 5.1, based on the safety requirements of the robotic arm ultrasound scanning task, define the following key safety sets:

[0024] Trajectory safety set: Let p(t) be the end position of the robot arm, p d (t) is the expected trajectory, and the trajectory error is defined as e p =||p(t)-p d (t)||, the safe set is: where d max is the maximum allowable error, ensuring that the robot arm's motion trajectory is within the specified range.

[0025] Contact force safety set, let F be the actual contact force, F d is the target contact force, and the contact force error e is defined F =|FF d |, the safe set is: Among them F max The maximum allowable contact force error ensures that the contact force between the probe and the human body is within a safe range;

[0026] Speed ​​safety set: Let v be the linear velocity of the end of the robot arm, and define the linear velocity safety set of the end of the robot arm as: where v max The maximum permissible speed is to prevent the robot arm from moving too fast and causing danger;

[0027] Collision safety set: Let d obs is the distance between the robot arm and objects other than the object to be scanned, and the safety set is: C coll ={x|h coll (x) = d obs -d min ≥0}, where d min Maintain a minimum safety distance to avoid collision between the robotic arm and the patient or surrounding equipment;

[0028] Step 5.2, for each security function h i (x) CBF conditions must be met: where α i (·) is a k-type function, usually α i (h) = k i h(k i >0), this condition ensures that the system state will not escape the safe set even if there is external interference;

[0029] The safety control input u is obtained by solving the optimization problem safe :min u ||uu RL || 2 , where u RL The action output by the reinforcement learning policy network ensures that the control input u is close to the reinforcement learning output u RL , and satisfy all safety constraints;

[0030] Step 6: Design a reward function, which includes trajectory constraint penalty, contact force error penalty, velocity error penalty, collision penalty, and penalty for violating the safety constraint of the control obstacle function;

[0031] The reward function adds the following penalty terms to the original task reward:

[0032] Trajectory constraint penalty: P path =λ path max(0,e p -d max ) 2 , where λ path is the trajectory penalty coefficient;

[0033] Contact force error penalty: P force =λ force max(0,e F -F max ) 2 , where λ force is the contact force penalty coefficient;

[0034] Speed ​​error penalty: P vel =λ vel max(0,||v||-v max ) 2 , where λ vel is the speed penalty coefficient;

[0035] Collision error penalty: where λ coll is the collision penalty coefficient;

[0036] Penalties for CBF violations: where λ CBF is the collision penalty coefficient;

[0037] Complete reward function: R total =R task -P path -P force -P vel -P coll -P CBF ;

[0038] Step 7: During training, add the security penalty term directly to the loss function; define the policy network loss function: Define the value network loss function: Among them, P safe =P path +P force +P vel +P coll +P CBF ,λ safe is the security penalty weight, θ is the policy network parameter, τ is the sampling trajectory, and γ is the discount factor;

[0039] By combining the objective function with the learning strategy, training starts with loose safety constraints and gradually tightens them. When approaching the safety boundary, exploration noise is reduced and unsafe state samples are filtered to ensure the safety and effectiveness of the training process.

[0040] Step 8: Use the simulation platform to train the deep reinforcement learning (DRL) model offline. By iteratively optimizing the policy network (determining actions) and the value network (evaluating state values), the model converges to an optimal policy.

[0041] Step 9: Deploy the trained and converged DRL model to the actual robotic arm control system. The model generates control instructions based on the real-time feedback from the sensor on the environmental status (such as position, posture, contact force, etc.) to drive the robotic arm to perform scanning. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 1 is a schematic diagram of a safety-constrained deep reinforcement learning robot-assisted ultrasound scanning system and method provided by an embodiment of the present invention; Specific implementation methods

[0043] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the embodiments described herein are only partial examples of the present invention and are not exhaustive. All other embodiments derived by ordinary technicians in this field based on the existing embodiments of the present invention without creative work are included within the scope of protection of the present invention.

[0044] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments:

[0045] refer to Figure 1 In this example, a deep reinforcement learning robot-assisted ultrasound scanning system and method based on safety constraints is provided, including the following steps:

[0046] S1, when building the Panda robot arm XML model in the MUJOCO simulation platform, for the connecting rod part, use <geom>Tags define shape types (such as cylinder, box), and perform geometric restoration with precise size parameters, while mass and inertia matrices are directly written into <inertial>Label;

[0047] Define the link by label, <joint>Tags define joints and set corresponding attribute values. At the same time, a virtual ultrasound probe model is added to the end of the robotic arm to simulate the movement and interaction of the probe during the actual scanning process.

[0048] S11, in the simulation environment, uses the supported basic geometric shapes to perform abstract modeling based on the actual clinical scanning scene, and accurately sets its size and spatial position through XML tags. <material>and <geom>Configure its physical parameters such as Young's modulus, density, stiffness and damping in the tag;

[0049] S12, in the XML file through <camera>The tag accurately configures the three-dimensional coordinates and Euler angles of the virtual camera to ensure that its spatial position corresponds to the real scene. At the same time, it sets the viewing angle parameters, adjusts the field of view, focal length and other properties, and then uses the rendering engine to generate the virtual scene image in real time.

[0050] S2, constructs an environment state space containing multi-dimensional information, as follows:

[0051] The robot's own state information: Real-time collection of joint angle data, mapping the joint space parameters to Cartesian space using a kinematics forward solution algorithm, and thus calculating the position and posture information of the end effector;

[0052] The acquisition of interaction state information relies on the combination of torque sensors and mechanical models. Real-time torque data is collected by torque sensors at the joints. Then, based on the dynamics model of the manipulator, the pseudo-inverse operation of the Jacobian matrix is ​​used: Establish a mapping relationship between joint torque and end contact force, and convert joint torque data into contact force values ​​between the ultrasound probe and the human body

[0053] Interaction status information: The torque of each joint of the robot arm is measured by the torque sensor, and the relationship between the end contact force and the torque of each joint is obtained. Calculate the contact force between the ultrasound probe and the human body, where is the pseudo-inverse of the Jacobian matrix of the robotic arm. This contact force directly affects the quality of ultrasound images and the comfort of the patient;

[0054] Among the task-related information, the scan path information is the core guide for the robot arm to perform ultrasound scanning tasks. It not only clearly defines the target area that the robot arm needs to cover, but also plans the execution order of the scan, ensuring that the ultrasound probe can systematically and efficiently scan key parts.

[0055] Safety state information is a key factor in ensuring the safety of robotic arm scanning. Safety state variables are defined based on control barrier functions (CBFs). The safety function values ​​calculated by CBFs can intuitively quantify the distance between the current state of the system and the safety boundary. These safety state variables are incorporated into the environmental state space, allowing the deep reinforcement learning model to perceive safety risks in real time when making decisions, thereby generating control strategies that meet both the scanning task requirements and the safety constraints.

[0056] S3, defines the motion space of the manipulator as the linear velocity and angular velocity vectors of the end effector in Cartesian space;

[0057] This vector contains three linear velocity components (v x ,v y ,v z ) and three angular velocity components (ω x ,ω y ,ω z ), which correspond to translation and rotation in three-dimensional space respectively. By planning the motion directly in the Cartesian coordinate system, the probe posture adjustment can be realized more intuitively, and it is convenient to combine visual feedback and safety constraints for trajectory planning;

[0058] S4, based on the proximal policy optimization (PPO) algorithm, introduced a deep reinforcement learning model. The model architecture consists of a policy network and a value network; The policy network uses fully connected layers or convolutional layers to extract features and perform nonlinear transformations on the environment state. Finally, through a probabilistic output layer such as Gaussian distribution, the state is mapped to a probability distribution in the action space, thereby achieving a quantitative expression of the possibility of different action selections.

[0059] The value network uses a similar network architecture to map state information into a one-dimensional real space. Its output layer usually uses a linear activation function to directly estimate the state value.

[0060] The two share state inputs during training, and optimize the policy gradient and value function estimation error respectively through the backpropagation algorithm, forming an end-to-end learning system based on gradient descent.

[0061] S5, use control barrier functions (CBFs) to constrain the motion of the manipulator, define the safety set and satisfy the CBFs inequality through control input;

[0062] S51, for the safety requirements of the robotic arm ultrasonic scanning task, defines the following key safety sets:

[0063] Trajectory safety set: Let p(t) be the end position of the robot arm, p d (t) is the expected trajectory, and the trajectory error is defined as e p =||p(t)-p d (t)||, the safe set is: where d max is the maximum allowable error, ensuring that the robot arm's motion trajectory is within the specified range;

[0064] Contact force safety set, let F be the actual contact force, F d is the target contact force, and the contact force error e is defined F =|FF d |, the safe set is: Among them F max The maximum allowable contact force error ensures that the contact force between the probe and the human body is within a safe range;

[0065] Speed ​​safety set: Let v be the linear velocity of the end of the robot arm, and define the linear velocity safety set of the end of the robot arm as: where v max The maximum permissible speed is to prevent the robot arm from moving too fast and causing danger;

[0066] Collision safety set: Let d obs is the distance between the robot arm and objects other than the object to be scanned, and the safety set is: C coll ={x|h coll (x) = d obs -d min ≥0}, where d min Maintain a minimum safety distance to avoid collision between the robotic arm and the patient or surrounding equipment;

[0067] S52, for each security function h i (x) CBF conditions must be met: where α i (·) is a k-type function, usually α i (h) = k i h(k i >0), this condition ensures that the system state will not escape the safe set even if there is external interference;

[0068] By solving the optimization problem min u ||uu RL || 2 Get safety control input u safe , where u RL The action output by the reinforcement learning policy network ensures that the control input u is close to the reinforcement learning output u RL , and satisfy all safety constraints;

[0069] S6, designing a reward function, wherein the reward function includes a trajectory constraint penalty, a contact force error penalty, a velocity error penalty, a collision penalty, and a penalty term for violating a safety constraint of a control obstacle function;

[0070] The reward function adds the following penalty terms to the original task reward:

[0071] Based on the preset scanning path and the actual position of the end of the robot arm, the spatial deviation between the two is calculated and a corresponding penalty value is assigned according to the degree of deviation. The greater the deviation, the heavier the penalty, so as to encourage the robot arm to fit the target trajectory. The trajectory constraint penalty is: P path =λ path max(0,e p -d max ) 2 , where λ path is the trajectory penalty coefficient;

[0072] The ideal contact force is compared with the actual measured contact force data, and the penalty intensity is set according to the error size to ensure that the contact force is maintained within the appropriate range. The contact force error penalty is: P force =λ force max(0,e F -F max ) 2 , where λ force is the contact force penalty coefficient;

[0073] Monitor the difference between the robot arm's running speed and the safe speed threshold, and impose penalties on overspeeding to avoid risks caused by excessive movement. Speed ​​error penalty: P vel =λ vel max(0,||v||-v max ) 2 , where λ vel is the speed penalty coefficient;

[0074] Real-time judgment of whether the robot arm collides with the surrounding environment or the human body. Once a collision is detected, a high penalty is immediately imposed to prevent dangerous behavior. Collision error penalty: where λ coll is the collision penalty coefficient;

[0075] Convert the degree of security boundary violation calculated by CBFs into a penalty value. CBFs violation penalty: where λ CBF is the collision penalty coefficient;

[0076] By adjusting the weight coefficients of each penalty term and balancing the importance of different constraints, the reward function can effectively guide the robot arm to learn the optimal strategy that meets both task requirements and safety regulations. The complete reward function is: R total =R task -P path -P force -P vel -P coll -P CBF ;

[0077] S7, during the training process, the security penalty term is directly added to the loss function; the policy network loss function is defined as: Define the value network loss function: Among them, P safe =P path +P force +P vel +P coll +P CBF ,λ safe is the security penalty weight, θ is the policy network parameter, τ is the sampling trajectory, and γ is the discount factor;

[0078] By combining the objective function with the learning strategy, training starts with loose safety constraints and gradually tightens them. When approaching the safety boundary, exploration noise is reduced and unsafe state samples are filtered to ensure the safety and effectiveness of the training process.

[0079] S8, when conducting offline training of the deep reinforcement learning (DRL) model in the simulation platform, first generates an initial state sequence based on the simulation environment. The policy network uses this state as input and outputs the action probability distribution through forward propagation, driving the robot arm to perform actions in the virtual scene. After receiving action feedback, the environment updates to the next state and calculates the corresponding reward value based on the reward function. The value network simultaneously evaluates the value of each state.

[0080] In each iteration, the policy gradient is calculated using algorithms such as Proximal Policy Optimization (PPO). Combined with the evaluation results of the value network, the policy network parameters are updated through the backpropagation algorithm so that the generated action probability distribution gradually approaches the optimal policy. At the same time, methods such as Temporal Difference (TD) learning are used to optimize the value network and reduce the error in state value estimation.

[0081] S9, deploying the trained DRL model to the actual robotic arm control system. The model generates control instructions based on the real-time feedback from the sensor on the environmental status (such as position, posture, contact force, etc.) to drive the robotic arm to perform scanning;

[0082] It should be noted that the above contents are only examples of preferred implementation methods of the present invention, and are not absolute limitations on the technical solutions of the present invention. Any person skilled in the art, based on an understanding of the core principles of the present technical solution, can make adaptive adjustments, structural modifications or equivalent replacements to the technical solutions for specific application scenarios based on the disclosed technical content. However, it should be emphasized that as long as such modifications or variations do not deviate from the essential features of the technical solutions of the present invention (i.e., the core logic of constraining the safety of the robotic arm by controlling the obstacle function and combining deep reinforcement learning to achieve safe scanning), or do not change the technical problems it solves and the technical effects it achieves, it should still be regarded as falling within the scope of protection of the technical solutions of the present invention.< / camera> < / geom> < / material> < / joint> < / inertial> < / geom> < / joint>

Claims

1. A safety-constrained deep reinforcement learning robot-assisted ultrasound scanning system and method, characterized in that: The following steps are involved: Step 1: Construct a robotic arm ultrasound scanning system, which includes a robotic arm, an ultrasound probe, a joint torque sensor, a visual sensor, and a control unit. The ultrasound probe is installed at the end of the robotic arm, the joint torque sensor is used to measure the contact force between the ultrasound probe and the human body, and the visual sensor is used to obtain visual information of the scanning area. Step 2: Establish an environmental state space, which includes the manipulator's joint angles, end position and posture, contact forces measured by force sensors, visual features and scanning paths acquired by vision sensors, and safety state variables defined based on the control obstacle function. Step 3: Design the motion space, which contains the linear and angular velocities of the manipulator end in Cartesian space (i.e., a three-dimensional rectangular coordinate system) to control the manipulator's motion to adjust the ultrasound probe's position and posture. Step 4: Build a deep reinforcement learning model based on the proximal policy optimization (PPO) algorithm, which includes a policy network and a value network. The policy network outputs an action probability distribution based on the input environment state, and the value network evaluates the value of the current state. Step 5: Use control barrier functions (CBFs) to impose safety constraints on the robot's motion. Define a safety set and satisfy the CBFs inequality through control input to ensure that the robot does not collide during the scanning process, the motion trajectory is within the specified range, and the contact force and running speed are within the safe range. Step 6: Design a reward function that includes trajectory constraint penalties, contact force error penalties, velocity error penalties, collision penalties, and penalties for violating the safety constraints of the control obstacle function. This function guides the robot arm to learn a strategy that both completes the ultrasound scanning task and satisfies safety constraints. Step 7: Incorporate safety constraints into the deep reinforcement learning training process and use a penalty function method to ensure that the actions generated by the policy network meet the safety constraints. Step 8: Train the deep reinforcement learning model in a simulation environment. By continuously iteratively optimizing the policy network and value network parameters, the robotic arm learns an efficient ultrasonic scanning strategy that meets safety constraints. Step 9: Deploy the trained deep reinforcement learning model to the actual robotic arm ultrasonic scanning system, control the robotic arm to perform ultrasonic scanning based on the real-time environmental status, and monitor safety constraints in real time. When a possible violation of the safety constraint is detected, adjust the robotic arm movement through the safety control strategy.

2. The safety-constrained deep reinforcement learning robotic arm ultrasonic scanning method according to claim 1, characterized in that: In the step of establishing the environmental state space, the image features acquired by the ultrasound probe are extracted through a convolutional neural network, and the visual features acquired by the visual sensor include information such as the outline of the scanning area and the position of key anatomical structures, which are obtained through a visual processing algorithm.

3. The safety-constrained deep reinforcement learning robotic arm ultrasonic scanning method according to claim 1, characterized in that: In the step of designing the reward function, the image quality reward is calculated based on indicators such as the clarity, contrast, and signal-to-noise ratio of the ultrasound image, and the scanning task completion reward is evaluated based on whether the robotic arm covers the predetermined scanning area and the integrity of the scanning path.

4. The safety-constrained deep reinforcement learning robotic arm ultrasonic scanning method according to claim 1, characterized in that: In the step of using control obstacle functions (CBFs) to safely constrain the motion of the robot arm, for trajectory constraints, an obstacle function is constructed based on the deviation between the motion trajectory of the robot end and the target trajectory; for contact force constraints, a control obstacle function is constructed based on the deviation between the contact force and the target contact force.

5. The safety-constrained deep reinforcement learning robotic arm ultrasonic scanning method according to claim 1, characterized in that: The safety constraints are integrated into the deep reinforcement learning training process. When the penalty function method is used, the penalty coefficient is dynamically adjusted according to the importance of the safety constraints and the stability during the training process. When the projected gradient method is used, the actions output by the policy network are projected into the safe feasible set by solving the optimization problem.

6. The safety-constrained deep reinforcement learning robotic arm ultrasonic scanning method according to claim 1, characterized in that: In the step of training the deep reinforcement learning model in a simulation environment, the simulation environment simulates the ultrasonic characteristics of human tissue, the robotic arm dynamics model, force sensor noise, visual sensor noise, and possible interference factors in an actual scanning scene.

7. The safety-constrained deep reinforcement learning robotic arm ultrasonic scanning method according to claim 1, characterized in that: After the step of deploying the trained deep reinforcement learning model to the actual robotic arm ultrasound scanning system, the process also includes regularly collecting actual scanning data and performing online fine-tuning on the deep reinforcement learning model to adapt to the individual differences of different patients and the uncertainties in actual scenarios.

8. A system for implementing the safety-constrained deep reinforcement learning robotic arm ultrasonic scanning method according to any one of claims 1 to 7, characterized in that: include: The data acquisition module is used to collect the joint angle, end position and posture of the robot arm, the image obtained by the ultrasonic probe, the contact force measured by the force sensor, and the visual information obtained by the visual sensor; The state construction module is used to construct the environment state space based on the collected data; Deep reinforcement learning model module, including policy network and value network, used to output actions based on the environment state and evaluate the state value; A reward calculation module is used to calculate the reward value based on the execution of the scanning task and the satisfaction of the security constraints; The safety constraint module constrains the motion of the robotic arm based on the control obstacle function and passes the safety constraint conditions to the deep reinforcement learning model module; The control module controls the movement of the robotic arm according to the actions output by the deep reinforcement learning model module and takes emergency control measures when safety constraints are violated.

9. The system according to claim 8, characterized in that It also includes a simulation training module, which is used to train the deep reinforcement learning model module in a simulation environment and optimize the model parameters.

Images