Automatic ultrasonic scanning system and method based on double-agent reinforcement learning

Abstract

The invention belongs to the technical field of intelligent robot control, and relates to an ultrasonic automatic scanning system and method based on double-agent reinforcement learning, and the scanning method comprises the following steps: (1) obtaining force and moment information collected by a six-dimensional force sensor and defining the input and output of double agents, (2) establishing the double agents, selecting a reinforcement learning algorithm, and (3) selecting a reinforcement learning algorithm; the method comprises the steps of (1) establishing an ultrasonic scanning simulation environment and completing intelligent agent reinforcement learning training, (3) deploying an optimized reinforcement learning PPO algorithm to a mechanical arm to provide automatic ultrasonic scanning control, and (4) testing an ultrasonic automatic scanning system and evaluating the quality of collected images by experts. According to the method, the double-agent reinforcement learning algorithm is adopted, the whole process is closed, the real-time effect is improved, meanwhile, the reinforcement learning model does not need to be adjusted for different parts of different patients, and the method has wide application prospects.

Description

Technical Field

[0001] The invention relates to an ultrasonic automatic scanning system and a method based on dual-agent reinforcement learning, belonging to the technical field of intelligent robot control. Background Art

[0002] Percutaneous puncture surgery is being used more and more widely in the treatment of liver cancer due to its advantages such as minimally invasive, less postoperative pain, low complications and fast recovery. The surgery usually includes two parts: image guidance and intraoperative puncture. The core task of image guidance is to accurately locate the target site of puncture. Ultrasound (US) has become the main image guidance tool for clinical liver puncture surgery due to its advantages such as non-invasiveness, real-time, no radiation and low equipment cost. However, traditional manual ultrasound scanning has some limitations, such as long scanning time, strong dependence on operator skills, poor operation consistency, and easy to cause operator fatigue. These problems may affect the accuracy of target site positioning, increase the risk of puncture operation, reduce patient comfort, and ultimately limit the overall effect of manual ultrasound scanning and the reliability of diagnosis. To meet these challenges, automatic ultrasound scanning technology combining robotics and artificial intelligence has emerged. Automatic ultrasound scanning not only improves the image guidance effect during surgery, but also improves the reliability of diagnosis by utilizing the high precision, high stability and repeatability of robots, and promotes the innovation and development of puncture treatment.

[0003] In recent years, some scholars have made some progress in the research of automatic ultrasound scanning. Huang Y et al. designed a fully autonomous carotid ultrasound scanning robot, and established a servo controller to learn the actual scanning process of doctors based on the ultrasound features of the carotid artery; Huang Q et al. used a depth camera to obtain the depth data and color data of the tissue surface to plan the autonomous scanning path and realize the automatic scanning of three-dimensional ultrasound imaging; Hannes Hase et al. built an automatic spine ultrasound scanning system that uses ultrasound images as input and combines reinforcement learning DQN and classification methods. Although ultrasound automatic scanning has made great progress, the existing methods mainly rely on depth cameras or ultrasound image information, and the proposed solutions are only applicable to the scanning of certain organs or certain parts. For example, the depth camera needs to be recalibrated after changing the scene, which has high requirements for space and equipment. In addition, the acquisition quality and processing speed of image data will also affect the real-time and reliability of the system. The automatic scanning system based on ultrasound images depends on the anatomical characteristics of the scanning part, and its scalability and universality are not strong. In fact, in ultrasound scanning, the pressure between the probe and the scanning surface has an important impact on the quality of ultrasound images. When the pressure is too low, a clear ultrasound image cannot be obtained. When the pressure is too high, it is easy to cause discomfort to the patient and even endanger personal safety. To this end, the researchers tried to add force sensors on both sides of the probe to achieve real-time monitoring of the probe pressure so that the pressure remains relatively constant and the safety of the system is guaranteed. In addition, a standard scanning plane is also very important for ultrasound examinations. When the probe is perpendicular to the scanning plane, more sound wave signals can be obtained and the imaging is clearer. When the probe deviates from the scanning plane, sound waves will be reflected or refracted, causing image distortion and affecting the accuracy of the diagnosis. Therefore, the operator also needs to maintain a vertical posture between the probe and the contact surface during the scanning process to minimize the torque. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide an ultrasonic automatic scanning system and method based on dual-agent reinforcement learning, taking into account the pressure and torque information between the probe and the scanning surface, and using different agents to process them respectively to improve the efficiency and accuracy of the system. The system uses a reinforcement learning algorithm to learn the adjustment rules of the probe during the scanning process, and uses a six-dimensional force sensor to collect the force and torque information between the probe and the contact surface during the scanning process. The force sensor is used as a key component to ensure the safety of the system and the multi-dimensional adjustment of the ultrasonic probe, and is combined with a robotic arm controller to realize the linkage adjustment of the system.

[0005] In order to achieve the above-mentioned invention purpose, the technical scheme adopted by the present invention is: an ultrasonic automatic scanning system based on dual-agent reinforcement learning, including an ultrasonic device, a force feedback module, a robotic arm, a control cabinet and a host, the ultrasonic device includes an ultrasonic scanner and an ultrasonic probe, and the ultrasonic scanner and the ultrasonic probe are connected by a coaxial cable; the force feedback module includes a six-dimensional force sensor and an ultrasonic clamp, wherein the two ends of the six-dimensional force sensor are detachably connected to the ultrasonic clamp and the end of the robotic arm respectively, the six-dimensional force sensor is connected to the host through a signal output line, and the ultrasonic probe is fixed together with the six-dimensional force sensor through the ultrasonic clamp; the control cabinet is connected to the host through a network cable, and is connected to the robotic arm through a signal output line; the host, a dual-agent reinforcement learning module, a robotic arm control module and a human-computer interaction module are built thereon; the six-dimensional force sensor collects force information during the scanning process, and completes the probe adjustment of the ultrasonic automatic scanning through the force information. In terms of data transmission, the data stream collected by the six-dimensional force sensor is one side. On the one hand, it is transmitted to the human-computer interaction interface for display, and on the other hand, it is transmitted to the dual-agent reinforcement learning module for processing; the dual agents process the force information and torque information respectively, generate corresponding action data and transmit them to the robotic arm control module to realize the control of the robotic arm; the robotic arm control module accepts the data transmitted by the dual-agent reinforcement learning module to complete the control of the physical robotic arm, and the relevant parameter information of the robotic arm is synchronously displayed in the human-computer interaction module during the control process; the human-computer interaction module includes two parts: a data display interface and a button panel. The data display interface displays the data stream collected by the six-dimensional force sensor and the actual posture data of the robotic arm; the button panel calls the robotic arm control module to adjust the posture of the robotic arm by itself. During the system scanning process, the ultrasonic image collected by the ultrasonic probe is displayed in real time on the ultrasonic scanner. After the scanning is completed, the quality of the collected ultrasonic image is manually judged, and the parameters of the dual-agent reinforcement learning module are optimized and adjusted according to the judgment results.

[0006] An automatic ultrasound scanning method based on dual-agent reinforcement learning comprises the following steps: S1. Obtain the force and torque information collected by the six-dimensional force sensor and define the input and output of the dual agent. Obtain the force and torque data collected by the six-dimensional force sensor mounted on the end of the robotic arm and input them into the dual agent, which are recorded as the force agent and the torque agent. The dual agents process the force information and torque information respectively. In the force agent, the action of the agent moving up or down is output according to the input force information, which corresponds to the up and down movement of the end of the robotic arm. Similarly, in the torque agent, the action of the agent deflection is output according to the torque information collected by the sensor, which corresponds to the rotation of the end of the robotic arm. The input and output of the dual agent are defined. S2. Establish dual agents, select reinforcement learning algorithms, build an ultrasound scanning simulation environment, and complete agent reinforcement learning training, which specifically includes the following sub-steps: S2.1 Selection and implementation of reinforcement learning algorithm, including the selection of reinforcement learning algorithm and network construction. In view of the advantages of high policy stability, suitability for dense feedback environment, strong convergence and strong adaptability of the proximal policy optimization algorithm, the PPO algorithm is selected as the reinforcement learning algorithm of the agent, including the actor network and the critic network; in the force agent, the Actor network takes the incoming force information as the state input, and selects the deterministic force agent action according to the strategy, that is, increase or decrease the force. The Critic network calculates the value of the action given by the Actor network, updates the agent's selection strategy, and repeats the above steps with the new strategy until it stops after a certain number of iterations. The strategy selection update of the torque agent is similar to that of the force agent. The Actor network takes the torque information between the probe and the scanning surface as the state input, selects the deterministic torque agent action, increases or decreases the torque, and the Critic network calculates the value of the selected action according to the action given by the Actor network, updates the agent's selection strategy, and repeats the above steps with the new strategy: S2.2 The construction of the Gym simulation environment for ultrasound scanning includes the selection of the simulation platform and the construction of the simulation environment. In reinforcement learning training, the simulation environment is crucial. The state space and reward mechanism can directly affect the strategies and behaviors learned by the agent, thereby affecting the final training results. The Gym library has the advantages of rich environment, unified interface, ease of use and extensive community support. Therefore, the reinforcement learning training environment Gym library is selected to realize the construction of the ultrasound scanning simulation environment. In order to make the ultrasound automatic scanning method independent of environmental information and widely used in different parts and organs, force and torque dual-agent training environments are constructed to learn the adjustment strategies. The construction of the environment includes designing a reasonable state space considering the force and torque factors that affect ultrasound imaging. (i=1, 2), action space And define the reward function based on the constructed ultrasound scanning simulation environment , where i=1 represents the simulated ultrasonic scanning force simulation environment, and i=2 represents the simulated ultrasonic scanning torque simulation environment. Similarly, the state space and action space of the environment correspond to the input and output dimensions of the agent. The specific settings of the reward function are shown below. After completing the relevant settings of the environment, the training strategy of the agent is also implemented. In the simulation environment simulating the scanning force, the force in the Z axis is adjusted to make the actual pressing force during the scanning process as close to the preset expected value as possible. The reward function An even function design is adopted near the preset pressure value. When the difference between the actual pressure and the preset pressure value is less than a certain range, it is considered that no further adjustment is required, and the agent is encouraged to approach the preset value, which is described by formula (1). (1) In the formula, Reward function for the agent, Indicates the actual measured pressure during scanning. Indicates the desired pressing force to be set. Represents a constant close to 0 to prevent the denominator from being 0; Similarly, during the actual scanning process, the ultrasonic probe is kept perpendicular to the contact surface, and the torque value is minimum at this time. In the setting of the torque agent, the torque values ​​of the two dimensions are adjusted to make the actual torque value during the scanning process as close to zero as possible. The reward function also adopts an even function design near zero, which is described by formula (2). (2) In the formula, represents the torque agent reward function, It indicates the torque generated at the end of the robot arm when the probe's forward direction is curved. It indicates the torque generated at the end of the robot arm when the probe is curved perpendicular to the forward direction in the scanning plane. represents the adjustment weight coefficient, and The role and meaning of c in the above sentence are the same; S2.3 Dual-agent training, including simulation data training and measured data training for dual-agents. After completing the construction of the reinforcement learning environment, simulation data training is performed on the dual-agents in the corresponding simulation environment. Random numbers are used to initialize the initial settings of the environment. Reward functions are collected according to the action selections of the agents in the simulation environment. When the results of the simulation data training reach the maximum predicted value through reward function curve observation and convergence, measured data training is performed. The multi-dimensional force information actually collected by the six-dimensional force sensor is used for training to obtain the optimized reinforcement learning PPO network parameters. The specific steps are as follows: S2.3.1 The reinforcement learning agent built using the PPO algorithm is trained in the training environment of learning force and learning torque respectively, and the initial state is randomly initialized within the specified reasonable range; S2.3.2Actor network selects actions based on the initial state of the agent to get the next state and calculates the value of this state; S2.3.3 Repeat steps S2.3.1 and S2.3.2 to iterate the agent’s selection strategy until the preset number of training times is completed; S2.3.4 inputs the force and torque information collected by the six-dimensional force sensor at the end of the robot arm in the current state into the Actor network corresponding to the dual agents, and obtains the action instructions corresponding to each agent; S2.3.5 outputs the motion command to the robot control module and converts it into the actual control of the robot. During the adjustment process, the torque value is adjusted first, and the force is adjusted after the torque adjustment is completed; S2.3.6 After one position adjustment is completed, proceed to the next position movement and adjustment; S2.3.7 Repeat the above steps S2.3.4-S2.3.6 until the preset end position is reached to obtain the optimization parameters of the reinforcement learning PPO algorithm; S3, deploy the optimized reinforcement learning PPO algorithm to the robotic arm to provide automatic ultrasonic scanning control; S4. Test the ultrasonic automatic scanning system and conduct expert evaluation on the quality of the acquired images. Select the ultrasonic automatic scanning area for system testing. Have experts make offline evaluation on the quality of the acquired ultrasonic images during the scanning process. If the image quality does not meet the expert evaluation standards, repeat the above steps S2.2 to S4, adjust the reward function and other parameters, and train and verify the adjusted results until the quality of the images acquired by the system passes the expert evaluation.

[0007] The beneficial effects of the present invention are: an ultrasonic automatic scanning system and method based on dual-agent reinforcement learning, wherein the scanning method includes the following steps: (1) obtaining force and torque information collected by a six-dimensional force sensor and defining the input and output of the dual-agent, (2) establishing a dual-agent, selecting a reinforcement learning algorithm, building an ultrasonic scanning simulation environment, and completing the training of the agent reinforcement learning, (3) deploying the optimized reinforcement learning PPO algorithm to the mechanical arm to provide automatic ultrasonic scanning control, and (4) testing the ultrasonic automatic scanning system and conducting expert evaluation of the quality of the collected image. The present invention adopts a dual-agent reinforcement learning algorithm, and can use simulation data for pre-training in a virtual environment. When the training result is stable at the predicted value and converges, the intelligent body is trained with measured data. Through data learning and optimization control strategies collected during the scanning process, the mechanical arm can be adjusted in real time according to the dynamic information of the target, reducing the influence of external signals on ultrasonic operation, realizing automatic movement, image acquisition and path planning, and the entire process is closed-loop, which improves the real-time effect. At the same time, there is no need to adjust the reinforcement learning model for different parts of different patients, and it has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a system principle block diagram of the present invention.

[0009] Figure 2 It is a flow chart of the steps of the method of the present invention.

[0010] Figure 3 It is a human-computer interaction module interface diagram of the system of the present invention. DETAILED DESCRIPTION

[0011] The present invention will be further described below in conjunction with the accompanying drawings.

[0012] like Figure 1 As shown, an ultrasonic automatic scanning system based on dual-agent reinforcement learning includes an ultrasonic device, a force feedback module, a mechanical arm, a control cabinet and a host, wherein the ultrasonic device includes an ultrasonic scanner and an ultrasonic probe, and the ultrasonic scanner and the ultrasonic probe are connected by a coaxial cable; the force feedback module includes a six-dimensional force sensor and an ultrasonic fixture, wherein the two ends of the six-dimensional force sensor are detachably connected to the ultrasonic fixture and the end of the mechanical arm respectively, the six-dimensional force sensor is connected to the host through a signal output line, and the ultrasonic probe is fixed to the six-dimensional force sensor through the ultrasonic fixture; the control cabinet is connected to the host through a network cable and is connected to the mechanical arm through a signal output line; the host is built with a dual-agent reinforcement learning module, a mechanical arm control module and a human-computer interaction module; the six-dimensional force sensor collects force information during the scanning process, and completes the probe adjustment of the ultrasonic automatic scanning through the force information. In terms of data transmission, the data stream collected by the six-dimensional force sensor is On the one hand, it is transmitted to the human-computer interaction interface for display, and on the other hand, it is transmitted to the dual-agent reinforcement learning module for processing; the dual agents process the force information and torque information respectively, generate corresponding action data and transmit them to the robotic arm control module to realize the control of the robotic arm; the robotic arm control module accepts the data transmitted by the dual-agent reinforcement learning module to complete the control of the physical robotic arm, and the relevant parameter information of the robotic arm is synchronously displayed in the human-computer interaction module during the control process; the human-computer interaction module includes two parts: a data display interface and a button panel. The data display interface displays the data stream collected by the six-dimensional force sensor and the actual posture data of the robotic arm; the button panel calls the robotic arm control module to adjust the posture of the robotic arm by itself. During the system scanning process, the ultrasonic image collected by the ultrasonic probe is displayed in real time on the ultrasonic scanner. After the scanning is completed, the quality of the collected ultrasonic image is manually judged, and the parameters of the dual-agent reinforcement learning module are optimized and adjusted according to the judgment results.

[0013] The dual-agent reinforcement learning module is built on the host using the Python language based on the PyTorch architecture. It consists of two reinforcement learning agents, which process the force information and torque information collected by the force sensor respectively, output the robotic arm logic control instructions based on the incoming data, and pass them to the robotic arm control module through the interface.

[0014] The robotic arm control module is implemented in C++ language, and the control of the robotic arm is achieved through instructions transmitted to the control cabinet by the host. The instructions of the robotic arm control module are converted into actual movements of the robotic arm by calling the built-in control interface. The data processed by the dual-agent reinforcement learning module is transmitted to the robotic arm control module and mapped into motion control of the robotic arm.

[0015] The human-computer interaction module is built based on Qt and mainly includes two parts: a display screen and a key panel. The display screen is used to display the actual state of the robot arm transmitted by the robot arm control module and the force information transmitted by the force feedback module; the key panel is used for the operator to control the robot arm by himself, including the control of the end position of the robot arm, the control of each joint of the robot arm, and the setting of the scanning area. The key panel can transmit instructions to the robot arm control module to realize the physical logic of the robot arm.

[0016] The dual-agent reinforcement learning module, human-computer interaction module and robotic arm control module are deployed on the same host. During the implementation of the system, the information collected by the six-dimensional force sensor will change when the ultrasonic probe contacts the scanning surface. These parameters are passed to the dual-agent for processing. The actions taken by the dual-agent on the incoming data are transmitted to the execution logic of the robotic arm through the interface to adjust the posture of the robotic arm. After that, the force between the ultrasonic probe and the scanning surface changes again, and the system forms a closed loop. In this process, the various joint parameters of the robotic arm and the force information collected by the six-dimensional force sensor are synchronously displayed on the display screen of the human-computer interaction module, and the scanning pressing force is also referenced in the control module. and the torque in two dimensions and The preset threshold value ensures the safety of the system. When the scanning force or torque exceeds the preset range, the system stops running. The robot control module corresponds to the human-computer interaction module. The user can also directly control the robot through the human-computer interaction module to make it reach the desired position.

[0017] like Figure 2 As shown, an automatic ultrasound scanning method based on dual-agent reinforcement learning includes the following steps: S1. Obtain the force and torque information collected by the six-dimensional force sensor and define the input and output of the dual agent. Obtain the force and torque data collected by the six-dimensional force sensor mounted on the end of the robotic arm and input them into the dual agent, which are recorded as the force agent and the torque agent. The dual agents process the force information and torque information respectively. In the force agent, the action of the agent moving up or down is output according to the input force information, which corresponds to the up and down movement of the end of the robotic arm. Similarly, in the torque agent, the action of the agent deflection is output according to the torque information collected by the sensor, which corresponds to the rotation of the end of the robotic arm. The input and output of the dual agent are defined. S2. Establish dual agents, select reinforcement learning algorithms, build an ultrasound scanning simulation environment, and complete agent reinforcement learning training, which specifically includes the following sub-steps: S2.1 Selection and implementation of reinforcement learning algorithm, including the selection of reinforcement learning algorithm and network construction. In view of the advantages of high policy stability, suitability for dense feedback environment, strong convergence and strong adaptability of the proximal policy optimization algorithm, the PPO algorithm is selected as the reinforcement learning algorithm of the agent, including the actor network and the critic network; in the force agent, the Actor network takes the incoming force information as the state input, and selects the deterministic force agent action according to the strategy, that is, increase or decrease the force. The Critic network calculates the value of the action given by the Actor network, updates the agent's selection strategy, and repeats the above steps with the new strategy until it stops after a certain number of iterations. The strategy selection update of the torque agent is similar to that of the force agent. The Actor network takes the torque information between the probe and the scanning surface as the state input, selects the deterministic torque agent action, increases or decreases the torque, and the Critic network calculates the value of the selected action according to the action given by the Actor network, updates the agent's selection strategy, and repeats the above steps with the new strategy: S2.2 The construction of the Gym simulation environment for ultrasound scanning includes the selection of the simulation platform and the construction of the simulation environment. In reinforcement learning training, the simulation environment is crucial. The state space and reward mechanism can directly affect the strategies and behaviors learned by the agent, thereby affecting the final training results. The Gym library has the advantages of rich environment, unified interface, ease of use and extensive community support. Therefore, the reinforcement learning training environment Gym library is selected to realize the construction of the ultrasound scanning simulation environment. In order to make the ultrasound automatic scanning method independent of environmental information and widely used in different parts and organs, force and torque dual-agent training environments are constructed to learn the adjustment strategies. The construction of the environment includes designing a reasonable state space considering the force and torque factors that affect ultrasound imaging. (i=1, 2), action space And define the reward function based on the constructed ultrasound scanning simulation environment , where i=1 represents the simulated ultrasonic scanning force simulation environment, and i=2 represents the simulated ultrasonic scanning torque simulation environment. Similarly, the state space and action space of the environment correspond to the input and output dimensions of the agent. The specific settings of the reward function are shown below. After completing the relevant settings of the environment, the training strategy of the agent is also implemented. In the simulation environment simulating the scanning force, the force in the Z axis is adjusted to make the actual pressing force during the scanning process as close to the preset expected value as possible. The reward function An even function design is adopted near the preset pressure value. When the difference between the actual pressure and the preset pressure value is less than a certain range, it is considered that no further adjustment is required, and the agent is encouraged to approach the preset value, which is described by formula (1). (1) In the formula, Reward function for the agent, Indicates the actual measured pressure during scanning. Indicates the desired pressing force to be set. Represents a constant close to 0 to prevent the denominator from being 0; Similarly, during the actual scanning process, the ultrasonic probe is kept perpendicular to the contact surface, and the torque value is minimum at this time. In the setting of the torque agent, the torque values ​​of the two dimensions are adjusted to make the actual torque value during the scanning process as close to zero as possible. The reward function also adopts an even function design near zero, which is described by formula (2). (2) In the formula, represents the torque agent reward function, It indicates the torque generated at the end of the robot arm when the probe's forward direction is curved. It indicates the torque generated at the end of the robot arm when the probe is curved perpendicular to the forward direction in the scanning plane. represents the adjustment weight coefficient, and The role and meaning of c in the above sentence are the same; S2.3 Dual-agent training, including simulation data training and measured data training for dual-agents. After completing the construction of the reinforcement learning environment, simulation data training is performed on the dual-agents in the corresponding simulation environment. Random numbers are used to initialize the initial settings of the environment. Reward functions are collected according to the action selections of the agents in the simulation environment. When the results of the simulation data training reach the maximum predicted value through reward function curve observation and convergence, measured data training is performed. The multi-dimensional force information actually collected by the six-dimensional force sensor is used for training to obtain the optimized reinforcement learning PPO network parameters. The specific steps are as follows: S2.3.1 The reinforcement learning agent built using the PPO algorithm is trained in the training environment of learning force and learning torque respectively, and the initial state is randomly initialized within the specified reasonable range; S2.3.2Actor network selects actions based on the initial state of the agent to get the next state and calculates the value of this state; S2.3.3 Repeat steps S2.3.1 and S2.3.2 to iterate the agent’s selection strategy until the preset number of training times is completed; S2.3.4 inputs the force and torque information collected by the six-dimensional force sensor at the end of the robot arm in the current state into the Actor network corresponding to the dual agents, and obtains the action instructions corresponding to each agent; S2.3.5 outputs the motion command to the robot control module and converts it into the actual control of the robot. During the adjustment process, the torque value is adjusted first, and the force is adjusted after the torque adjustment is completed; S2.3.6 After one position adjustment is completed, proceed to the next position movement and adjustment; S2.3.7 Repeat the above steps S2.3.4-S2.3.6 until the preset end position is reached to obtain the optimization parameters of the reinforcement learning PPO algorithm; S3, deploy the optimized reinforcement learning PPO algorithm to the robotic arm to provide automatic ultrasonic scanning control; S4. Test the ultrasonic automatic scanning system and conduct expert evaluation on the quality of the acquired images. Select the ultrasonic automatic scanning area for system testing. Have experts make offline evaluation on the quality of the acquired ultrasonic images during the scanning process. If the image quality does not meet the expert evaluation standards, repeat the above steps S2.2 to S4, adjust the reward function and other parameters, and train and verify the adjusted results until the quality of the images acquired by the system passes the expert evaluation.

Claims

1. An ultrasonic automatic scanning system based on dual-agent reinforcement learning, comprising an ultrasonic device, a force feedback module, a robotic arm, a control cabinet and a host, characterized in that: The ultrasonic equipment comprises an ultrasonic scanner and an ultrasonic probe, which are connected to each other via a coaxial cable; the force feedback module comprises a six-dimensional force sensor and an ultrasonic clamp, wherein two ends of the six-dimensional force sensor are detachably connected to the ultrasonic clamp and the end of the robotic arm respectively, the six-dimensional force sensor is connected to the host via a signal output line, and the ultrasonic probe is fixed to the six-dimensional force sensor via the ultrasonic clamp; the control cabinet is connected to the host via a network cable and to the robotic arm via a signal output line; the host is constructed with a dual-agent reinforcement learning module, a robotic arm control module and a human-computer interaction module; the six-dimensional force sensor collects force information during the scanning process, and uses the force information to complete the probe adjustment of the ultrasonic automatic scanning. In terms of data transmission, the data stream collected by the six-dimensional force sensor is transmitted to the human-computer interaction interface display on the one hand, and to the dual-agent on the other hand. The dual-agent reinforcement learning module processes the force information and torque information respectively, generates corresponding action data and transmits it to the robotic arm control module to control the robotic arm; the robotic arm control module receives the data transmitted by the dual-agent reinforcement learning module to complete the control of the physical robotic arm, and the relevant parameter information of the robotic arm is synchronously displayed in the human-computer interaction module during the control process; the human-computer interaction module includes two parts: a data display interface and a button panel. The data display interface displays the data stream collected by the six-dimensional force sensor and the actual posture data of the robotic arm; the button panel calls the robotic arm control module to adjust the posture of the robotic arm by itself. During the system scanning process, the ultrasonic image collected by the ultrasonic probe is displayed in real time on the ultrasonic scanner. After the scanning is completed, the quality of the collected ultrasonic image is manually judged, and the parameters of the dual-agent reinforcement learning module are optimized and adjusted according to the judgment results.

2. An automatic ultrasonic scanning method based on dual-agent reinforcement learning, characterized in that: Using the system of claim 1 to perform automatic ultrasonic scanning based on dual-agent reinforcement learning, comprising the following steps: S1. Obtain the force and torque information collected by the six-dimensional force sensor and define the input and output of the dual agent. Obtain the force and torque data collected by the six-dimensional force sensor mounted on the end of the robotic arm and input them into the dual agent, which are recorded as the force agent and the torque agent. The dual agents process the force information and torque information respectively. In the force agent, the action of the agent moving up or down is output according to the input force information, which corresponds to the up and down movement of the end of the robotic arm. Similarly, in the torque agent, the action of the agent deflection is output according to the torque information collected by the sensor, which corresponds to the rotation of the end of the robotic arm. The input and output of the dual agent are defined. S2. Establish dual agents, select reinforcement learning algorithms, build an ultrasound scanning simulation environment, and complete agent reinforcement learning training, which specifically includes the following sub-steps: S2.1 Selection and implementation of reinforcement learning algorithm, including the selection of reinforcement learning algorithm and network construction. In view of the advantages of high policy stability, suitability for dense feedback environment, strong convergence and strong adaptability of the proximal policy optimization algorithm, the PPO algorithm is selected as the reinforcement learning algorithm of the agent, including the actor network and the critic network; in the force agent, the Actor network takes the incoming force information as the state input, and selects the deterministic force agent action according to the strategy, that is, increase or decrease the force. The Critic network calculates the value of the action given by the Actor network, updates the agent's selection strategy, and repeats the above steps with the new strategy until it stops after a certain number of iterations. The strategy selection update of the torque agent is similar to that of the force agent. The Actor network takes the torque information between the probe and the scanning surface as the state input, selects the deterministic torque agent action, increases or decreases the torque, and the Critic network calculates the value of the selected action according to the action given by the Actor network, updates the agent's selection strategy, and repeats the above steps with the new strategy: S2.2 The construction of the Gym simulation environment for ultrasound scanning includes the selection of the simulation platform and the construction of the simulation environment. In reinforcement learning training, the simulation environment is crucial. The state space and reward mechanism can directly affect the strategies and behaviors learned by the agent, thereby affecting the final training results. The Gym library has the advantages of rich environment, unified interface, ease of use and extensive community support. Therefore, the reinforcement learning training environment Gym library is selected to realize the construction of the ultrasound scanning simulation environment. In order to make the ultrasound automatic scanning method independent of environmental information and widely used in different parts and organs, force and torque dual-agent training environments are constructed to learn the adjustment strategies. The construction of the environment includes designing a reasonable state space considering the force and torque factors that affect ultrasound imaging. (i=1, 2), action space And define the reward function based on the constructed ultrasound scanning simulation environment , where i=1 represents the simulated ultrasonic scanning force simulation environment, and i=2 represents the simulated ultrasonic scanning torque simulation environment. Similarly, the state space and action space of the environment correspond to the input and output dimensions of the agent. The specific settings of the reward function are shown below. After completing the relevant settings of the environment, the training strategy of the agent is also implemented. In the simulation environment simulating the scanning force, the force in the Z axis is adjusted to make the actual pressing force during the scanning process as close to the preset expected value as possible. The reward function An even function design is adopted near the preset pressure value. When the difference between the actual pressure and the preset pressure value is less than a certain range, it is considered that no further adjustment is required, and the agent is encouraged to approach the preset value, which is described by formula (1). （1）； In the formula, Reward function for the agent, Indicates the actual measured pressure during scanning. Indicates the desired pressing force to be set. Represents a constant close to 0 to prevent the denominator from being 0; Similarly, during the actual scanning process, the ultrasonic probe is kept perpendicular to the contact surface, and the torque value is minimum at this time. In the setting of the torque agent, the torque values ​​of the two dimensions are adjusted to make the actual torque value during the scanning process as close to zero as possible. The reward function also adopts an even function design near zero, which is described by formula (2). （2）； In the formula, represents the torque agent reward function, It indicates the torque generated at the end of the robot arm when the probe's forward direction is curved. It indicates the torque generated at the end of the robot arm when the probe is curved perpendicular to the forward direction in the scanning plane. represents the adjustment weight coefficient, and The role and meaning of c in the above sentence are the same; S2.3 Dual-agent training, including simulation data training and measured data training for dual-agents. After completing the construction of the reinforcement learning environment, simulation data training is performed on the dual-agents in the corresponding simulation environment. Random numbers are used to initialize the initial settings of the environment. Reward functions are collected according to the action selections of the agents in the simulation environment. When the results of the simulation data training reach the maximum predicted value through reward function curve observation and convergence, measured data training is performed. The multi-dimensional force information actually collected by the six-dimensional force sensor is used for training to obtain the optimized reinforcement learning PPO network parameters. The specific steps are as follows: S2.3.1 The reinforcement learning agent built using the PPO algorithm is trained in the training environment of learning force and learning torque respectively, and the initial state is randomly initialized within the specified reasonable range; S2.3.2Actor network selects actions based on the initial state of the agent to get the next state and calculates the value of this state; S2.3.3 Repeat steps S2.3.1 and S2.3.2 to iterate the agent’s selection strategy until the preset number of training times is completed; S2.3.4 inputs the force and torque information collected by the six-dimensional force sensor at the end of the robot arm in the current state into the Actor network corresponding to the dual agents, and obtains the action instructions corresponding to each agent; S2.3.5 outputs the motion command to the robot control module and converts it into the actual control of the robot. During the adjustment process, the torque value is adjusted first, and the force is adjusted after the torque adjustment is completed; S2.3.6 After one position adjustment is completed, proceed to the next position movement and adjustment; S2.3.7 Repeat the above steps S2.3.4-S2.3.6 until the preset end position is reached to obtain the optimization parameters of the reinforcement learning PPO algorithm; S3, deploy the optimized reinforcement learning PPO algorithm to the robotic arm to provide automatic ultrasonic scanning control; S4. Test the ultrasonic automatic scanning system and conduct expert evaluation on the quality of the acquired images. Select the ultrasonic automatic scanning area for system testing. Have experts make offline evaluation on the quality of the acquired ultrasonic images during the scanning process. If the image quality does not meet the expert evaluation standards, repeat the above steps S2.2 to S4, adjust the reward function and other parameters, and train and verify the adjusted results until the quality of the images acquired by the system passes the expert evaluation.

Images