Multi-modal interaction system and method based on man-machine cooperation

Abstract

The invention provides a multi-modal interaction system based on man-machine collaboration, a training method, a training device, equipment and a medium, a trained deep reinforcement learning model is adopted, ultrasonic image, touch force and visual image features are fused, the fused features are processed by using a PPO algorithm, corresponding control instruction data are obtained, and the control instruction data are sent to a user. The mechanical arm is controlled to carry the ultrasonic probe to act, complete autonomous ultrasonic imaging of a soft, movable and unmarked action target is achieved, robustness is guaranteed, and meanwhile flexibility is improved.

Description

Technical Field

[0001] The present application relates to the field of medical technology, and in particular to a multimodal interaction system, training method, apparatus, equipment and medium based on human-computer collaboration. Background Art

[0002] Completing robotic autonomous ultrasound imaging through human-robot collaboration is a challenging task because the robot requires strong analytical capabilities to make intelligent decisions in complex spatial relationships.

[0003] While previous studies have demonstrated the effectiveness of equipping robots with visual and tactile perception capabilities for tasks involving complex contact interactions, different forms of data have different impacts on task execution, and processing multiple sensor data introduces significant complexity.

[0004] Numerous studies have demonstrated the feasibility and effectiveness of ultrasound image-based robot navigation decision-making. However, most methods rely on a single data modality for modeling and fail to fully utilize diverse sensor information for decision-making, resulting in limited flexibility and robustness. Summary of the Invention

[0005] This application proposes a multimodal interaction system, training method, device, equipment and medium based on human-computer collaboration, which can solve one of the problems existing in the background technology.

[0006] To achieve the above objectives, this application adopts the following technical solutions:

[0007] In a first aspect, a multimodal interaction system based on human-computer collaboration is provided, the interaction system comprising:

[0008] A robotic arm equipped with an ultrasonic probe and a force sensor, the force sensor being used to sense the tactile force between the ultrasonic probe and an object; a camera that captures the robotic arm and the object; and

[0009] A computer is used to obtain an ultrasonic image sequence through the ultrasonic probe, a tactile force sequence through the force sensor, and a visual image sequence through the camera, wherein the ultrasonic image sequence reflects the position of a scanning target, the tactile force sequence reflects the tactile force between the ultrasonic probe and the target, and the visual image sequence reflects the posture of the robotic arm and the relative position between the ultrasonic probe and the target; based on a pre-trained deep reinforcement learning model, feature extraction and feature fusion are performed on the ultrasonic image sequence, the tactile force sequence, and the visual image sequence to obtain fused features; and the proximal strategy optimization (PPO) algorithm is used to process the fused features to obtain control instruction data for controlling the movement of the robotic arm.

[0010] Based on the above technical solution, a trained deep reinforcement learning model is used to fuse ultrasonic image, tactile force and visual image features, and the PPO algorithm is used to process the fused features to obtain the corresponding control command data to control the movement of the robotic arm carrying the ultrasonic probe, realizing fully autonomous ultrasonic imaging of soft, movable and unmarked targets, while ensuring robustness and improving flexibility.

[0011] In addition, experimental verification has found that adding tactile force to the fusion features can not only improve the control accuracy, but also help accelerate model convergence and improve training efficiency.

[0012] In a possible design manner of the first aspect, the deep reinforcement learning model includes:

[0013] a multimodal feature fusion module, configured to extract and fuse features of the ultrasound image sequence, the tactile force sequence, and the visual image sequence to obtain fused features; and

[0014] A PPO agent module is used to process the fusion features using a proximal strategy optimization PPO algorithm to obtain the control instruction data.

[0015] The reward function of the PPO agent module is defined by the distance and contact maintenance between the ultrasound probe and the target, the ultrasound image quality, and the tactile force.

[0016] In a possible design method of the first aspect, the multimodal feature fusion module includes: an ultrasonic feature extraction module, a tactile feature extraction module and a visual feature extraction module, the ultrasonic feature extraction module includes: a UNet-type convolutional neural network CNN sub-module designed in sequence, several convolution layers and a feature encoding layer, the tactile feature extraction module includes: a recurrent neural network RNN ​​sub-module designed in sequence, a long short-term memory network LSTM layer and a temporal encoding layer, and the visual feature extraction module includes: a UNet-type CNN sub-module designed in sequence, several convolution layers and a downsampling layer.

[0017] In a possible design manner of the first aspect, the computer is further used to: build a virtual simulation platform to pre-train the deep reinforcement learning model.

[0018] In a possible design manner of the first aspect, the computer is further used to: after the pre-training, in a real environment, fine-tune the parameters of the deep reinforcement learning model based on the structural similarity index SSIM between the standard ultrasound image and the proxy ultrasound image obtained by imaging using the ultrasound scanning system.

[0019] Based on the above technical solution, the model is first pre-trained on a virtual simulation platform, and then the model parameters are fine-tuned in a real environment. This not only improves the generalization ability of the model in a real environment, but also reduces the actual training requirements and accelerates the transition from simulation to reality.

[0020] In a second aspect, a training method for a deep reinforcement learning model is provided, the training method comprising:

[0021] Obtaining an ultrasound image sequence, a tactile force sequence, and a visual image sequence, wherein the visual image sequence includes a robotic arm and an action target within a shooting range, the robotic arm being equipped with an ultrasound probe and a force sensor, the ultrasound image sequence reflecting the position of the scanned target, the tactile force sequence reflecting the tactile force between the ultrasound probe and the action target, and the visual image sequence reflecting the posture of the robotic arm and the relative position between the ultrasound probe and the action target;

[0022] performing feature extraction and feature fusion on the ultrasound image sequence, the tactile force sequence, and the visual image sequence to obtain fusion features; and

[0023] The proximal policy optimization (PPO) algorithm is used to process the fusion features to obtain control instruction data for controlling the movement of the robotic arm.

[0024] In a possible design of the second aspect, the training method is based on the constructed virtual simulation platform.

[0025] In a third aspect, a training device for a deep reinforcement learning model is provided, the training device comprising:

[0026] an acquisition unit, configured to obtain an ultrasound image sequence, a tactile force sequence, and a visual image sequence, wherein the visual image sequence includes a robotic arm and an action target within a shooting range, the robotic arm being equipped with an ultrasound probe and a force sensor, the ultrasound image sequence reflecting the position of the scanned target, the tactile force sequence reflecting the tactile force between the ultrasound probe and the action target, and the visual image sequence reflecting the posture of the robotic arm and the relative position between the ultrasound probe and the action target;

[0027] a fusion unit, configured to extract and fuse features of the ultrasound image sequence, the tactile force sequence, and the visual image sequence to obtain fusion features; and

[0028] A processing unit is used to process the fusion features using a proximal policy optimization (PPO) algorithm to obtain control instruction data for controlling the movement of the robotic arm.

[0029] In a fourth aspect, an electronic device is provided, comprising: a processor, and a memory coupled to the processor, the memory being used to store a computer program; and the processor being used to execute the computer program stored in the memory, so that the electronic device performs the training method as any possible implementation in the second aspect.

[0030] In a fifth aspect, a computer-readable storage medium is provided, comprising a computer program or instructions, which, when executed on a computer, causes the computer to execute the training method of any possible implementation of the second aspect.

[0031] In a sixth aspect, a computer program product is provided, comprising: a computer program or instructions, which, when the computer program or instructions are run on a computer, causes the computer to execute the training method of any possible implementation of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0033] Figure 1 is a structural diagram of the ultrasonic scanning system provided in Example 1 of the present application;

[0034] Figure 2 Schematic diagram of a multimodal autonomous ultrasound scanning system provided in Example 2 of the present application, wherein: (a) a 6-DOF robotic arm; (b) an ultrasound instrument, including an ultrasound probe; (c) a force sensor; (d) a camera; (e) a thyroid prosthesis; and (f) a computer.

[0035] Figure 3 This is a structural diagram of a multimodal autonomous ultrasonic scanning system solution provided in Example 2 of the present application;

[0036] Figure 4 This is a schematic diagram of autonomous navigation based on multimodal feature fusion provided in Example 2 of the present application;

[0037] Figure 5 This is a comparison of different baselines in a simulation environment provided by Example 2 of this application, including (a) training curve comparison; (b) scanning success rate comparison;

[0038] Figure 6This is an evaluation of autonomous ultrasound scanning in a real environment provided by Example 2 of the present application, including (a) comparison of training curves; (b) robustness evaluation of scanning success rates in different environments;

[0039] Figure 7 This is a snapshot of the trajectory of the ultrasound scanning process provided in Example 2 of the present application, wherein: (a) normal mode: the robot-controlled probe reaches the surface of the soft target, performs scanning, and obtains the required standard ultrasound image; (b) visual impairment caused by artifacts and conclusions. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0041] Example 1

[0042] like Figure 1 As shown, the multimodal interaction system based on human-computer collaboration of the first embodiment of the present application includes:

[0043] a robotic arm 103 equipped with an ultrasonic probe 101 and a force sensor 102, wherein the force sensor 102 is used to sense the tactile force between the ultrasonic probe 101 and the target 100; a camera 104 that captures the robotic arm 103 and the target 100; and

[0044] The computer 105 is used to obtain an ultrasonic image sequence through the ultrasonic probe 101, obtain a tactile force sequence through the force sensor 102, and obtain a visual image sequence through the camera 104, wherein the ultrasonic image sequence reflects the position of the scanning target, the tactile force sequence reflects the tactile force between the ultrasonic probe 101 and the action target 100, and the visual image sequence reflects the posture of the robotic arm 103 and the relative position between the ultrasonic probe 101 and the action target 100; based on a pre-trained deep reinforcement learning model, feature extraction and feature fusion are performed on the ultrasonic image sequence, the tactile force sequence and the visual image sequence to obtain fusion features; and the proximal strategy optimization PPO algorithm is used to process the fusion features to obtain the control instruction data.

[0045] Specifically, the ultrasonic probe 101 and the ultrasonic instrument body together constitute the ultrasonic instrument. The ultrasonic probe 101 serves as a sensing component, transmitting the sensed data to the ultrasonic instrument body for imaging, thereby obtaining a corresponding ultrasonic image sequence.

[0046] The force sensor 102 can convert the force value into a related electrical signal. In specific applications, it can be implemented using a strain gauge, diaphragm, or other form of force sensor. When the ultrasonic probe 101 is placed at the end of the robotic arm 103, the force sensor 102 can be placed between the end of the robotic arm 103 and the main body of the robotic arm. Then, when the ultrasonic probe 101 contacts the target 100, the relative position between the end of the robotic arm 103 and the main body of the robotic arm 103 changes. The force sensor 102 can capture this relative position change, which essentially reflects the tactile force between the ultrasonic probe 101 and the target 100, and then converts it into a related electrical signal, namely a tactile force sequence.

[0047] The robotic arm 103 can employ a multi-degree-of-freedom robotic arm structure. By adjusting the position of the robotic arm 103, the ultrasound probe 101 can be positioned at a desired location. The position of the robotic arm 103 is generally expressed as position P and attitude θ, and the position adjustment amount of the robotic arm 103 can be expressed as ΔP and Δθ, respectively.

[0048] The camera 104 can be stably placed on one side of the target 100, and the robotic arm 103 and the target 100 can be included in the shooting range. In this way, the camera 104 can capture the posture of the robotic arm 103 and the relative position relationship between the ultrasound probe 101 and the target 100 by shooting the obtained visual image sequence.

[0049] After obtaining the ultrasound image sequence, tactile force sequence, and visual image sequence, the computer 105 processes these sequences based on a pre-trained deep reinforcement learning model, and ultimately obtains control instruction data for controlling the movement of the robotic arm 103 .

[0050] The deep reinforcement learning model can specifically include two networks (or modules), one is a multimodal feature fusion network, and the other is a PPO agent network.

[0051] The multimodal feature fusion network is used to extract and fuse features from ultrasound image sequences, tactile force sequences, and visual image sequences to obtain fused features. Multimodal feature extraction can be achieved using convolutional neural networks (CNNs), recurrent neural networks (RNNs), and long short-term memory networks (LSTMs) with corresponding convolutional layers, feature encoding layers, temporal encoding layers, and downsampling layers. Multimodal feature fusion can be achieved using attention mechanisms, residual connections, and multi-scale feature fusion. Of course, multimodal features also need to be normalized and standardized to make the different modal features suitable for subsequent fusion.

[0052] The Proximal Policy Optimization (PPO) agent network is a specific implementation of an intelligent agent within the Markov Decision Process (MDP) framework. An MDP consists of the following elements:

[0053] State space (States): denoted as S, represents the state of the environment in which the agent is located.

[0054] Action space (Actions): denoted as A, represents the possible actions that can be performed in each state.

[0055] Transition Function: The probability distribution of transitioning from state s to a new state s' after executing action a, denoted as P(s'│s,a).

[0056] Reward Function: After each action is performed, an immediate reward will be obtained, denoted as r(s,a).

[0057] Discount Factor: denoted as γ∈[0,1), which is used to balance the importance of short-term rewards and long-term rewards.

[0058] The goal of the agent is to find an optimal policy π* that maximizes the expected cumulative reward during its interactions with the environment. One of the core concepts of reinforcement learning is how to search for or optimize the policy π(a│s). This can be achieved using methods such as value-based, policy-based, and actor-critic hybrid approaches.

[0059] PPO belongs to the school of thought that directly updates policy parameters using gradients, also known as policy gradient methods, and can handle problems in continuous action spaces. The PPO algorithm is an online algorithm that generates data using the current policy and uses this data to update the policy. The principle of PPO is to minimize the cost function at each policy update step while ensuring that the difference between the new policy and the old policy is not too large.

[0060] In this embodiment, the reward function R of the PPO agent module is t The distance and contact between the ultrasound probe and the target can be maintained by distance , ultrasound image quality R us and the tactile force R force The definitions are as follows:

[0061] R t =ω1R distance +ω2R us +ω3R force

[0062] Among them, ω1, ω2 and ω3 are weights.

[0063] In order to reduce the actual training requirements, a virtual simulation platform can be built on the computer 105 to pre-train the deep reinforcement learning model.

[0064] After this pre-training, to accelerate the transition from simulation to real-world implementation, the deep reinforcement learning model can be fine-tuned in real-world environments based on the Structural Similarity Index (SSIM). Specifically, the SSIM between standard ultrasound images and images generated using the ultrasound scanning system can be used to fine-tune the parameters of the deep reinforcement learning model.

[0065] The above process of simulation pre-training followed by real-world fine-tuning can also improve the generalization ability of the model in real environments.

[0066] This embodiment also provides a training method and training device for a deep reinforcement learning model.

[0067] Wherein, the training method includes:

[0068] Obtaining an ultrasound image sequence, a tactile force sequence, and a visual image sequence, wherein the visual image sequence includes a robotic arm and an action target within a shooting range, the robotic arm being equipped with an ultrasound probe and a force sensor, the ultrasound image sequence reflecting the position of the scanned target, the tactile force sequence reflecting the tactile force between the ultrasound probe and the action target, and the visual image sequence reflecting the posture of the robotic arm and the relative position between the ultrasound probe and the action target;

[0069] performing feature extraction and feature fusion on the ultrasound image sequence, the tactile force sequence, and the visual image sequence to obtain fusion features; and

[0070] The proximal policy optimization (PPO) algorithm is used to process the fusion features to obtain control instruction data for controlling the movement of the robotic arm.

[0071] The training device comprises:

[0072] an acquisition unit, configured to obtain an ultrasound image sequence, a tactile force sequence, and a visual image sequence, wherein the visual image sequence includes a robotic arm and an action target within a shooting range, the robotic arm being equipped with an ultrasound probe and a force sensor, the ultrasound image sequence reflecting the position of the scanned target, the tactile force sequence reflecting the tactile force between the ultrasound probe and the action target, and the visual image sequence reflecting the posture of the robotic arm and the relative position between the ultrasound probe and the action target;

[0073] a fusion unit, configured to extract and fuse features of the ultrasound image sequence, the tactile force sequence, and the visual image sequence to obtain fusion features; and

[0074] A processing unit is used to process the fusion features using a proximal policy optimization (PPO) algorithm to obtain control instruction data for controlling the movement of the robotic arm.

[0075] The content of the training method and training device in specific implementation is similar to the above and will not be repeated here.

[0076] Example 2

[0077] The following is an illustrative example of the ultrasonic scanning system and training method of the embodiment of the present application through a specific application example.

[0078] The present embodiment provides a fully automatic ultrasonic robot scanning method based on a multimodal model of human-machine collaboration, which allows the robot to comprehensively utilize various sensor information sources and make autonomous decisions to perform ultrasonic scanning while ensuring the reliability and stability of the scanning task, thereby avoiding the need for human intervention to obtain the best ultrasonic image.

[0079] This example draws inspiration from the ultrasound scanning process performed by human doctors, integrating visual and tactile information into an ultrasound robotic system and exploring the impact of different information modalities on the task. The multimodal deep reinforcement learning (DRL) framework proposed in this example can integrate real-time visual feedback and tactile perception, and directly output 6D posture decisions to control the ultrasound probe, thereby achieving fully autonomous ultrasound imaging of soft, movable, and unmarked targets.

[0080] Based on this framework, relevant devices are deployed in actual scenarios, such as Figure 2 As shown, a 6-DOF UR5 robotic arm is used, with a six-axis force sensor and an ultrasonic probe mounted on the end effector. The force sensor is connected to the robotic arm control box, outputting six-axis force data at a frequency of 60Hz. The ultrasonic probe is connected to an ultrasound machine, capable of transmitting ultrasound image data at a frequency of 30Hz. Furthermore, a Realsense D435i camera is installed directly in front of the robotic arm and the model, providing real-time scene imaging at a frequency of 60Hz. A gigabit switch is used to transmit all data to the host LAN, which then outputs corresponding actions to control the robot's motion.

[0081] This work is motivated by the increasing demand for intelligent human-computer interaction in medical applications. By increasing the degree of automation in traditional medical scanning procedures, such as ultrasound scanning, the efficiency of medical scanning can be greatly improved.

[0082] To address the above problems, we first proposed a multimodal deep reinforcement learning model. The model structure is as follows: Figure 3 As shown, the model includes the following modules:

[0083] The visual positioning module is used to locate the target and obtain the collected images.

[0084] The tactile feedback module is used to adjust the scanning force during the scanning process and obtain information about the contact force of the object.

[0085] The simulation transfer module is used to extend the navigation strategies learned in the simulator to real-world applications, improving the generalization ability of the model in real environments.

[0086] Based on the autonomous navigation module of multimodal feature fusion, the robotic arm carrying the ultrasound probe autonomously navigates to the standard scanning plane through visual information and force sensing information.

[0087] The visual positioning module primarily functions for visual observation and target positioning. Visual perception provides both environmental observation and accurate target positioning. Scene images captured by external cameras contain key information such as the robot's posture and the distance between the probe and the scanning surface. Visual perception guides the robot to the target and navigates to the optimal position for optimal ultrasound images. By stacking ultrasound image sequences taken at different time points, this information provides information about the position of the scanned target at different times. These potential positional relationships guide the robot's navigation decisions and help it achieve the optimal posture for optimal ultrasound images. The six-dimensional data captured by the force sensor measures the pressure between the probe and the contact surface, ensuring safe contact forces and avoiding excessive tissue compression, thereby ensuring safe and stable robot control. This information is crucial to the robot's overall decision-making process during ultrasound scanning. In the first stage of ultrasound probe control, visual perception input guides the robot to the target and navigates to the optimal position for optimal ultrasound images.

[0088] During the implementation of this solution, the safety strategy of human-computer interaction was fully considered. The six-dimensional data provided by the force sensor in tactile perception can ensure precise control of the probe force, so that the robotic arm can safely and effectively reach the scanning plane with the assistance of visual information. In the subsequent scanning process, the safety of the robotic arm, probe and user is also taken into consideration, greatly reducing the possibility of accidental collisions among the three.

[0089] The tactile feedback module obtains tactile feedback through the force control sensor module in the manipulator's tactile sense. The main function of tactile feedback is to obtain contact force information and achieve stability force control.

[0090] Tactile perception provides information about contact forces on objects, enabling robots to perform tasks requiring precision and dexterity, such as manipulating objects of varying shapes, sizes, and textures. Tactile perception is particularly important when the ultrasound probe makes minute adjustments after contact with the target, as it provides six-dimensional data about the force applied by the probe, ensuring safe and stable robot control.

[0091] In the second stage of ultrasound probe control, when the probe is scanned along the target surface, tactile perception serves as the basis for evaluating the scanning task, and the positional relationships within the underlying ultrasound image can guide the robot's navigation decisions to achieve the optimal posture for obtaining the best ultrasound image.

[0092] Simulation transfer includes a two-stage framework of simulation and migration from simulation to real environment, which enables this solution to be applied in real life. This solution pre-trains a multimodal deep reinforcement learning model by building a simulation platform that is highly similar to the real environment. The model can integrate visual and tactile information to control the ultrasound probe. Through domain randomization and a large amount of interactive training in the simulation environment, the model learns to make decisions in complex scenarios. This pre-trained model is then transferred to the real environment, and its performance is further optimized through additional fine-tuning to adapt to the specific conditions of the real world. This approach not only improves the generalization ability of the model in the real environment, but also reduces the need for actual training and accelerates the transition from simulation to reality.

[0093] In summary, this scheme demonstrates a two-stage learning framework for transfer from simulation to reality. By pre-training in a simulation environment and fine-tuning in a real environment, a multimodal autonomous ultrasound scanning system can effectively perform tasks in the real world, improving the efficiency of human-computer interaction and the quality of medical diagnosis.

[0094] In autonomous navigation based on multimodal feature fusion, this solution constructs a deep reinforcement learning framework that can integrate real-time visual feedback and tactile perception, and directly outputs the 6D posture decision for controlling the ultrasound probe. First, a parallel deep neural network is used to extract the data features of ultrasound images, visual perception, and tactile perception respectively, and then these feature vectors are merged to form a multimodal fusion feature representation vector for the robot's decision-making process. This method enables the robot to navigate and control the ultrasound probe more accurately to obtain high-quality ultrasound images. On this basis, the simulation-based training environment can assist the robotic arm to make safer and more accurate movements in real-world scenarios. The overall framework is as follows: Figure 4 As shown in Figure 2, the entire multimodal information fusion process is demonstrated.

[0095] This solution effectively integrates visual and tactile information through technical means, improving the accuracy and efficiency of autonomous ultrasound scanning systems. By processing and integrating data from different modalities using deep learning models, the system can make better decisions in complex environments, playing a vital role in medical diagnosis.

[0096] Specifically, the autonomous navigation method based on multimodal feature fusion includes the following steps:

[0097] Step 1: Obtain ultrasound, vision, and tactile modal information data through a camera, ultrasound probe, and robotic arm.

[0098] Step 2: A multi-feature fusion network is used to fuse the three multimodal data obtained in step 1 to obtain the fused feature data of the robot in the ultrasonic scanning task.

[0099] Step three: Input the acquired fusion feature data into the pre-trained multimodal deep reinforcement learning model to obtain the robotic arm control command data.

[0100] In this step, the pre-trained multimodal deep reinforcement learning model is used to perform strategy optimization on the fused feature data obtained in step 2 to obtain the optimal control action.

[0101] The multimodal deep reinforcement learning model includes the following modules:

[0102] Ultrasound feature extraction module, used to process ultrasound image sequences and extract high-level features, including: UNet-type CNN submodule, 4 convolutional layers and feature encoding layer.

[0103] The visual feature extraction module is used to process visual image sequences, including: CNN submodule, 3 convolutional layers and downsampling layers.

[0104] The tactile feature extraction module is used to process force sequence data, including: RNN submodule, LSTM layer and time series encoding layer.

[0105] The feature fusion module is used to concatenate the trimodal features into a 576-dimensional feature vector.

[0106] The PPO agent module is used to output control instructions based on the PPO algorithm.

[0107] Step 4: Control the robotic arm to move / navigate according to the control instruction data obtained in step 3.

[0108] The training method (pre-training method) of the multimodal deep reinforcement learning model includes the following steps:

[0109] In step 1, the environment of the ultrasonic robot system is initialized, and the Markov decision process (MDP) framework is defined. The input is the PyBullet virtual environment configuration parameters and the initial posture of the robot. The posture data includes position and attitude. The output is the initial multimodal observation state, and a 300×300×4 ultrasound image sequence, a 200×200×4 visual image sequence and a 6×16 six-axis force sensor data are obtained.

[0110] In this paper, the ultrasonic robot system is represented as a Markov decision process (MDP) for continuous space tasks. MDP is a framework for sequential decision problems in a fully observable random environment, consisting of a Markov transition model and additional rewards. At time step t, the MDP consists of a set of states S, a set of actions A, a state transition function P(s′|s,a), a reward function R(s) and a discount factor γ∈(0,1]. To obtain the optimal policy π(a,s), we parameterize the policy as π θ (a|s), where θ represents the policy parameters, so the goal of this task is to maximize the expected discounted reward J(θ):

[0111]

[0112] In order to maximize the expected discounted reward, a policy gradient-based method is used to optimize θ. The gradient formula is as follows, where Q πθ Indicates that after executing action a in state s, following strategy π θ The expected value of the cumulative discounted return that can be obtained is:

[0113]

[0114] In step 2, the force, ultrasound, and visual image data from step 1 are fed into a pre-built multimodal deep reinforcement learning model to obtain the fused features required by the reinforcement learning model. The fused features serve as input data for the PPO agent network.

[0115] The pre-built multimodal deep reinforcement learning model consists of three parallel deep neural network modules, each processing high-dimensional data from ultrasound, visual perception, and tactile perception. Two of these modules utilize a convolutional neural network (CNN) similar to the Unet architecture to process ultrasound image sequences (input size 300×300×4) and grayscale visual image sequences (input size 200×200×4), respectively. The former outputs a 256-dimensional feature vector, while the latter outputs a 128-dimensional feature vector. The third module, for tactile perception, utilizes a three-layer recurrent neural network (RNN) to process a 6×16-dimensional six-axis force sequence and output a 64-dimensional feature vector.

[0116] The observation space in the environment consists of three parts: an ultrasound image sequence with a size of 300 × 300 × 4, a visual image sequence with a size of 200 × 200 × 4, and a force sequence with a size of 6 × 16. The dimension of each image is represented by the pixel intensity value P i Indicates P i ∈[0,255), the higher the value, the brighter the pixel. Each set of force data F=[f x ,f y ,f z ,r x ,r y,r z )] represents the tactile perception of the ultrasound probe, where [fx, f y , f z ] and [r x ,r y ,r z ] represent the forces and moments acting on the probe in the x, y, and z directions, respectively. Because the observation data contains multiple modalities, a multimodal feature fusion network is constructed to integrate the high-dimensional information from different modalities into a compact feature representation. This fused representation serves as the actual state input to the PPO agent network.

[0117] In step 3, the PPO agent network optimizes the strategy parameters based on the fusion features input in step 2 to obtain the action parameters and the actual action to be performed.

[0118] For the action space, the robot's control action is based on the position offset ΔP of the end effector and the posture adjustment Δθ. Both belong to the action space A. If the output action exceeds the action space A, they will be clipped. Assume that the position of the end effector at time t is P t , then the expected end-effector posture that the robot should achieve at the next time step t+1 is:

[0119] P t+1 =P t +ΔP·R(θ)·e x

[0120] Where R(θ) is the training rotation matrix generated by the posture adjustment Δθ, e x is the unit vector in the x direction.

[0121] In step 4, based on the current state feedback obtained in step 3, such as the distance d between the probe and the target, the contact stability, the clarity of the ultrasound image Q, and the magnitude of the contact force F, a reward function is calculated for policy update in the PPO agent network.

[0122] During the training process of the ultrasound robot system, the robot continuously observes and interacts with the environment, using environmental feedback to iterate its strategy. To this end, it is necessary to design effective reward feedback to enable the robot to learn customized scanning tasks. In the robotic scanning task, the main factors affecting the robot's decision-making include the distance between the probe and the target and the contact maintenance R distanc , ultrasound image quality R ultrasoun (abbreviated as R us ) and the magnitude of the tactile force R force In reward function design, these factors are combined to calculate the reward the robot receives from a single decision using constant weights ω1, ω2, ω3:

[0123] Rt =ω1R distance +ω2R us +ω3R force

[0124] Before the probe makes contact with the target, the robot is encouraged to approach the target by using the difference between the Euclidean distance between the probe and the target position after the action decision and the Euclidean distance before the decision as a reward. Once the probe reaches the target and an ultrasound image is available, the distance reward is canceled to prevent the probe from remaining stationary. Instead, the probe is encouraged to adjust its position to explore better ultrasound images:

[0125]

[0126] represents the Euclidean distance, where p targe Indicates the target location, represents the position of the probe in the previous time step, Indicates the location of the probe at the current time step.

[0127] After the probe contacts the target, we use the structural similarity index (SSIM) to evaluate the quality of the current ultrasound image:

[0128]

[0129] First, the structural similarity between images x and y is calculated based on the evaluation of brightness, contrast and structure. μx and μy are the average values ​​of x and y respectively. σ x and σ y Represents the standard deviation. xy represents the covariance between x and y. c1 and c2 are constants used to stabilize the division and avoid division by zero.

[0130] Calculate the current ultrasound image ultrasound (us t ) and our predefined ultrasound standard image ultrasound (us std ) as a reward. The more similar the ultrasound image obtained by the robot after performing the action is to our predefined standard image, the higher the reward. This encourages the robot to continuously explore and obtain better ultrasound images:

[0131]

[0132] where ω e represents the constant reward obtained in the presence of ultrasound. This motivates the probe to maintain contact with the surface and continue moving. usis the ultrasound reward coefficient, which rewards the probe for exploring the surrounding area and obtaining better ultrasound images. During the probe's exploration of the target surface, it is crucial to ensure that the robot applies appropriate force to achieve clear ultrasound imaging. Therefore, we can design a force reward system that includes positive and negative rewards to encourage the robot to control the applied force within a safe range:

[0133]

[0134] Among them F tz represents the force applied by the probe along the z direction at time t. Constant ω f Represents the weighting factor or coefficient used in the force reward design.

[0135] Step 5: If, during the training and testing process, the processing is terminated under the following conditions: (1) the task is completed, (2) the probe exceeds the specified area of ​​interest, (3) the force exceeds the maximum limit, or (4) the number of steps exceeds the maximum limit. For condition (1), since the constructed ultrasound dataset does not fully cover the imaging space, the task completion criteria in the simulator are simplified by evaluating the probe posture and the distance between the probe and the soft tissue target point. In the real environment, we use SSIM as the evaluation metric. If the ultrasound imaging state value exceeds or equals the predefined threshold, it means that the task is completed. The pre-trained multimodal reinforcement learning control model is obtained. Otherwise, for condition (2), the probe posture space is restricted and jump to step 3 to ensure that the probe remains within the camera's field of view to reduce invalid space exploration. For condition (3), the maximum force feedback is set to 20N to ensure experimental safety. For condition (4), the maximum number of steps is set to 64 to reduce invalid repeated scans.

[0136] Step 6: Build the virtual platform Pybullet and pre-train the strategy in simulation.

[0137] Our simulator experiments investigate the effectiveness of our model in learning relevant feature representations from different modal combinations and how these representations can improve the success rate of ultrasound scanning tasks. Ablation experiments are conducted on the simulator, comparing the full model with three baseline conditions: no visual perception, no tactile perception, and both visual and tactile perception. Furthermore, to enhance and validate the model's generalization capabilities, the initial positions of the robot's end effector and the target object in the model are randomly initialized within the camera's field of view during training and testing. For each baseline experiment, the agent training details are as described in the previous section. Training is performed using eight random seeds, and the mean and standard deviation of the rewards for the first 3,000 episodes are reported.

[0138] from Figure 5As can be seen from the agent training curve plot shown in (a), the performance of all three baseline models is lower than that of the full model. The full model shows the best performance in the ultrasound scanning task, and obtains the maximum reward in each episode after effective training for 3000 episodes. This shows that the agent has learned the ability to move the probe to the target location based on scene perception. Figure 5 As shown in (b), the model containing only visual perception still performs well. Although it is not as good as the full model in terms of obtaining the maximum reward, it obtains a reward close to task completion. The trend of the training curve shows that this model also has room for improvement, highlighting that the full model converges faster than the model lacking tactile perception. In contrast, the baselines lacking visual perception and visual and tactile perception show the worst results and are barely able to complete the task. This is because the first stage of the robotic ultrasound scanning process (moving the probe to the target surface) relies heavily on visual perception. Without visual perception, it is difficult for the probe to reach the target surface and it is impossible to obtain an ultrasound image. From these experimental results, it can be analyzed that vision and touch each play different roles in the ultrasound scanning task. Visual perception is indispensable, while tactile perception helps accelerate model convergence and improve training efficiency. This further demonstrates the effectiveness of this approach in fusing visual and tactile perception.

[0139] After training, the complete model was further compared with three baselines to see how well they performed the ultrasound scanning task in the test environment. To this end, 20 random target test experiments were conducted, in which the end effector pose of the robotic arm and the position of the deformable target were randomly initialized within a specified spatial range. If the final position deviation between the end effector and the target was less than 5mm, the task was defined as successful. This scheme calculated the success rate of completing the task and the average deviation from the target pose. The experimental results are shown in Figure 2. Figure 5 (b) shows the complete model achieving the highest success rate of 90% and the smallest average deviation of 2.31mm. On the other hand, baselines lacking either visual or tactile perception performed worse than the complete model. The model lacking both vision and tactile perception performed the worst, with the robot's decisions moving randomly in space and deviating significantly from the target. The lack of tactile feedback, however, did not affect the model's performance as severely as with visual feedback, achieving a success rate of 60%. Furthermore, the experiments revealed that most failures of the model without tactile feedback during task execution were due to difficulty controlling force. Although tactile information was not incorporated into the model input, force was still monitored in real time, and when the force exceeded the maximum limit, the task would fail. The experimental results further demonstrate that the lack of vision and tactile perception negatively impacts the task, highlighting the reinforcing role of the multimodal features learned by the complete model in task completion.

[0140] Step seven: transfer the model to the real environment for fine-tuning to improve the generalization performance of the model.

[0141] This proposal verifies the feasibility of the multimodal ultrasound scanning system through ablation experiments in a simulator. The model trained in the simulator is transferred to the real hardware as a pre-trained model to further evaluate the generalization and robustness of the model. Although the simulation environment consistent with the real environment is carefully constructed, some errors are inevitable when performing tasks in the real environment. To mitigate the impact of these errors, the probe is placed closer to the model to reduce the number of training iterations in the real environment. In addition, real-world data was collected and an additional 10,000 interactive training iterations were performed to optimize the entire process of ultrasound probe scanning, as described in the previous section. To demonstrate the effectiveness of the pre-trained model transfer, the impact of the pre-trained model on the real training process is compared. We performed eight random seed training runs and reported the mean and standard deviation of the rewards. The training curve is shown in Figure 2. Figure 6 (a) As shown in the figure, the model trained based on the pre-trained model can learn and converge quickly in fewer training iterations. This is because the agent has acquired a certain degree of understanding of visual and tactile feedback through extensive training in the simulator, indicating that our pre-trained model exhibits good generalization ability. In contrast, without a pre-trained model, the agent needs to learn the policy from scratch, resulting in slower learning and poorer generalization ability in fewer training iterations.

[0142] Since the learning curve of the model without pre-training is that the average reward increases with the number of training iterations, it is challenging to collect a large amount of synchronized data in the real environment, which poses a significant challenge to training in the real environment. However, the method of using pre-trained model transfer can effectively solve this problem. In the testing phase of the real environment, since the ultimate goal of the ultrasound robot system is to obtain a standard ultrasound image, a qualitative evaluation was performed using the structural similarity index (SSIM). The SSIM between the ultrasound image acquired by the agent and the predefined standard ultrasound image was calculated. If the SSIM is greater than or equal to 0.55, the task is considered successful because it indicates that this is a high-quality standard ultrasound image. 20 random target placement experiments were performed in the ideal scene, and force feedback was recorded throughout the process and the SSIM of the final image was measured to qualitatively evaluate the performance of the model. In addition, visual interference such as artifacts, partial occlusion, and center occlusion was introduced, and experiments were conducted under the same settings as the ideal scene to further evaluate the robustness of our model to visual feedback interference. The results are shown in Figure 2. Figure 6 (b) shows the bar graph representing the success rate (SSIM ≥ 0.55) and the line graph representing the average SSIM of the ultrasound image. The results show that in experiments involving 20 random target positions, the model achieved an 80% success rate and an average SSIM of 0.60 ± 0.19 under normal conditions. The ultrasound scanning process is visualized as follows Figure 7(a). First, the robotic arm controls the probe to reach the target surface, then performs real-time ultrasound imaging analysis to scan the target surface. Finally, the probe is navigated to obtain the desired standard ultrasound image. These experiments demonstrate that the proposed ultrasound robot is capable of autonomously controlling the probe and obtaining high-quality ultrasound images during the scanning process. It is worth noting that due to the maximum number of steps the robotic arm can take per scan, in some experimental settings, the robotic arm may become trapped in a local optimum and fail to obtain the optimal ultrasound image.

[0143] By increasing the maximum step size limit or manually adjusting the initial position of the robotic arm, the robotic arm may eventually successfully meet the experimental standards. Although the ultrasound probe navigation failed in some placement experiments, resulting in a low SSIM score for the acquired ultrasound image, we believe that this is due to insufficient data collected in the real environment. By manually adjusting the initial position of the robotic arm, higher quality ultrasound images can be obtained. In addition, since the probe relies on visual guidance to reach the target surface to obtain ultrasound images, in scenarios with visual interference, the probe may have difficulty reaching the target surface, resulting in the inability to obtain ultrasound images, and ultimately leading to the failure of the experiment. Therefore, the performance of the system in visual interference scenarios was further evaluated, and the method of manually adjusting the posture was used to minimize the impact of visual interference. Figure 7 As shown in (b), compared with the normal mode, the model performance has declined in the presence of artifacts and partial occlusions, but it still performs well. Especially in the experiment with artifacts, although the average SSIM of the ultrasound image is reduced to 0.56±0.13 compared with the normal case, the experimental success rate is still as high as 75%, indicating that the model is highly robust to visual interference. Interestingly, even in the presence of central occlusion in the visual image, the model still has the potential to complete the task. This is because in some experiments, even with central occlusion, the probe can still touch the surface of the body model. In this case, the robot relies on ultrasound imaging status and tactile perception for navigation, which may make it possible to complete the task. Inspired by this, in scenarios characterized by visual interference, such as Figure 7 As shown in (b), the posture of the robotic arm can be manually adjusted to make the probe as close as possible to the area of ​​interest on the target surface, thereby improving the success rate of task completion.

[0144] In summary, this program has at least the following beneficial effects:

[0145] First, for the precise decision-making of robots in complex spaces in the field of medical ultrasound imaging, a multimodal human-machine fusion interaction framework that integrates visual and tactile information is proposed. Multi-sensor data fusion is used to achieve complementary advantages and obtain rich environmental information. Through a multimodal deep reinforcement learning model, real-time control of the 6D posture of the ultrasound probe is achieved, and the robot has the ability to autonomously navigate to the standard scanning plane in a dynamic environment.

[0146] Second, the ultrasound robot can safely diagnose patient diseases. During manual ultrasound scanning, ultrasound technicians rely on visual perception to guide probe positioning and find the right position, while using tactile feedback force sensors to ensure precise control of the probe force, thereby ensuring the safety of robot control. With high-precision sensor information and deep neural networks, the ultrasound robot can effectively navigate and control the ultrasound probe, thereby achieving more stable and accurate ultrasound diagnosis. Using ultrasound robots in medical applications can improve the quality and efficiency of medical diagnosis and reduce the physical burden on human operators. Under the premise of ensuring the safety of patients and operators, efficient remote operation and collaborative assistance are achieved, ensuring the safety of human-machine collaborative operation, and effectively transferring models trained in simulators to real environments.

[0147] The embodiments of the present application also provide an electronic device, comprising: a processor, and a memory coupled with the processor, the memory being configured to store a computer program; and the processor being configured to execute the computer program stored in the memory, so that the electronic device performs the method according to any one of the preceding embodiments.

[0148] The electronic device can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The electronic device can include, but is not limited to, a processor and a memory.

[0149] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The processor is the control center of the electronic device, and is connected to various parts of the entire device through various interfaces and lines.

[0150] The memory can be configured to store the computer program, and the processor can be configured to realize various functions of the electronic device by running or executing the computer program stored in the memory and calling data stored in the memory.

[0151] The memory may primarily include a program storage area and a data storage area. The program storage area may store an operating system, at least one application required for a function, and the like; the data storage area may store data generated based on the use of the mobile phone, and the like. Furthermore, the memory may include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0152] The embodiment of the present application also provides a storage medium, which is a computer-readable storage medium, and the computer program is stored in the computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0153] An embodiment of the present application further provides a computer program product, including: a computer program or instructions, which, when executed on a computer, causes the computer to execute any of the above-mentioned possible implementation methods.

[0154] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications are also considered to be within the scope of protection of the present application.

Claims

1. A multimodal interaction system based on human-computer collaboration, characterized in that: The interactive system includes: A robotic arm equipped with an ultrasonic probe and a force sensor, the force sensor being used to sense the tactile force between the ultrasonic probe and an object; a camera that captures the robotic arm and the object; and A computer is used to obtain an ultrasonic image sequence through the ultrasonic probe, a tactile force sequence through the force sensor, and a visual image sequence through the camera, wherein the ultrasonic image sequence reflects the position of a scanning target, the tactile force sequence reflects the tactile force between the ultrasonic probe and the target, and the visual image sequence reflects the posture of the robotic arm and the relative position between the ultrasonic probe and the target; based on a pre-trained deep reinforcement learning model, feature extraction and feature fusion are performed on the ultrasonic image sequence, the tactile force sequence, and the visual image sequence to obtain fused features; and the proximal strategy optimization (PPO) algorithm is used to process the fused features to obtain control instruction data for controlling the movement of the robotic arm.

2. The interactive system according to claim 1, wherein: The deep reinforcement learning model includes: a multimodal feature fusion module, configured to extract and fuse features of the ultrasound image sequence, the tactile force sequence, and the visual image sequence to obtain fused features; and A PPO agent module is used to process the fusion features using a proximal strategy optimization PPO algorithm to obtain the control instruction data. The reward function of the PPO agent module is defined by the distance and contact maintenance between the ultrasound probe and the target, the ultrasound image quality, and the tactile force.

3. The interactive system according to claim 2, wherein: The multimodal feature fusion module includes: an ultrasonic feature extraction module, a tactile feature extraction module and a visual feature extraction module. The ultrasonic feature extraction module includes: a UNet-type convolutional neural network (CNN) submodule, several convolution layers and a feature encoding layer designed in sequence. The tactile feature extraction module includes: a recurrent neural network (RNN) submodule, a long short-term memory (LSTM) layer and a temporal encoding layer designed in sequence. The visual feature extraction module includes: a UNet-type CNN submodule, several convolution layers and a downsampling layer designed in sequence.

4. The interactive system according to claim 1, wherein: The computer is also used to: build a virtual simulation platform to pre-train the deep reinforcement learning model.

5. The interactive system according to claim 2, wherein: The computer is further used to: after the pre-training, in a real environment, fine-tune the parameters of the deep reinforcement learning model based on the structural similarity index SSIM between the standard ultrasound image and the proxy ultrasound image obtained by imaging using the ultrasound scanning system.

6. A training method for a deep reinforcement learning model, characterized in that: The training method comprises: Obtaining an ultrasound image sequence, a tactile force sequence, and a visual image sequence, wherein the visual image sequence includes a robotic arm and an action target within a shooting range, the robotic arm being equipped with an ultrasound probe and a force sensor, the ultrasound image sequence reflecting the position of the scanned target, the tactile force sequence reflecting the tactile force between the ultrasound probe and the action target, and the visual image sequence reflecting the posture of the robotic arm and the relative position between the ultrasound probe and the action target; performing feature extraction and feature fusion on the ultrasound image sequence, the tactile force sequence, and the visual image sequence to obtain fusion features; and The fusion features are processed using a proximal policy optimization (PPO) algorithm to obtain control instruction data for controlling the movement of the robotic arm.

7. The training method according to claim 6, wherein: The training method is based on the constructed virtual simulation platform.

8. A training device for a deep reinforcement learning model, characterized in that: The training device comprises: an acquisition unit, configured to obtain an ultrasound image sequence, a tactile force sequence, and a visual image sequence, wherein the visual image sequence includes a robotic arm and an action target within a shooting range, the robotic arm being equipped with an ultrasound probe and a force sensor, the ultrasound image sequence reflecting the position of the scanned target, the tactile force sequence reflecting the tactile force between the ultrasound probe and the action target, and the visual image sequence reflecting the posture of the robotic arm and the relative position between the ultrasound probe and the action target; a fusion unit, configured to extract and fuse features of the ultrasound image sequence, the tactile force sequence, and the visual image sequence to obtain fusion features; and A processing unit is used to process the fusion features using a proximal policy optimization (PPO) algorithm to obtain control instruction data for controlling the movement of the robotic arm.

9. An electronic device, characterized in that: The electronic device includes: a processor, and a memory coupled to the processor, The memory is used to store computer programs; and The processor is configured to execute the computer program stored in the memory, so that the electronic device performs the training method according to claim 6 or 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a computer program or instructions, which, when executed on a computer, causes the computer to execute the training method according to claim 6 or 7 .

Images