Robot autonomous ultrasound scanning skill strategy generation method and device, and storage medium

Abstract

The method, device and storage medium for generating a robot autonomous ultrasound scanning skill strategy provided by the embodiments of the present disclosure comprise: collecting multi-modal data in a robot ultrasound scanning process, inputting the multi-modal data to a multi-modal feature fusion network for learning to obtain fusion features of the multi-modal data; constructing a virtual training environment of a robot ultrasound scanning skill, performing randomization processing on texture rendering of the virtual training environment and dynamics parameters related to ultrasound scanning operation by using an autonomous domain randomization method, to obtain diversified robot ultrasound scanning training environments; constructing an autonomous ultrasound scanning skill strategy model, training the autonomous ultrasound scanning skill strategy model by combining an imitation learning method and a reinforcement learning method and using the fusion features, and using the autonomous ultrasound scanning skill strategy model to generate an autonomous ultrasound scanning skill strategy. The present disclosure can realize an accurate and efficient ultrasound scanning task under the condition of no human intervention.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of medical robots, and in particular to a method and device for generating a robot autonomous ultrasound scanning skill strategy and a storage medium. BACKGROUND

[0002] Ultrasound scanning is an indispensable diagnostic method in modern medical examination, and is widely used in clinical practice. With the development of remote medical consultation technology, it is gradually possible to diagnose diseases through remote ultrasound scanning. A currently feasible solution is for a remote doctor expert to control the movement of an ultrasound probe through teleoperation to achieve ultrasound scanning of the corresponding organ part of the scanned object. However, the inevitable time delay in remote teleoperation causes the movement of the doctor's teleoperation handle to not immediately affect the ultrasound probe, resulting in poor diagnosis experience and low efficiency. On the other hand, the level of professional skill of the doctor and the degree of familiarity with the teleoperation device also affect the results of remote ultrasound scanning. Therefore, it is extremely necessary to design an intelligent autonomous ultrasound scanning skill strategy generation device to enable the robot to autonomously complete ultrasound scanning of each part of the scanned object with high adaptability. SUMMARY

[0003] The present disclosure aims to solve one of the above problems.

[0004] To this end, the robot autonomous ultrasound scanning skill strategy generation method provided by the embodiments of the present disclosure comprises:

[0005] Collecting multi-modal data in the process of robot ultrasound scanning, inputting the multi-modal data to a multi-modal feature fusion network for learning to obtain fusion features of the multi-modal data;

[0006] Constructing a virtual training environment for robot ultrasound scanning skill, using an autonomous domain randomization method to randomly process the texture rendering of the virtual training environment and the dynamics parameters related to ultrasound scanning operation, to obtain diversified robot ultrasound scanning training environment; constructing an autonomous ultrasound scanning skill strategy model, combining imitation learning and reinforcement learning methods and using the fusion features to train the autonomous ultrasound scanning skill strategy model, to obtain a trained autonomous ultrasound scanning skill strategy model for generating an autonomous ultrasound scanning skill strategy.

[0007] The robot autonomous ultrasound scanning skill strategy generation method provided by the first aspect of the embodiments of the present disclosure has the following features and advantages:

[0008] The method for generating a robot autonomous ultrasonic scanning skill strategy provided by the first aspect of the present disclosure collects and fuses multi-modal sensing information in an ultrasonic scanning process to obtain a robust feature expression, which can better overcome the problem of weak feature expression robustness caused by factors such as occlusion in the autonomous operation process of ultrasonic scanning. The ultrasonic scanning skill learning method fusing imitation learning and reinforcement learning can efficiently complete the learning of the ultrasonic scanning skill, and the learned skill can better adapt to the intelligent ultrasonic scanning requirements for the body parts of the scanning object.

[0009] In some embodiments, the multi-modal data includes an RGB image, a depth image, a tactile image, and a six-dimensional force / torque when the robot ultrasonic probe contacts the scanning object in the robot ultrasonic scanning process.

[0010] In some embodiments, the multi-modal feature fusion network includes a multi-modal fusion sub-network, and an RGB image feature extraction sub-network, a depth image feature extraction sub-network, a tactile feature extraction sub-network, and a six-dimensional force / torque feature extraction sub-network connected in parallel with the input end of the multi-modal fusion sub-network.

[0011] In some embodiments, the input of the RGB image feature extraction subnetwork is an RGB image of 240x240x3, and the output is a 32-dimensional RGB image feature vector, the RGB image feature extraction subnetwork has 4 first convolutional layers and 1 first fully connected layer, the number of channels of each first convolutional layer is 16, 32, 64, and 32 respectively, the sliding step of each first convolutional layer is 2, the convolution kernel size is 2x2, the number of neurons of the first fully connected layer is 32, and ReLU activation function is used; the input of the depth image feature extraction subnetwork is a depth image of 240x240x1, and the output is a 32-dimensional depth image feature vector, the depth image feature extraction subnetwork has 4 second convolutional layers and 1 second fully connected layer, the number of channels of each second convolutional layer is 16, 32, 64, and 32 respectively, the sliding step of each second convolutional layer is 2, the convolution kernel size is 2x2, the number of neurons of the second fully connected layer is 64, and ReLU activation function is used; the input of the tactile image feature extraction subnetwork is a tactile image of 128x128x3, and the output is a 32-dimensional tactile image feature vector, the tactile image feature extraction subnetwork has 4 third convolutional layers and 1 third fully connected layer, the number of channels of each third convolutional layer is 16, 32, 64, and 32 respectively, the sliding step of each third convolutional layer is 2, the convolution kernel size is 2x2, the number of neurons of the third fully connected layer is 64, and ReLU activation function is used; the input of the six-dimensional force / torque feature extraction subnetwork is an 18-dimensional vector obtained by splicing the spatial six-dimensional force / torque of continuous 3 time points, and the output is a 32-dimensional force feature vector, the six-dimensional force / torque feature extraction subnetwork has 3 fourth fully connected layers, the number of neurons of each fourth fully connected layer is 128, 128, and 64 respectively, and ReLU activation function is used; the multi-modal fusion subnetwork has 1 fifth fully connected layer, the number of neurons is 128, and ReLU activation function is used; the extracted RGB image feature vector, depth image feature vector, tactile image feature vector, and force feature vector with a dimension of 32 are connected together and input to the fifth fully connected layer to obtain a 128-dimensional fusion feature vector.

[0012] In some embodiments, before the virtual training environment for building the robot ultrasound scanning skill, the method further comprises: after the collected multi-modal data is completed and aligned, training the multi-modal data fusion network by using a self-supervised learning method.

[0013] In some embodiments, when training the multi-modal data fusion network, an ultrasound probe tip pose prediction subnetwork, an ultrasound probe tip force prediction subnetwork, and an ultrasound image successful acquisition classification subnetwork are arranged in parallel after the multi-modal fusion subnetwork, and are respectively used to output a predicted pose of the tip of the ultrasound probe, a predicted force applied to the tip of the ultrasound probe, and a classification result of whether an ultrasound image is successfully acquired; then, learning is performed in a self-supervised learning manner to predict the pose and force of the tip of the ultrasound probe and whether the ultrasound probe successfully acquires an ultrasound image, that is, to minimize the following formula:

[0014]

[0015] In the formula, L is a loss function; D is a sample data set collected, including teaching samples collected in a teaching operation ultrasound scanning task process and interaction samples generated by autonomous interaction of a robot and an environment, each sample including six-dimensional pose data of a tip of an ultrasound probe, six-dimensional force / torque, a classification result of whether an ultrasound image is successfully acquired, a haptic image, an RGB image, and a depth image, d s represents a classification result of whether an ultrasound image is successfully acquired corresponding to the sample, d p represents six-dimensional pose of the tip of the ultrasound probe in the sample, d f represents six-dimensional force / torque of the tip of the ultrasound probe in the sample; d' s d' p d' f respectively represent outputs of the ultrasound image successful acquisition classification subnetwork, the ultrasound probe tip pose prediction subnetwork, and the ultrasound probe tip force prediction subnetwork; and a1, a2, and a3 are weight coefficients. represents a sample collected from the sample data set D, d s d p d f is a calculated expected value.

[0016] In some embodiments, the autonomous ultrasound scanning skill strategy model includes the trained multi-modal feature fusion network, and an ultrasound probe tip pose subnetwork and an ultrasound probe tip force subnetwork connected in parallel after the multi-modal fusion subnetwork in the multi-modal feature fusion network; the ultrasound probe tip pose subnetwork and the ultrasound probe tip force subnetwork respectively output six-dimensional pose of the ultrasound probe and six-dimensional force / torque applied to the tip of the ultrasound probe.

[0017] In some embodiments, the autonomous ultrasound scanning skill model is trained by combining imitation learning and reinforcement learning methods and using the fusion features, and specifically includes:

[0018] The teaching samples are collected, and the teaching samples are discretized to construct a teaching pool. The samples in the teaching pool are state transition pairs (s). t ,a t ,r t ,s t+1 ), s t Let a represent the state at time t. t r represents the teaching action performed at time t. t s represents the reward value obtained at time t. t+1 Indicates that in state s t Perform action a t The state that is reached later; at the same time, an experience pool is constructed, and the data stored in the experience pool is the interaction sample;

[0019] In the reinforcement learning part, the robot autonomously explores and learns in the environment based on a given reward value. The reward value is determined by a reward function in the form of sparse rewards, specifically as follows: if the robot can complete the ultrasound scanning task, it is given a reward value m = 1; if the robot fails to complete the ultrasound scanning task, it is given a reward value n = -1. The interaction samples are stored in the experience pool, and the autonomous ultrasound scanning skill strategy model is trained by collecting training samples from the teaching pool and the experience pool respectively using an offline reinforcement learning method until the autonomous ultrasound scanning skill strategy learning converges.

[0020] The robot autonomous ultrasonic scanning skill strategy generation device provided in the second aspect of this disclosure includes:

[0021] The fusion feature generation module is used to collect multimodal data during the ultrasonic scanning process of the robot, and input it into the multimodal feature fusion network for learning to obtain the fusion features of the multimodal data;

[0022] An autonomous ultrasonic scanning skill strategy generation module is used to construct a virtual training environment for robot ultrasonic scanning. An autonomous domain randomization method is employed to randomize the texture rendering and dynamic parameters related to ultrasonic scanning operations in the virtual training environment, resulting in diverse robot ultrasonic scanning training environments. An autonomous ultrasonic scanning skill strategy model is constructed, and trained using imitation learning and reinforcement learning methods and the fusion features, yielding a trained autonomous ultrasonic scanning skill strategy model used to generate autonomous ultrasonic scanning skill strategies.

[0023] A computer-readable storage medium is provided in a third aspect embodiment of this disclosure, the computer-readable storage medium storing computer instructions for causing the computer to execute the above-described robot autonomous ultrasonic scanning skill strategy generation method. Attached Figure Description

[0024] Figure 1 A flowchart of a robot autonomous ultrasound scanning skill strategy generation method provided by the first aspect of the present disclosure is shown.

[0025] Figure 2 A layout diagram of a collection platform corresponding to the generation method provided by the first aspect of the present disclosure is shown.

[0026] Figure 3 A structure diagram of a multi-modal feature fusion network in the generation method provided by the first aspect of the present disclosure is shown.

[0027] Figure 4 A structure diagram of an autonomous ultrasound scanning skill model in the generation method provided by the first aspect of the present disclosure is shown.

[0028] Figure 5 A structure diagram of a robot autonomous ultrasound scanning skill strategy generation device provided by the second aspect of the present disclosure is shown.

[0029] Figure 6 A structure diagram of an electronic device provided by the third aspect of the present disclosure is shown.

[0030] In the figure, 1-platform, 2-scan object, 30-robotic arm, 31-six-axis force / torque sensor, 32-ultrasound probe, 33-color depth camera, 34-tactile sensor, 4-mobile platform, 5-color depth camera. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0032] On the contrary, the present application covers any alternative, modification, equivalent method and scheme made on the essence and scope of the present application as defined by the claims. Further, in order to make the public better understand the present application, some specific details are described in detail in the following detailed description of the present application. The present application can also be completely understood without the description of these details by those skilled in the art.

[0033] Referring to Figure 1 The robot autonomous ultrasound scanning skill strategy generation method provided by the first aspect of the present disclosure comprises:

[0034] The multi-modal data in the robot ultrasound scanning process is collected and input to the multi-modal feature fusion network for learning to obtain stable and robust fusion features of the multi-modal data;

[0035] The virtual training environment for constructing robot ultrasound scanning skills is constructed, and the texture rendering and the dynamic parameters related to the ultrasound scanning operation of the virtual training environment are randomized by using the autonomous domain randomization method to obtain diversified robot ultrasound scanning training environments; an autonomous ultrasound scanning skill strategy model is constructed, and the autonomous ultrasound scanning skill strategy model is trained by combining the imitation learning and reinforcement learning methods and using the fused features to obtain the trained autonomous ultrasound scanning skill strategy model, which is used to generate the autonomous ultrasound scanning skill strategy.

[0036] In some embodiments, the collected multi-modal data in the robot ultrasound scanning process includes visual data, tactile data, and spatial force data when the robot end ultrasound probe contacts the scanned object.

[0037] In some embodiments, the visual data includes RGB images and depth images of the operation environment acquired in the robot ultrasound scanning process; the tactile data includes tactile images in the robot ultrasound scanning process; and the spatial force data includes six-dimensional force / torque information when the robot end ultrasound probe contacts the scanned object.

[0038] In one embodiment, the multi-modal data is collected by using a collection platform as shown in Figure 2 Specifically, the mechanical arm 30 of the robot is driven by a moving platform 4 and is arranged near the scanned object 2, and the scanned object 2 is located on the platform 1; a first color depth camera 5 is arranged in the scanning environment to collect full-scene RGB images and full-scene depth images. An ultrasound probe 32 is arranged at the end of the mechanical arm 30 of the robot, and a tactile sensor 34 is arranged at the end of the ultrasound probe 32 to collect tactile images; on the mechanical arm 30, a six-dimensional force / torque sensor 31 and a second color depth camera 33 are arranged to collect six-dimensional force / torque information, RGB images, and depth images when the robot end ultrasound probe 32 contacts the scanned object. Wherein, the scanning personnel demonstrate the ultrasound scanning process for different parts of the scanned object 2 by means of the teleoperation handle with force feedback, and record the RGB images, depth images, six-dimensional pose data of the end of the ultrasound probe, six-dimensional force / torque information, and tactile images collected in the process.

[0039] In some embodiments, before the multi-modal feature fusion, the following data processing steps are further included:

[0040] The collected full-scene RGB images are subjected to semantic segmentation to obtain a semantic segmentation map, and then the depth information provided by the collected full-scene depth images and the spatial coordinate change method are used to obtain the position coordinates of each scanned part of the scanned object 2 in the robot coordinate system, so as to facilitate the subsequent ultrasound probe to move to the position coordinates for ultrasound scanning.

[0041] In some embodiments, before the multi-modal feature fusion is performed, the following data processing steps are further included:

[0042] The collected visual data, tactile data and spatial force data when the robot end ultrasonic probe contacts the scanned object are completed and aligned, so that the learned multi-modal features are more accurate.

[0043] In some embodiments, referring to Figure 3 , the multi-modal feature fusion network includes an RGB image feature extraction subnetwork, a depth image feature extraction subnetwork, a tactile feature extraction subnetwork, a six-dimensional force / torque feature extraction subnetwork, and a multi-modal fusion subnetwork. The input of the RGB image feature extraction subnetwork is a 240x240x3 RGB image, and the output is a 32-dimensional RGB image feature vector. The RGB image feature extraction subnetwork has 4 first convolutional layers and 1 first fully connected layer. The channel numbers of each first convolutional layer are 16, 32, 64, and 32, respectively. The sliding step of each first convolutional layer is 2, and there is no pooling layer. The convolution kernel size is 2x2. The number of neurons in the first fully connected layer is 32, and ReLU activation function is used. The input of the depth image feature extraction subnetwork is a 240x240x1 depth image, and the output is a 32-dimensional depth image feature vector. The depth image feature extraction subnetwork has 4 second convolutional layers and 1 second fully connected layer. The channel numbers of each second convolutional layer are 16, 32, 64, and 32, respectively. The sliding step of each second convolutional layer is 2, and there is no pooling layer. The convolution kernel size is 2x2. The number of neurons in the second fully connected layer is 64, and ReLU activation function is used. The input of the tactile image feature extraction subnetwork is a 128x128x3 tactile image, and the output is a 32-dimensional tactile image feature vector. The tactile image feature extraction subnetwork has 4 third convolutional layers and 1 third fully connected layer. The channel numbers of each third convolutional layer are 16, 32, 64, and 32, respectively. The sliding step of each third convolutional layer is 2, and there is no pooling layer. The convolution kernel size is 2x2. The number of neurons in the third fully connected layer is 64, and ReLU activation function is used. The input of the six-dimensional force / torque feature extraction subnetwork is an 18-dimensional vector obtained by concatenating the spatial six-dimensional force / torque of the last 3 (or other number) time points, and the output is a 32-dimensional force feature vector. The six-dimensional force / torque feature extraction subnetwork has 3 fourth fully connected layers. The number of neurons in each fourth fully connected layer is 128, 128, and 64, respectively. ReLU activation function is used. The multi-modal fusion subnetwork has 1 fifth fully connected layer with 128 neurons, and ReLU activation function is used. The extracted 32-dimensional RGB image feature vector, depth image feature vector, tactile image feature vector, and force feature vector are concatenated and input into the fifth fully connected layer to obtain a 128-dimensional fusion feature vector.

[0044] Further, referring to Figure 3 , in order to obtain an effective and robust fusion feature vector, the multi-modal feature fusion network is trained, and during the training process, an ultrasonic probe end position prediction sub-network, an ultrasonic probe end force prediction sub-network and an ultrasonic image successful acquisition classification sub-network are arranged in parallel after the multi-modal fusion sub-network; wherein the ultrasonic probe end position prediction sub-network has 3 layers of the sixth full connection layer, the neuron numbers of the 3 layers of the sixth full connection layer are 128, 256 and 6 respectively, all using the ReLU activation function, and the output dimension is 6, which respectively corresponds to the six-dimensional position of the ultrasonic probe end to be reached, i.e. the predicted position of the ultrasonic probe end; the ultrasonic probe end force prediction sub-network has 3 layers of the seventh full connection layer, the neuron numbers of the 3 layers of the seventh full connection layer are 128, 256 and 6 respectively, all using the ReLU activation function, and the output dimension is 6, which respectively corresponds to the six-dimensional force / torque that the ultrasonic probe end should exert, i.e. the predicted force exerted on the ultrasonic probe end; the ultrasonic image successful acquisition classification sub-network has 3 layers of the eighth full connection layer, the neuron numbers of the 3 layers of the eighth full connection layer are 128, 256 and 1 respectively, wherein the first two layers use the ReLU activation function, and the third layer uses the Sigmoid activation function, and the output is the classification result "1" or "0" of whether the ultrasonic image is successfully acquired, "1" indicating that the ultrasonic probe successfully acquires the ultrasonic image, and "0" indicating that the ultrasonic probe does not successfully acquire the ultrasonic image, which is generally judged by whether the ultrasonic scanning instrument successfully displays the ultrasonic image.

[0045] Further, the self-supervised learning method is used to predict whether the ultrasonic probe successfully acquires the ultrasonic image data, the end position of the ultrasonic probe and the end force, and the neural network parameters of the multi-modal feature fusion network are trained and learned, and after the training converges, a robust multi-modal feature vector is obtained, and optionally, the training process is realized by minimizing the following loss function:

[0046]

[0047] In the formula, L is a loss function; D is a sample data set collected, including teaching samples collected in a teaching demonstration process and interaction samples generated by a robot and an environment in autonomous interaction, each sample including a tactile image, an RGB image, a depth image, a label result d s of whether an ultrasonic image is successfully acquired, ultrasonic probe end six-dimensional position data d p and six-dimensional force / torque d f . s , p , frespectively represent the outputs of the ultrasound image successful acquisition classification subnetwork, the ultrasound probe tip pose prediction subnetwork, and the ultrasound probe tip force prediction subnetwork; a1, a2, a3 are weight coefficients (one acceptable solution is a1=0.3, a2=0.3, a3=0.3); represents the d-th sample collected from the sample data set D s represents the d-th sample collected from the sample data set D p represents the d-th sample collected from the sample data set D f represents the d-th sample collected from the sample data set D

[0048] In some embodiments, a virtual training environment for robot ultrasound scanning is constructed, a self-domain randomization method is adopted, and a curriculum learning mechanism is fused to randomize the texture rendering of the virtual training environment and the dynamic parameters related to ultrasound scanning operations, thereby obtaining diversified robot ultrasound scanning training environments, which specifically include:

[0049] First, the light and object surface texture in the virtual training environment are randomly rendered, and the randomization degree is autonomously selected according to the performance of the ultrasound scanning skill strategy learning, that is, when the performance of the ultrasound scanning skill strategy improves, the randomization degree is increased, otherwise the randomization degree is reduced; after the success rate of the ultrasound scanning skill strategy learning meets the requirements, the virtual training environment rendering and the dynamic parameters related to the ultrasound scanning operation (such as the friction coefficient) are simultaneously randomized by adding random noise, so that the ultrasound scanning skill strategy learns in this kind of self-domain randomization environment to improve the efficiency of the ultrasound scanning skill strategy learning. The domain randomization method adopted in the embodiments of the present disclosure is realized in an autonomous manner, that is, considering that if the domain randomization degree is too large in the domain randomization process, the ultrasound scanning skill strategy learning is difficult to converge; if the domain randomization degree is too small, it is not conducive to the improvement of the performance of the ultrasound scanning skill strategy, therefore, the domain randomization degree is automatically adjusted based on the evaluation of the current ultrasound scanning skill strategy performance during the ultrasound scanning skill strategy learning process; considering that simultaneously randomizing the environment texture rendering and the dynamic parameters of two types of factors leads to difficulty in ultrasound scanning skill strategy learning, therefore, based on the curriculum learning mechanism, the environment texture rendering part is first randomized, and after the success rate of the ultrasound scanning skill strategy learning meets the requirements, the environment texture rendering and the dynamic parameters are simultaneously randomized.

[0050] In some embodiments, referring to Figure 4The autonomous ultrasound scanning skill strategy model takes the multi-modal fusion features as input and takes the actions of the autonomous ultrasound scanning skill strategy as output. The output actions of the autonomous ultrasound scanning skill strategy include a six-dimensional pose to which an ultrasound probe at the end of a robot should reach and a six-dimensional force / torque that should be applied. The autonomous ultrasound scanning skill strategy model network specifically includes a trained multi-modal feature fusion network and an ultrasound probe end pose sub-network and an ultrasound probe end force sub-network connected in parallel to the multi-modal fusion sub-network of the multi-modal feature fusion network. The ultrasound probe end pose sub-network is used to output a six-dimensional pose of the ultrasound probe to control the ultrasound probe on the robot to move to a scanning position for ultrasound scanning. The ultrasound probe end force sub-network is used to output a six-dimensional force / torque applied to the end of the ultrasound probe to ensure the quality of ultrasound imaging. During the training of the autonomous ultrasound scanning skill strategy model, only the neural network parameters of the ultrasound probe end pose sub-network and the ultrasound probe end force sub-network are learned and updated, and the neural network parameters of the multi-modal feature fusion network are all frozen.

[0051] In some embodiments, the autonomous ultrasound scanning skill strategy model is trained in combination with the imitation learning and reinforcement learning methods and using fusion features to obtain a trained autonomous ultrasound scanning skill strategy model, specifically including:

[0052] The demonstration sample is collected by demonstrating the ultrasound scanning task, and the collected demonstration sample is discretized to construct a demonstration pool. The samples in the demonstration pool are state transition pairs (s t ,a t ,r t ,s t+1 ), where s t represents the state at time t, a t represents the demonstration action performed by the demonstrator at time t, r t represents the reward value obtained at time t, and s t+1 represents the state reached after performing action a t at state s t , i.e., the state at time t+1. At the same time, an experience pool is constructed, and the data stored in the experience pool is the interaction data in the exploration process of the robot and the environment, i.e., the interaction sample.

[0053] The reinforcement learning part needs the robot to autonomously explore and learn in the environment based on a given reward value, wherein the reward value is determined by a reward function defined in the form of sparse rewards, specifically as follows: if the robot can complete the ultrasonic scanning task, it is given a reward value m = 1, and if the robot fails to complete the ultrasonic scanning task, it is given a reward value n = -1; the interaction data of the robot and the environment during the exploration process is stored in an experience pool, and an offline reinforcement learning method is used to collect training samples from the demonstration pool and the experience pool to train the autonomous ultrasonic scanning skill policy model, and finally after the skill policy learning converges, the learned skill policy is deployed to the real environment to complete the ultrasonic scanning task on the scanned object.

[0054] Referring to Figure 5 The robot autonomous ultrasonic scanning skill policy generation device provided by the second aspect of the present disclosure comprises:

[0055] The fusion feature generation module is configured to collect multi-modal data in the ultrasonic scanning process of the robot, and input the multi-modal data to a multi-modal feature fusion network for learning to obtain fusion features of the multi-modal data.

[0056] The autonomous ultrasonic scanning skill policy generation module is configured to construct a virtual training environment for ultrasonic scanning of the robot, randomly process the texture and dynamics parameters related to the ultrasonic scanning operation of the virtual training environment by using an autonomous domain randomization method, to obtain a diversified ultrasonic scanning training environment for the robot; construct an autonomous ultrasonic scanning skill policy model, combine imitation learning and reinforcement learning methods, and train the autonomous ultrasonic scanning skill policy model by using the fusion features, to obtain a trained autonomous ultrasonic scanning skill policy model, which is used to generate an autonomous ultrasonic scanning skill policy.

[0057] To implement the above-mentioned embodiments, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to perform the robot autonomous ultrasonic scanning skill policy generation method of the above-mentioned embodiments.

[0058] Reference will be made to Figure 6 which shows a structural schematic diagram of an electronic device 100 suitable for implementing the embodiments of the present disclosure. It should be noted that Figure 6 The electronic device shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0059] As Figure 6As shown, the electronic device 100 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 101 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 102 or loaded into a random access memory (RAM) 103 from a storage device 108. Various programs and data required for the operation of the electronic device 100 are also stored in the RAM 103. The processing device 101, the ROM 102, and the RAM 103 are connected to each other through a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.

[0060] Generally, the following devices can be connected to the I / O interface 105: input devices 106 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, etc.; output devices 107 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 108 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 109. The communication devices 109 can allow the electronic device 100 to communicate with other devices wirelessly or wired to exchange data. Although Figure 6 The electronic device 100 is shown with various devices, but it should be understood that all of the illustrated devices are not required to implement or have the electronic device. More or less devices can alternatively be implemented.

[0061] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the present embodiments include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 109, or installed from the storage devices 108, or installed from the ROM 102. When the computer program is executed by the processing device 101, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0062] It should be noted that the computer readable medium described above in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer readable medium can be transmitted by any suitable medium, including but not limited to a wire, a cable, an RF (radio frequency) or the like, or any suitable combination of the above.

[0063] The computer readable medium described above can be contained in the electronic device described above; or can exist separately and not be assembled into the electronic device.

[0064] The computer readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to: collect multi-modal data in a robotic ultrasound scanning process, input to a multi-modal feature fusion network for learning to obtain fusion features of the multi-modal data; construct a virtual training environment for robotic ultrasound scanning, and use a domain randomization method to randomize texture rendering and dynamic parameters related to ultrasound scanning operation of the virtual training environment, to obtain diversified robotic ultrasound scanning training environments; construct an autonomous ultrasound scanning skill strategy model, combine imitation learning and reinforcement learning methods, and use the fusion features to train the autonomous ultrasound scanning skill strategy model, to obtain a trained autonomous ultrasound scanning skill strategy model, which is used to generate an autonomous ultrasound scanning skill strategy.

[0065] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++, Python, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0066] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0067] In addition, the terms "first", "second", etc. are used only for the purpose of description and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0068] Any process or method descriptions or descriptions of the flow diagrams in the specification or otherwise described herein can be understood as representing code modules, segments, or portions of code which include one or more executable instructions for implementing specific logic functions (or steps) in the process, and the various embodiments of the application can include additional or fewer functions (or steps) in the process, and the functions (or steps) can be performed in the sequence shown or in other sequences, in an alternate order, or in parallel, depending on the implementation and the functions (or steps) involved.

[0069] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of instructions to implement logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, processor- based system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a computer- readable storage medium or a computer-readable signal medium. The computer- readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection having one or more wires (electrical connections), a portable computer diskette (a magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

[0070] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, the various steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or their combinations can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0071] Those skilled in the art can understand that all or part of the steps carried out by the above-mentioned embodiments can be completed by programs instructing related hardware, and the developed programs can be stored in a computer-readable storage medium, and the programs include one or a combination of the steps of the method embodiments when executed.

[0072] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing module, or each of the units can be physically present separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0073] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method for robot autonomous ultrasound scan skill policy generation, the method comprising: The method comprises the following steps: Collecting multi-modal data in a robot ultrasonic scanning process, inputting the multi-modal data into a multi-modal feature fusion network for learning to obtain fusion features of the multi-modal data; Constructing a virtual training environment for robot ultrasonic scanning skills, using an autonomous domain randomization method to randomize texture rendering and dynamics parameters related to ultrasonic scanning operations in the virtual training environment to obtain diversified robot ultrasonic scanning training environments; constructing an autonomous ultrasonic scanning skill strategy model, combining imitation learning and reinforcement learning methods and using the fusion features to train the autonomous ultrasonic scanning skill strategy model to obtain a trained autonomous ultrasonic scanning skill strategy model for generating an autonomous ultrasonic scanning skill strategy; The multi-modal data includes RGB images, depth images, tactile images, and six-dimensional forces / torques when the robot ultrasonic probe contacts the scanning object during the robot ultrasonic scanning process; The multi-modal feature fusion network includes a multi-modal fusion sub-network, and RGB image feature extraction sub-networks, depth image feature extraction sub-networks, tactile feature extraction sub-networks, and six-dimensional force / torque feature extraction sub-networks connected in parallel with the input end of the multi-modal fusion sub-network; When training the multi-modal data fusion network, an ultrasonic probe end pose prediction sub-network, an ultrasonic probe end force prediction sub-network, and an ultrasonic image successful acquisition classification sub-network are set in parallel after the multi-modal fusion sub-network, respectively used to output the predicted pose of the ultrasonic probe end, the predicted force applied to the ultrasonic probe end, and the classification result of whether the ultrasonic image is successfully acquired; then the learning is performed by predicting the pose and force of the ultrasonic probe end and whether the ultrasonic probe successfully acquires the ultrasonic image in a self-supervised learning form, i.e., minimizing the following formula: In the formula, L is a loss function; D is a sample data set collected, including teaching samples collected in a teaching operation ultrasonic scanning task process and interaction samples generated by autonomous interaction of a robot and an environment, each sample including six-dimensional pose data of an ultrasonic probe tip, six-dimensional force / torque, a classification result of whether an ultrasonic image is successfully collected, a tactile image, an RGB image and a depth image, d s represents a classification result of whether an ultrasonic image is successfully collected corresponding to the sample, d p represents six-dimensional pose of the ultrasonic probe tip in the sample, d f represents six-dimensional force / torque of the ultrasonic probe tip in the sample; d s p f respectively represent outputs of the ultrasonic image successful collection classification subnetwork, the ultrasonic probe tip pose prediction subnetwork and the ultrasonic probe tip force prediction subnetwork; α1, α2, α3 are weight coefficients respectively; represents a sample d s p f calculates an expected value;​​​​ The autonomous ultrasonic scanning skill strategy model includes the trained multi-modal feature fusion network, and an ultrasonic probe end pose sub-network and an ultrasonic probe end force sub-network connected in parallel after the multi-modal fusion sub-network in the multi-modal feature fusion network; the ultrasonic probe end pose sub-network and the ultrasonic probe end force sub-network respectively output the six-dimensional pose of the ultrasonic probe and the six-dimensional force / torque applied to the ultrasonic probe end.

2. The robotic autonomous ultrasound scan skill policy generation method of claim 1, wherein, The input of the RGB image feature extraction subnetwork is an RGB image of 240*240*3, and the output is an RGB image feature vector of 32 dimensions. The RGB image feature extraction subnetwork has four first convolutional layers and one first fully connected layer. The channel numbers of each first convolutional layer are 16, 32, 64 and 32 respectively. The sliding step length of each first convolutional layer is 2, and the convolution kernel size is 2*2. The number of neurons of the first fully connected layer is 32, and all use ReLU activation functions. The input of the depth image feature extraction subnetwork is a depth image of 240*240*1, and the output is a depth image feature vector of 32 dimensions. The depth image feature extraction subnetwork has four second convolutional layers and one second fully connected layer. The channel numbers of each second convolutional layer are 16, 32, 64 and 32 respectively. The sliding step length of each second convolutional layer is 2, and the convolution kernel size is 2*2. The number of neurons of the second fully connected layer is 64, and all use ReLU activation functions. The input of the tactile image feature extraction subnetwork is a tactile image of 128*128*3, and the output is a tactile image feature vector of 32 dimensions. The tactile image feature extraction subnetwork has four third convolutional layers and one third fully connected layer. The channel numbers of each third convolutional layer are 16, 32, 64 and 32 respectively. The sliding step length of each third convolutional layer is 2, and the convolution kernel size is 2*2. The number of neurons of the third fully connected layer is 64, and all use ReLU activation functions. The input of the six-dimensional force / torque feature extraction subnetwork is an 18-dimensional vector obtained by splicing the spatial six-dimensional force / torque of the continuous three time points, and the output is a force feature vector of 32 dimensions. The six-dimensional force / torque feature extraction subnetwork has three fourth fully connected layers. The number of neurons of each fourth fully connected layer is 128, 128 and 64 respectively, and all use ReLU activation functions. The multi-modal fusion subnetwork has one fifth fully connected layer with 128 neurons, all using ReLU activation functions. The extracted RGB image feature vector, depth image feature vector, tactile image feature vector and force feature vector with dimensions of 32 are connected together and input into the fifth fully connected layer to obtain a fusion feature vector of 128 dimensions.

3. The robotic autonomous ultrasound scan skill policy generation method of claim 2, wherein, Before the virtual training environment for constructing the robot ultrasound scanning skill, the multi-modal data collected is completed and aligned, and the multi-modal data fusion network is trained by using a self-supervised learning method.

4. The robotic autonomous ultrasound scan skill policy generation method of claim 1, wherein, The self-supervised learning method is combined with the reinforcement learning method, and the fusion feature is used to train the autonomous ultrasound scanning skill strategy model, specifically including: The teaching sample is collected, the teaching sample is discretized, a teaching pool is constructed, and the samples in the teaching pool are state transition pairs (s t , a t , r t , s t+1 ) , wherein s t represents a state at time t, a t represents a teaching action performed at time t, r t represents a reward value obtained at time t, and s t+1 represents a state reached after performing the action a t on the state s t ; and an experience pool is constructed, and data stored in the experience pool is the interaction sample; The reinforcement learning part is that the robot autonomously explores and learns in the environment based on a given reward value, wherein the reward value is determined by a reward function in the form of sparse reward, specifically as follows: if the robot can complete the ultrasonic scanning task, it is given a reward value m=1, and if the robot fails to complete the ultrasonic scanning task, it is given a reward value n=-1; the interaction samples are stored in the experience pool, and an offline reinforcement learning method is used to collect training samples from the teaching pool and the experience pool respectively to train the autonomous ultrasonic scanning skill policy model until the autonomous ultrasonic scanning skill policy learning converges.

5. A robotic autonomous ultrasound scanning skill policy generation apparatus based on the method of any one of claims 1 to 4, characterized by Comprise: The fusion feature generation module is used for collecting multi-modal data in the process of ultrasonic scanning of the robot, and inputting the multi-modal data to a multi-modal feature fusion network for learning to obtain fusion features of the multi-modal data; The autonomous ultrasonic scanning skill policy generation module is used for constructing a virtual training environment for ultrasonic scanning of the robot, using an autonomous domain randomization method to randomize texture rendering and dynamics parameters related to ultrasonic scanning operation of the virtual training environment to obtain diversified robot ultrasonic scanning training environments; constructing an autonomous ultrasonic scanning skill policy model, combining imitation learning and reinforcement learning methods and using the fusion features to train the autonomous ultrasonic scanning skill policy model to obtain a trained autonomous ultrasonic scanning skill policy model for generating an autonomous ultrasonic scanning skill policy.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the computer to execute the robot autonomous ultrasonic scanning skill policy generation method of any one of claims 1-4.

Images