Robotic breast ultrasound probe automatic adjustment method and system based on reinforcement learning

Abstract

The present application relates to the technical field of robot breast ultrasound, in particular to an automatic adjustment method and system of a robot breast ultrasound probe based on reinforcement learning; the present application constructs a Markov decision model and an expert demonstration experience pool, a real-time interaction experience pool and a training data pool, inputs an ultrasound image in real time, adopts a width reinforcement learning method, thereby outputs reinforcement learning data, and then adjusts the probe to a target posture according to the reinforcement learning data, so as to efficiently adjust the ultrasound probe to obtain high-quality ultrasound images.

Description

Technical Field

[0001] This invention relates to the field of robotic breast ultrasound technology, and in particular to an automatic adjustment method and system for a robotic breast ultrasound probe based on reinforcement learning. Background Technology

[0002] Breast cancer is a deadly disease, with over 2 million new cases diagnosed and more than 684,000 deaths annually. Breast ultrasound and X-ray examinations can enable early detection and treatment. Ultrasound is more effective than X-rays in detecting breast cancer. Furthermore, ultrasound offers advantages in terms of price, safety, and real-time visualization of key areas. Therefore, handheld ultrasound is widely used worldwide for breast cancer screening. However, handheld ultrasound operation is highly dependent on the doctor's experience, and the procedure is time-consuming. Prolonged scanning of ultrasound images can cause doctor fatigue, thus reducing the accuracy of breast cancer diagnosis. Robotic breast ultrasound can standardize and simplify ultrasound examinations, thereby reducing the workload of doctors.

[0003] Robotic breast ultrasound generally involves two steps. The first step is preoperative path planning based on individual breast differences and automatic path tracking via a robotic arm. On the vertical projection plane of the breast, several parallel and equidistant lines are used to generate individual boundaries, which are then projected onto the breast surface to obtain the final scanning path. Typically, RGB-D or CT / MRI images are used to generate the scanning path, ensuring complete breast coverage. During path tracking, confidence map-based or force-based control can be used to ensure the probe contacts the scanning area. The second step is precise probe adjustment around the region of interest (ROI). This step is fundamental to robotic breast ultrasound systems and is of great importance. Currently, there is no mature solution for precise probe adjustment in robotic autonomous ultrasound examinations. Summary of the Invention

[0004] The main technical problem solved by this invention is to provide an automatic adjustment method for a robotic breast ultrasound probe based on reinforcement learning. This method constructs a Markov decision model, an expert teaching experience pool, a real-time interactive experience pool, and a training data pool. Ultrasound images are input in real time, and a width-based reinforcement learning method is used to output reinforcement learning data. Then, based on the reinforcement learning data, the probe is adjusted to the target posture, achieving efficient adjustment of the ultrasound probe to obtain high-quality ultrasound images. This invention also provides an automatic adjustment system for a robotic breast ultrasound probe based on reinforcement learning.

[0005] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is: to provide an automatic adjustment method for a robotic breast ultrasound probe based on reinforcement learning, comprising the following steps:

[0006] Step S1: Construct a Markov decision model;

[0007] Step S2: Establish an expert teaching experience pool, a real-time interactive experience pool, and a training data pool;

[0008] Step S3: Input ultrasound images in real time, and use the width reinforcement learning method to output reinforcement learning data;

[0009] Step S4: Adjust the probe to the target pose based on the reinforcement learning data.

[0010] As an improvement of the present invention, in step S2, the expert teaching experience pool is used to store expert teaching data, the real-time interactive experience pool is used to store data generated by reinforcement learning, and the training data pool samples data from the expert teaching experience pool and the real-time interactive experience pool.

[0011] As a further improvement of the present invention, in step S3, three task neural networks are constructed using a pre-trained MobileNet-v2 as a feature extractor. The first task neural network is for expert evaluation of the quality of real-time ultrasound images, the second task neural network is for classifying whether there are lesions in the ultrasound images, and the third task neural network is for segmenting the location of lesions in the ultrasound images.

[0012] As a further improvement of the present invention, the structure of the first task neural network consists of two convolutional layers (3×3 cores with 128 channels and 3×3 cores with 64 channels) and three fully connected layers.

[0013] As a further improvement of the present invention, the structure of the second task neural network is the same as that of the first task neural network.

[0014] As a further improvement of the present invention, a pre-trained MobileNet-v2 backbone is used as its encoder, and the decoder consists of five convolutional blocks with channel numbers of 256, 128, 64, 32, and 16, respectively. During training, each ultrasound image is reconstructed into... The ultrasound images are then mapped to feature maps of size 14×14×256, and these feature maps are then subjected to different convolution operations.

[0015] As a further improvement of the present invention, in the third task neural network, the upsampling kernel is used to restore the spatial resolution of the segmentation map S to 224×224 pixels.

[0016] As a further improvement of the present invention, in step S3, the data output for reinforcement learning includes termination status, expert score, whether there is a lesion, and lesion segmentation.

[0017] An automatic adjustment system for a robotic breast ultrasound probe based on reinforcement learning includes:

[0018] The building module is used to build Markov decision models, expert teaching experience pools, real-time interactive experience pools, and training data pools.

[0019] A lightweight, multi-tasking neural module is used to learn the expert's grading ability to determine the presence of lesions and output segmentation results;

[0020] The confidence map processor is used to obtain contact information between the ultrasound probe and the breast and output expert scores for ultrasound image quality, lesion information, confidence maps, lesion segmentation results, and the terminal status of reinforcement learning.

[0021] An auxiliary classifier is used to learn classification and reinforce the learning of the terminal state.

[0022] The beneficial effects of this invention are as follows: Compared with the prior art, this invention constructs a Markov decision model and an expert teaching experience pool, a real-time interactive experience pool and a training data pool, inputs ultrasound images in real time, adopts a width reinforcement learning method, and outputs reinforcement learning data. Then, based on the reinforcement learning data, the probe is adjusted to the target posture, thereby achieving efficient adjustment of the ultrasound probe to obtain high-quality ultrasound images. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the steps of the automatic adjustment method of the present invention;

[0024] Figure 2 This is a block diagram of the automatic adjustment system of the present invention;

[0025] Figure 3 This is a block diagram of the lightweight, multi-tasking real-time ultrasound image preprocessing system of the present invention.

[0026] Figure 4 This is a diagram showing the motion posture of the ultrasonic probe of the present invention;

[0027] Figure 5 This is a schematic diagram of ultrasound image collection according to the present invention. Detailed Implementation

[0028] Please refer to Figures 1 to 5 The automatic adjustment method for a robotic breast ultrasound probe based on reinforcement learning of the present invention includes the following steps:

[0029] Step S1: Construct a Markov decision model;

[0030] Step S2: Establish an expert teaching experience pool, a real-time interactive experience pool, and a training data pool;

[0031] Step S3: Input ultrasound images in real time, and use the width reinforcement learning method to output reinforcement learning data;

[0032] Step S4: Adjust the probe to the target pose based on the reinforcement learning data.

[0033] This invention constructs a Markov decision model, an expert teaching experience pool, a real-time interactive experience pool, and a training data pool. Ultrasound images are input in real time, and a width reinforcement learning method is used to output reinforcement learning data. Then, based on the reinforcement learning data, the probe is adjusted to the target posture, thereby achieving efficient adjustment of the ultrasound probe to obtain high-quality ultrasound images.

[0034] In step S2, the expert teaching experience pool is used to store expert teaching data, the real-time interactive experience pool is used to store data generated by reinforcement learning, and the training data pool samples data from the expert teaching experience pool and the real-time interactive experience pool.

[0035] In step S3, using a pre-trained MobileNet-v2 as a feature extractor, three task neural networks are constructed. The first task neural network performs expert evaluation of the real-time ultrasound image quality, the second task neural network classifies whether lesions exist in the ultrasound images, and the third task neural network segments the location of lesions in the ultrasound images. The structure of the first task neural network consists of two convolutional layers (3×3 kernels with 128 channels and 3×3 kernels with 64 channels) and three fully connected layers. The structure of the second task neural network is the same as that of the first task neural network. The pre-trained MobileNet-v2 backbone is used as its encoder, and the decoder consists of five upper convolutional blocks with channel numbers of 256, 128, 64, 32, and 16, respectively. During training, each ultrasound image is reconstructed into... The ultrasound image is then mapped to a feature map of size 14×14×256, and these feature maps are then subjected to different convolution operations. In the third task neural network, an upsampling kernel is used to restore the spatial resolution of the segmentation map S to 224×224 pixels.

[0036] In step S3, the output reinforcement learning data includes the termination state, expert score, whether there is a lesion, and lesion segmentation.

[0037] This invention also provides an automatic adjustment system for a robotic breast ultrasound probe based on reinforcement learning, comprising:

[0038] The building module is used to build Markov decision models, expert teaching experience pools, real-time interactive experience pools, and training data pools.

[0039] A lightweight, multi-tasking neural module is used to learn the expert's grading ability to determine the presence of lesions and output segmentation results;

[0040] The confidence map processor is used to obtain contact information between the ultrasound probe and the breast and output expert scores for ultrasound image quality, lesion information, confidence maps, lesion segmentation results, and the terminal status of reinforcement learning.

[0041] An auxiliary classifier is used to learn classification and reinforce the learning of the terminal state.

[0042] Specifically, the automatic adjustment system of the present invention includes a lightweight multi-task neural network module (LiMuNN), a confidence graph processor, and an auxiliary classifier for determining the terminal state of reinforcement learning. LiMuNN is used to learn the expert's rating ability to determine the presence of lesions and output the segmentation results. The confidence graph processor can clearly obtain the contact information between the ultrasound probe and the breast. LiMuCA outputs the expert's score on the ultrasound image quality, lesion information, confidence graph, lesion segmentation results, and the terminal state of reinforcement learning.

[0043] This invention utilizes a pre-trained MobileNet-v2 as a feature extractor, and subsequently constructs three task-specific neural networks, such as... Figure 3 As shown, Task 1 performs expert evaluation of real-time ultrasound image quality. The network structure consists of two convolutional layers (3×3 kernels with 128 channels and 3×3 kernels with 64 channels) and three fully connected layers. Task 2 classifies the presence of lesions, with the same network structure as Task 1. Task 3 segments lesion locations. A U-Net-based network is used, with a pre-trained MobileNet-v2 backbone as its encoder. The decoder consists of five upper convolutional blocks with channel numbers of 256, 128, 64, 32, and 16, respectively. During training, each ultrasound image is reconstructed into... They are then mapped to feature maps of size 14×14×256, and these feature maps are then subjected to different convolution operations. For task 3, lesion location segmentation, the upsampling kernel is used to restore the spatial resolution of the segmentation map S to 224×224 pixels. The cost function of LiMuCA is as follows:

[0044]

[0045] in, α is the loss value for label prediction, N is the total number of samples, j is the pixel position in the ultrasound image, the loss function is the cross-entropy loss function, and α is the loss value for label prediction. k (k = 1, ..., 3) are the weights of different tasks. The parameters of the multi-task network are obtained using gradient descent.

[0046]

[0047] Within this invention, a confidence map measures the uncertainty of an ultrasound image at the pixel level. Its pixels C(i,j) are mapped from the original image I via a random walk, ranging from 0 to 1. The confidence map highlights attenuated and shadowed areas of the ultrasound image, used to assess probe-skin contact. The confidence-weighted boundary center μ of the confidence map provides a standard for measuring how the image is divided, and is calculated in two directions as follows:

[0048]

[0049]

[0050] in, C represents the total confidence level and the average confidence level. m The method for calculating which part of the sensor is in contact with the skin is as follows:

[0051]

[0052] Among them, K i and K j This refers to the length and height of the confidence plot C. The extracted parameters (μ) i μ i C m These data are saved for subsequent auxiliary classification.

[0053] Within this invention, an auxiliary network is used for highly imbalanced classification. Since the target state is extremely rare compared to other states, this classification problem is highly imbalanced and difficult to handle using ordinary methods. BLS (CS-BLS) is an efficient highly imbalanced classification method with significant performance in medical applications; therefore, we use it to predict the target state. CS-BLS uses a weighted cross-entropy loss function and employs four weight calculation methods. The weight calculation methods for the four CS-BLS methods (Log-CS-BLS, Lin-CS-BLS, Sqr-CS-BLS, and EN-CS-BLS) are listed below:

[0054]

[0055]

[0056] Where, ω Log ω Lin ω Sqr ω EN These are the weight calculation methods for the four methods, where N is the total number of samples in the imbalanced training set. c It is the number of classes in the training set, and ε is the number of valid samples in class c. c Defined as sample volume, it is calculated as follows:

[0057]

[0058] β∈[0,1) is a hyperparameter.

[0059] The input to the network in this invention is a statistical analysis of real-time ultrasound images using LiMuCA and confidence plots, respectively, representing expert scores y1 and C. m ,μ i μ j The boundary center (S) of the classification result y2 and the real-time segmentation result i S j ), (S i S j The calculation is as follows:

[0060]

[0061] in, The output of the auxiliary network is a two-dimensional vector used to predict the target state.

[0062] In this invention, the probe adjustment task is modeled as a Markov decision problem (MDP), and then the probe adjustment experience of experts on the breast model is collected in real time. Finally, a width reinforcement learning algorithm (BRLfD) based on expert teaching guidance is proposed to solve the MDP problem with near-optimal policy.

[0063] 1) Markov decision model (MDP) for breast ultrasound probe adjustment

[0064] The components in this MDP are defined as follows:

[0065] 1. Movement space: The breast ultrasound probe controller can control the probe movement by pressing down or lifting it, or by rotating it left or right within a plane. Figure 5 Four probe actions are defined, where action 0 and action 2 represent the ultrasound probe being depressed and raised to a constant height, respectively. Action 1 and action 3 represent the probe rotating to the left and right to a constant angle, respectively.

[0066] A = {0, 1, 2, 3} (8)

[0067] 2. Status and Observation: When the ultrasound probe is adjusted to a new position, a new ultrasound image I can be observed at time point t. t The image is then processed using LiMuCA, the output is extracted and concatenated into a state s. t During the data acquisition process, the randomness of the state comes from the differences in lesions among the subjects and the interference between the probe and the breast model.

[0068] 3. Reward function: When the reinforcement learning agent enters a new state s after time step t. t At that time, it will also receive a reward r. t This represents the image quality of the current frame. Several factors influence breast ultrasound image quality. Considering the state extracted from LiMuCA, we incorporate its output into a reward function, which is the sum of weights for the following factors, the first of which considers the influence of the confidence score: The second item is to assess the centroid of the ultrasound image confidence map: The third item is the experts' assessment of the current situation. t Observation rating: The last item is based on the location of the lesion observed by the agent:

[0069] In r4, the two terms represent whether a lesion exists in the current frame. δ1, δ2, ..., δ5 are constant hyperparameters in the reward function. The reward function is designed as follows:

[0070]

[0071] Among them, S t The current state is represented by k, and the target pose of the ultrasound image is represented by G. k1, k2, k3, and k4 are constant hyperparameters.

[0072] Within this study, expert experience was collected and extracted. During the BRLfD training process, 16 different subjects were used to obtain the BUS probe adjustment strategy. In each study subject, experts actually scanned different lesions in the breast model with the probe, collected the position through an optical tracking system with passive markers, collected ultrasound images at a constant frequency, and then classified the breast ultrasound probe adjustment posture according to their actual posture.

[0073] like Figure 5 As shown, an expert uses an ultrasound probe to press on a breast model. The system acquires ultrasound images and saves them on a fixed workstation. The expert adjusts the system based on the real-time displayed images to obtain higher quality images. It can be seen from the figure that the expert's strategy is not optimal, with redundant movements throughout the probe adjustment process. Postures 3 and 4 in the second row and postures 5 and 6 in the third row are redundant. If we were very familiar with probe adjustment, we would use posture 3 (pressing) three times to reach the target posture. BRLfD can solve this problem by finding a near-optimal route. After the teaching process, the expert's teaching experience is saved in the expert teaching pool D as an MDP pattern (s, a, r, s′).

[0074] Within this invention, a wide reinforcement learning algorithm (BRLfD) based on expert instruction guidance is proposed. Wide reinforcement learning (BRL) is an improvement upon DQN, and due to the application of wide learning (BLS), it is highly effective in practical applications. Furthermore, considering the guidance of expert experience, we propose an improved BRL method, namely, constructing an expert instruction pool D and employing wide learning to approximate the action-state value function of reinforcement learning.

[0075] like Figure 2 As shown, the proposed BRLfD consists of two BLS networks used to approximate the action state function Q. BLS and target action state function, action state function The learning process of BRLfD is described as follows: It includes three experience pools: the human teaching pool D, the experience pool M obtained through reinforcement learning agents, and the training pool T.

[0076] The first step is the collection of human experience. Notably, for each object, several expert-taught trajectories are collected and stored in D until it is satisfied. Then, samples from the training pool T are used to train Q. BLS At that time, the target The calculation method is as follows:

[0077]

[0078] Among them, W - It was learned The weights are determined by the iteration index. When the iteration index is greater than L1 and less than L2, samples are taken from D and M. The index value of D is shown below.

[0079]

[0080] When interacting with a breast cancer model, the following methods were adopted: -Greedy algorithm for selecting actions:

[0081] a t =argmax a∈A Q(s t ,a;W) (12)

[0082] Where W is Q BLS The learned weights.

[0083] The following steps involve applying gradient-based width learning to approximate the evaluation of the Q-function Q. BLS First, map the state s to the feature space E. i .

[0084] E i =φ i (sW ei +βei (13)

[0085] Where, φ i Let E represent the i-th linear activation function. Then, combine all feature nodes into E. n E n ≡[E1, ..., E n ], where n is the number of feature nodes in the set, and then E n Through the nonlinear activation function ξ j Processing to generate enhanced node H j .

[0086]

[0087] Then, combine all the enhanced nodes into H. m H m ≡[H1, ..., H m ], where m is the number of groups of augmenting nodes, and H is the number of augmenting nodes. m and feature node E n Combination Therefore, the weights of BLS can be obtained through ridge regression.

[0088]

[0089] To save computational resources, gradient descent is used in BRLfD to obtain BLS.

[0090] Finally, parameter synchronization is performed, i.e., every k Q iterations. BLS Training, Q BLS The structure and parameters will be copied China and Israel have completed synchronization.

[0091] In summary, the proposed L-PA method is as follows.

[0092]

[0093]

[0094] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A system for automatic adjustment of a robotic breast ultrasound probe based on reinforcement learning, characterized in that, The automatic adjustment system comprises: a construction module for constructing a Markov decision model, an expert demonstration experience pool, a real-time interaction experience pool and a training data pool; the expert demonstration experience pool is used for storing expert demonstration data, the real-time interaction experience pool is used for storing data generated by reinforcement learning, and the training data pool samples data from the expert demonstration experience pool and the real-time interaction experience pool; a lightweight multi-task neural module for learning the hierarchical ability of an expert to determine whether a lesion exists and outputting a segmentation result; a pre-trained MobileNet-v2 is used as a feature extractor to construct three task neural networks, a first task neural network is used for expert evaluation of the quality of an ultrasound image, a second task neural network is used for classification of whether a lesion exists in the ultrasound image, and a third task neural network is used for segmentation of a lesion in the ultrasound image; a confidence map processor for obtaining contact information between an ultrasound probe and a breast and outputting an expert score of the quality of an ultrasound image, a confidence map C and a lesion segmentation result; the confidence map C measures the uncertainty of an ultrasound image at a pixel level, a pixel C(i,j) of the confidence map C is mapped from an ultrasound image through random walk, and ranges from 0 to 1; the confidence map C obtains contact information between an ultrasound probe and a breast by highlighting attenuated and shadowed parts of an ultrasound image; a confidence weighted boundary center μ of the confidence map C gives a standard for measuring image segmentation, and the calculation method is as follows: wherein, represents the total confidence; Average confidence C m Indicates which part of the ultrasound probe is in contact with the skin, calculated as follows: where K i and K j refer to the length and height of the confidence map C; the extracted parameters μ i , μ j and C m are saved for subsequent assisted classification; an auxiliary classifier for obtaining a target pose of an ultrasound probe according to an expert score of the quality of an ultrasound image, a confidence map C and a lesion segmentation result; and the automatic adjustment system adjusts an ultrasound probe to the target pose.

Images