A quantitative ultrasound localization microscopy imaging method based on attention learning mechanism

Abstract

The present invention provides a quantitative ultrasound localization microscopy imaging method based on an attention learning mechanism. Aiming at the shortcomings of the existing ultrasound localization microscopy imaging technology, an end-to-end deep convolutional neural network combining attention mechanism, residual learning and upsampling operation is adopted. A quantitative ultrasound localization microscopy imaging model is obtained based on the deep convolutional neural network training. The microbubble amplitude and position information in each frame of the original ultrasound image are accurately restored through the model. Finally, the restored results of all frames are superimposed to obtain a quantitative ultrasound localization microscopy imaging image corresponding to the original ultrasound image. The method combines deep learning technology with ultrasound localization microscopy imaging, significantly improves the ability to analyze microbubble amplitude signals in ultrasound imaging, and at the same time reduces the computational complexity of ultrasound localization microscopy imaging, avoids parameter dependence, and is suitable for quantitative ultrasound localization microscopy imaging.

Description

Technical Field

[0001] The present invention belongs to the technical field of ultrasonic imaging, and in particular relates to a quantitative ultrasonic positioning microscopic imaging method based on an attention learning mechanism. Background Art

[0002] Ultrasonic imaging can achieve fast, radiation-free imaging of biological tissues non-invasively and has been widely used. However, due to the limitations of acoustic diffraction theory, the spatial resolution of ultrasonic imaging is not high. In order to solve this problem, ultrasonic localization microscopy technology was proposed. Compared with traditional ultrasonic imaging, ultrasonic localization microscopy can increase its imaging spatial resolution by an order of magnitude and achieve ultra-high resolution ultrasonic imaging. However, the current ultrasonic localization microscopy technology only focuses on locating the position information of microbubbles in the imaging area, but cannot accurately restore its amplitude information, that is, it cannot achieve quantitative imaging. From a certain perspective, this limits the further application of ultrasonic localization microscopy technology.

[0003] Deep learning has important application potential in the field of super-resolution reconstruction of images. In short, deep learning methods can learn the end-to-end mapping between low-resolution images and high-resolution images by training a large number of paired data sets. Super-resolution reconstruction can be achieved based on the trained learning model. In deep learning-based methods, the attention mechanism has been successfully used to improve the efficiency and accuracy of important feature processing. Compared with traditional convolutional neural networks, this mechanism can perform targeted learning and extraction based on the importance of features. Summary of the invention

[0004] In order to solve the above problems, an imaging method combining deep learning technology and ultrasonic positioning microscopy is provided. The present invention adopts the following technical solutions:

[0005] The present invention provides a quantitative ultrasound localization microscopy imaging method based on an attention learning mechanism, which is characterized in that it comprises the following steps: step S1, constructing a deep convolutional neural network based on an attention mechanism; step S2, training the deep convolutional neural network to obtain a quantitative ultrasound localization microscopy imaging model; step S3, obtaining an original ultrasound image sequence to be imaged, each frame of the original ultrasound image containing randomly distributed microbubbles; step S4, inputting the original ultrasound image into the quantitative ultrasound localization microscopy imaging model, thereby accurately restoring the amplitude and position information of the microbubbles in each frame of the original ultrasound image; step S5, superimposing the restored results of all frames to obtain a quantitative ultrasound localization microscopy imaging image corresponding to the original ultrasound image, wherein the deep convolutional neural network comprises a first convolutional layer, a multi-scale attention mechanism module and a sub-pixel convolutional layer.

[0006] The quantitative ultrasound localization microscopy imaging method based on the attention learning mechanism provided by the present invention may also have such a technical feature, wherein step S2 includes the following sub-steps: step S2-1, obtaining training data and training labels based on the ultrasound imaging system; step S2-2, constructing a loss function based on MSE combined with L1 regularization and structural similarity index:

[0007]

[0008] Where θ is the training label, is the predicted output image of the network, N is the number of small batch images that the network optimizes each time, and λ1 and λ2 are weighting coefficients; step S2-3, taking the training data and training labels as input and output respectively, and using the loss function to optimize the deep convolutional neural network using the Adam optimization algorithm to traverse and train, thereby obtaining a quantitative ultrasound positioning microscopy imaging model.

[0009] The quantitative ultrasound localization microscopy imaging method based on the attention learning mechanism provided by the present invention may also have such a technical feature, wherein, in the optimization traversal training, the training cycle is 500, the number of small batch images N of each optimization traversal of the network is 50, and the initial learning rate is 0.001.

[0010] The quantitative ultrasound localization microscopic imaging method based on the attention learning mechanism provided by the present invention may also have such a technical feature, wherein the process of acquiring training data is as follows: Step S2-1-1, based on the ultrasound imaging system, a single microbubble in the imaging area of ​​the original image is imaged multiple times, and the average lateral half-height width FHWM of the single microbubble in the multiple imaging results is calculated. x The average longitudinal half-width FHWM y ; Step S2-1-2, based on the average horizontal half-height width FHWM x The average longitudinal half-width FHWM y , calculate the lateral standard deviation δ x and the longitudinal standard deviation δ y :

[0011]

[0012] Step S2-1-3, based on the horizontal standard deviation δ x The point spread function model of the ultrasound imaging system is obtained by calculating the longitudinal standard deviation:

[0013]

[0014] Where H(x,y) represents the point spread function, σ x and σ yare the lateral standard deviation and the longitudinal standard deviation, f represents the ultrasonic emission frequency, and c represents the speed of sound; step S2-1-4, generating n randomly distributed microbubbles of different brightness in the grid area of ​​the first predetermined pixel; step S2-1-5, using the forward model to simulate the ultrasonic imaging process by convolving the actual positions of n microbubbles of different brightness according to the point spread function model H(x, y) to obtain a noise-free microscopic simulation image; step S2-1-6, adding Gaussian white noise of different signal-to-noise ratios to the noise-free microscopic simulation image to obtain training data, and mapping the microbubble distribution information in the noise-free microscopic simulation image to the grid area of ​​the second predetermined pixel as a training label.

[0015] The quantitative ultrasound localization microscopy imaging method based on the attention learning mechanism provided by the present invention may also have such a technical feature, wherein the first predetermined pixel is 64x64 and the second predetermined pixel is 256x256.

[0016] The quantitative ultrasound localization microscopy imaging method based on the attention learning mechanism provided by the present invention may also have such a technical feature, wherein the value range of the Gaussian white noise is 5dB to 25dB.

[0017] The quantitative ultrasound localization microscopy imaging method based on the attention learning mechanism provided by the present invention may also have the following technical features: the convolution layer is used to extract shallow features of the image, the sub-pixel convolution layer is used to implement upsampling operations, and the multi-scale attention mechanism module is used to extract deep effective features of the data. The convolution module includes two channel space attention modules and a multi-scale fusion attention module. The convolution layer is connected to the multi-scale attention mechanism module for global residual learning.

[0018] The quantitative ultrasound localization microscopy imaging method based on the attention learning mechanism provided by the present invention may also have such a technical feature, wherein the channel space attention module is composed of a channel attention module and a spatial attention module stacked together, the channel attention module learns the weighting coefficient for each channel of the feature map, considering all areas of the image, and the spatial attention module learns the weighting coefficient for each different area in the feature map, considering all channels of the feature map.

[0019] The quantitative ultrasound positioning microscopy imaging method based on the attention learning mechanism provided by the present invention may also have such technical features, wherein the multi-scale fusion attention module is composed of a multi-scale attention module and a spatiotemporal fusion attention module, the multi-scale attention module includes convolution operations with different receptive fields, channel attention convolutions and spatial attention convolutions, the spatiotemporal fusion attention module is composed of the channel attention and the spatial attention with jump connection superposition, and jump connection is performed between every two modules for local residual learning.

[0020] Function and Effect of the Invention

[0021] According to the quantitative ultrasound localization microscopy method based on the attention learning mechanism of the present invention, in view of the shortcomings of the existing ultrasound localization microscopy technology, an end-to-end deep convolutional neural network combining attention mechanism, residual learning and upsampling operation is adopted, and a quantitative ultrasound localization microscopy model is obtained based on the deep convolutional neural network training, and the microbubble amplitude and position information in each frame of the original ultrasound image are accurately restored through the model, and finally the restored results of all frames are superimposed to obtain the quantitative ultrasound localization microscopy image corresponding to the original ultrasound image. Due to the introduction of the multi-scale attention mechanism module, the network can extract microbubble amplitude information at different scales, solving the problem of large differences in microbubble amplitude; and due to the addition of jump connections in the network and the application of local residual structure and global residual structure, the training error is reduced, thereby avoiding the defect that the deeper network is difficult to optimize and train due to gradient vanishing or gradient explosion.

[0022] The quantitative ultrasound localization microscopy method based on the attention learning mechanism of the present invention combines deep learning technology with ultrasound localization microscopy, significantly improving the ability to analyze microbubble amplitude signals in ultrasound imaging, while reducing the computational complexity of ultrasound localization microscopy and avoiding parameter dependence, and is suitable for quantitative ultrasound localization microscopy. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flow chart of a quantitative ultrasound localization microscopic imaging method based on an attention learning mechanism in an embodiment of the present invention;

[0024] Figure 2 is a schematic diagram of the structure of a deep convolutional neural network based on an attention mechanism in an embodiment of the present invention;

[0025] Figure 3 is a schematic diagram of simulated imaging based on a simulated blood vessel model in an embodiment of the present invention;

[0026] Figure 4 is an ultrasound localization microscopic imaging result diagram obtained by reconstructing a simulated blood vessel model based on the method of the present invention in an embodiment of the present invention; and

[0027] Figure 5 It is the amplitude comparison result of the ultrasound positioning microscopic image reconstructed from the simulated blood vessel model based on the method of the present invention in the embodiment of the present invention. DETAILED DESCRIPTION

[0028] In view of the shortcomings of the existing ultrasound localization microscopy technology, the present invention proposes a quantitative ultrasound localization microscopy method based on the attention deep learning mechanism, which significantly improves the analysis ability of microbubble amplitude signals in ultrasound imaging, while reducing the computational complexity of ultrasound localization microscopy, avoiding parameter dependence, and is suitable for quantitative ultrasound localization microscopy.

[0029] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the quantitative ultrasound localization microscopy imaging method based on the attention learning mechanism of the present invention is specifically described below in combination with embodiments and drawings.

[0030] <Example>

[0031] Figure 1 It is a flow chart of a quantitative ultrasound localization microscopy imaging method based on attention learning mechanism in an embodiment of the present invention.

[0032] like Figure 1 As shown, the quantitative ultrasound localization microscopy imaging method based on attention learning mechanism includes the following steps:

[0033] Step S1, construct a deep convolutional neural network based on the attention mechanism.

[0034] In this embodiment, an end-to-end deep convolutional neural network combining attention mechanism, residual learning and upsampling operation is used for quantitative ultra-high resolution reconstruction of ultrasound images.

[0035] Figure 2 It is a schematic diagram of the structure of a deep convolutional neural network based on the attention mechanism in an embodiment of the present invention.

[0036] like Figure 2 As shown in Figure 3, the deep convolutional neural network consists of three parts.

[0037] Among them, the first part consists of a convolutional layer with a larger convolution kernel, which is used to extract shallow features of the image.

[0038] The second part is the main body of the deep convolutional neural network, namely the multi-scale attention mechanism module, which is used to extract deep and effective features of the data. This part includes two channel spatial attention modules (Channel Spatial Attention Block) and a multi-scale fusion attention module (Multi-Scale Channel-Spatial Fusion Block).

[0039] The channel-spatial attention module is composed of a stack of channel-attention modules and spatial-attention modules. The channel-attention module learns weighting coefficients for each channel of the feature map, taking into account all regions of the image; the spatial-attention module learns weighting coefficients for each different region in the feature map, taking into account all channels of the feature map.

[0040] The multi-scale fusion attention module is composed of a multi-scale attention module and a spatiotemporal fusion attention module. The multi-scale attention module contains convolution operations with different receptive fields, channel attention convolutions, and spatial attention convolutions. The spatiotemporal fusion attention module is composed of a skip connection superposition of channel attention and spatial attention, and a skip connection is made between every two modules for local residual learning.

[0041] In this embodiment, the convolution layer of the first part is connected to the convolution layer before the third part for global residual learning. The third part is a sub-pixel convolution layer for implementing upsampling operation.

[0042] Step S2, training a deep convolutional neural network to obtain a quantitative ultrasound localization microscopic imaging model. Specifically:

[0043] Step S2-1, acquiring training data and training labels based on the ultrasound imaging system.

[0044] The process of obtaining the training data and training labels in this step is as follows:

[0045] Step S2-1-1, based on the ultrasonic imaging system, a single microbubble in the imaging area of ​​the original image is imaged multiple times, and the average lateral half-height width FHWM of the single microbubble in the multiple imaging results is calculated. x The average longitudinal half-width FHWM y ;

[0046] Step S2-1-2, based on the average horizontal half-height width FHWM x The average longitudinal half-width FHWM y , calculate the lateral standard deviation δ x and the longitudinal standard deviation δ y :

[0047]

[0048] Step S2-1-3, based on the horizontal standard deviation δ x The point spread function model of the ultrasound imaging system is obtained by calculating the longitudinal standard deviation:

[0049]

[0050] Where H(x,y) represents the point spread function, σ x and σy are the transverse standard deviation and the longitudinal standard deviation, f represents the ultrasonic emission frequency, and c represents the sound speed;

[0051] Step S2-1-4, generating n randomly distributed microbubbles of different brightness in a 64x64 grid area;

[0052] Step S2-1-5, using the forward model to simulate the ultrasonic imaging process by performing convolution at the real positions of n microbubbles with different brightness according to the point spread function model H(x, y), and obtaining a noise-free microscopic simulation image;

[0053] Step S2-1-6, in order to increase the robustness of the training model, Gaussian white noise (5-25dB) with different signal-to-noise ratios (SNR) is added to part of the noise-free microscopic simulation image to obtain training data, and the microbubble distribution information in the noise-free microscopic simulation image is mapped to a 256x256 pixel grid area as a training label.

[0054] Figure 3 It is a schematic diagram of quantitative ultrasound positioning microscopic imaging based on a simulated blood vessel model in an embodiment of the present invention.

[0055] Figure 3 (a) is the mean image; (b) is the real distribution of simulated blood vessels in the simulated blood vessel model; (d)-(e) are enlarged images of the boxed areas in (a)-(b), respectively.

[0056] In the process of acquiring the above training data, this embodiment is based on Figure 3 The simulated blood vessel model shown in (b) is subjected to simulated ultrasound imaging to obtain a sequence of ultrasound images at different times. Each frame of the image contains multiple point scatterers (microbubbles) of different amplitudes, which are randomly distributed in the simulated blood vessel model area. The specific simulation process is as follows:

[0057] During each ultrasound imaging process, 1-5 microbubbles with different amplitude information are randomly activated in the simulated vascular model area, and the ultrasound imaging is simulated by convolving the point spread function of the imaging system obtained by the above calculation; then, the positions of 1-5 microbubbles in the vascular model are randomly changed, and the simulated imaging is performed again; in order to describe the movement of microbubbles in the vascular model, the above imaging process is repeated 2000 times to obtain 2000 frames of original ultrasound images. After obtaining all the original ultrasound images, in order to simulate the influence of noise in the actual experiment, 20dB Gaussian white noise is added to the generated ultrasound images one by one.

[0058] Unlike the traditional ultrasound localization microscopy imaging method, the method proposed in the present invention directly maps the original low-resolution ultrasound image to an ultra-high-resolution ultrasound image with a real microbubble distribution (position and amplitude) through the network, so the loss function has a great influence on the network training. The use of the mean square error (MSE) loss commonly used in image processing usually makes the output result too smooth, and it is impossible to accurately evaluate the recovery effect of the microbubble amplitude information, which is not suitable for quantitative microbubble reconstruction tasks. Therefore, this embodiment uses the loss function of MSE combined with L1 regularization and structural similarity index (SSIM) to train the network:

[0059] Step S2-2, construct the loss function based on MSE combined with L1 regularization and structural similarity index:

[0060]

[0061] Where θ is the training label, is the predicted output image of the network, N is the number of mini-batch images that the network traverses each time it is optimized, and λ1 and λ2 are weighting coefficients.

[0062] Step S2-3, using the training data and training labels as input and output respectively, using the loss function to optimize the deep convolutional neural network using the Adam optimization algorithm to perform traversal training, thereby obtaining a quantitative ultrasound positioning microscopy imaging model.

[0063] In this embodiment, the Adam optimization algorithm is used to train a total of 500 cycles (500 times of traversing the training set) on a training set containing 5000 pairs of original ultrasound images and corresponding labels, wherein the small batch size N of each optimization traversal of the network is 50, and the initial learning rate is 0.001. In order to make the error converge effectively and stably, this embodiment uses a learning rate decay strategy (i.e., reducing the learning rate when the training error is stable).

[0064] Step S3, obtaining a sequence of original ultrasound images to be imaged, each frame of the original ultrasound image containing randomly distributed microbubbles.

[0065] Step S4, inputting the original ultrasound image into the quantitative ultrasound localization microscopic imaging model, so as to accurately restore the amplitude and position information of the microbubbles in each frame of the original ultrasound image.

[0066] Step S5, superimposing the restoration results of all frames to obtain a quantitative ultrasound localization microscopic imaging image corresponding to the original ultrasound image.

[0067] Figure 4 is an ultrasound localization microscopic imaging result diagram obtained by reconstructing a simulated blood vessel model based on the method of the present invention in an embodiment of the present invention, and Figure 5It is the amplitude comparison result of the ultrasound positioning microscopic image reconstructed from the simulated blood vessel model based on the method of the present invention in the embodiment of the present invention.

[0068] In this embodiment, the trained quantitative ultrasound localization microscopic imaging model is used to analyze the ultrasound image acquired based on the simulated blood vessel, and the obtained microscopic imaging results are as follows: Figure 4 (c) Figure 4 (f) Figure 4 (c) Enlarged image of the boxed area.

[0069] The amplitude results of microbubbles obtained based on the quantitative ultrasound localization microscopy imaging model of the present invention are as follows: Figure 5 As shown. Among them, Figure 5 (a) corresponds to Figure 3 (b) Figure 4 Amplitude curve of the horizontal white dashed line in (c); Figure 5 (b) corresponds to Figure 3 (b) Figure 4 Amplitude curve of the vertical white dashed line in (c).

[0070] Example Function and Effect

[0071] According to the quantitative ultrasound localization microscopy method based on the attention learning mechanism provided in this embodiment, in view of the shortcomings of the existing ultrasound localization microscopy technology, an end-to-end deep convolutional neural network combining attention mechanism, residual learning and upsampling operation is adopted, and a quantitative ultrasound localization microscopy model is obtained based on the deep convolutional neural network training, through which the microbubble amplitude and position information in each frame of the original ultrasound image are accurately restored, and finally the restored results of all frames are superimposed to obtain a quantitative ultrasound localization microscopy image corresponding to the original ultrasound image. This method combines deep learning technology with ultrasound localization microscopy, significantly improves the ability to analyze microbubble amplitude signals in ultrasound imaging, and at the same time reduces the computational complexity of ultrasound localization microscopy, avoids parameter dependence, and is suitable for quantitative ultrasound localization microscopy.

[0072] In the embodiment, due to the introduction of a multi-scale attention mechanism module, the network can extract microbubble amplitude information at different scales, thereby solving the problem of large differences in microbubble amplitudes; and due to the addition of jump connections in the network and the application of local residual structures and global residual structures, the training error is reduced, thereby avoiding the defect that deeper networks are difficult to optimize training due to gradient vanishing or gradient exploding reasons.

[0073] In the embodiment, a loss function combining MSE, l1 regularization and structural similarity index (SSIM) is constructed to solve the problems of using the mean square error (MSE) loss function commonly used in the prior art to train the deep learning model, resulting in the reconstructed image being too smooth and the inability to accurately evaluate the microbubble amplitude recovery effect. This makes the quantitative ultrasound localization microscopy imaging model better suitable for the microbubble quantitative reconstruction task.

[0074] The above embodiments are only used to illustrate specific implementation modes of the present invention, and the present invention is not limited to the description scope of the above embodiments.

Claims

1. A quantitative ultrasound localization microscopy imaging method based on attention learning mechanism, characterized in that: The following steps are involved: Step S1, constructing a deep convolutional neural network based on the attention mechanism; Step S2, training the deep convolutional neural network to obtain a quantitative ultrasound localization microscopy imaging model; Step S3, obtaining a sequence of original ultrasound images to be imaged, each frame of the original ultrasound image containing randomly distributed microbubbles; Step S4, inputting the original ultrasound image into the quantitative ultrasound localization microscopy imaging model, so as to restore the amplitude and position information of the microbubbles in each frame of the original ultrasound image; Step S5, superimposing the restoration results of all frames to obtain a quantitative ultrasound localization microscopic imaging image corresponding to the original ultrasound image, The deep convolutional neural network includes a convolutional layer, a multi-scale attention mechanism module and a sub-pixel convolutional layer. The convolutional layer is used to extract shallow features of the image. The sub-pixel convolution layer is used to implement upsampling operation. The multi-scale attention mechanism module is used to extract deep and effective features of data. The convolution module includes two channel space attention modules and a multi-scale fusion attention module. The convolutional layer is connected to the multi-scale attention mechanism module for global residual learning. The channel space attention module is composed of a channel attention module and a spatial attention module stacked together. The channel attention module learns weighting coefficients for each channel of the feature map, considering all regions of the image. The spatial attention module learns weighting coefficients for each different region in the feature map, considering all channels of the feature map. The multi-scale fusion attention module is composed of a multi-scale attention module and a spatiotemporal fusion attention module. The multi-scale attention module includes convolution operations with different receptive fields, channel attention convolutions, and spatial attention convolutions. The spatiotemporal fusion attention module is composed of a skip connection superposition of channel attention and spatial attention, and a skip connection is performed between every two modules for local residual learning.

2. The quantitative ultrasound localization microscopy imaging method based on attention learning mechanism according to claim 1, Features: Wherein, the step S2 includes the following sub-steps: Step S2-1, acquiring training data and training labels based on an ultrasound imaging system; Step S2-2, construct the loss function based on MSE combined with L1 regularization and structural similarity index: Where θ is the training label, is the predicted output image of the network, N is the number of small batch images traversed by the network each time, and λ1 and λ2 are weighting coefficients; Step S2-3, using the training data and the training labels as input and output respectively, using the loss function to optimize the deep convolutional neural network using the Adam optimization algorithm to perform traversal training, thereby obtaining the quantitative ultrasound localization microscopy imaging model.

3. The quantitative ultrasound localization microscopy imaging method based on attention learning mechanism according to claim 2, characterized in that: in, In the optimization traversal training, the training cycle is 500, the number of small batch images N of each optimization traversal of the network is 50, and the initial learning rate is 0.

001.

4. The quantitative ultrasound localization microscopy imaging method based on attention learning mechanism according to claim 2, Features: The process of obtaining the training data is as follows: Step S2-1-1, based on the ultrasound imaging system, a single microbubble in the imaging area of ​​the original image is imaged multiple times, and the average lateral half-height width FHWM of the single microbubble in the multiple imaging results is calculated. x The average longitudinal half-width FHWM y ; Step S2-1-2, based on the average lateral half-height width FHWM x The average longitudinal half-width FHWM y , calculate the lateral standard deviation δ x and the longitudinal standard deviation δ y : Step S2-1-3, based on the lateral standard deviation δ x The point spread function model of the ultrasound imaging system is obtained with the longitudinal standard deviation calculation: Where H(x,y) represents the point spread function, σ x and σ y are the transverse standard deviation and the longitudinal standard deviation, f represents the ultrasonic emission frequency, and c represents the sound speed; Step S2-1-4, generating n randomly distributed microbubbles of different brightness in a grid area of ​​a first predetermined pixel; Step S2-1-5, using the forward model to simulate the ultrasonic imaging process by performing convolution at the real positions of the n microbubbles with different brightness according to the point spread function model H(x, y), to obtain a noise-free microscopic simulation image; Step S2-1-6, adding Gaussian white noise with different signal-to-noise ratios to the noise-free microscopic simulation image to obtain the training data, and mapping the microbubble distribution information in the noise-free microscopic simulation image to a grid area of ​​a second predetermined pixel as the training label.

5. The quantitative ultrasound localization microscopy imaging method based on attention learning mechanism according to claim 4, characterized in that: in, The first predetermined pixel is 64x64, and the second predetermined pixel is 256x256.

6. The quantitative ultrasound localization microscopy imaging method based on attention learning mechanism according to claim 4, characterized in that: in, The value range of the Gaussian white noise is 5-25dB.

Images