A fast ultrasound localization microscopy imaging method based on generative adversarial network

By generating adversarial networks and residual convolutional neural networks, combining multi-scale structure similarity loss and generation adversarial loss, the problems of long imaging time and low resolution in ultrasonic positioning microscopy technology are solved, and fast and high resolution are achieved Ultrasound localization microscopy reduces computational complexity and parameter dependence.

CN114897689BActive Publication Date: 2024-11-15FUDAN UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210460857.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-28
Publication Date
2024-11-15
Estimated Expiration
2042-04-28

AI Technical Summary

Technical Problem

When existing ultrasonic positioning microscopy imaging technology achieves ultra-high resolution imaging in living organisms, the imaging time takes a long time and the image resolution is limited. The single-molecular positioning strategy increases the probability of microbubble overlapping and reduces accuracy.

Method used

Using a fast ultrasonic positioning microscopy imaging method based on generative adversarial network, the generator is constructed through the Pix2pix framework and the residual convolutional neural network, and the multi-scale structure similarity loss, L1 regular loss and generative adversarial loss are trained to achieve low-resolution images Mapping to high-resolution images reduces imaging frame count and improves resolution.

Benefits of technology

Fast ultrasound positioning microscopy is achieved, with the imaging quality close to full positioning, reducing computational complexity and parameter dependence, and improving the performance and temporal resolution of ultrasound imaging technology.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114897689B_ABST
    Figure CN114897689B_ABST
Patent Text Reader

Abstract

The present invention provides a fast ultrasound localization microscopy imaging method based on a generative adversarial network, comprising the following steps: constructing an initial imaging model based on a generative adversarial network of a Pix2pix framework, in which a residual convolutional neural network is used as a generator and a PatchGAN structure is used as a discriminator; obtaining a low-resolution original ultrasound image as a network input, and obtaining a corresponding high-resolution ultrasound localization microscopy image as a training label, thereby respectively constructing a training set and a test set; inputting the training set into the initial imaging model, and using a loss function and a training label for training to generate a fast super-resolution imaging model; obtaining an ultra-high resolution ultrasound image corresponding to the original ultrasound image to be tested based on the fast super-resolution imaging model. The method does not require additional calculations or manual parameter adjustment, reduces the complexity of calculations in the imaging reconstruction process, avoids parameter dependence, greatly improves the performance of existing ultrasound localization microscopy imaging technology, and is more suitable for fast ultrasound localization microscopy imaging.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of ultrasound localization microscopy, and in particular relates to a fast ultrasound localization microscopy method based on generative adversarial networks (GAN). Background Art

[0002] Ultrasound Localization Microscopy (ULM) technology has been proposed and applied. Compared with traditional ultrasound imaging, by combining single-molecule localization technology with ultrasound contrast agents, ULM breaks through the acoustic diffraction limit and can observe structural information in vivo at the subwavelength scale.

[0003] However, it is still extremely challenging to use ULM for in vivo ultra-high resolution ultrasound imaging. The main reason is that ULM is based on a single-molecule localization strategy to achieve super-resolution imaging. Based on the single-molecule localization method, although the spatial resolution of ultrasound imaging can be greatly improved, it also greatly increases the time required for imaging. In order to overcome this limitation, a feasible method is to increase the number of microbubbles in a single frame, thereby reducing the number of frames required for ultra-high resolution reconstruction to increase the imaging speed. However, this method increases the probability of microbubble overlap to a certain extent and reduces the accuracy of single-molecule localization. Although the imaging speed can be improved, the image resolution is also sacrificed.

[0004] Deep learning is an emerging technology that has good performance in super-resolution image reconstruction tasks. In short, the deep learning method uses low-resolution images as network input and high-resolution images as training labels, and iteratively trains the network to learn the mapping relationship between low-resolution images and high-resolution images. At present, traditional deep convolutional networks have been used for super-resolution reconstruction of ultrasound images. However, this method still achieves super-high resolution imaging through a single-frame positioning strategy, and the imaging quality still depends on the number of positioning frames.

[0005] Compared with traditional convolutional networks, the training process of generative adversarial networks not only takes minimizing pixel-level differences as the optimization goal, but also considers the deep evaluation network indicators. Through adversarial training between networks, the generated images are closer to reality. In short, the generative adversarial network consists of a generator and a discriminator. The generator uses a low-resolution image as input to obtain an output image, and uses the output image as the discriminator input to obtain the discriminator score. The generator is iteratively trained according to the discriminator score. Therefore, the present invention proposes an ultrasound positioning imaging technology based on a generative adversarial network. Summary of the invention

[0006] In order to solve the above problems, an imaging method for realizing ultra-high resolution reconstruction of ultrasound images by generating adversarial networks is provided. The present invention adopts the following technical solutions:

[0007] The present invention provides a fast ultrasound localization microscopic imaging method based on a generative adversarial network, which is used for fast ultrasound localization of an original ultrasound image to obtain a super-resolution ultrasound image, and is characterized in that it comprises: step S1, constructing an initial imaging model based on a generative adversarial network; step S2, obtaining a low-resolution original ultrasound image as a network input, and obtaining a corresponding high-resolution ultrasound localization microscopic image as a training label, thereby respectively constructing a training set and a test set; step S3, inputting the training set into the initial imaging model, training and optimizing the model using a loss function and training labels until the training is completed, and obtaining a fast super-resolution imaging model; step S4, inputting the original ultrasound image to be tested into the fast super-resolution imaging model for fast ultrasound localization, thereby obtaining a corresponding ultra-high resolution ultrasound image, wherein the generative adversarial network uses Pix2pix as a basic framework, uses a residual convolutional neural network as a generator network, and uses a PatchGAN structure as a discriminator network.

[0008] The present invention provides a fast ultrasound localization microscopy imaging method based on a generative adversarial network, which may also have the following technical features: the discriminator network is used to provide generative adversarial loss for model training, the generator network includes 6 residual modules for residual learning and 2 upsampling modules for upsampling operations, each residual module includes 2 convolutional layers and corresponding batch normalization layers and activation functions, and each upsampling module consists of a transposed convolutional layer, a batch normalization layer and an activation function.

[0009] The present invention provides a rapid ultrasound localization microscopic imaging method based on a generative adversarial network, which may also have such a technical feature, wherein the process of acquiring the original ultrasound image is: performing ultrasound imaging on the imaging object based on an ultrasound contrast agent to obtain a group of ultrasound original image sequences, wherein each frame of the ultrasound original image contains a plurality of sparsely distributed microbubbles randomly distributed in the imaging area; randomly selecting 20 consecutive frames from this group of ultrasound original image sequences for superposition, and obtaining a low-resolution original ultrasound image after normalization, wherein the original ultrasound image is 512*512 pixels.

[0010] The present invention provides a rapid ultrasound localization microscopy imaging method based on a generative adversarial network, which may also have the following technical features: the training label construction process is as follows: single-molecule localization is performed on each frame of the ultrasound original image, that is, the independent microbubbles in each frame of the image are centrally located by a Gaussian fitting method and a centroid method to obtain the localization frame corresponding to the microbubble, 95% of the localization frames in all the localization frames are superimposed, and after normalization and upsampling by a predetermined multiple, an ultrasound localization microscopy image is obtained as a training label, and the ultrasound localization microscopy image is 2048*2048 pixels.

[0011] The present invention provides a method for rapid ultrasound localization microscopy based on a generative adversarial network, which may also have such a technical feature, wherein, in the training of the initial imaging model, the loss function of the generator network training consists of a multi-scale structural similarity loss, an L1 regularization loss and a generative adversarial loss:

[0012] L G =α×L1(G(LR l ), HR L )+β×[1-L SSIM (G(LR l ), HR L )]+-logD(G(LR l ), LR l )

[0013] In the formula, L1(·) represents L1 loss, G(·) represents the generator network, and LR l Represents the original ultrasound image obtained by superimposing 1 frame of images, HR L represents the ultrasound localization microscopic image obtained by superimposing L frame localization frames, L SSIM (·) represents the multi-scale structural similarity loss, D(·) represents the discriminator network, α and β represent the coefficients of L1 regularization loss and multi-scale structural similarity loss, and the loss function of the discriminator adopts generative adversarial loss:

[0014] L D = -logD(HR L , L.R. l )-log(1-D(G(LR l ), LR l )).

[0015] The present invention provides a fast ultrasound localization microscopy imaging method based on a generative adversarial network, which may also have the following technical features: optimization is to use the Adam optimization gradient descent algorithm to train the original ultrasound image and the corresponding ultrasound localization microscopy image for 50 cycles, that is, traversing the training set 50 times, the data batch size N of each cycle is 4, and the initial learning rate is 1×10.

[0016] The present invention provides a fast ultrasound localization microscopy imaging method based on a generative adversarial network, which may also have a technical feature, wherein, during the optimization process, a learning rate decay strategy is adopted to gradually reduce the learning rate to 0, thereby quickly approaching the optimal parameters and making the loss function converge to the minimum at the fastest speed.

[0017] Function and Effect of the Invention

[0018] According to a fast ultrasound localization microscopic imaging method based on a generative adversarial network of the present invention, Pix2pix is ​​used as the basic framework of the generative adversarial network, thereby constructing a fast super-resolution ultrasound imaging model with pixel-level advantages compared to the existing deep learning network. By inputting the low-resolution original ultrasound image into the fast super-resolution ultrasound imaging model for reconstruction, the corresponding imaging quality can be obtained, which is comparable to the ultra-high resolution ultrasound image with full positioning. In addition, in the generative adversarial network, the generator network uses a residual convolutional neural network to replace the U-Net network of the original Pix2pix framework, and realizes the fusion of shallow and deep features at the same scale through jump links. While greatly reducing the training error, it also overcomes the defect of scattered feature loss caused by multiple pooling of high-dimensional information extracted by the U-Net network.

[0019] The rapid ultrasound localization microscopy imaging method based on a generative adversarial network of the present invention does not require any additional calculations or manual parameter adjustments, reduces the complexity of calculations in the imaging reconstruction process, avoids parameter dependence, greatly improves the performance of existing ultrasound localization microscopy imaging technology, and is more suitable for rapid ultrasound localization microscopy imaging. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of a fast ultrasound localization microscopic imaging method based on a generative adversarial network in an embodiment of the present invention;

[0021] Figure 2 is a schematic diagram of the structure of a generative adversarial network in an embodiment of the present invention;

[0022] Figure 3 is a schematic diagram of the structure of a generator network in an embodiment of the present invention;

[0023] Figure 4 is a schematic diagram of an ultrasonic image model used in a simulation experiment in an embodiment of the present invention;

[0024] Figure 5 is a schematic diagram of an ultrasound localization microscopic image in an embodiment of the present invention;

[0025] Figure 6 It is a schematic diagram of a mean image generated by superimposing a small number of original ultrasound image frames and an ultrasound localization microscopic imaging result obtained based on a fast super-resolution imaging model in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] Inspired by the ultrasound localization microscopy technology based on single-molecule localization, the present invention proposes to use a generative adversarial network combined with an end-to-end generator network based on a residual module for super-resolution ultrasound image reconstruction, that is, to obtain an ultra-high resolution ultrasound image with imaging quality comparable to that of full localization by reconstructing a small amount of original ultrasound images, thereby realizing rapid ultrasound localization microscopy.

[0027] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following is a specific description of a rapid ultrasound localization microscopy method based on a generative adversarial network of the present invention in combination with embodiments and drawings.

[0028] <Example>

[0029] Figure 1 It is a flow chart of a fast ultrasound localization microscopic imaging method based on a generative adversarial network in an embodiment of the present invention.

[0030] like Figure 1 As shown, the rapid ultrasound localization microscopy imaging method based on the generative adversarial network of this embodiment includes the following steps:

[0031] Step S1, constructing an initial imaging model based on a generative adversarial network, where the generative adversarial network consists of a generator network and a discriminator network.

[0032] Figure 2 Schematic diagram of the structure of a generative adversarial network in an embodiment of the present invention.

[0033] Considering that the Pix2pix framework has a pixel-level advantage in end-to-end supervised learning tasks in the generative adversarial network, this embodiment uses Pix2pix as the basic framework of the generative adversarial network (such as Figure 2 shown).

[0034] Since the microbubbles in the original ultrasound image have scattered characteristics, the original Pix2pix framework selects U-Net as the generator network to extract multi-scale information. The generator extracts high-dimensional information through multiple pooling, which easily leads to the loss of scattered features. In order to overcome this limitation, Figure 2 and Figure 3 As shown, this embodiment uses a residual convolutional neural network as the generator network, and realizes the fusion of shallow and deep features at the same scale through skip links, so that the training error is greatly reduced and converges to a small value. At the same time, since the original ultrasound images collected still have the problem of low resolution, this embodiment also adds an upsampling module to the generator network to achieve large-scale reconstructed image output. Specifically,

[0035] Figure 3 It is a schematic diagram of the structure of the generator network in an embodiment of the present invention.

[0036] like Figure 3As shown, the generator network of this embodiment is mainly composed of 6 residual modules and 2 upsampling modules. Each residual module contains 2 convolutional layers and their corresponding batch normalization layers and activation functions for residual learning. The two upsampling modules are composed of transposed convolutional layers, batch normalization layers and activation functions to implement upsampling operations.

[0037] The discriminator network adopts the PatchGAN structure to provide adversarial loss for network training.

[0038] Step S2, obtaining a low-resolution original ultrasound image as a network input, and obtaining a corresponding high-resolution ultrasound localization microscopic image as a training label, thereby constructing a training set and a test set respectively.

[0039] Figure 4 Schematic diagram of an ultrasonic image model used in a simulation experiment in an embodiment of the present invention.

[0040] In this embodiment, based on Figure 4 The ultrasound image model shown in the figure combines the ultrasound point spread function to simulate ultrasound imaging. The process is as follows:

[0041] Step S2-1, based on ultrasound contrast agent Figure 4 In the ultrasound image model shown, 30 positions are randomly selected in the structural model as the real positions of the microbubbles, and the impulse function at the real positions of the microbubbles is convolved with the point spread function corresponding to the ultrasound system to simulate the ultrasound imaging process. The process is repeated 500 times to obtain a 500-frame ultrasound original image sequence.

[0042] Step S2-2, randomly selecting 20 consecutive frames from the 500-frame original ultrasound image sequence for superposition, and then normalizing to obtain a low-resolution original ultrasound image, the size of which is 512*512 pixels.

[0043] Figure 5 Schematic diagram of ultrasound positioning microscopic image in an embodiment of the present invention.

[0044] Step S2-3, single-molecule localization is performed on each frame of the original ultrasound image, that is, the independent microbubbles in each frame of the image are centrally located by Gaussian fitting method and center of gravity method to obtain the localization frame corresponding to the microbubble, 95% of the localization frames are superimposed, and after normalization, four times upsampling is performed to obtain an ultrasound localization microscopic image as a training label.

[0045] In this embodiment, all original ultrasound images are subjected to single-molecule localization by Gaussian fitting method to obtain 500 single-molecule localization frames, and 475 frames are randomly selected and superimposed to obtain densely localized super-resolution ultrasound localization microscopic images (such as Figure 5 The image size is 2048*2048 pixels.

[0046] Step S3, input the training set into the initial imaging model, use the loss function and training labels to train and optimize the model until the training is completed to obtain a fast super-resolution imaging model, and use the test set for testing.

[0047] In this embodiment, in order to simulate the influence of noise in actual experiments, background noise is added to the original ultrasound images in the training set one by one.

[0048] In the iterative optimization training process, the loss function has a great influence on network training, and its minimization direction is the method for optimizing model performance. L2 loss is widely used in image processing tasks, but this type of pixel-level loss, which only aims to improve the peak signal-to-noise ratio (PSNR), often makes the output image too blurry and smooth, and performs poorly in super-resolution reconstruction tasks.

[0049] In order to improve the output image quality, this embodiment combines three loss functions, namely, multi-scale structural similarity loss (MS-SSIM), L1 regularization loss, and generative adversarial loss, to jointly train the generator network, namely:

[0050] L G =α×L1(G(LR l ), HR L )+β×[1-L SSIM (G(LR l ), HR L )]+-logD(G(LR l ), LR l )

[0051] In the formula, L1(·) represents L1 loss, G(·) represents the generator network, and LR l Represents the original ultrasound image obtained by superimposing 1 frame of images, HR L represents the ultrasound positioning microscopic image obtained by superimposing L frame positioning frames, L SSIM (·) represents the multi-scale structural similarity loss, D(·) represents the discriminator network, α and β represent the coefficients of L1 loss and structural similarity loss. In this embodiment, α and β are set to 16 and 84 respectively to weigh the weight of each loss function.

[0052] The loss function of the discriminator network uses the generative adversarial loss function:

[0053] L D = -logD(HR L , L.R. l )-log(1-D(G(LR l ), LR l )).

[0054] In this embodiment, the Adam optimized gradient descent algorithm is used to train 1500 original ultrasound images and corresponding ultrasound localization microscopic images for a total of 500 cycles, that is, the training set is traversed 50 times, wherein the batch size N of the traversed data in each cycle is 4, and the initial learning rate is 1×10-5. In order to make the loss function converge to the minimum at the fastest speed, the present invention uses a learning rate decay strategy to gradually reduce the learning rate to 0, thereby gradually approaching the optimal parameters.

[0055] Step S4: input the original ultrasound image to be measured into a fast super-resolution imaging model for fast ultrasound positioning, thereby obtaining a corresponding ultra-high resolution ultrasound image.

[0056] Figure 6 It is a schematic diagram of a mean image generated by superimposing a small number of original ultrasound image frames and an ultrasound localization microscopic imaging result obtained based on a fast super-resolution imaging model in an embodiment of the present invention.

[0057] Depend on Figure 6 It can be seen from the ultra-high resolution ultrasound images that the method proposed in this embodiment can effectively improve the spatial resolution of ultrasound imaging and realize ultrasound localization microscopy; at the same time, the method can greatly reduce the number of original image frames required for ultrasound localization microscopy, reduce the time required for image acquisition, and avoid computational redundancy caused by single-molecule localization. Therefore, based on this method, the temporal resolution of imaging can also be effectively improved, rapid imaging can be achieved, and the application of ultrasound imaging technology in in vivo experiments becomes possible.

[0058] Example Function and Effect

[0059] According to the fast ultrasound localization microscopy method based on a generative adversarial network provided in this embodiment, a fast super-resolution ultrasound imaging model is constructed and trained based on a generative adversarial network. By inputting the low-resolution original ultrasound image into the fast super-resolution ultrasound imaging model for reconstruction, the corresponding imaging quality is comparable to that of a fully positioned ultra-high resolution ultrasound image, thereby realizing fast ultrasound localization microscopy.

[0060] In the embodiment, since the generative adversarial network uses Pix2pix as the basic framework, the end-to-end supervised learning task of this embodiment has a pixel-level advantage over the existing deep learning network. Also, since the generator network of this embodiment uses a residual convolutional neural network to replace the U-Net network of the original Pix2pix framework, the fusion of shallow and deep features is achieved at the same scale through jump links. While greatly reducing the training error, it also overcomes the defect of scattered feature loss caused by multiple pooling of high-dimensional information extracted by the U-Net network. At the same time, considering the low resolution of the original ultrasound image collected, an upsampling module is also added to the generator network in the embodiment to achieve the output of large-size reconstructed images.

[0061] In addition, different from the L2 regularized loss function used in the existing model training, the embodiment adopts a combination of L1 regularized loss function, multi-scale structural similarity function and generative adversarial loss to jointly train the model, thereby avoiding the problem that the reconstructed image is too smooth and blurred and cannot retain the image detail information well due to the L2 regularized loss function, so that the trained imaging model is more suitable for ultrasonic positioning microscopy imaging tasks.

[0062] In summary, the fast ultrasound localization microscopy imaging method based on generative adversarial networks in this embodiment does not require any additional calculations or manual parameter adjustments, reduces the complexity of calculations in the imaging process, avoids parameter dependence, and greatly improves the performance of existing ultrasound localization microscopy imaging technology.

[0063] The above embodiments are only used to illustrate specific implementation modes of the present invention, and the present invention is not limited to the description scope of the above embodiments.

Claims

1. A fast ultrasound localization microscopic imaging method based on a generative adversarial network, which is used to perform fast ultrasound localization on an original ultrasound image to obtain a super-resolution ultrasound image, characterized in that: include: Step S1, constructing an initial imaging model based on a generative adversarial network; Step S2, obtaining a low-resolution original ultrasound image as a network input, and obtaining a corresponding high-resolution ultrasound localization microscopic image as a training label, thereby constructing a training set and a test set respectively; Step S3, inputting the training set into the initial imaging model, using the loss function and the training labels to train and optimize the model until the training is completed, thereby obtaining a fast super-resolution imaging model; Step S4, inputting the original ultrasound image to be measured into the fast super-resolution imaging model for fast ultrasound positioning, thereby obtaining a corresponding super-resolution ultrasound image. The generative adversarial network uses Pix2pix as the basic framework, a residual convolutional neural network as the generator network, and a PatchGAN structure as the discriminator network. The discriminator network is used to provide generative adversarial loss for model training. The generator network includes 6 residual modules for residual learning and 2 upsampling modules for implementing upsampling operations. Each of the residual modules includes 2 convolutional layers and corresponding batch normalization layers and activation functions. Each upsampling module consists of a transposed convolutional layer, a batch normalization layer and an activation function. The acquisition process of the original ultrasound image is as follows: performing ultrasound imaging on the imaging object based on the ultrasound contrast agent to obtain a set of ultrasound original image sequences, wherein each frame of the ultrasound original image contains a plurality of sparsely distributed microbubbles randomly distributed in the imaging area; randomly selecting 20 consecutive frames from the set of ultrasound original image sequences for superposition, and then obtaining a low-resolution original ultrasound image after normalization, wherein the original ultrasound image is 512*512 pixels, The training label construction process is as follows: single-molecule localization is performed on each frame of the original ultrasound image, that is, the independent microbubbles in each frame of the image are centrally located by Gaussian fitting method and centroid method to obtain the localization frame corresponding to the microbubble, 95% of the localization frames are superimposed, and after normalization and upsampling by a predetermined multiple, an ultrasound localization microscopic image is obtained as the training label, and the ultrasound localization microscopic image is 2048*2048 pixels. In the training of the initial imaging model, the loss function of the generator network training consists of multi-scale structural similarity loss, L1 regularization loss and generative adversarial loss: L G =α×L1(G(LR l ),HR L )+β×[1-L SSIM (G(LR l ),HR L )]+-logD(G(LR l ),LR l ) In the formula, L1(·) represents L1 loss, G(·) represents the generator network, and LR l represents the original ultrasound image obtained by superimposing l frames of images, HR L represents the ultrasound localization microscopic image obtained by superimposing L frame localization frames, L SSIM (·) represents the multi-scale structural similarity loss, D(·) represents the discriminator network, α and β represent the coefficients of L1 regularization loss and multi-scale structural similarity loss, The loss function of the discriminator adopts the generation of adversarial loss: L D =-logD(HR L ,LR l )-log(1-D(G(LR l ),LR l ))。 2. The method for rapid ultrasound localization microscopy based on generative adversarial networks according to claim 1, characterized in that: in, The optimization is to use the Adam optimization gradient descent algorithm to train the original ultrasound image and the corresponding ultrasound localization microscopy image for 50 cycles, that is, to traverse the training set 50 times. The data batch size N of each cycle is 4, and the initial learning rate is 1×10 -5 .

3. The method for rapid ultrasound localization microscopy based on generative adversarial networks according to claim 2, characterized in that: in, In the optimization process, a learning rate decay strategy is adopted to gradually reduce the learning rate to 0, so as to quickly approach the optimal parameters and make the loss function converge to the minimum at the fastest speed.