A deep learning based fast super-resolution ultrasound imaging method
By combining deep learning's generative adversarial networks and CNN+Transformer networks, the problems of microvascular resolution and speed in ultrasound imaging were solved, achieving efficient super-resolution microvascular imaging and improving image quality and speed.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ANHUI MEDICAL UNIV
- Filing Date
- 2025-08-21
- Publication Date
- 2026-06-05
AI Technical Summary
Existing ultrasound imaging techniques have limited resolution in microvascular imaging, and the imaging process is complex, making it difficult to clearly distinguish tiny targets and improve imaging speed.
We employ a deep learning-based super-resolution generative adversarial network and CNN+Transformer network architecture, combined with ultrasound localization microscopy technology. We obtain microbubble trajectories through Gaussian fitting and tracking algorithms, improve image resolution through adversarial training of the generator and discriminator, and achieve end-to-end super-resolution reconstruction through CNN+Transformer network.
It significantly improves the spatial resolution and imaging speed of microvascular images, enhances the realism and detail retention of images, and simplifies the imaging process.
Smart Images
Figure CN122155945A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of super-resolution ultrasound imaging, specifically to a rapid super-resolution ultrasound imaging method based on deep learning. Background Technology
[0002] Ultrasound imaging technology has been widely adopted due to its excellent temporal resolution (tens of frames per second) and its advantages such as low cost, real-time operation, and no radiation. Its imaging principle involves using an ultrasound probe to emit ultrasound waves into the human body, and then obtaining the image information needed for clinical diagnosis based on the echo signals generated by the reflection of these ultrasound signals in human tissues. However, its drawbacks are also obvious. From the perspective of the imaging principle, the diffraction effect of ultrasound waves limits its resolving power, making it impossible to clearly distinguish tiny targets smaller than half a wavelength. Therefore, ultrasound imaging has limitations in spatial resolution.
[0003] Deep learning has demonstrated significant advantages in image super-resolution reconstruction in recent years, especially generative adversarial networks (GANs) and Transformer networks, which have unique value in image detail recovery and modeling sequence dependencies. However, existing research rarely combines these two technologies organically for super-resolution imaging of microvessels to improve imaging resolution and generation speed.
[0004] Ultrasound localization microscopy (ULM) is a cutting-edge imaging technology that breaks through the acoustic diffraction limit and achieves subwavelength imaging of microvessels. In ultrasound imaging based on ULM, an ultrasound contrast agent (such as microbubbles) is first injected into the experimental subject. Hundreds to thousands of frames per second (ultra-high-speed imaging) are acquired using ultrafast ultrasound technology to capture the rapid movement of microbubbles within the blood vessels. The position of individual contrast agents is obtained through a localization algorithm, and these images are then superimposed to form a high-resolution image. During in vivo measurements, motion correction can be used to avoid motion artifacts. This allows us to obtain the desired vascular structure map, and a tracking algorithm can then be used to obtain the direction and velocity of blood flow within the vessels.
[0005] The spatial resolution and structural reconstruction quality of current ULM images are still limited by factors such as image acquisition density, microbubble localization accuracy, and reconstruction algorithm performance. These limitations hinder their ability to fully reveal the integrity and complexity of microvascular networks. Furthermore, the entire ULM processing procedure is overly cumbersome and complex, with each step affecting the final image quality to varying degrees. For example, excessively high microbubble concentrations lead to inaccurate localization and low resolution, while excessively low concentrations require more time to collect a sufficient number of samples. Summary of the Invention
[0006] The purpose of this invention is to provide a rapid super-resolution ultrasound imaging method based on deep learning to solve the problems mentioned in the background art.
[0007] To achieve the above objectives, the present invention provides the following technical solution:
[0008] A fast super-resolution ultrasound imaging method based on deep learning includes the following steps:
[0009] Step S1: Obtain microvascular images using an ultrasound-guided microscope, as detailed below:
[0010] Step S101, Acquisition of ultrasound contrast agent data: After the contrast agent is injected into the experimental body, the ultrasound positioning microscope uses ultrafast ultrasound technology to acquire hundreds to thousands of images per second, thereby capturing the trajectory of the microbubbles moving rapidly in the blood vessels.
[0011] Step S102: First, a positioning algorithm is used to fit a Gaussian function to each microbubble in each frame of the image to determine the center position. The positioning accuracy of this step can reach the subwavelength level (e.g., tens of micrometers), realizing accurate estimation of microbubble coordinates. The microbubble moves along the blood flow. By considering the time correlation of the microbubble movement, the trajectory that gives the vascular structure view can be found through the tracking algorithm. In addition, the tracking provides velocity information, that is, the direction and velocity of the blood flow.
[0012] Step S103 involves quantitatively analyzing the spatial distribution density and flow velocity of the microbubble contrast agent in vivo, based on localization and tracking, to create a visual representation of the process.
[0013] Step S2: Construct a super-resolution generative adversarial network to enhance the resolution of microvascular images, as detailed below:
[0014] Step 201: The super-resolution generative adversarial network architecture consists of two core components: a generator and a discriminator. High-quality image generation is achieved through adversarial training.
[0015] The generator network uses the Parametric ReLU activation function and includes a B residual module, two 3×3 convolutional kernels and a convolutional layer with 64 feature maps, a batch normalization layer, and a subpixel convolutional layer with 3×3 kernels and 256 feature maps.
[0016] The discriminator network consists of 8 convolutional layers, using the LeakyReLU activation function (α = 0.2), 3×3 convolutional kernels, and the number of features increases sequentially from 64 to 512. Finally, it is connected by two fully connected layers and a sigmoid activation function.
[0017] Step 202: The network training uses the Adam optimizer with a batch size of 16 and a learning rate of 0.00001. It is implemented based on the PyTorch framework, and the hardware configuration is a 64-bit operating system, an Intel Core i5-10200H 2.40GHz CPU and an NVIDIA GeForce GTX 1650 graphics card. The training is carried out for 500 epochs until the loss function converges. The datasets used for network training and testing are computer-simulated microvascular images and rat ultrasound microvascular images.
[0018] Step 203: The ultrasound positioning microscope microvascular image uses an innovative perceptual loss function, which is composed of a weighted sum of content loss and adversarial loss. The content loss is calculated using an improved mean square error, taking into account the upsampling factor r and image size normalization. The adversarial loss is calculated by the logarithm of the probability that the generated image is judged as a real image, which significantly improves the realism and detail preservation of the image reconstruction.
[0019] The perceptual loss function is shown in the following formula:
[0020]
[0021] In equation (1), For content loss function, To counteract the loss function, the weighted sum of the two represents the perceptual loss function.
[0022] Content loss MSE is calculated as follows:
[0023]
[0024] In equation (2), The calculation focuses on the mean squared error loss for the super-resolution task, where r represents the upsampling factor, W and H represent the width and height of the low-resolution input image, and rW and rH represent the width and height of the generated high-resolution image. I represents the pixel value at position (x, y) of a true high-resolution image. LR This represents the input low-resolution image. This represents the pixel value at position (x, y) of the high-resolution image output by the generator. This represents the normalization coefficient.
[0025] The countermeasure loss is calculated as follows:
[0026]
[0027] In equation (3), The calculation is the adversarial loss value of the generator. I is the probability that the reconstructed image is a natural HR image. LRRepresents the input low-resolution image.
[0028] Step 204, using the ultrasonic localization microscope microvascular image as the input image, and using the pre-trained super-resolution generative adversarial network to generate an image with a clearer resolution.
[0029] Step S3, build a CNN+Transformer network architecture, use the high-resolution image output by the super-resolution generative adversarial network as the label, and use the B-mode ultrasound microbubble image sequence as the input to train the network to quickly generate super-resolution microvascular images.
[0030] Step S301, first construct two convolutional layers, each followed by a ReLU activation function. The first layer uses a configuration of 3 input channels, 3×3 convolutional kernel and 64 output channels to extract low-frequency features from the low-resolution input image; the second layer is used after being processed by the linear long-distance attention module, and is configured with 64 input channels, 3×3 convolutional kernel and 64 output channels to eliminate redundant low-frequency features.
[0031] Step S302, deploy a linear long-distance attention module, each module contains two sub-modules: local feature extraction and group-by-group multi-scale processing, and both use residual connections.
[0032] Step S3021, local feature extraction is achieved through two shifted convolutional layers, with a ReLU activation inserted in the middle. Specifically, the input features are evenly divided into five groups. The first four groups are shifted in the left, right, up, and down directions respectively and then processed by 1×1 convolution, and the last group retains the original features.
[0033] Step S3022, the group-by-group multi-scale module first performs a preprocessing process on the input features. The number of channels, length, and width of the input feature map (X∈R C×H×W ), are divided into K groups by channel: The number of channels in each group is C / K. At the same time, each group of features is processed independently, and the window size M k is different, and then the SA of the Kth group of features is calculated according to the window size. The calculation cost of the SA of the K groups is approximately Finally, the SAs calculated for different groups are connected.
[0034] Step S3023, implement linear self-attention in the grouped multi-scale self-attention module. First, flatten each group of features X k into a 2D sequence, that is, X k ∈R (H×W)×(C / K) , the sequence length n = H×W, the feature dimension d = C / K, and two groups of low-rank projection matrices are introduced using the method of linear projection. One projects the key to k dimensions (k << n); the other projects the value to k dimensions, and finally calculates the linear attention.
[0035] Step S303: A switchable parallel upsampling module is used to generate a high-resolution image. The parameters of the parallel upsampling module are fixed at 64 input channels, a 3×3 filter size, and 64 output channels. The unknown model can be trained by simultaneously executing three different scales (i.e., ×2, ×3, and ×4) through switching. This module mainly maps the obtained low-frequency features to high-frequency features. Finally, a signal convolution layer is performed to obtain a high-quality predicted image through the obtained high-frequency features. The number of input and output channels is 3, and the filter size is 3×3.
[0036] Step S304: The system is trained using computer simulation data of ultrasound positioning microscope and 3000 images from a publicly available experimental dataset. The Adam optimizer is configured (learning rate 0.0001), and the system is run for 500 epochs on an NVIDIA GTX 1650 graphics card. The super-resolution generative adversarial network-enhanced images are used as supervision labels, and the B-mode ultrasound microbubble image sequence is used as input to achieve end-to-end super-resolution reconstruction.
[0037] Compared with the prior art, the beneficial effects of the present invention are:
[0038] 1. This invention uses a super-resolution generative adversarial network to significantly improve the resolution of microvascular images from ULM, which is much greater than the resolution improvement of a typical convolutional neural network.
[0039] 2. This invention uses images with increased resolution from the SRGAN network as training labels and quickly forms the entire image, thereby improving the imaging speed of the entire process while ensuring image quality.
[0040] 3. The CNN+Transformer structure used in this invention is based on a new self-attention mechanism. Compared with the traditional ULM imaging algorithm, it can effectively improve the spatial resolution of ultrasound imaging, achieve super-resolution ultrasound imaging, and accelerate the imaging speed. Attached Figure Description
[0041] Figure 1 This is a diagram of the dual-stage super-resolution imaging architecture of the present invention; Figure 2 This is a flowchart of the fast super-resolution imaging process of the present invention; Figure 3 This is a diagram of the SRGAN network generator architecture of the present invention; Figure 4 This is a diagram of the SRGAN network discriminator architecture of the present invention; Figure 5 This is a diagram of the CNN+Transform network architecture of the present invention; Figure 6This is a diagram illustrating the imaging effect of the present invention. Detailed Implementation
[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0043] Example 1
[0044] This invention addresses the limitations of spatial resolution and the complexity of the imaging process in ultrasound-guided microscopy by fusing a generative adversarial network (GAN) with a CNN+Transformer hybrid architecture to achieve efficient super-resolution reconstruction of microvascular networks. Due to the involvement of deep learning model deployment and real-time image processing, the following technical objectives require hardware architecture to achieve:
[0045] Subwavelength-level positioning accuracy: By combining Gaussian fitting with deep learning, the positioning accuracy of microbubbles is improved to the tens of micrometer level;
[0046] Multi-scale feature fusion: Utilizing grouped multi-scale self-attention to capture the local and global dependencies of vascular structures;
[0047] End-to-end accelerated imaging: Using super-resolution generative adversarial network-enhanced images as labels and B-mode ultrasound microbubble image sequences as input, a direct mapping from low-resolution input to super-resolution output is achieved.
[0048] Based on the above technical requirements, this embodiment adopts the following... Figure 1 The "two-stage super-resolution imaging architecture" shown:
[0049] Microbubble trajectory processing module: Composed of an ultrasound probe, a high-speed acquisition card (frame rate ≥1000fps) and a GPU accelerator, it calculates the center coordinates of microbubbles in real time through a Gaussian fitting algorithm and outputs vascular trajectory and blood flow velocity information.
[0050] like Figure 3-4 As shown, the super-resolution generative adversarial network enhancement module uses B residual blocks cascaded subpixel convolutions for the generator and an 8-layer CNN for the discriminator. The perceptual loss function (Equation 1) is used to optimize the realism of the generated image.
[0051] like Figure 5 As shown, the CNN+Transformer reconstruction module includes a linear long-distance attention module (containing shift convolution and grouped multi-scale self-attention modules) and a parallel upsampling layer, which achieves efficient feature extraction through a linear self-attention mechanism.
[0052] like Figure 2 As shown, the fast super-resolution imaging process includes:
[0053] Step S1: Microbubble trajectory acquisition and localization. Microbubble motion sequences are acquired at 2000fps. Gaussian function fitting is applied to each frame of the image to determine the center position of the microbubble, with localization accuracy reaching the subwavelength level (e.g., 50µm).
[0054] Step S2: SRGAN resolution enhancement. Input the original image from the ultrasonic positioning microscope into the generator and optimize it using the perceptual loss function (Equation 1):
[0055]
[0056] The content loss (Equation 2) is calculated as follows:
[0057]
[0058] The adversarial loss (Equation 3) is calculated as follows:
[0059]
[0060] Step S3: End-to-end training of CNN+Transformer. Input the B-mode ultrasound microbubble image sequence into the network. In the LLAB module, GMSA divides the input features into K groups according to channels for independent processing. The computational complexity is:
[0061]
[0062] Efficient attention calculation is achieved through a linear projection matrix.
[0063] Example 2
[0064] like Figure 6 As shown, in the application of cerebral blood flow microvascular imaging:
[0065] Data preparation:
[0066] Using 3000 simulated images and rat microvascular images, with an input resolution of 256×256, SRGAN outputs 512×512 labeled images.
[0067] Implementation of key parameters:
[0068] In the perceptual loss function, the upsampling factor r = 4, the content loss weight is 1, and the adversarial loss weight is 0.001.
[0069] The GMSA module is set to have 8 groups and the window size M_k is set to [4, 8, 16].
[0070] The linear projection matrix has a dimension k = 32 (much smaller than the sequence length n = H × W).
[0071] Effect verification:
[0072] On the rat brain microvessel data test set, the PSNR reached 34.2dB, which is 5.1dB higher than the traditional ULM method;
[0073] The imaging speed has been increased from 30 minutes / frame to 8 minutes / frame.
[0074] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0075] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A fast super-resolution ultrasound imaging method based on deep learning, characterized in that, Includes the following steps: Step S1: Obtain microvascular images using an ultrasound-guided microscope; Step S2: Construct a super-resolution generative adversarial network to enhance the resolution of microvascular images; Step S3: Build a CNN+Transformer network architecture, using high-resolution images output by the super-resolution generative adversarial network as labels and B-mode ultrasound microbubble image sequences as input to train the network and quickly generate super-resolution microvascular images.
2. The fast super-resolution ultrasound imaging method based on deep learning according to claim 1, characterized in that: Step S1, specifically, includes the following steps for acquiring microvascular images: Step S101: Acquire ultrasound contrast agent data and capture the movement trajectory of microbubbles in blood vessels; Step S102: Gaussian function fitting is used to locate the microbubbles, and a vascular structure view is generated by a tracking algorithm; Step S103: Quantitatively analyze the spatial distribution density and flow velocity of microbubbles to generate a visual image.
3. The rapid super-resolution ultrasound imaging method based on deep learning according to claim 1, characterized in that: In step S2, the super-resolution generative adversarial network includes a generator and a discriminator, wherein: The generator uses the Parametric ReLU activation function, B residual modules, and subpixel convolutional layers; The discriminator comprises multiple convolutional layers, a LeakyReLU activation function, and a sigmoid output layer.
4. The fast super-resolution ultrasound imaging method based on deep learning according to claim 1, characterized in that: The super-resolution generative adversarial network was trained using the Adam optimizer with a learning rate of 0.00001, a batch size of 16, and for 500 epochs.
5. The fast super-resolution ultrasound imaging method based on deep learning according to claim 1, characterized in that: The perceptual loss function of the super-resolution generative adversarial network is: in, Represented as a content loss function, This is represented as the adversarial loss function.
6. The fast super-resolution ultrasound imaging method based on deep learning according to claim 1, characterized in that: The CNN+Transformer network architecture includes: Two convolutional layers and a ReLU activation function are used to extract low-frequency features; Multiple linear long-range attention modules are used to extract local and global features; The parallel upsampling module supports multi-scale upsampling (×2, ×3, ×4).
7. The fast super-resolution ultrasound imaging method based on deep learning according to claim 1, characterized in that: The plurality of linear long-range attention modules include: The local feature extraction module employs shifted convolutional layers and the ReLU activation function; Self-attention is calculated by dynamically adjusting the window size of each multi-scale module.
8. The fast super-resolution ultrasound imaging method based on deep learning according to claim 1, characterized in that: The group-by-group multi-scale module uses a low-rank projection matrix to approximate self-attention in order to reduce computational complexity.
9. The fast super-resolution ultrasound imaging method based on deep learning according to claim 1, characterized in that: The dataset used in this method includes computer-simulated microvascular images and rat ultrasound microvascular images.
10. An ultrasound imaging system, characterized in that, include: An ultrasonic probe is used to acquire microbubble data; A processor for performing the method according to any one of claims 1-9; A display used to output super-resolution ultrasound images.