A fast ultrasound super-resolution imaging method, storage medium and processor
By combining bidirectional long short-term memory networks and convolutional neural networks in deep learning, the problems of slow imaging speed and computational complexity in ultrasound localization microscopy have been solved, achieving fast and high-quality dynamic super-resolution imaging suitable for clinical applications.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN INST OF ADVANCED TECH
- Filing Date
- 2023-12-25
- Publication Date
- 2026-05-15
AI Technical Summary
In existing ultrasound-guided microscopic imaging methods, the sparse distribution of microbubbles leads to slow imaging speed, and the imaging performance depends on the reconstruction parameters, resulting in high computational complexity and making it difficult to use flexibly in clinical applications.
A deep learning approach combining bidirectional long short-term memory networks and convolutional neural networks is employed. By learning the motion trajectory of microbubbles and combining singular value decomposition and particle tracking algorithms, dynamic super-resolution videos are rapidly reconstructed, supplemented by traditional super-resolution blood flow images.
It improves imaging speed, reduces data acquisition time, lowers computational burden, and enables high-quality dynamic super-resolution imaging, making it suitable for clinical applications.
Smart Images
Figure CN117788543B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of ultrasound imaging technology, and in particular relates to a rapid ultrasound super-resolution imaging method, storage medium, and processor. Background Technology
[0002] In ultrasound localization microscopy (ULM), the localization method used directly impacts imaging performance. Current ULM imaging methods typically resolve microbubbles in the vascular system using single-emitter localization methods, such as Gaussian fitting and centroid methods. When using these methods, it's necessary to reduce the likelihood of the distance between two microbubbles falling below the resolvable limit; that is, microbubbles in each frame must be sufficiently sparse to allow for the identification of individual centers. Therefore, a low microbubble concentration is usually required, resulting in longer data acquisition times needed to accumulate a sufficient number of microbubble localization frames. This limits the application of ULM in rapid biomedical imaging.
[0003] One approach to improving imaging speed is to use time-dependent techniques, such as super-resolution optical wave imaging (SOFI). Bar-Zion et al. achieved rapid vascular ultrasound imaging with enhanced spatial resolution by utilizing the temporal fluctuations of acoustic signals. However, in SOFI-based ULM methods, spatial resolution is often limited to lower-order cumulants, resulting in limited improvement in spatial resolution compared to single-emitter localization methods. Considering that microbubbles are typically sparsely distributed in each frame compared to the imaging region, a compressed sensing-based localization method (called CS-ULM) has recently been proposed and successfully applied to ULM. In previous work by Luo et al., by applying CS to the US imaging model, it was demonstrated that high-density overlapping microbubbles can be effectively identified. Therefore, it reduces the number of imaging frames required to accumulate sufficient microbubbles, thereby improving data acquisition speed.
[0004] However, a fundamental problem exists when using CS-ULM: imaging performance depends on the reconstruction parameters. Unfortunately, there is currently no clear consensus on how to find the "optimal" parameters, which requires user expertise and increases processing complexity. Furthermore, CS-ULM suffers from a high computational burden due to its complex iterative process, making it inflexible for clinical applications. Existing technologies have inherent problems. Summary of the Invention
[0005] The purpose of this application is to provide a fast ultrasound super-resolution imaging method, storage medium, and processor, aiming to solve the technical problem that the super-resolution imaging speed in the prior art is too slow.
[0006] On the one hand, this application provides a rapid ultrasound super-resolution imaging method, the method comprising the following steps:
[0007] s1. Each image in the image sequence is represented as a two-dimensional array and learned through a bidirectional long short-term memory network to learn the microbubble motion trajectory in a certain number of future frames and a certain number of past frames;
[0008] s2. The bidirectional long short-term memory network outputs the learned feature tensor into the convolutional neural network to extract feature information;
[0009] s3. The convolutional neural network outputs two files expressing the confidence of the predicted structure based on the feature information;
[0010] s4. Binarize each element in the built-in confidence graph of the file, set the parameters with confidence scores less than a set threshold to zero, and set the parameters with confidence scores greater than a set threshold to one;
[0011] s5. Perform uncertainty weighting calculation on each pixel in the binary image according to the confidence level, with the weighting factor being α. The higher the confidence level, the larger the weighting factor α; so that the pixels in the binary image appear as spots through the weighted calculation.
[0012] s6. Multiply the pixels of the blob-like binary image by a two-dimensional Gaussian function with a standard deviation equal to one-half of the original pixel value to complete the trajectory reconstruction.
[0013] s7. The output reconstructed frame sequence is a dynamic super-resolution video.
[0014] Preferably, the steps preceding step s1 include:
[0015] q1. Inject an appropriate concentration of ultrasound contrast agent containing microparticles with scattering properties into the imaging target;
[0016] q2. Acquire ultrasound signals using a linear array ultrafast plane wave composite imaging algorithm;
[0017] q3. Based on the fact that the coherence of tissue signals is greater than that of the microparticles of flowing ultrasound contrast agents, an image sequence reflecting the flow of ultrasound contrast agent microparticles is extracted through singular value decomposition.
[0018] Preferably, the ultrasound contrast agent is one of the following: attenuated bacteria capable of producing gas vesicles in vivo, phagocytic macrophages, and drug-loaded microvesicles.
[0019] Preferably, the acquisition process in step q2 includes the following sequential steps:
[0020] q21. The acquired ultrasonic radio frequency signals are beamformed and then orthogonally demodulated to obtain the orthogonally demodulated signal;
[0021] q22. Perform cross-correlation on the collected data and remove frames with significant changes;
[0022] q23. Take the part with relatively gentle motion as the ultrasonic signal.
[0023] Preferably, after step q3, the following steps are further included:
[0024] q4. Extract the image sequence frame by frame to construct an image sequence in the form of a frame sequence.
[0025] Preferably, the step s7 is followed by the following step:
[0026] s8. By using the nearest neighbor matching algorithm in particle tracking velocimetry, the microbubble coordinates in the dynamic super-resolution video are paired frame by frame to calculate the distance of microbubble motion between adjacent frames, thereby obtaining the microbubble motion velocity, which is the blood flow velocity in the dynamic super-resolution video.
[0027] On the other hand, this application also provides a storage medium storing program files capable of implementing the above-described rapid ultrasound super-resolution imaging method.
[0028] On the other hand, this application also provides a processor for running a program, wherein the program executes the above-described rapid ultrasound super-resolution imaging method during runtime.
[0029] This application proposes a rapid ultrasound super-resolution imaging method that uses microparticles with scattering properties instead of traditional contrast agents. The acquired data is input into a deep learning network for microparticle localization and tracking, and dynamic super-resolution video is quickly generated. This is supplemented by traditional super-resolution blood flow images, which effectively improves the imaging speed of super-resolution imaging. Attached Figure Description
[0030] Figure 1 This is a basic flowchart of the rapid ultrasound super-resolution imaging method provided in Embodiment 1 of this application;
[0031] Figure 2 This is a schematic diagram of the deep learning network structure used in this application;
[0032] Figure 3 This is the image sequence acquired in this application. Detailed Implementation
[0033] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0034] The specific implementation of this application will be described in detail below with reference to specific embodiments:
[0035] Example 1:
[0036] Figure 1 The implementation flow of the rapid ultrasound super-resolution imaging method provided in Embodiment 1 of this application is shown. Figure 2 A deep learning network consisting of a bidirectional long short-term memory network and a convolutional neural network is shown. For ease of explanation, only the parts relevant to the embodiments of this application are shown, and are described in detail below:
[0037] On the one hand, this application provides a rapid ultrasound super-resolution imaging method, the method comprising the following steps:
[0038] s1. Each image in the image sequence is represented as a two-dimensional array and learned through a bidirectional long short-term memory network to learn the microbubble motion trajectory in a certain number of future frames and a certain number of past frames;
[0039] s2. The bidirectional long short-term memory network outputs the learned feature tensor into the convolutional neural network to extract feature information;
[0040] s3. The convolutional neural network outputs two files expressing the confidence of the predicted structure based on the feature information;
[0041] In practical implementation, existing ultrasound super-resolution reconstruction patents all use convolutional neural networks (CNNs) such as U-Net and ResNet residual structures, which cannot take into account the influence of contextual data on the prediction results. The deep learning network in this application is a combination of a recurrent neural network variant and a convolutional neural network. Compared with ordinary networks, the bidirectional long-short-period network can better learn and understand the contextual information in the sequence, enabling high-quality learning with fewer input frames and improving imaging speed. Furthermore, combining the bidirectional long-short-period network with a convolutional neural network enhances the overall feature extraction capability of the network in this patent, making the neural network in this patent more powerful in feature extraction than ordinary recurrent neural networks (RNNs) or long-short-period networks (LSTMs), thus improving the network's learning and prediction capabilities for the input sequence.
[0042] As attached Figure 2 As shown, in the deep learning network structure of this application, the entire image represents the time direction from top to bottom and the image processing direction from left to right. The overall process is as follows: the low-resolution image is processed by SVD and then fed into the deep learning neural network. First, it enters a bidirectional long and short periodic network LSTM to obtain the relationship between contexts, then it enters a convolutional neural network CNN to extract feature values, and finally outputs a super-resolution image sequence, i.e., dynamic super-resolution video.
[0043] Specifically, low-resolution data is processed through singular value decomposition to obtain a localization map. This localization map is then processed by a bidirectional long short-period network (LSTM), where each cell is a processing module that learns and understands information from its preceding and following cells. This effectively avoids the inability to consider the influence of contextual data on the prediction results. After the data is derived from the LSTM, it is then processed by a convolutional neural network (CNN) to extract feature values for reconstruction, resulting in a super-resolution image sequence.
[0044] s4. Binarize each element in the built-in confidence graph of the file, set the parameters with confidence scores less than a set threshold to zero, and set the parameters with confidence scores greater than a set threshold to one;
[0045] s5. Perform uncertainty weighting calculation on each pixel in the binary image according to the confidence level, with the weighting factor being α. The higher the confidence level, the larger the weighting factor α; so that the pixels in the binary image appear as spots through the weighted calculation.
[0046] s6. Multiply the pixels of the blob-like binary image by a two-dimensional Gaussian function with a standard deviation equal to one-half of the original pixel value to complete the trajectory reconstruction.
[0047] s7. The output reconstructed frame sequence is a dynamic super-resolution video.
[0048] The deep learning network in this application adopts a sequence-to-sequence input-output structure, which enables it to generate reconstructed videos of dynamic processes. This provides an additional way to observe vascular structures through dynamic video compared to other super-resolution reconstruction methods, which is beneficial for assisting medical personnel in making better medical diagnoses of target blood vessels.
[0049] In practical implementation, the above algorithm is specifically expressed as follows:
[0050]
[0051]
[0052] Where B is a binary array, and x and y represent the x and y coordinates of a pixel, respectively. i ,y i ) represents the coordinate position of the i-th data point. I(x) i ,y i ) is (x i ,y i The confidence level value for the position. `threshold` is the set threshold. `α` is the weighting factor, and `e` is the natural constant.
[0053] This calculation can reduce pixel intensity where network confidence is low, while maintaining high pixel intensity in other areas.
[0054] The smaller the confidence level I(x,y), the more pixel values in B(x,y) equal to 0, and the smaller the uncertainty weighting factor. The first two parameters cause the output pixel values to decrease. In the Gaussian function part, the confidence level I(x,y) is the standard deviation, which affects the pixel distribution of the processed image. The smaller the standard deviation, the smaller the width of the function, and the sharper the shape of the function becomes, which can be regarded as each pixel shrinking towards the center. In summary, this calculation can reduce pixel intensity in areas with low network confidence while maintaining high pixel intensity in other areas.
[0055] The output reconstructed frame sequence is a dynamic super-resolution video, from which hemodynamic parameters can be obtained. Finally, by superimposing a small number of reconstructed frames, a complete super-resolution reconstructed image can be obtained to achieve fast super-resolution assisted imaging.
[0056] Preferably, the steps preceding step s1 include:
[0057] q1. Inject an appropriate concentration of ultrasound contrast agent containing microparticles with scattering properties into the imaging target;
[0058] q2. Acquire ultrasound signals using a linear array ultrafast plane wave composite imaging algorithm;
[0059] q3. Based on the fact that the coherence of tissue signals is greater than that of the microparticles of flowing ultrasound contrast agents, an image sequence reflecting the flow of ultrasound contrast agent microparticles is extracted through singular value decomposition.
[0060] Preferably, the ultrasound contrast agent is one of the following: attenuated bacteria capable of producing gas vesicles in vivo, phagocytic macrophages, and drug-loaded microvesicles.
[0061] Preferably, the acquisition process in step q2 includes the following sequential steps:
[0062] q21. The acquired ultrasonic radio frequency signals are beamformed and then orthogonally demodulated to obtain the orthogonally demodulated signal;
[0063] q22. Perform cross-correlation on the collected data and remove frames with significant changes;
[0064] q23. Take the part with relatively gentle motion as the ultrasonic signal.
[0065] Preferably, after step q3, the following steps are further included:
[0066] q4. Extract the image sequence frame by frame to construct an image sequence in the form of a frame sequence.
[0067] Preferably, the step s7 is followed by the following step:
[0068] s8. By using the nearest neighbor matching algorithm in particle tracking velocimetry, the microbubble coordinates in the dynamic super-resolution video are paired frame by frame to calculate the distance of microbubble motion between adjacent frames, thereby obtaining the microbubble motion velocity, which is the blood flow velocity in the dynamic super-resolution video.
[0069] Specifically, the output reconstructed frame sequence is a dynamic super-resolution video from which hemodynamic parameters can be obtained. Finally, by superimposing a small number of reconstructed frames, a complete super-resolution reconstructed image can be obtained to achieve fast super-resolution assisted imaging.
[0070] For hemodynamic parameters extracted from dynamic super-resolution videos, blood flow velocity is calculated by measuring the distance traveled dynamically per unit time through the dynamic trajectory video of microbubbles. Furthermore, the wider the blood flow in the image, the wider the surface blood vessels, thereby understanding the degree of vascular stenosis and judging blood flow.
[0071] Example 2:
[0072] On the other hand, this application also provides a storage medium storing program files capable of implementing the above-described rapid ultrasound super-resolution imaging method.
[0073] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as ROM / RAM, disk, optical disk, etc.
[0074] Example 3:
[0075] On the other hand, this application also provides a processor for running a program, wherein the program executes the above-described rapid ultrasound super-resolution imaging method during runtime.
[0076] In the embodiments of this application, the rapid ultrasound super-resolution imaging method can be implemented by corresponding hardware or software units. Each unit can be an independent hardware or software unit, or it can be integrated into a single hardware or software unit, which is not intended to limit this application. Specific implementation methods for each unit can be found in the description of Embodiment 1, and will not be repeated here.
[0077] Example 4:
[0078] On the other hand, combined with the appendix Figure 1-3 The present invention will be further described below.
[0079] In practical implementation, the experimental setup of this application includes a programmable ultrasound imaging platform (Verasonics Vantage 256, Kirkland, WA, USA) and a linear array transducer L9-4 for ultrasound data acquisition. The ultrasonic linear array has 128 elements, a transducer center frequency of 6.5 MHz, a bandwidth of 4-9 MHz, transmits one signal cycle at a time, and has a mechanical index of 0.1. Ultrafast composite plane wave imaging is used, with the plane wave deflection angle divided into five equal angles from -3 degrees to 3 degrees, and an effective pulse repetition rate (PRF) of 800 Hz. A total of 1000 frames of ultrasound image sequences were acquired for each profile. After data acquisition, the original RF data was beamformed into IQ data using the DAS beamforming method, with the beamforming data grid size set to half a wavelength. Finally, the IQ data was saved and post-processed using MATLAB (2020a, MathWorks, Natick, MA, USA). The acquired image sequences are shown below. Figure 3 As shown.
[0080] The animals used were healthy SD rats (approximately 250g). Before the experiment, the rats were anesthetized with 2% isoflurane oxygen. After deep anesthesia, they were placed on a heating pad to maintain a body temperature of 37 degrees Celsius. The leg hair was removed using an animal hair removal instrument and hair removal cream. Then, the tumor tissue suspension was injected subcutaneously. After slowly withdrawing the needle, pressure was applied for two minutes. After the animal woke up, it was placed in a cage.
[0081] The acquired ultrasound radio frequency (RF) signals are beamformed and orthogonally demodulated to obtain orthogonal demodulated signals (IQ signals). The acquired data are cross-correlated, and frames with large variations are removed, retaining only the more stable components. Subsequently, singular value decomposition is performed on the IQ signals to separate tissue signals from microbubble signals flowing with blood flow. The raw ultrasound IQ signal can be represented as:
[0082] S = C(n) x ,n z ,n t )+B(n x ,n z ,n t )+N(n x ,n z ,n t );
[0083] Where C represents the tissue signal, B represents the microbubble signal, N represents the noise signal, x represents the lateral dimension (along the transducer array), z represents the imaging depth along the transducer axis, and t represents the acquisition time. The original data matrix S(n x ,n z ,n t ) is transformed into a two-dimensional spatiotemporal matrix form S(n) by time series transformation in the form of a Cascorati matrix rearranged.x ×n z ,n t Then S(n) x ×n z ,n t Singular value decomposition yields:
[0084] S=UΔV * ;
[0085] Where Δ is a form of (n) x ×n z ,n t The matrix is an off-diagonal matrix, where U and V are orthogonal matrices with their own dimensions (n, n). x ×n z ,n x ×n z ) and (n t ,n t ), * represents the conjugate transpose. The columns of matrices U and V correspond to the spatial (left) and temporal (right) singular vectors of matrix S, respectively. By setting a threshold to remove points with large singular values, tissue and microbubble signals can be separated.
[0086] After interpolating the microbubble images of each frame, the point spread function (PSF) of the ultrasound system is normalized and cross-correlated with the microbubble signal in two dimensions to obtain the cross-correlation coefficient between the image and the PSF. Then, unnecessary noise is removed by setting an appropriate threshold.
[0087]
[0088] Where f represents the microbubble image and t represents the PSF template image. and represents the mean of the microbubble image and the PSF template image, respectively, and c represents the normalized two-dimensional cross-correlation coefficient. By setting an appropriate threshold to remove unnecessary noise and extracting the spatial coordinates of the centroids of the cross-correlated images one by one, individual microbubbles can be located. Finally, the microbubble positions of all acquired frames are summed to obtain the final super-resolution image.
[0089] Finally, the nearest neighbor matching algorithm (Kuhn-Munkras matching algorithm) in particle tracking velocimetry (PTV) is used to pair the microbubble coordinates frame by frame. This allows the calculation of the distance of microbubble movement between adjacent frames, thus obtaining the microbubble movement velocity, which is the blood flow velocity in the imaging body.
[0090] Meanwhile, this application utilizes deep learning for rapid super-resolution assisted imaging. The network part employs a variant of recurrent neural network (RNN)—bidirectional long short-term memory network (LSTM) and convolutional neural network (CNN)—to perform rapid super-resolution imaging of microbubble signals after singular value decomposition (SVD) while maintaining a certain spatial resolution.
[0091] In summary, compared to traditional ultrasound super-resolution methods, achieving seamless tracking and reconstruction with higher accuracy often requires maintaining a high sampling rate and capturing a large number of sample frames, resulting in long reconstruction times. The fast ultrasound super-resolution imaging method proposed in this application, based on deep learning, can predict microbubble motion trajectories through neural networks, enabling super-resolution reconstruction with fewer sample frames and a lower sampling rate. This results in shorter reconstruction times, and hardware acceleration can further improve reconstruction speed.
[0092] Meanwhile, traditional super-resolution techniques must consider the possibility of trajectory breakpoints, trajectory offsets, and motion blur caused by image drift during microbubble trajectory reconstruction. However, the deep learning-based super-resolution method in this application, due to the high generalization ability and stability of the trained model, demonstrates excellent tracking capability for microbubbles whose overall structure remains within the localization map, and is almost unaffected by image drift or rotation.
[0093] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A rapid ultrasound super-resolution imaging method, characterized in that, The method includes the following steps: s1. Each image in the image sequence is represented as a two-dimensional array and learned through a bidirectional long short-term memory network to learn the microbubble motion trajectory in a certain number of future frames and a certain number of past frames; s2. The bidirectional long short-term memory network outputs the learned feature tensor into the convolutional neural network to extract feature information; s3. The convolutional neural network outputs two files expressing the confidence of the predicted structure based on the feature information; s4. Binarize each element in the built-in confidence graph of the file, set the parameters with confidence scores less than a set threshold to zero, and set the parameters with confidence scores greater than a set threshold to one; s5. Perform uncertainty weighting calculation on each pixel in the binary image according to the confidence level, with the weighting factor being α. The higher the confidence level, the larger the weighting factor α; so that the pixels in the binary image appear as spots through the weighted calculation. s6. Multiply the pixels of the blob-like binary image by a two-dimensional Gaussian function with a standard deviation equal to one-half of the original pixel value to complete the trajectory reconstruction. s7. The output reconstructed frame sequence is a dynamic super-resolution video.
2. The method as described in claim 1, characterized in that, The steps preceding step s1 include: q1. Inject an appropriate concentration of ultrasound contrast agent containing microparticles with scattering properties into the imaging target; q2. Acquire ultrasound signals using a linear array ultrafast plane wave composite imaging algorithm; q3. Based on the fact that the coherence of tissue signals is greater than that of the microparticles of flowing ultrasound contrast agents, an image sequence reflecting the flow of ultrasound contrast agent microparticles is extracted through singular value decomposition.
3. The method as described in claim 2, characterized in that, The ultrasound contrast agent is one of the following: attenuated bacteria capable of producing gas vesicles in vivo, phagocytic macrophages, and drug-loaded microvesicles.
4. The method as described in claim 1, characterized in that, The data acquisition process in step q2 includes the following sequential steps: q21. The acquired ultrasonic radio frequency signals are beamformed and then orthogonally demodulated to obtain the orthogonally demodulated signal; q22. Perform cross-correlation on the collected data and remove frames with significant changes; q23. Take the part with relatively gentle motion as the ultrasonic signal.
5. The method as described in claim 4, characterized in that, Following step q3, the following steps are also included: q4. Extract the image sequence frame by frame to construct an image sequence in the form of a frame sequence.
6. The method as described in claim 5, characterized in that, The following steps are included after step s7: s8. By using the nearest neighbor matching algorithm in particle tracking velocimetry, the microbubble coordinates in the dynamic super-resolution video are paired frame by frame to calculate the distance of microbubble motion between adjacent frames, thereby obtaining the microbubble motion velocity, which is the blood flow velocity in the dynamic super-resolution video.
7. A storage medium, characterized in that, The storage medium stores program files capable of implementing the rapid ultrasound super-resolution imaging method according to any one of claims 1 to 6.
8. A processor, characterized in that, The processor is used to run a program, wherein the program executes the rapid ultrasound super-resolution imaging method according to any one of claims 1 to 6.