Method and system for arbitrary scale cardiac cine MIR super-resolution reconstruction

Abstract

The application belongs to the technical field of medical image processing. A method and system for arbitrary scale cardiac cine MIR super-resolution reconstruction are provided. The diffusion prior feature and the key target frame feature are used as the input of spatial perception implicit attention. The output of the diffusion prior feature, the key target frame feature and the spatial perception implicit attention is used as the input of time perception implicit attention. The output of the time perception implicit attention is processed by a decoder to obtain a super-resolution reconstruction result corresponding to 2N+1 continuous video frame sequences. The problems of fixed scale reconstruction, spatial resolution limitation caused by insufficient frequency domain information, motion artifacts and high computational overhead caused by multiple iteration steps in the diffusion model generation process in the existing super-resolution method are solved, and more accurate super-resolution reconstruction of cardiac cine MRI is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, in particular to an arbitrary scale heart cine MIR super-resolution reconstruction method, an arbitrary scale heart cine MIR super-resolution reconstruction system, a computer device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] Cardiac magnetic resonance imaging (CMR) has advantages in quantitatively evaluating the morphology, motion and blood flow conditions of the heart. High-resolution (HR) cardiac cine magnetic resonance images can dynamically present the motion process of the heart. However, due to the limited acquisition window of k-space (frequency domain) filling, sufficient frequency domain information cannot be obtained, which limits the spatial resolution of the image. In addition, long-time scanning may cause involuntary movement of the patient, which is prone to motion artifacts, thereby further reducing the clarity and diagnostic value of the image. Super-resolution (SR) technology can realize the reconstruction of the low-resolution (LR) MR image obtained by scanning, and improve the resolution of the image.

[0004] The existing cardiac cine MIR super-resolution reconstruction has the following problems:

[0005] (1) Limitation of fixed scale super-resolution. Existing MRI super-resolution methods can usually only reconstruct for fixed up-sampling scales (such as 2x, 3x, 4x); therefore, these methods have to train and store corresponding neural network models for each up-sampling multiple separately, which makes it difficult to flexibly and efficiently adapt to different imaging conditions and specific clinical requirements.

[0006] (2) Spatial resolution limited by insufficient frequency domain information. In the process of cardiac cine MRI imaging, the acquisition window of k-space filling is limited, and this acquisition limitation directly leads to insufficient frequency domain information, thereby limiting the spatial resolution of the image. Insufficient frequency domain information will cause the details of the image to be blurred, making it difficult to clearly display the key features of the cardiac structure, which poses a technical challenge to the quantitative analysis of cardiac function and the early diagnosis of lesions.

[0007] (3) The impact of motion artifacts on image quality. Long-term scanning may cause motion artifacts due to involuntary movement of the patient. The presence of motion artifacts not only affects the clarity of the image, but may also obscure key lesion information, thereby reducing the diagnostic value of the image. This is especially true in cardiac cine MRI, where capturing accurate dynamic motion processes is crucial. The interference of motion artifacts poses additional technical challenges to reconstructing high-quality dynamic images.

[0008] (4) Limitations of existing diffusion models. Diffusion models have demonstrated outstanding capabilities in the field of image enhancement, especially in generating high-fidelity images. However, in MRI super-resolution tasks, diffusion models still have some important limitations. On the one hand, diffusion models usually require a multi-step iterative generation process, which results in a long reconstruction time and is difficult to meet the requirements of clinical application scenarios that require real-time performance. On the other hand, due to the limited ability to control details, the super-resolution images generated by the diffusion model may have deficiencies in texture representation, especially when processing complex data such as cardiac cine MRI, which can easily lead to the problem that the generated images are not realistic enough. Summary of the Invention

[0009] In order to solve the problems of fixed-scale reconstruction in existing super-resolution methods, the spatial resolution limitation caused by insufficient frequency domain information, the influence of motion artifacts, and the high computational overhead caused by multiple iterative steps in the diffusion model generation process, the present invention provides a super-resolution reconstruction method and system for cardiac Cine MIR at any scale, which achieves more accurate super-resolution reconstruction of cardiac Cine MRI.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions:

[0011] In a first aspect, the present invention provides a method for super-resolution reconstruction of cardiac Cine MIR at any scale.

[0012] A super-resolution reconstruction method for cardiac Cine MIR at any scale, including the following steps:

[0013] Get 2N+1 consecutive video frame sequences I τ-N ,…I τ …,I τ+N , where N is a positive integer greater than or equal to 1;

[0014] The encoder is used to extract the boundary reference frame I τ-N and I τ+N Boundary characteristics;

[0015] The denoising diffusion model is used to denoise the intermediate frame sequence I τ-N+1 ,…I τ …,I τ+N-1After processing, the diffusion prior feature corresponding to each intermediate frame is obtained, and the key target frame feature is obtained according to the diffusion prior and the boundary feature;

[0016] The diffusion prior feature and the key target frame feature are taken as the input of spatial perception implicit attention, the output of the diffusion prior feature, the key target frame feature and the spatial perception implicit attention is taken as the input of time perception implicit attention, and the output of the time perception implicit attention is processed by using a decoder to obtain the super-resolution reconstruction result corresponding to the 2N+1 continuous video frame sequence.

[0017] In the present application, the training process includes a pre-training stage and a training stage, the pre-training stage uses an encoder to extract the boundary features of boundary reference frames I τ-N and I τ+N , and a denoising diffusion model is used to process intermediate frames I τ ; the training stage uses an encoder to extract the boundary features of boundary reference frames I τ-N and I τ+N , and a denoising diffusion model is used to process intermediate frame sequence I τ-N+1 ,…I τ …,I τ+N-1 .

[0018] In the second aspect, the present application provides an arbitrary scale cardiac cine MIR super-resolution reconstruction system.

[0019] An arbitrary scale cardiac cine MIR super-resolution reconstruction system comprises:

[0020] A data acquisition unit is configured to acquire 2N+1 continuous video frame sequence I τ-N ,…I τ …,I τ+N , wherein N is a positive integer greater than or equal to 1;

[0021] A first processing unit is configured to use an encoder to extract the boundary features of boundary reference frames I τ-N and I τ+N .

[0022] A second processing unit is configured to use a denoising diffusion model to process intermediate frame sequence I τ-N+1 ,…I τ …,I τ+N-1 , and obtain the diffusion prior feature corresponding to each intermediate frame, and obtain the key target frame feature according to the diffusion prior and the boundary feature.

[0023] The reconstruction result generation unit is configured to: take the diffusion prior feature and the key target frame feature as input of spatial perception implicit attention, take the diffusion prior feature, the key target frame feature and output of the spatial perception implicit attention as input of time perception implicit attention, and obtain the super-resolution reconstruction result corresponding to the 2N+1 continuous video frame sequence by processing output of the time perception implicit attention using a decoder.

[0024] In a third aspect, the present application provides a computer device, comprising: a processor and a computer readable storage medium;

[0025] The processor is adapted to execute the computer program.

[0026] The computer readable storage medium has a computer program stored therein, and the computer program is executed by the processor to implement the arbitrary scale cardiac cine MIR super-resolution reconstruction method according to the first aspect of the present application.

[0027] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is adapted to be loaded and executed by a processor to implement the arbitrary scale cardiac cine MIR super-resolution reconstruction method according to the first aspect of the present application.

[0028] Compared with the prior art, the present application has the following beneficial effects:

[0029] 1. The implicit neural representation based on diffusion prior (DP-INR) method proposed in the present application successfully realizes super-resolution reconstruction of arbitrary scales by introducing an implicit neural network for modeling. The present application does not need to train a network model for each fixed multiple separately, but adopts a unified model framework to flexibly perform super-resolution reconstruction of different scales according to the requirements of input images. This means that whether the target is 2 times, 3 times or higher multiple resolution enhancement, the same network model can be used for processing, greatly improving the universality and operation efficiency of the model, and better adapting to different imaging requirements and clinical scenarios.

[0030] 2. In the process of cardiac cine MRI imaging, due to the limitation of scanning time and k-space filling, sufficient frequency domain information cannot be collected, which directly affects the spatial resolution of the image, resulting in blurred image details, especially in the display of cardiac structure and motion characteristics. Therefore, the present application designs an image super-resolution reconstruction method to improve the spatial resolution of the image in the image domain, clearly presents the detailed features of the heart, and further improves the accuracy of quantitative analysis of cardiac function and the effect of early lesion diagnosis.

[0031] 3. For the influence of motion artifacts on the quality of cardiac cine MRI images, the application combines the spatio-temporal relationship between dynamic frames, effectively alleviating the artifact problem caused by patient involuntary movement through spatio-temporal modeling technology; In the image reconstruction process, the application introduces a spatio-temporal implicit attention mechanism, which can not only accurately capture the dynamic changes between frames, but also effectively separate and eliminate the interference of motion artifacts on the heart structure and motion; Through this method, the motion coherence of the reconstructed image is significantly improved, and the key cardiac details are clearly presented, greatly improving the diagnostic value of the image.

[0032] 4. The traditional diffusion model generation process requires multiple iterations, not only the calculation cost is large, but also the reconstruction time is significantly increased, which limits its feasibility in clinical real-time application; The application introduces a skip sampling strategy, uses a step jump at a key step in the diffusion process, reduces the number of iterations, extracts key prior information from the intermediate step, avoids the complete generation process, effectively shortens the generation time, and preserves the high-quality image features of the diffusion model; In view of the deficiency of the diffusion model in detail recovery, the application combines implicit neural representation and spatio-temporal modeling technology to accurately model the spatio-temporal structure of cardiac cine MRI, significantly improving the detail recovery capability and the realism of the image.

[0033] The advantages of the additional aspects of the application will be partially given in the following description, partially become obvious from the following description, or be known by the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0034] The drawings accompanying the specification of this application form a part thereof, serve to provide further understanding of the application, and together with the description of the exemplary embodiments of the application and its description serve to explain the application, and do not constitute an improper limitation of the application.

[0035] Figure 1 The overall structure diagram of any scale cardiac cine MIR super-resolution reconstruction method provided for embodiment 1 of the application;

[0036] Figure 2 The working schematic diagram of space-aware implicit attention provided for embodiment 1 of the application;

[0037] Figure 3 The working schematic diagram of time-aware implicit attention provided for embodiment 1 of the application;

[0038] Figure 4 The comparison diagram of visualization results of various methods provided for embodiment 1 of the application;

[0039] Figure 5 The comparison diagram of visualization results of other methods in the time dimension provided for embodiment 1 of the application;

[0040] Figure 6 A comparison chart of visualization results out of scale provided for embodiment 1 of the present application and other methods;

[0041] Figure 7 A schematic diagram of an arbitrary scale cardiac cine MIR super-resolution reconstruction system provided for embodiment 2 of the present application;

[0042] Figure 8 A schematic diagram of a computer device provided for embodiment 3 of the present application. DETAILED DESCRIPTION

[0043] The present application will be further described below with reference to the accompanying drawings and embodiments.

[0044] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0045] The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0046] Embodiment 1:

[0047] The present implementation proposes an arbitrary scale cardiac cine MIR super-resolution reconstruction method, and designs an implicit neural representation method based on diffusion prior (DP-INR) for arbitrary scale cardiac cine MRI super-resolution reconstruction. DP-INR contains two key components: diffusion module (DM) and spatio-temporal implicit attention (ST-IA), and the purpose of designing DP-INR is to integrate DM and ST-IA together, so as to make full use of the advantages of DM and realize MRI reconstruction of arbitrary scale.

[0048] In order to construct a continuous time sequence with limited spatial resolution as input, the present application uses a sliding window video frame reading mechanism, and the sliding window size is 2N+1 and the step is 1. Therefore, the input of the network is composed of 2N+1 continuous time frames: [τ-N,τ+N], and the overall structure of the DP-INR proposed by the present application is as shown in Figure 1 As can be seen from the figure, the method proposed by the present application adopts a phased training strategy: pre-training phase and training phase.

[0049] In the pre-training phase, DP-INR uses the intermediate sequence frame I τ to obtain the diffusion prior Z0 to guide the ST-IA to decode the spatial coordinates and learn the motion trajectory, while the boundary reference frame I τ-N ,I τ+NContext information is extracted by the encoder and provided for the target frame. In the training phase, the full LR video sequence {I τ-N ,…,I τ+N} is used to optimize the parameters of the model to enable the model to more accurately reconstruct the high-resolution continuous-time sequence, specifically, using the first and last frames as boundary references, but focusing on the processing of the middle frame sequence [τ-N-1,τ+N-1], the model better understands and processes the temporal and spatial position relationship between the middle sequence frames and the boundary reference frames through the corresponding time-embedded position encoding (TPE) of each frame, thereby more effectively learning the motion trajectory.

[0050] In addition, the traditional DM needs to be trained through a large number of time steps, and the present application uses appropriate update rules to "skip" some steps to obtain accurate results using fewer iterations.

[0051] In the present application, preferably, two cardiac cine data, CMR x Recon public data set and In-house clinical data set, are used. For the CMR x Recon data set, the original k-space data of 44 subjects is selected and divided into a training set (30 subjects), a validation set (5 subjects) and a test set (9 subjects). The In-house data set contains 34 subjects, which are divided into a training set / validation set / test set according to 23 / 4 / 7. Both data sets contain 6 slices of cine data for each subject, but the CMR x Recon data set contains 12 time frames for each slice, while the In-house data set contains 15 time frames. The In-house data set is obtained by full sampling short-axis cardiac MR scanning using the TRUFI sequence, with the parameters of repetition time (TR) = 47.88 ms, echo time (TE) = 1.5 ms, slice thickness = 8.0 mm and field of view (FOV) = 344 mm.

[0052] In the present application, a diffusion model is introduced to shape specific prior features. As shown in the upper left corner of Figure 1 The denoising diffusion model (DDM) used in the present application defines a diffusion procedure based on Y∈{I τ-N-1 ,…,I τ+N-1}: Z T →Z T-1 →…→Z1→Z0. This inverse process (inference process) needs to learn a denoising process, usually called "Joint Distribution", which can be explained as follows:

[0053]

[0054] where Z t represents the prior at time step t, Z0 is the desired result, and Z T is the noisy forward process (diffusion process) from Z

[0055]

[0056] To better handle noise and missing values in the data, the present application applies singular value decomposition (SVD) to project the original data Z t and Y into a new, more compact pixel space, which reduces redundancy, highlights important features, and improves data representation, the data representation in the pixel space is given by:

[0057]

[0058] where, is used to represent the projection process, and further, for equation (1) and equation (2), the following definitions are given:

[0059]

[0060] where, represents Gaussian noise, σ T is the noise at time step T, and s represents the singular value.

[0061] The classic spatiotemporal SR method often has difficulty in effectively processing information in both spatial and temporal dimensions at the same time, which may lead to confusion or loss of information, and in the study and processing of complex spatiotemporal data, spatiotemporal implicit neural representation has become an innovative and effective method, which aims to capture and understand the inherent patterns and characteristics of the interweaving of space and time, unlike previous implicit representations, the present application proposes to learn parameters that can better integrate local regions with the help of attention mechanism.

[0062] Spatial-aware implicit attention (SIA), specifically, for any target LR time sequence frame, SIA focuses attention on both visual features and coordinate information, thereby more accurately capturing information essential for sampling at any scale. Specifically, in processing visual features, SIA can highlight features that have significant semantic and structural importance at different scales, and for coordinate information, it can perceive changes in spatial position and adjust the strategy of upsampling accordingly, the specific structure of SIA is as shown in Figure 2 .

[0063] Unlike previous self-attention, SIA does not take visual features as queries, but takes coordinate information as queries. Specifically, the query space coordinates are up-sampled, and then the distance between the query coordinates and the nearest prior coordinates is calculated , which can be represented as:

[0064]

[0065] where Δ is used to represent the solution of the coordinate distance, Z is derived from the prior feature representing the nearest query coordinate potential motion, C z is the spatial coordinates of the potential motion Z, and the sampling process of Q, K and V in SIA is defined as follows:

[0066]

[0067] where S represents the sampling operation for the target, and ↑ represents up-sampling, LR and HR represent the processed features in the low-dimensional space and the high-dimensional space respectively. Since the precise sampling and up-sampling strategies for different dimensional features are introduced, the information difference between low-dimensional and high-dimensional features is effectively balanced, thereby optimizing the representation and transmission of features, so as to alleviate the resolution gap between them. Since the query, key and value are obtained, further, SIA can be represented as:

[0068]

[0069] where ψ is a multi-layer perception (MLP) used to predict pixel values, [·] is used to represent the connection operation, represents matrix multiplication, and ω is a time position encoding (TPE) designed to introduce time-related position weights to the model, thereby affecting the subsequent attention calculation and pixel value prediction process, and its value is related to the position of the target frame in the sequence.

[0070] Time-aware implicit attention (TIA), specifically, when dealing with complex spatio-temporal data, the information of the time dimension is also crucial. In order to better learn the dynamic changes and features of the time dimension, the present application proposes a time-aware implicit attention. As Figure 3 shown, the core idea of TIA is to focus on the key target frame features and the change trend of the previous and subsequent reference frames, and further learn the motion trajectory along the time axis, thereby continuously enhancing the understanding and representation ability of the model for time information.

[0071] It is difficult to directly rely on the network to build the decoding characteristics of the target frame, because the network not only needs to learn the movement rule between continuous time frames, but also needs to grasp the context information. Therefore, the application proposes to assign adaptive weights TPE for different time frames, which is used to introduce the information of time dimension, so that the network can focus more accurately on the target frame and its context, thereby more effectively capturing the dynamic patterns in the time series. Formally, TIA can be defined as:

[0072]

[0073] where, and The sampling process of and is similar to formula (7), ω is the weight TPE, I br derived from the boundary reference frame I τ-N and I τ+B . Specifically, after the concatenation of and and TPE, the fused motion-related rich information is obtained by means of MLP, which also supplements the low-frequency components in the target frame , further:

[0074]

[0075] where, and are forward motion estimation and backward motion estimation, respectively. Next, the motion flow maps and the boundary reference frame I br perform motion flow guided grid warping. The estimation of motion flow plays a key role in supplementing the low-frequency components of the target frame. However, considering the possible imperfections of flow estimation, this may lead to inaccurate dynamic information. In order to make up for this deficiency, the application extracts prior features in the last step for local feature enhancement, which also supplements the high-frequency components in the target frame. The specific steps are as follows:

[0076]

[0077] where, are backward motion features and forward motion features, respectively, S represents the sampling operation, φ is used to represent grid warping, and δ is the convolution layer.

[0078] In this implementation, the multi-layer perceptron (MLP) is directly integrated into the ST-IA, which is different from the traditional attention mode. The previous attention mechanism adopts a Softmax function in the calculation, which has certain nonlinear characteristics, while the ST-IA in the present application deliberately ignores this when calculating attention, mainly because nonlinear calculation may ignore the information characteristics of other areas or sacrifice the ability to model long-distance dependencies, which may cause the model to be more prone to overfitting in some scenarios. However, the introduction of MLP cleverly makes up for the lack of model expression ability caused by the abandonment of nonlinearity. The powerful nonlinear mapping ability of MLP can effectively capture complex patterns and features in the data, enhancing the learning and extraction ability of the model.

[0079] The present application adopts two loss functions to optimize the performance of the model: L1-pixel loss and DC loss.

[0080] L1-pixel loss is a commonly used loss function for measuring the pixel-level difference between the predicted image and the real image. Its definition is as follows:

[0081]

[0082] where, represents the i-th frame of the real image, represents the i-th frame of the predicted image, 2N-1 represents the total number of image frames involved in the calculation each iteration, and ||·||1 is the L1 norm.

[0083] DC loss is used to ensure that the model maintains consistency with the original data when reconstructing the image, and its definition is as follows:

[0084]

[0085] where, denotes the Fourier transform operation, and DC loss emphasizes maintaining the consistency of image data in the frequency domain, thereby improving the quality of the reconstructed image.

[0086] Combining the above two loss functions, the total loss function of the present application is defined as follows:

[0087]

[0088] where, λ1 and λ2 are weight parameters used to balance the contribution of the two losses in the total loss.

[0089] The present application implements the proposed method on a NVIDIA A100 GPU workstation. The sliding window is used to read in consecutive frames, and the window size is set to 5, i.e. 2N+1=5. The Adam optimizer is used for network training for 60000 iterations. In the two-stage training strategy of the present application, the scaling factor of the first stage (the first 45000 iterations) is set to 4, and the scaling factor of the second stage (the last 15000 iterations) is flexibly selected in the range of [2, 4]. The initial learning rate is 10 -4 , the momentum decay coefficient is set to [0.9, 0.999], and the learning rate is decayed to 10 -7 after every 15000 iterations. The batch size is set to 16. The time step T in the diffusion model is 5. In addition, we set two hyperparameters in the training loss: λ1 is set to 1, and λ2 is set to 0.001. For the decoder, the present application sets the hidden layer dimension to [64, 64, 256, 256].

[0090] The present application uses two indicators, peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), to evaluate the performance of the model. In addition, the present application also uses normalized mean square error (NMSE) for comparison to measure the performance of the model in the MR reconstruction task. Figure 4 In the present application, the qualitative visual comparison of the present application and other SR methods on two data sets is provided, with a scale of 4x. The top, middle and bottom rows are the SR results under the CMRxRecon and In-house data sets. The second and third rows are the local enlarged areas in the first row box and the corresponding error maps, respectively. The more texture in the error map, the worse the quality of the reconstructed SR result. It can be seen that the proposed method of the present application exhibits superior performance in cardiac cine MRI tissue reconstruction, and the cardiac structure of the present application is clearer compared with other methods. At the same time, Figure 5 The visualization results also show that the DP-INR of the present application can more accurately capture spatial and temporal detail information. In addition, Figure 6 The present application shows the SR reconstruction results of the present application and other arbitrary scale SR methods at a scale of 5.3x. By selecting a non-integer scaling factor, the present application can better evaluate the ability of the model to accurately reconstruct details and structures at uncommon scales, proving the feasibility of the present application in handling arbitrary scale scenarios.

[0091] Quantitative comparisons of DP-INR with other reconstruction methods are presented in Tables 1 and 2. On the in-house and CMR×Recon datasets, DP-INR achieves competitive performance compared to other state-of-the-art models. Table 1 (based on in-scale data) and Table 2 (based on out-of-scale data) show that our method achieves the best results across all metrics, demonstrating its advancement in modeling cardiac cycle motion and maximizing motion recovery using spatiotemporal awareness.

[0092] Table 1: Quantitative comparison with SOTA on CMR×Recon and in-house datasets (in scale), with the best performance highlighted in bold, and evaluation metrics including PSNR (dB), SSIM and NMSE (×10 -2 ).

[0093]

[0094]

[0095] Table 2: Quantitative comparison with SOTA on CMR×Recon and in-house datasets (out of scale), with the best performance highlighted in bold, and evaluation metrics including PSNR (dB), SSIM, and NMSE (×10 -2 ).

[0096]

[0097] The present invention can effectively improve the spatial resolution of images, making image details clearer, especially in the display of cardiac structure and dynamic motion process. Compared with traditional super-resolution methods, the technology of the present invention can achieve higher image clarity, improve the visualization effect of cardiac structure and the accuracy of motion process, which is of great significance for cardiac function analysis, lesion detection and clinical diagnosis. On multiple cardiac cine MRI data sets, the reconstructed images of the present invention have a peak signal-to-noise ratio (PSNR) improved by about 1-5.1dB and a structural similarity index (SSIM) improved by about 0.001-0.11 compared with traditional methods, and the maximum reduction in normalized mean square error (NMSE) reached 7.5 (×10 -2 ), showing higher image quality.

[0098] The traditional diffusion model needs multiple iterations to complete image generation, resulting in large amount of calculation and long time. By introducing jump sampling and using the intermediate step prior information of the diffusion model, the present application can significantly shorten the reconstruction time. This method avoids the lengthy iteration in the complete diffusion process, reduces the calculation overhead, thereby improving the reconstruction speed, meeting the requirements of real-time and rapid diagnosis in clinic; compared with the traditional diffusion model, the present application reduces the reconstruction time by 40%-60%, greatly improving the efficiency in clinical use.

[0099] The present application effectively solves the influence of motion artifacts on image quality by combining space-time modeling technology. The space-time implicit attention mechanism can more finely model the dynamic process of cardiac motion, suppress motion artifacts and improve the stability and detail performance of dynamic images. The present application uses a scale-free super-resolution method, which not only improves the training and storage efficiency, but also can flexibly adapt to different imaging conditions and clinical needs. The present application does not need to train separate models for different up-sampling multiples, greatly saving storage space and computing resources.

[0100] Optionally, in other implementations, the Transformer can be used for the recovery of frequency domain information. In the reconstruction of cardiac cine MRI images, the Transformer can infer the high-frequency information of the spatial image from the missing frequency domain information through the self-attention mechanism. The Transformer can automatically capture the detail recovery pattern by learning the global features in the image, thereby making up for the lack of frequency domain information and improving the image resolution.

[0101] Optionally, in other implementations, time series modeling is combined with image reconstruction to improve image quality. In cardiac cine MRI images, the inter-frame time series information is very important for recovering details. By introducing a time series information model (such as a long short-term memory network (LSTM) or a convolution-based time series modeling method), the missing image details can be recovered from the time dimension, and the spatial resolution can be improved. On the basis of time series modeling, combined with motion compensation technology, the influence of motion artifacts can be reduced by capturing motion information and correcting the image, further improving the effect of super-resolution reconstruction.

[0102] Optionally, in other implementations, a physical constraint deep learning framework can be designed by combining traditional physical modeling methods (such as compressed sensing, extended conjugate gradient method, etc.). By modeling the physical constraints (such as sampling pattern, noise model, etc.) in the image recovery process, the network can perform super-resolution reconstruction in the case of insufficient frequency domain information, but another solution needs to be designed for the problem of any scale.

[0103] Embodiment 2:

[0104] AsFigure 7 As shown, the present implementation provides an arbitrary scale cardiac cine MIR super-resolution reconstruction system, comprising:

[0105] A data acquisition unit configured to acquire a sequence of 2N+1 continuous video frames I τ-N ,…I τ …,I τ+N , wherein N is a positive integer greater than or equal to 1;

[0106] A first processing unit configured to extract boundary features of boundary reference frames I τ-N and I τ+N using an encoder;

[0107] A second processing unit configured to obtain diffusion prior features corresponding to each intermediate frame by processing the sequence of intermediate frames I τ-N+1 ,…I τ …,I τ+N-1 using a denoising diffusion model, and obtain key target frame features according to the diffusion priors and the boundary features;

[0108] A reconstruction result generation unit configured to take the diffusion prior features and the key target frame features as inputs of spatial-aware implicit attention, take outputs of the diffusion prior features, the key target frame features, and the spatial-aware implicit attention as inputs of time-aware implicit attention, and obtain a super-resolution reconstruction result corresponding to the sequence of 2N+1 continuous video frames by processing outputs of the time-aware implicit attention using a decoder.

[0109] The specific working processes of the above-mentioned units are described in Embodiment 1, which will not be repeated here.

[0110] It can be understood that the above-mentioned units can be combined into one or several other units respectively or entirely, or some of the units can be further split into a plurality of units with smaller functions to constitute, which can achieve the same operation without affecting the implementation of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions, and the functions of one unit can also be realized by multiple units, or the functions of multiple units can be realized by one unit. In other embodiments of the present application, the system can also include other units, and these functions can also be realized by other units in actual application, and can be realized by multiple units.

[0111] According to another embodiment of the present application, the system described in the embodiment can be constructed and the method of the embodiment 1 of the present application can be implemented by running a computer program (including program codes) capable of performing each step involved in the corresponding method described in the embodiment 1 on a general computing device such as a computer including processing elements and storage elements such as a Central Processing Unit (CPU), a Random Access Memory (RAM), a Read Only Memory (ROM), and the like, the computer program can be recorded on a computer readable recording medium and loaded into the above computing device through the computer readable recording medium and run therein.

[0112] Embodiment 3

[0113] As shown in Figure 8 , the present implementation provides an electronic device including a processor 1001, a communication interface 1002, and a computer readable storage medium 1003. Wherein the processor 1001, the communication interface 1002 and the computer readable storage medium 1003 can be connected through the bus or other ways.

[0114] Among them, the communication interface 1002 is used to receive and send data, the computer readable storage medium 1003 can be stored in the memory of the electronic device, the computer readable storage medium 1003 is used to store computer programs, the computer programs include program instructions, the processor 1001 is used to execute the program instructions stored in the computer readable storage medium 1003.

[0115] The processor 1001 (or CPU (Central Processing Unit)) is the computing core and control core of the electronic device, which is suitable for implementing one or more instructions, and is particularly suitable for loading and executing one or more instructions to realize the corresponding method flow or corresponding function.

[0116] The processor 1001 is configured to perform the following process:

[0117] Obtain 2N+1 continuous video frame sequences I τ-N ,…I τ …,I τ+N , Wherein N is a positive integer greater than or equal to 1;

[0118] Extract the boundary features of the boundary reference frames I τ-N and I τ+N using the encoder;

[0119] The intermediate frame sequence I τ-N+1 ,…Iτ …,I τ+N-1 After processing, the diffusion prior features corresponding to each intermediate frame are obtained, and the key target frame features are obtained based on the diffusion prior features and the boundary features;

[0120] The diffusion prior features and the key target frame features are used as the input of spatial perception implicit attention, and the output of the diffusion prior features, the key target frame features and the spatial perception implicit attention is used as the input of temporal perception implicit attention. The output of the temporal perception implicit attention is processed by a decoder to obtain a super-resolution reconstruction result corresponding to 2N+1 continuous video frame sequences.

[0121] The specific working process is described in Example 1 and will not be repeated here.

[0122] Example 4:

[0123] This implementation provides a computer-readable storage medium (Memory). This computer-readable storage medium is a memory device in an electronic device that is used to store programs and data. It is understood that the computer-readable storage medium herein can include both built-in storage media in the electronic device and, of course, extended storage media supported by the electronic device. The computer-readable storage medium provides storage space that stores the processing system of the electronic device.

[0124] Furthermore, the storage space also stores one or more instructions suitable for being loaded and executed by the processor. These instructions may be one or more computer programs (including program code). It should be noted that the computer-readable storage medium herein may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage; optionally, it may be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0125] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes the one or more instructions stored in the computer-readable storage medium to implement the following process:

[0126] Get 2N+1 consecutive video frame sequences I τ-N ,…I τ …,I τ+N , where N is a positive integer greater than or equal to 1;

[0127] The encoder is used to extract the boundary reference frame I τ-N and I τ+N Boundary characteristics;

[0128] The denoising diffusion model is used to denoise the intermediate frame sequence I τ-N+1 ,…Iτ …,I τ+N-1 After processing, the diffusion prior feature corresponding to each intermediate frame is obtained, and the key target frame feature is obtained according to the diffusion prior and the boundary feature;

[0129] The diffusion prior feature and the key target frame feature are taken as inputs of spatial perception implicit attention, and the diffusion prior feature, the key target frame feature, and the output of the spatial perception implicit attention are taken as inputs of time perception implicit attention. The output of the time perception implicit attention is processed by a decoder to obtain a super-resolution reconstruction result corresponding to a sequence of 2N+1 continuous video frames.

[0130] The specific working process is described in Embodiment 1 and will not be repeated here.

[0131] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in the present application can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Professional technical objects can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0132] In the above embodiments, all or part can be realized by software, hardware, firmware, or any combination thereof. When realized by software, all or part can be realized in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. Computer instructions can be stored in a computer-readable storage medium or transmitted by a computer-readable storage medium. Computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (e.g., coaxial cable, optical fiber, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) methods. The computer-readable storage medium can be any available medium that the computer can access or a data processing device such as a server, data center, etc. containing one or more available media sets. The available media can be magnetic media (e.g., floppy disk, hard disk, magnetic tape), optical media (e.g., DVD), or semiconductor media (e.g., solid state disk (SSD)), etc.

[0133] The above merely provides the preferred embodiments of the present application, and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the principles and technical scope of the present application shall fall into the scope of the present application.

Claims

1. A method for cardiac cine MIR super-resolution reconstruction of any scale, characterized in that, The method comprises the following processes: acquiring a sequence of consecutive video frames wherein N is a positive integer greater than or equal to 1; Extracting boundary reference frames using encoders and boundary features of Adopting a denoising diffusion model to a sequence of intermediate frames After processing, the diffusion prior feature corresponding to each intermediate frame is obtained, and according to the diffusion prior and the boundary feature, the key target frame feature is obtained. The diffusion prior feature and the key target frame feature are used as the input of spatial perception implicit attention, and the output of the diffusion prior feature, the key target frame feature and the spatial perception implicit attention is used as the input of temporal perception implicit attention. The decoder is used to process the output of the temporal perception implicit attention to obtain Super-resolution reconstruction results corresponding to a continuous sequence of video frames; Applying a denoising diffusion model to a sequence of intermediate frames processing, including: A denoising diffusion model is employed to define a diffusion schedule with as a precondition: A forward process with increasing noise from to and is applied to project the original data into a new, more compact pixel space using singular value decomposition. The output of the time-aware implicit attention includes: wherein, , are respectively a backward motion feature and a forward motion feature, is a convolution layer, is a diffusion prior feature; , wherein, denotes a sampling operation, is used to represent mesh warping, , is a weighted TPE, is a multi-layer perceptron, are the query vector, key vector and value vector of temporal-aware implicit attention, respectively.

2. The arbitrary scale cardiac cine MIR super-resolution reconstruction method according to claim 1, wherein, The training process includes a pre-training phase and a training phase. The pre-training phase uses an encoder to extract a boundary reference frame. and The boundary features of the intermediate frames are also analyzed by using the denoising diffusion model. Processing; the training stage uses an encoder to extract the boundary reference frame and The boundary features of the intermediate frame sequence are also analyzed by using the denoising diffusion model. to be processed.

3. The arbitrary scale cardiac cine MIR super-resolution reconstruction method according to claim 1, wherein, The output of the spatial-aware implicit attention includes: wherein, is used to represent the connection operation, represents the matrix multiplication, is the time position encoding, , , and is the corresponding multi-layer perception, , is the relative offset of the closest prior coordinate to the query coordinate , is the diffusion prior feature, is the key target frame feature, represents the sampling operation for the target, represents the up-sampling, , respectively represent that the processed feature is in a low-dimensional space, a high-dimensional space.

4. The arbitrary scale cardiac cine MIR super-resolution reconstruction method according to any one of claims 1-3, wherein, The loss function is the sum of the L1-pixel loss and the DC loss.

5. An arbitrary scale cardiac cine MIR super-resolution reconstruction system, characterized by, The arbitrary scale cardiac cine MIR super-resolution reconstruction method according to any one of claims 1-4, comprising: The data acquisition unit is configured to acquire a sequence of N continuous video frames wherein N is a positive integer greater than or equal to 1. a first processing unit configured to extract boundary features of a boundary reference frame using an encoder and a second processing unit configured to extract boundary features of a boundary target frame using a decoder The second processing unit is configured to adopt a de-noising diffusion model to process the intermediate frame sequence After processing, the diffusion prior feature corresponding to each intermediate frame is obtained, and the key target frame feature is obtained according to the diffusion prior and the boundary feature. The reconstruction result generation unit is configured to: take the diffusion prior feature and the key target frame feature as input of spatial perception implicit attention, take output of the diffusion prior feature, the key target frame feature and the spatial perception implicit attention as input of time perception implicit attention, and obtain a super-resolution reconstruction result corresponding to a sequence of continuous video frames after processing output of the time perception implicit attention by using a decoder. a super-resolution reconstruction result corresponding to a sequence of continuous video frames.

6. A computer device, comprising: It comprises: a processor and a computer readable storage medium; a processor adapted to execute a computer program; a computer readable storage medium, wherein a computer program is stored in the computer readable storage medium, and the computer program is executed by the processor to realize the arbitrary scale cardiac cine MIR super-resolution reconstruction method according to any one of claims 1 to 4.

7. A computer readable storage medium characterized by The computer readable storage medium stores a computer program, and the computer program is adapted to be loaded and executed by the processor to realize the arbitrary scale cardiac cine MIR super-resolution reconstruction method according to any one of claims 1 to 4.

Images