Heart LGE image generation method and device based on diffusion model

Abstract

The invention provides a heart LGE image generation method and device based on a diffusion model. The method comprises the following steps: carrying out data preprocessing on a CINE sequence and a T1 image; mapping the T1 image to a submerged space by using a variational auto-encoder, and performing channel splicing in the submerged space to generate a submerged vector of T1; extracting space-time embedding from the CINE sequence through a Video MAE encoder, and generating a control vector in the diffusion step; the latent vector of T1 and the control vector in the diffusion step are spliced according to channel dimensions to be sent into the U-Net model, and cross-attention injection CINE features are used as dynamic information in a multi-scale layer of the U-Net model; and training the model, and reasoning and generating a heart LGE image based on the trained model. The application of the contrast agent can be obviously reduced or replaced clinically, the cost is reduced, and the contraindication risk is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a cardiac LGE image generation method and device based on a diffusion model. BACKGROUND

[0002] Late Gadolinium Enhancement (LGE) cardiac magnetic resonance has important clinical value for evaluating myocardial fibrosis, scarring and ischemic injury, but it relies on gadolinium-based contrast agents, which have the problems of administration cost, contraindications and potential deposition risks. The clinical routine non-contrast sequences include: CINE sequence: provides spatiotemporal motion information of the complete cardiac cycle (myocardial contraction / dilation, heart wall thickness change, etc.); T1 mapping: provides histology-related structural and tissue parameter information.

[0003] The existing method only uses a single condition (such as T1 or a single-phase image), does not explicitly model the timing dynamics of CINE, and the scarring boundary and myocardial motion-related artifacts are not reconstructed well, therefore, how to solve the problem of simultaneously fusing the spatiotemporal features of CINE and the structural latent features of T1 in the diffusion generation framework, and realizing robust conditional injection in the latent space, so that the synthesized LGE image is significantly improved in lesion presentation, structural similarity and subjective realism, has become a problem to be solved at present. SUMMARY

[0004] In view of the defects in the prior art, the purpose of the present application is to provide a cardiac LGE image generation method and device based on a diffusion model, which can significantly reduce or replace the use of contrast agents in clinical practice, cover the scenarios of contraindication or shortage of contrast agents, reduce costs and reduce the risk of contraindications.

[0005] To solve the above problems, the technical scheme of the present application is as follows:

[0006] A cardiac LGE image generation method based on a diffusion model, comprising the following steps:

[0007] Data preprocessing is performed on the CINE sequence and T1 image;

[0008] The T1 image is mapped to the latent space using a variational autoencoder, and a channel splicing is performed in the latent space to generate a T1 latent vector;

[0009] The CINE sequence is extracted through a VideoMAE encoder to extract spatiotemporal embedding, and a control vector in the diffusion step is generated;

[0010] The T1 latent vector and the control vector in the diffusion step are spliced according to the channel dimension and sent into a U-Net model, and the CINE features are injected as dynamic information in the multi-scale layer of the U-Net model with cross-attention;

[0011] training the model, and inferring to generate the cardiac LGE image based on the trained model.

[0012] Preferably, the step of data preprocessing the CINE sequence and the T1 image comprises: fixing the CINE sequence to 30 frames, and supplementing with black frames if insufficient; aligning the T1 image and the LGE image on the slice level; and uniformly scaling all images to a size of 256*256 and performing intensity normalization processing.

[0013] Preferably, the step of mapping the T1 image to the latent space using the variational autoencoder, performing channel splicing in the latent space, and generating the T1 latent vector comprises: using the variational autoencoder to encode the T1 image into the latent vector, and using public large-scale pre-training weights to enhance the representation and convergence stability.

[0014] Preferably, the step of extracting the spatiotemporal embedding of the CINE sequence through the VideoMAE encoder to generate the control vector in the diffusion step comprises: inputting the CINE sequence of the same subject into the VideoMAE encoder to extract the spatiotemporal embedding feature representing the myocardial motion pattern and the time sequence change; and pre-training the VideoMAE encoder using public large-scale CINE sequence data.

[0015] Preferably, the step of concatenating the T1 latent vector and the control vector in the diffusion step in the channel dimension and inputting them into the U-Net model, and injecting the CINE feature as the dynamic information in the form of cross-attention in the multi-scale layer of the U-Net model comprises: concatenating the T1 latent vector and the control vector in the diffusion step in the channel dimension and inputting them into the U-Net model, and inserting cross-attention blocks in the intermediate layers of the U-Net model to inject the dynamic information in the form of cross-attention.

[0016] Further, the application also provides a cardiac LGE image generation device based on a diffusion model, comprising an input unit, a VAE encoding unit, a VideoMAE encoding unit, a noise prediction U-Net unit, and a VAE decoding unit, wherein the input unit is used to receive a CINE sequence and a T1 image; the VAE encoding unit is used to map the T1 image to a latent space; the VideoMAE encoding unit is used to extract spatiotemporal embedding after self-supervised pre-training of the CINE; the noise prediction U-Net unit is used to fuse the spatiotemporal embedding through cross-attention in the intermediate layer, and perform denoising prediction in the latent space; and the VAE decoding unit is used to output a synthetic LGE image.

[0017] Preferably, the CINE features are injected in the form of cross-attention in the U-Net model, K / V, with the U-Net intermediate representation as Q, so that the cardiac dynamics is guided as a condition on each diffusion step.

[0018] Compared with the prior art, the present application uses a diffusion model in the VAE latent space to generate LGE images by denoising with CINE and T1 as conditional inputs, which can significantly reduce or replace the use of contrast agents in clinical practice, cover the scenarios of contrast contraindications or shortages, reduce costs and reduce the risk of contraindications. The present application provides a non-invasive and convenient imaging solution for the evaluation of myocardial fibrosis, myocarditis and other diseases. In addition, the generated LGE images can assist clinicians in making more accurate treatment decisions, optimizing risk stratification and treatment pathways, and reducing medium and long-term adverse cardiovascular events. BRIEF DESCRIPTION OF DRAWINGS

[0019] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments, made with reference to the accompanying drawings:

[0020] Figure 1 A flow chart of the method for generating a cardiac LGE image based on a diffusion model according to the present application is shown in the figure;

[0021] Figure 2 A schematic diagram of the injection position of cross-attention in the U-Net according to the present application is shown in the figure;

[0022] Figure 3 A structure block diagram of the device for generating a cardiac LGE image based on a diffusion model according to the present application is shown in the figure. DETAILED DESCRIPTION

[0023] The present application will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present application. These all belong to the protection scope of the present application.

[0024] Specifically, the present application provides a method for generating a cardiac LGE image based on a diffusion model, as shown in the figure, the method comprises the following steps: Figure 1

[0025] S1: data preprocessing of CINE sequence and T1 image;

[0026] Specifically, the CINE sequence is fixedly cropped to 30 frames, and if it is insufficient, it is supplemented with black frames. The T1 image and the LGE image are registered and aligned at the slice level. All images are uniformly scaled to a size of 256x256 and intensity normalized. ​

[0027] S2: mapping the T1 image to a latent space using a variational autoencoder, channel concatenating in the latent space to generate a latent vector of T1;

[0028] Specifically, the T1 mapping image is encoded into a latent vector using a variational autoencoder (VAE), and the variational autoencoder can use publicly available large-scale pre-trained weights to enhance the representation and convergence stability; the encoded T1 feature latent vector and the noise latent vector are concatenated in the channel dimension and input into the noise prediction U-Net.

[0029] S3: extracting a spatiotemporal embedding of the CINE sequence through a VideoMAE encoder to generate a control vector in the diffusion step;

[0030] Specifically, a Video Masked Autoencoder (VideoMAE) is used, and the CINE sequence of the same subject is input into the VideoMAE encoder to extract spatiotemporal embedding features representing myocardial motion patterns and temporal changes; the VideoMAE encoder can be pre-trained using publicly available large-scale CINE sequence data.

[0031] S4: concatenating the latent vector of T1 and the control vector in the diffusion step in the channel dimension and inputting it into the U-Net model, and injecting CINE features as dynamic information in the multi-scale layer of the U-Net model through cross-attention;

[0032] Specifically, the latent vector of T1 and the control vector in the diffusion step are concatenated in the channel dimension and input into the U-Net model, and dynamic information is injected in the multi-scale layer of the U-Net model through cross-attention (Q is the current layer feature, and K / V is the spatiotemporal embedding of CINE), realizing the collaborative constraint of structure (T1) and motion (CINE). As shown in Figure 2 the cross-attention block (Cross-Attention) is inserted in several intermediate layers of the U-Net model, and the cross-attention (Cross-Attention) is injected as K / V, and the U-Net intermediate representation is Q, so that the cardiac dynamics is guided as a condition in each diffusion step, wherein Q: the current layer U-Net feature; K / V: obtained by projecting the CINE spatiotemporal embedding; and the conditional response is obtained and fused with the main feature through residual / connection; the injection strategy can be repeated at multiple scales to improve the regulation of myocardial motion and structural details at different scales.

[0033] S5: training the model and generating a cardiac LGE image based on the trained model.

[0034] Specifically, the training adopts a diffusion denoising objective, and auxiliary losses related to the task can be introduced in the verification and effect display stage. The standard diffusion denoising objective ( - prediction or - prediction) as the main loss ; Optimizer with hyperparameters: AdamW; Learning rate ; Training rounds 100-300 epochs; Diffusion steps are set according to computing resources.

[0035] In the inference stage, only CINE and T1 of the same subject need to be provided to synthesize quasi-LGE images with high consistency with real LGE, which can be used for clinical evaluation and subsequent quantitative analysis.

[0036] Further, the application also provides a cardiac LGE image generation device based on a diffusion model, as shown in the figure, the device comprises an input unit 1, a VAE encoding unit 2, a VideoMAE encoding unit 3, a noise prediction U-Net unit 4 and a VAE decoding unit 5. Figure 3

[0037] The input unit 1 is used for receiving CINE sequence and T1 image; the VAE encoding unit 2 is used for mapping the T1 image to the latent space; the VideoMAE encoding unit 3 is used for extracting the spatio-temporal embedding after self-supervised pre-training of CINE; the noise prediction U-Net unit 4 is used for fusing the spatio-temporal embedding through cross-attention in the intermediate layer and performing denoising prediction in the latent space; and the VAE decoding unit 5 is used for outputting the synthesized LGE image.

[0038] The image is mapped to the latent space using a variational autoencoder (VAE), and the diffusion process is performed in the latent space; the T1 mapping is encoded into a latent vector through the same VAE, and then input into the noise prediction U-Net after being spliced with the noise latent vector in the channel dimension; the CINE sequence extracts the spatio-temporal embedding through the VideoMAE encoder, and the spatio-temporal embedding is injected into the U-Net as K / V in the form of cross-attention, and the intermediate representation of the U-Net is Q, so that the cardiac dynamics is guided to each diffusion step as a condition; the training adopts the diffusion denoising target, and the task-related auxiliary loss can be introduced in the verification and effect display stage.

[0039] The specific embodiments of the application are described above. It should be understood that the application is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essential content of the application. In the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.​

Claims

1. A method for generating cardiac LGE images based on a diffusion model, characterized in that, The method includes the following steps: Data preprocessing of CINE sequences and T1 images; The T1 image is mapped to the latent space using a variational autoencoder, and channel stitching is performed in the latent space to generate the latent vector of T1. The spatiotemporal embedding of the CINE sequence is extracted using a VideoMAE encoder to generate the control vector in the diffusion step. The latent vector of T1 and the control vector in the diffusion step are concatenated along the channel dimension and fed into the U-Net model. CINE features are injected across attention as dynamic information in the multi-scale layers of the U-Net model. The model is trained, and cardiac LGE images are generated based on the trained model.

2. The method for generating cardiac LGE images based on a diffusion model according to claim 1, characterized in that, The steps for data preprocessing of the CINE sequence and T1 image specifically include: cropping the CINE sequence to 30 frames, and padding with black frames if necessary; registering and aligning the T1 image and LGE image at the slice level; and scaling all images to a uniform size of 256×256 and performing intensity normalization.

3. The method for generating cardiac LGE images based on a diffusion model according to claim 1, characterized in that, The steps of mapping the T1 image to the latent space using a variational autoencoder, performing channel stitching in the latent space, and generating the latent vector of T1 specifically include: encoding the T1 mapped image into a latent vector using a variational autoencoder, wherein the variational autoencoder employs publicly available large-scale pre-trained weights to enhance representation and convergence stability.

4. The method for generating cardiac LGE images based on a diffusion model according to claim 1, characterized in that, The step of extracting the spatiotemporal embedding of the CINE sequence through the VideoMAE encoder to generate the control vector in the diffusion step specifically includes: inputting the CINE sequence of the same subject into the VideoMAE encoder to extract the spatiotemporal embedding features that characterize the myocardial motion pattern and temporal changes; the VideoMAE encoder is pre-trained using publicly available large-scale CINE sequence data.

5. The method for generating cardiac LGE images based on a diffusion model according to claim 1, characterized in that, The step of concatenating the latent vector of T1 and the control vector in the diffusion step according to the channel dimension and feeding them into the U-Net model, and injecting CINE features as dynamic information in the multi-scale layers of the U-Net model in the form of cross-attention, specifically includes: concatenating the latent vector of T1 and the control vector in the diffusion step according to the channel dimension and feeding them into the U-Net model, and inserting cross-attention blocks in several intermediate layers of the U-Net model to inject dynamic information in the form of cross-attention.

6. A method and apparatus for generating cardiac LGE images based on a diffusion model, characterized in that, The device includes an input unit, a VAE encoding unit, a VideoMAE encoding unit, a noise prediction U-Net unit, and a VAE decoding unit. The input unit receives a CINE sequence and a T1 image. The VAE encoding unit maps the T1 image to a latent space. The VideoMAE encoding unit extracts spatiotemporal embeddings from the CINE sequence after self-supervised pre-training. The noise prediction U-Net unit performs denoising prediction in the latent space by fusing spatiotemporal embeddings across attention in an intermediate layer. The VAE decoding unit outputs a synthesized LGE image.

7. The cardiac LGE image generation device based on a diffusion model according to claim 6, characterized in that, In the U-Net model, CINE features are injected as K / V in the form of cross-attention, and U-Net intermediate representations are used as Q, thereby applying cardiac dynamics as a conditional guide to each diffusion step.

Images