Method for converting multi-stage non-iodine contrast CT image into coronary artery CTA image

Abstract

The invention provides a method for converting a multi-stage non-iodine contrast CT image into a coronary artery CTA image, and the method comprises the steps: obtaining an NCCT volume and a CTA volume which are matched, carrying out the data preprocessing, and constructing a training data set; respectively constructing an NCCT encoder-decoder network and a CTA encoder-decoder network, and carrying out pre-training based on the training data set; a mark reconstruction network is constructed and trained, a masked low-resolution CTA mark is reconstructed under the condition of an NCCT token obtained by a trained encoder, and then a low-resolution CTA volume is reconstructed by a trained decoder; a super-resolution network is constructed and trained, and a high-resolution CTA image is generated from the low-resolution CTA image; a high resolution CTA volume is generated from the low resolution NCCT volume using the trained multi-stage model. The method has the beneficial effects that the problem of posterior crash is avoided, the decoding speed during reasoning is higher, and the training process is more stable; in the low-resolution stage, the model can learn multi-scale context information, and in the high-resolution stage, the model focuses on recovering details, more fine structures are reserved in final output, and clinical requirements can be met.

Description

Technical Field

[0001] The invention relates to the technical field of medical image processing, and in particular to a method for converting a multi-stage non-iodine contrast agent CT image into a coronary artery CTA image. Background Art

[0002] As a major challenge to global public health, the incidence of coronary heart disease is increasing with the aging of the population and changes in lifestyle. Coronary CTA (computed tomography angiography) is a non-invasive method for evaluating cardiovascular conditions. It has high sensitivity and specificity for the diagnosis of coronary heart disease and can effectively identify coronary artery lesions. However, the risks of using iodinated contrast agents in CTA, such as contrast agent extravasation, acute kidney injury, and allergic reactions, limit its widespread application. These risks not only affect patient health, but also increase medical costs. Therefore, developing a CTA imaging technology that does not require iodine contrast agents and converting non-iodine contrast CT (NCCT) images into coronary CTA images to reduce risks and maintain high diagnostic performance is an urgent problem to be solved in the field of medical imaging.

[0003] So far, some research results have been achieved in the cross-modal conversion technology of images, mainly including autoregressive models, variational autoencoders (VAE) and GAN. The autoregressive model mainly uses its sequence modeling ability to gradually construct an image by predicting the probability distribution of the next element in the sequence; VAE generates new data points by learning the potential distribution of data, providing a continuous potential space; GAN generates high-quality images by adversarially training the generator and discriminator, which is suitable for processing dual-domain image conversion and uses deep neural networks to find the mapping relationship between images in different domains.

[0004] However, these methods are often subject to various limitations. Autoregressive models are slow to generate and very expensive for high-resolution image generation; VAEs often produce suboptimal sample quality and suffer from the problem of posterior collapse; the training process of GANs can be unstable, and adversarial training between the generator and the discriminator can lead to mode collapse. In addition, due to the significant difference in data distribution between medical images and natural images, directly applying GAN models in the field of natural images to medical imaging may encounter poor performance. In summary, existing methods have defects and there is no satisfactory solution yet.

[0005] The model of the multi-stage method provided by the present invention includes an encoder-decoder network, a label reconstruction network and a super-resolution network, wherein the encoder-decoder network based on VQ-VAE compresses the low-resolution volume into discrete labels, and its vector quantization step avoids the problem of VAE posterior collapse; the label reconstruction network based on the Transformer network completes the reconstruction by predicting the masked low-resolution labels, and has a faster decoding speed during inference and a higher generated fidelity than the autoregressive method; constructing a super-resolution network based on a diffusion model is more stable than the GAN training process, can generate images with rich details, and can meet clinical requirements. Summary of the invention

[0006] In order to solve the above technical problems, the present invention aims to provide a multi-stage method for converting non-iodine contrast agent CT images into coronary artery CTA images.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A multi-stage method for converting a non-iodine contrast agent CT image to a coronary artery CTA image comprises the following steps:

[0009] Step S1, obtaining the matched NCCT volume and CTA volume, performing data preprocessing, and constructing a training data set;

[0010] Step S2, constructing an NCCT encoder-decoder network and a CTA encoder-decoder network respectively, and performing pre-training based on the training data set obtained in step S1;

[0011] Step S3, constructing and training a marker reconstruction network, reconstructing the masked low-resolution CTA markers conditioned on the NCCT tokens obtained from the encoder trained in step S2, and then reconstructing the low-resolution CTA volume using the decoder trained in step S2;

[0012] Step S4, constructing a super-resolution network and training it to generate a high-resolution CTA image from a low-resolution CTA image;

[0013] Step S5: Generate a high-resolution CTA volume from the low-resolution NCCT volume using the multi-stage model trained in steps S2, S3, and S4.

[0014] Furthermore, in step S1, the matched NCCT volume and CTA volume are from the same patient, the NCCT volume includes a series of NCCT images, and the CTA includes a series of CTA images.

[0015] Furthermore, in step S1, data preprocessing and construction of a training data set are specifically as follows:

[0016] Step S1-1, normalize all image data to [-1, 1], and unify the volume size to 512×512×128;

[0017] Step S1-2, registering the NCCT volume and the CTA volume to ensure that each slice of the two volumes corresponds to each other;

[0018] Step S1-3: downsample all images to a size of 128×128 to obtain matched low-resolution NCCT volumes and low-resolution CTA volumes as training data sets.

[0019] Furthermore, in step S1, the method for registering the NCCT volume and the CTA volume includes manual registration and registration based on a registration network.

[0020] Furthermore, in step S2, the NCCT encoder-decoder network obtains low-resolution NCCT markers from the input low-resolution NCCT volume and reconstructs the low-resolution NCCT volume; the CTA encoder-decoder network obtains low-resolution CTA markers from the input low-resolution CTA volume and reconstructs the low-resolution CTA volume.

[0021] Furthermore, in step S2, both the NCCT encoder-decoder network and the CTA encoder-decoder network are based on VQVAE, including an encoder, a codebook and a decoder.

[0022] Furthermore, in step S2, pre-training is performed based on the training data set obtained in step S1, including the following steps:

[0023] Step S2-1, pass the input data through the encoder to obtain a potential representation;

[0024] Step S2-2, compare the potential representation with the vectors in the codebook, find the closest vector as the image label;

[0025] Step S2-3, the decoder reconstructs the image according to the image mark;

[0026] Step S2-4: Calculate the loss function and use the gradient descent method to update the encoder, decoder and codebook.

[0027] Furthermore, in step S3, a marker reconstruction network is constructed and trained, and the masked low-resolution CTA markers are reconstructed based on the NCCT tokens obtained from the encoder trained in step S2, and then the low-resolution CTA volume is reconstructed by the decoder trained in step S2, including the following steps:

[0028] Step S3-1, inputting the low-resolution CTA volume into the CTA encoder trained in step S2 to obtain a low-resolution CTA marker;

[0029] Step S3-2, inputting the low-resolution NCCT volume into the NCCT encoder trained in step S2 to obtain a low-resolution NCCT label;

[0030] Step S3-3, randomly masking a certain proportion of low-resolution CTA markers to obtain masked low-resolution CTA markers;

[0031] Step S3-4, inputting the masked low-resolution CTA marker and the low-resolution NCCT marker into the Transformer network, reconstructing the masked low-resolution CTA marker, and updating the Transformer network parameters by minimizing the negative log-likelihood of the masked marker;

[0032] Step S3-5: input the reconstructed low-resolution CTA marker into the CTA decoder trained in step S2 to obtain a reconstructed low-resolution CTA volume.

[0033] Furthermore, in step S4, a super-resolution network is constructed and trained to generate a high-resolution CTA image from a low-resolution CTA image, including the following steps:

[0034] Step S4-1, taking a low-resolution CTA image in the low-resolution CTA volume, and upsampling it to obtain an upsampled CTA image with a size of 512×512;

[0035] Step S4-2: during the diffusion process, for the corresponding high-resolution CTA image, gradually adding noise in each diffusion step to blur the image;

[0036] Step S4-3, during the reverse diffusion process, for each diffusion step, the super-resolution network takes the noisy high-resolution CTA, the diffusion step moment and the upsampled CTA image as a condition as input, predicts the noise of the diffusion step, and updates the super-resolution network parameters by calculating the error between the predicted noise and the actual noise as the loss function.

[0037] Furthermore, in step S4, the super-resolution network is based on a diffusion model, and the structure is a Unet network.

[0038] Furthermore, in step S5, the high-resolution CTA volume is generated from the low-resolution NCCT volume using the multi-stage model trained in steps S2, S3, and S4, including the following steps:

[0039] Step S5-1, inputting the low-resolution NCCT volume into the NCCT encoder trained in step S2 to obtain a low-resolution NCCT label;

[0040] Step S5-2, starting from the blank low-resolution CTA marker, input it and the low-resolution NCCT marker into the Transformer network trained in step 3 to reconstruct the masked low-resolution CTA marker;

[0041] Step S5-3, inputting the reconstructed low-resolution CTA marker into the CTA decoder trained in step 2 to obtain a reconstructed low-resolution CTA volume;

[0042] Step S5-4, taking a low-resolution CTA image in the reconstructed low-resolution CTA volume, and upsampling it to obtain an upsampled CTA image with a size of 512×512;

[0043] Step S5-5, an image randomly sampled from a Gaussian distribution is used as the initial inverse diffusion image, the noise of the current diffusion step is predicted by the high-resolution network trained in step 4, and the CTA image of the previous step is obtained by calculating the CTA image of the current diffusion step, the diffusion step moment and the predicted noise, and the iteration is repeated until a reconstructed high-resolution CTA image is obtained.

[0044] Step S5-6: Repeat operations 4)-5) for each low-resolution CTA image to finally obtain a high-resolution CTA volume.

[0045] Compared with the closest prior art, the technical solution provided by the present invention has the following beneficial effects:

[0046] The present invention compresses low-resolution volumes into discrete tags by constructing an encoder-decoder network based on VQ-VAE. Its vector quantization step avoids the problem of posterior collapse and has strong representation capabilities, which can better adapt to downstream generation tasks.

[0047] The present invention constructs a marker reconstruction network based on a Transformer network and completes reconstruction by predicting masked low-resolution markers, which has a faster decoding speed during inference and high generated fidelity.

[0048] The present invention generates high-resolution CTA images from low-resolution CTA images by constructing a super-resolution network based on a diffusion model. The training process is more stable and images with rich details can be generated.

[0049] The present invention proposes a multi-stage conversion method of non-iodine contrast agent CT images to coronary artery CTA images. In the low-resolution stage, the model can learn multi-scale contextual information, while in the high-resolution stage it focuses on restoring details, retaining more fine structures in the final output, which can meet clinical needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flow chart of a method for converting a multi-stage non-iodine contrast agent CT image to a coronary artery CTA image provided by an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of an NCCT encoder-decoder network and a CTA encoder-decoder network architecture provided by an embodiment of the present invention;

[0052] Figure 3 A schematic diagram of a marking and reconstruction network architecture provided by an embodiment of the present invention;

[0053] Figure 4 A schematic diagram of a super-resolution network architecture provided by an embodiment of the present invention;

[0054] Figure 5 A schematic diagram of a multi-stage model architecture provided in an embodiment of the present invention, which enables generation of a high-resolution CTA volume from a low-resolution NCCT volume. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Embodiment 1

[0057] Figure 1 A flow chart of a method for converting a multi-stage non-iodine contrast agent CT image to a coronary artery CTA image provided by an embodiment of the present invention is shown below. Figure 1 , explaining each step in detail.

[0058] Step S110: Acquire the matched NCCT volume and CTA volume, perform data preprocessing, and construct a training data set.

[0059] The matched NCCT volume and CTA volume are from the same patient, the NCCT volume includes a series of NCCT images, and the CTA includes a series of CTA images.

[0060] Data preprocessing includes normalizing all image data to [-1, 1] and unifying the volume size to 512×512×128.

[0061] The NCCT volume and the CTA volume are registered to ensure that the structures in each slice of the two volumes are corresponding. The method of registering the NCCT volume and the CTA volume is to first perform manual registration so that the volumes of the two modalities are consistent on the z-axis; then perform registration based on an unsupervised registration network.

[0062] Furthermore, the registration based on the unsupervised registration network is based on a CNN network, the input is the volume of two modalities, one as a fixed volume and the other as a moving volume, and the output is the deformation field from the moving volume to the fixed volume. The loss function is the error between the deformed moving volume and the fixed volume.

[0063] All images were downsampled to 128 × 128 size, and matched low-resolution NCCT volumes and low-resolution CTA volumes were obtained as training datasets.

[0064] Step S120: construct an NCCT encoder-decoder network and a CTA encoder-decoder network respectively, and perform pre-training based on a training data set.

[0065] For details, see Figure 2 , which is a schematic diagram of the NCCT encoder-decoder network and CTA encoder-decoder network architecture provided by an embodiment of the present invention. The NCCT encoder-decoder network obtains low-resolution NCCT markers from an input low-resolution NCCT volume and reconstructs a low-resolution NCCT volume; the CTA encoder-decoder network obtains low-resolution CTA markers from an input low-resolution CTA volume and reconstructs a low-resolution CTA volume.

[0066] Furthermore, both the NCCT encoder-decoder network and the CTA encoder-decoder network are based on VQVAE, including the parameters of the encoder, codebook, and decoder.

[0067] Further, pre-training is performed based on the training data set obtained in S110, specifically:

[0068] 1) Pass the input data through the encoder to obtain a potential representation;

[0069] 2) Compare the latent representation with the vectors in the codebook and find the closest vector as the image tag;

[0070] 3) The decoder reconstructs the image based on the image markers;

[0071] 4) Calculate the loss function and use the gradient descent method to update the encoder, decoder and codebook.

[0072] Specifically, the loss function of the encoder-decoder network includes three terms:

[0073]

[0074] Among them, the first item represents the reconstruction loss, which is used to optimize the encoder and decoder; the second item is used to optimize the codebook; and the third item is used to keep the update speed of the encoder consistent with the update speed of the codebook.

[0075] Step S130, construct a marker reconstruction network and train it, reconstruct the masked low-resolution CTA markers based on the NCCT tokens obtained from the encoder trained in step S120, and then reconstruct the low-resolution CTA volume using the decoder trained in step S120.

[0076] For details, see Figure 3 FIG. 1 is a schematic diagram of a network architecture for label reconstruction provided by an embodiment of the present invention. Specifically:

[0077] 1) Inputting the low-resolution CTA volume into the CTA encoder trained in step S120 to obtain a low-resolution CTA marker;

[0078] 2) inputting the low-resolution NCCT volume into the NCCT encoder trained in step S120 to obtain a low-resolution NCCT label;

[0079] 3) randomly masking a certain proportion of low-resolution CTA markers to obtain masked low-resolution CTA markers;

[0080] 4) Input the masked low-resolution CTA marker and the low-resolution NCCT marker into the Transformer network, reconstruct the masked low-resolution CTA marker, and update the Transformer network parameters by minimizing the negative log-likelihood of the masked marker;

[0081] 5) Input the reconstructed low-resolution CTA marker into the CTA decoder trained in step S120 to obtain a reconstructed low-resolution CTA volume.

[0082] Specifically, the loss function of the Transformer network is:

[0083]

[0084] in, Represents the masked low-resolution CTA token, which is input into a Transformer network to predict the probability of each masked token

[0085] Step S140: construct a super-resolution network and perform training to generate a high-resolution CTA image from a low-resolution CTA image. Figure 4 FIG. 1 is a schematic diagram of a super-resolution network architecture provided by an embodiment of the present invention. Specifically:

[0086] 1) Take a low-resolution CTA image in the low-resolution CTA volume and upsample it to obtain an upsampled CTA image of size 512×512;

[0087] 2) During the diffusion process, for the corresponding high-resolution CTA image, noise is gradually added in each diffusion step, making the image blurred;

[0088] 3) During the inverse diffusion process, for each diffusion step, the super-resolution network takes the noisy high-resolution CTA, the diffusion step time and the upsampled CTA image as the condition as input, predicts the noise of the diffusion step, and updates the super-resolution network parameters by calculating the error between the predicted noise and the actual noise as the loss function.

[0089] Furthermore, the super-resolution network is based on the diffusion model and its structure is a Unet network.

[0090] Specifically, upsampling uses bilinear interpolation.

[0091] Specifically, the diffusion process is a 0 Gradually introduce Gaussian noise and iterate T steps to generate a noisy image x T The noisy CTA image x generated by the diffusion step at time t t It is expressed as:

[0092]

[0093] Among them, x t represents the noisy CTA image corresponding to the diffusion step time t, represents the noise parameter, which is α t The cumulative product of t Represents the actual noise added at diffusion step time t, which obeys the Gaussian distribution ε~N(0,I).

[0094] Specifically, the inverse diffusion process is a process of gradually denoising by calculating the predicted noise, and the loss function is:

[0095]

[0096] Where c represents the upsampled CTA image as condition.

[0097] Step S150: Generate a high-resolution CTA volume from a low-resolution NCCT volume using the trained multi-stage model.

[0098] For details, see Figure 5 FIG. 1 is a schematic diagram of a multi-stage model architecture provided by an embodiment of the present invention. Specifically:

[0099] 1) inputting the low-resolution NCCT volume into the NCCT encoder trained in step S120 to obtain a low-resolution NCCT label;

[0100] 2) Starting from the blank low-resolution CTA marker, it and the low-resolution NCCT marker are input into the Transformer network trained in step S130 to reconstruct the masked low-resolution CTA marker;

[0101] 3) Inputting the reconstructed low-resolution CTA marker into the CTA decoder trained in step S120 to obtain a reconstructed low-resolution CTA volume;

[0102] 4) taking a low-resolution CTA image in the reconstructed low-resolution CTA volume, and upsampling it to obtain an upsampled CTA image of size 512×512;

[0103] 5) An image randomly sampled from a Gaussian distribution is used as an initial inverse diffusion image. The noise of the current diffusion step is predicted by the high-resolution network trained in step S140, and the CTA image of the previous step is obtained by calculating the CTA image of the current diffusion step, the diffusion step moment and the predicted noise. The iteration is repeated until a reconstructed high-resolution CTA image is obtained.

[0104] 6) Repeat steps 4) to 5) for each low-resolution CTA image to finally obtain a high-resolution CTA volume.

[0105] Specifically, the following reparameterization technique is used to calculate the CTA image x of the previous step in the current diffusion step t: t-1 :

[0106]

[0107] where σ t z means known x t Under the distribution of x t-1 When t>1, z~N(0,I), otherwise z=0. After T steps of iteration, the denoised high-resolution CTA image can be obtained.

[0108] Through the above implementation, we can generate a high-resolution CTA volume from an original low-resolution NCCT volume.

[0109] The matters not described in detail in the present invention are all known technologies to those skilled in the art.

[0110] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical schemes composed of any combination of the above-mentioned technical features. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also regarded as the protection scope of the present invention.

Claims

1. A multi-stage method for converting non-iodine contrast agent CT images to coronary artery CTA images, characterized in that: The steps include: Step S1, obtaining the matched NCCT volume and CTA volume, performing data preprocessing, and constructing a training data set; Step S2, constructing an NCCT encoder-decoder network and a CTA encoder-decoder network respectively, and performing pre-training based on the training data set obtained in step S1; Step S3, constructing and training a marker reconstruction network, reconstructing the masked low-resolution CTA markers conditioned on the NCCT tokens obtained from the encoder trained in step S2, and then reconstructing the low-resolution CTA volume using the decoder trained in step S2; Step S4, constructing a super-resolution network and training it to generate a high-resolution CTA image from a low-resolution CTA image; Step S5: Generate a high-resolution CTA volume from the low-resolution NCCT volume using the multi-stage model trained in steps S2, S3, and S4.

2. The method for converting a multi-stage non-iodine contrast agent CT image to a coronary artery CTA image according to claim 1, characterized in that: In step S1, the specific steps are: Step S1-1, normalize all image data to [-1, 1], and unify the volume size to 512×512×128; Step S1-2, registering the NCCT volume and the CTA volume to ensure that each slice of the two volumes corresponds to each other; Step S1-3: downsample all images to a size of 128×128 to obtain matched low-resolution NCCT volumes and low-resolution CTA volumes as training data sets.

3. According to the multi-stage method for converting non-iodine contrast agent CT images to coronary artery CTA images according to claim 1, in step S2, the NCCT encoder-decoder network and the CTA encoder-decoder network are both based on VQVAE, including an encoder, a codebook and a decoder, and are pre-trained by the following steps: Step S2-1, pass the input data through the encoder to obtain a potential representation; Step S2-2, compare the potential representation with the vectors in the codebook, find the closest vector as the image label; Step S2-3, the decoder reconstructs the image according to the image mark; Step S2-4: Calculate the loss function and use the gradient descent method to update the encoder, decoder and codebook.

4. The method for converting a multi-stage non-iodine contrast agent CT image to a coronary artery CTA image according to claim 1, characterized in that: In step S3, the specific steps are: Step S3-1, inputting the low-resolution CTA volume into the CTA encoder trained in step S2 to obtain a low-resolution CTA marker; Step S3-2, inputting the low-resolution NCCT volume into the NCCT encoder trained in step S2 to obtain a low-resolution NCCT label; Step S3-3, randomly masking a certain proportion of low-resolution CTA markers to obtain masked low-resolution CTA markers; Step S3-4, inputting the masked low-resolution CTA marker and the low-resolution NCCT marker into the Transformer network, reconstructing the masked low-resolution CTA marker, and updating the Transformer network parameters by minimizing the negative log-likelihood of the masked marker; Step S3-5: input the reconstructed low-resolution CTA marker into the CTA decoder trained in step S2 to obtain a reconstructed low-resolution CTA volume.

5. The method for converting a multi-stage non-iodine contrast agent CT image to a coronary artery CTA image according to claim 1, characterized in that: In step S4, the specific steps are: Step S4-1, taking a low-resolution CTA image in the low-resolution CTA volume, and upsampling it to obtain an upsampled CTA image with a size of 512×512; Step S4-2: during the diffusion process, for the corresponding high-resolution CTA image, gradually adding noise in each diffusion step to blur the image; Step S4-3, during the reverse diffusion process, for each diffusion step, the super-resolution network takes the noisy high-resolution CTA, the diffusion step moment and the upsampled CTA image as a condition as input, predicts the noise of the diffusion step, and updates the super-resolution network parameters by calculating the error between the predicted noise and the actual noise as the loss function.

6. The method for converting a multi-stage non-iodine contrast agent CT image to a coronary artery CTA image according to claim 1, characterized in that: In step S5, the specific steps are: Step S5-1, inputting the low-resolution NCCT volume into the NCCT encoder trained in step S2 to obtain a low-resolution NCCT label; Step S5-2, starting from the blank low-resolution CTA marker, input it and the low-resolution NCCT marker into the Transformer network trained in step S3 to reconstruct the masked low-resolution CTA marker; Step S5-3, inputting the reconstructed low-resolution CTA marker into the CTA decoder trained in step S2 to obtain a reconstructed low-resolution CTA volume; Step S5-4, taking a low-resolution CTA image in the reconstructed low-resolution CTA volume, and upsampling it to obtain an upsampled CTA image with a size of 512×512; Step S5-5, using an image randomly sampled from a Gaussian distribution as an initial inverse diffusion image, predicting the noise of the current diffusion step through the high-resolution network trained in step S4, and obtaining the CTA image of the previous step from the CTA image of the current diffusion step, the diffusion step moment and the predicted noise, and repeating the iteration until a reconstructed high-resolution CTA image is obtained; Step S5-6: Repeat the operations of step S4 and step S5 for each low-resolution CTA image to finally obtain a high-resolution CTA volume.

Images