Image conversion method from computed tomography to magnetic resonance imaging

Abstract

The invention discloses an image conversion method from computed tomography to magnetic resonance imaging. The image conversion method comprises the following steps: step 1, data acquisition; 2, preprocessing the image; step 3, feature extraction; step 4, generating an MRI image; step 5, true and false discrimination; step 6, verifying circulation consistency; 7, model training; and 8, generating an image. The method has the characteristics of high efficiency, high fidelity, no need of registration, robustness and the like, can quickly generate a high-quality MRI image, retains the structural information of an original CT image, and meets the requirements of medical diagnosis.

Description

Technical Field

[0001] The present invention relates to the technical field of magnetic resonance imaging, and in particular to a method for converting computer tomography (CT) images to magnetic resonance imaging (MRI) images. Background Art

[0002] Traditional generative adversarial networks (GANs) consist of two main components: a generator and a discriminator. The goal of the generator is to produce data that is as close to the real data distribution as possible, while the discriminator tries to distinguish the generated data from the real data. In this setting, the discriminator is only used to provide adversarial feedback to the generator, and its learned data representation is usually discarded after training.

[0003] Existing magnetic resonance imaging technology has some limitations in imaging time, equipment cost, sensitivity to metal implants and motion artifacts, and image detail information, which limits its use in certain application scenarios. For example, MRI imaging takes a long time and requires patients to remain still for a long time, which is challenging for some patients who cannot cooperate. In addition, the high cost of MRI equipment has led to a relatively low penetration rate, making it difficult to meet the needs of all patients. Summary of the invention

[0004] In order to overcome the above technical problems, the purpose of the present invention is to provide a method for converting computed tomography to magnetic resonance imaging. The method has the characteristics of high efficiency, high fidelity, no need for registration and robustness, and can quickly generate high-quality MRI images and retain the structural information of the original CT images to meet the needs of medical diagnosis.

[0005] The technical solution adopted by the present invention is:

[0006] A method for converting computer tomography to magnetic resonance imaging comprises the following steps:

[0007] Step 1: Data collection;

[0008] First, a CT scan is performed on the patient to obtain CT image data of the patient;

[0009] Step 2: Image preprocessing;

[0010] Preprocess the CT images, including denoising and standardization operations;

[0011] Step 3: Feature extraction;

[0012] The preprocessed CT image is input into the feature extraction network; the feature extraction network is composed of a multi-layer convolutional neural network, which can extract important feature information from the CT image, and the feature information is texture, shape and edge;

[0013] Step 4: MRI image generation;

[0014] The feature information extracted by the feature extraction network is input into the image generation network; the feature information is converted into an MRI image, and an image having features similar to the real MRI image is generated;

[0015] Step 5: Authenticity determination;

[0016] The generated MRI image is input into the authenticity discrimination network to determine whether the input image is a real MRI image or a fake image generated by the image generation network;

[0017] Step 6: Cycle consistency verification;

[0018] The generated MRI image is input into the feature extraction network again, and the CT image is regenerated through the image generation network; the regenerated CT image is compared with the original CT image to ensure the consistency of the image content; the cycle consistency verification helps to ensure that the structural information of the original CT image is not lost during the style transfer process.

[0019] Step 7: Model training;

[0020] The feature extraction network, image generation network, and authenticity discrimination network are jointly trained through adversarial learning strategy and cycle consistency loss function. The adversarial learning strategy enables the image generation network to generate images that are difficult to be distinguished by the authenticity discrimination network, while training the authenticity discrimination network to accurately distinguish between real images and fake images; the cycle consistency loss ensures that the generated MRI images maintain consistency in content after the CT images are regenerated by the feature extraction network and the image generation network;

[0021] Step 8: Image generation;

[0022] After the training is completed, the trained model is used to convert CT images into MRI images. The converted MRI images will have similar visual features and structural information as the real MRI images.

[0023] The converted MRI images can be used for medical diagnosis and research.

[0024] In step 3, the feature extraction network includes an initial convolution layer, a normalization and LeakyReLU activation function, and a downsampling convolution block;

[0025] The initial convolution layer includes a convolution layer with 64 3x3 convolution kernels, a step size of 1, and no padding, to perform preliminary feature extraction on the input image;

[0026] This layer is followed by normalization and LeakyReLU activation functions to add nonlinear processing capabilities to the input image after preliminary feature extraction;

[0027] The normalization and LeakyReLU activation function are followed by downsampling convolution blocks. There are four downsampling convolution blocks, each of which includes a convolution layer with a size of 4x4 and a step size of 2, which is used to downsample and extract higher-level features. Each convolution layer is followed by a batch normalization layer and a LeakyReLU activation function to maintain the training stability of the network and introduce nonlinearity.

[0028] In step 4, the structure of the image generation network is symmetrical to the feature extraction network but in the opposite direction, and includes multiple convolutional layers, which gradually map the features extracted by the feature extraction network to the target domain, that is, the visual representation of the MRI image. The network includes convolutional layers and upsampling layers, which gradually enlarge the feature map to the resolution of the original image.

[0029] In step 5, the authenticity discrimination network is used to discriminate the authenticity of the image. The network consists of 4 convolutional layers, followed by a batch normalization layer and a LeakyReLU activation function, and finally a fully connected layer, which converts the high-dimensional feature vector output by the convolutional layer into a single discrimination score, indicating the probability that the input image is a real image.

[0030] In step 7, an adversarial learning strategy is used to train the image generation network and the authenticity discrimination network to compete, wherein the image generation network is committed to generating images that are highly similar to actual magnetic resonance imaging (MRI) to mislead the authenticity discrimination network into mistaking them for real MRI images; at the same time, the authenticity discrimination network strives to distinguish between images output by the image generation network and real MRI images, thereby improving its ability to identify real and generated images.

[0031] The expression of the adversarial learning strategy is:

[0032]

[0033] Among them, x represents the data distribution p data The real CT image, y represents the image from the data distribution p data The real MRI image, G x→y is the image generation network from CT to MRI, E x→y is the feature extraction network from CT to MRI, D y It is a combination of MRI to CT feature extraction network and authenticity discrimination network. During training, the feature extraction network and the authenticity discrimination network are trained together.

[0034] In step 7, the cycle consistency loss is used to ensure that the information of the original image can be retained during the conversion from the CT image to the MRI image and then converted back to the CT image, thereby maintaining consistency in content.

[0035] The expression of cycle consistency loss is:

[0036]

[0037] Among them, G x→y is the image generation network from CT to MRI, G y→x It is the corresponding inverse generator from MRI to CT.

[0038] Comprehensive loss function, used for overall training of the entire network;

[0039] Combining the above loss terms, the final optimization target of the model is the total loss function, which is the weighted sum of each loss term:

[0040] L total =λ adv L adv +λ cyc L cyc

[0041] Among them, λ adv and λ cyc They represent the weight coefficients of adversarial loss and cycle consistency loss, respectively, which are used to balance the contribution of different loss terms to the total loss.

[0042] Beneficial effects of the present invention:

[0043] The method steps and model of the present invention provide an efficient, high-quality, registration-free CT to MRI image conversion method, which can effectively solve the limitations of existing MRI image acquisition and expand the application scope of MRI images, and has important clinical application value.

[0044] The present invention uses a feature extraction network, and the model can effectively extract features related to MRI images, such as texture and shape, from CT images of cervical cancer patients. These features are used to guide the image generation network to generate high-quality MRI images and ensure that the generated images are consistent with the original CT images in content, thereby achieving unsupervised image conversion from CT to MRI.

[0045] The present invention uses an image generation network, which can convert the feature information extracted by the feature extraction network into the visual representation of the MRI image in the target domain. The network gradually restores the resolution and detail information of the image through a series of convolution and upsampling operations, and finally generates a realistic MRI image.

[0046] The present invention uses a true-false discrimination network, which can evaluate the quality of generated MRI images and provide adversarial feedback to the image generation network. The true-false discrimination network learns to distinguish between real MRI images and generated MRI images, thereby guiding the image generation network to continuously optimize its generation results, making them more realistic and natural.

[0047] Through the adversarial learning strategy, the model of the present invention can effectively improve the realism of generated MRI images, so that the generative network can generate images that are more difficult to distinguish from real images, while enhancing the ability of the discriminative network, thereby improving the overall quality of the generated images.

[0048] Through cycle consistency loss, the model of the present invention can ensure that the structural information of the image is preserved during the style conversion process, so that the converted image is consistent with the original image in content, thereby enhancing the reliability of image conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of the overall model architecture of the present invention.

[0050] Figure 2 It is a schematic diagram of generating MRI visualization results according to the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] like Figure 1 As shown, a method for converting a computer tomography image to a magnetic resonance imaging image comprises the following steps:

[0053] Step 1: Data collection;

[0054] First, the patient is subjected to a CT scan to obtain CT image data of the patient; the CT scan can provide high-resolution anatomical structure information, such as the shape and position of bones and organs.

[0055] Step 2: Image preprocessing;

[0056] The CT images are preprocessed, including denoising and standardization operations; the denoising operation effectively removes random noise in the CT images by using the non-local mean filtering algorithm, thereby improving the clarity of the image. The standardization operation ensures the consistency of the grayscale value of the image in the entire data set, which helps to eliminate the image differences caused by different scanning devices or conditions, and provides a unified benchmark for subsequent image analysis and processing.

[0057] To improve image quality and reduce errors in subsequent conversion processes.

[0058] Step 3: Feature extraction;

[0059] The preprocessed CT image is input into the feature extraction network; the feature extraction network consists of a multi-layer convolutional neural network, which can extract important feature information from the CT image, including texture, shape and edge; these feature information will be used for subsequent MRI image generation.

[0060] Step 4: MRI image generation;

[0061] The feature information extracted by the feature extraction network is input into the image generation network; the feature information is converted into an MRI image, and an image having features similar to the real MRI image is generated;

[0062] The image generation network is also composed of multi-layer convolutional neural networks, but its structure is opposite to that of the feature extraction network.

[0063] Step 5: Authenticity determination;

[0064] The generated MRI image is input into the authenticity discrimination network to determine whether the input image is a real MRI image or a fake image generated by the image generation network;

[0065] The authenticity discrimination network is composed of a multi-layer convolutional neural network. The training goal of the authenticity discrimination network is to improve its discrimination ability, thereby promoting the image generation network to generate more realistic MRI images.

[0066] Step 6: Cycle consistency verification;

[0067] The generated MRI image is input into the feature extraction network again, and the CT image is regenerated through the image generation network; the regenerated CT image is compared with the original CT image to ensure the consistency of the image content; the cycle consistency verification helps to ensure that the structural information of the original CT image is not lost during the style transfer process.

[0068] Step 7: Model training;

[0069] The feature extraction network, image generation network, and authenticity discrimination network are jointly trained through adversarial learning strategy and cycle consistency loss function. The adversarial learning strategy enables the image generation network to generate images that are difficult to be distinguished by the authenticity discrimination network, while training the authenticity discrimination network to accurately distinguish between real images and fake images; the cycle consistency loss ensures that the generated MRI images maintain consistency in content after the CT images are regenerated by the feature extraction network and the image generation network;

[0070] Step 8: Image generation;

[0071] After the training is completed, the trained model is used to convert CT images into MRI images. The converted MRI images will have similar visual features and structural information as the real MRI images.

[0072] The converted MRI images can be used for medical diagnosis and research.

[0073] The present invention is composed of a feature extraction network, an image generation network and a true-false discrimination network. The feature extraction network is used for feature extraction and provides input information for the image generation network and the true-false discrimination network. The design of the model makes the model structure more compact, reduces the complexity of the model, and the feature extraction network is trained more effectively.

[0074] The model of the present invention adopts a shared feature extraction network architecture; the feature extraction network consists of a multi-layer convolutional network, which is designed to extract useful feature representations from the input image.

[0075] Specifically, this shared architecture includes the following key components.

[0076] Initial convolution layer: A convolution layer with 64 3x3 kernels, a stride of 1, and no padding. The purpose is to perform preliminary feature extraction on the input image.

[0077] This layer is followed by batch normalization and LeakyReLU activation function to increase nonlinear processing capabilities.

[0078] Batch Normalization (BN) and LeakyReLU activation functions are commonly used processing layers in deep learning. They usually follow the convolutional layer to enhance the training efficiency and performance of the model. Here is how to combine expressions to illustrate these operations:

[0079] First, the purpose of batch normalization is to normalize the output of the convolutional layer so that the distribution of each feature map is more stable, which helps to speed up the training process and reduce the gradient disappearance problem. The basic steps of batch normalization can be expressed as:

[0080] Calculate the mean (μ) and variance (σ) of each feature map in the current batch 2 ):

[0081]

[0082]

[0083] Where m is the batch size, x i is each element in the feature map.

[0084] Normalize the feature map:

[0085]

[0086] Here, ò is a small constant used for numerical stability.

[0087] The normalized feature map is scaled and translated by learnable parameters γ and β:

[0088]

[0089] Next, the LeakyReLU activation function is used to introduce nonlinearity, and its expression is:

[0090]

[0091] Here, α is a small positive number called the leakage coefficient (usually set to 0.01).

[0092] Combining these two operations, we can apply them to the output (O) of the convolutional layer. Assuming (O) is the output of the convolutional layer, then the output (O) after batch normalization and LeakyReLU activation function processing activated ) can be expressed as:

[0093] Batch Normalization:

[0094]

[0095]

[0096] LeakyReLU activation:

[0097] O activated =f(O bn )

[0098] Finally, O activated It will be used as the input of the next layer, which not only maintains the nonlinearity of the features, but also improves the stability of training through batch normalization.

[0099] Downsampling convolution block: Next are four downsampling convolution blocks, each of which includes a convolution layer with a kernel size of 4x4 and a stride of 2, which doubles with depth to downsample and extract higher-level features. Each convolution layer is followed by a batch normalization layer and a LeakyReLU activation function to maintain the training stability of the network and introduce nonlinearity.

[0100] The above parts together constitute a feature extraction network to extract important feature information from CT images, such as texture, shape, and edge, etc. These feature information are the basis for MRI image generation.

[0101] The design of the image generation network adopts a symmetrical strategy with the feature extraction network, but in the opposite direction, focusing mainly on mapping (converting) the features extracted by the feature extraction network into specific visual representations of the target domain (MRI images).

[0102] The authenticity discrimination network is used to discriminate the authenticity of an image. The network consists of four convolutional layers, followed by a batch normalization layer and a LeakyReLU activation function. Finally, there is a fully connected layer that converts the high-dimensional feature vector output by the convolutional layer into a single discriminant score, which represents the probability that the input image is a real image. The process is as follows:

[0103] Assume that the feature vector output by the convolutional layer is F, and its dimension is [b,h,w,c], where b is the batch size, h and w are the height and width of the feature map, and c is the number of channels of the feature map.

[0104] First, flatten the feature map F into a one-dimensional vector F flat :

[0105] F flat =Flatten(F)

[0106] The dimension of the flattened vector is [b, d], where d = h × w × c.

[0107] Then, the fully connected layer transforms F through the weight matrix W and the bias vector b. flat Convert to discriminant score S:

[0108] S=W·F flat +b

[0109] Here, the dimension of W is [d, 1] and the dimension of b is [1].

[0110] Finally, the Sigmoid activation function is usually used to convert the discriminant score (S) into a probability (P), which represents the probability that the input image is a real image:

[0111]

[0112] The fully connected layer converts the high-dimensional feature vector into a single discriminant score through matrix multiplication and bias addition, and outputs the probability value through the Sigmoid function.

[0113] Model Training

[0114] In the present invention, the model training adopts an adversarial learning strategy, which aims to find the optimal network parameters to achieve high-quality conversion of unsupervised cervical cancer CT images to MRI images.

[0115] Among them, the feature extraction network and the authenticity discrimination network are jointly trained, and the loss function adopted by the overall model is composed of the following parts, which promotes the performance of the model in generating realistic images and learning robust features.

[0116] Adversarial loss is used to train the image generation network and the authenticity discrimination network to compete, where the image generation network is committed to generating images that are highly similar to actual magnetic resonance imaging (MRI) to mislead the authenticity discrimination network into mistaking them for real MRI images; at the same time, the authenticity discrimination network strives to distinguish between the images output by the image generation network and the real MRI images, thereby improving its ability to identify real and generated images.

[0117] This loss term enables the image production network to produce images that are difficult to be distinguished by the authenticity discrimination network, while training the authenticity discrimination network to accurately distinguish between the input real images and the fake images generated by the authenticity discrimination network.

[0118] The formalization of adversarial loss is obtained by considering the objective functions of the generator and discriminator in the generative adversarial network. Specifically, it is derived based on the two principles of maximizing the probability that the discriminator correctly classifies real and generated images, and maximizing the probability that the generator generates images that are misclassified by the discriminator.

[0119]

[0120] Among them, x represents the data distribution p data The real CT image, y represents the image from the data distribution p data The real MRI image, G x→y is the image generation network from CT to MRI, E x→y is the feature extraction network from CT to MRI, D y It is a combination of MRI to CT feature extraction network and authenticity discrimination network. During training, the feature extraction network and the authenticity discrimination network are trained together.

[0121] Cycle consistency loss is used to ensure that the information of the original image can be retained when converting from CT images to MRI images and then back to CT images, thereby maintaining content consistency. This loss function improves the authenticity and reliability of the converted image by forcing the generator to maintain the stability of the image content during the conversion process, making the generated MRI image not only visually realistic, but also consistent with the original CT image in terms of anatomical structure.

[0122] This loss helps ensure that the structural information of the original image is not lost during the style transfer process. It is expressed in the following formula. It is mainly based on the concept of cycle consistency. It measures the difference between the image before and after conversion through mathematical norms, and ensures the performance of the conversion network by minimizing this difference.

[0123]

[0124] Among them, G x→y is the image generation network from CT to MRI, G y→x It is the corresponding inverse generator from MRI to CT.

[0125] Comprehensive loss function, used for overall training of the entire network.

[0126] Combining the above loss terms, the final optimization target of the model is the total loss function, which is the weighted sum of each loss term:

[0127] L total =λ adv L adv +λ cyc L cyc

[0128] Among them, λ adv and λ cyc They represent the weight coefficients of adversarial loss and cycle consistency loss, respectively, which are used to balance the contribution of different loss terms to the total loss.

[0129] Example:

[0130] This study used imaging data of 50 patients with cervical cancer from Xijing Hospital, of which 45 patients were used as training sets. After processing, a total of 2844 CT images and 857 MRI images were obtained for model training. Another 5 patients were selected as test sets, and after non-rigid registration processing, 279 pairs of CT-MRI images were obtained as test sets. The results will be evaluated from the perspectives of indicators and visualization. And the results will be compared with the PIX2PIX model.

[0131] Table 1 Results of generating MRI indicators

[0132] SSIM(%) PSNR MAE PIX2PIX 63.21±6.10 17.32±2.20 23.26±3.39 This research model 71.99±7.6 20.16±1.76 20.31±4.72

[0133] From the indicator results, we can see that compared with the PIX2PIX model, the model in this study has improved SSIM by 13%, PSNR by 16.40%, and MAE by 12.68%. Figure 2 It can be seen that the model proposed in this study can better perform modality conversion from CT to MRI, and the generated MRI images contain more and richer detailed information.

Claims

1. A method for converting computer tomography to magnetic resonance imaging, characterized in that: The following steps are involved: Step 1: Data collection; First, a CT scan is performed on the patient to obtain CT image data of the patient; Step 2: Image preprocessing; Preprocess the CT images, including denoising and standardization operations; Step 3: Feature extraction; Input the preprocessed CT image into the feature extraction network; Extract important feature information from CT images, including texture, shape and edge; Step 4: MRI image generation; The feature information extracted by the feature extraction network is input into the image generation network; the feature information is converted into an MRI image, and an image having features similar to the real MRI image is generated; Step 5: Authenticity determination; Input the generated MRI image into the authenticity discrimination network; Used to determine whether the input image is a real MRI image or a fake image generated by the image generation network; Step 6: Cycle consistency verification; The generated MRI image is input into the feature extraction network again, and the CT image is regenerated through the image generation network; the regenerated CT image is compared with the original CT image to ensure the consistency of the image content; the cycle consistency verification helps to ensure that the structural information of the original CT image is not lost during the style transfer process. Step 7: Model training; The feature extraction network, image generation network, and authenticity discrimination network are jointly trained through adversarial learning strategy and cycle consistency loss function. Step 8: Image generation; After the training is completed, the trained model is used to convert CT images into MRI images. The converted MRI images will have similar visual features and structural information as the real MRI images.

2. The method for converting computer tomography to magnetic resonance imaging according to claim 1, characterized in that: In step 3, the feature extraction network includes an initial convolution layer, a normalization and LeakyReLU activation function, and a downsampling convolution block; The initial convolution layer includes a convolution layer with 64 3x3 convolution kernels, a step size of 1, and no padding, to perform preliminary feature extraction on the input image; This layer is followed by normalization and LeakyReLU activation functions to add nonlinear processing capabilities to the input image after preliminary feature extraction; The normalization and LeakyReLU activation function are followed by downsampling convolution blocks. There are four downsampling convolution blocks, each of which includes a convolution layer with a size of 4x4 and a step size of 2, which is used to downsample and extract higher-level features. Each convolution layer is followed by a batch normalization layer and a LeakyReLU activation function to maintain the training stability of the network and introduce nonlinearity.

3. The method for converting computer tomography to magnetic resonance imaging according to claim 1, characterized in that: In step 4, the structure of the image generation network is symmetrical to the feature extraction network but in the opposite direction, and includes multiple convolutional layers, which gradually map the features extracted by the feature extraction network to the target domain, that is, the visual representation of the MRI image. The network includes convolutional layers and upsampling layers, which gradually enlarge the feature map to the resolution of the original image.

4. The method for converting computer tomography to magnetic resonance imaging according to claim 1, characterized in that: In step 5, the authenticity discrimination network is used to discriminate the authenticity of the image. The network consists of 4 convolutional layers, followed by a batch normalization layer and a LeakyReLU activation function, and finally a fully connected layer, which converts the high-dimensional feature vector output by the convolutional layer into a single discrimination score, indicating the probability that the input image is a real image.

5. The method for converting computer tomography to magnetic resonance imaging according to claim 1, characterized in that: In step 7, an adversarial learning strategy is used to train the image generation network and the authenticity discrimination network to compete, wherein the image generation network is committed to generating images that are highly similar to actual magnetic resonance imaging (MRI) to mislead the authenticity discrimination network into mistaking them for real MRI images; at the same time, the authenticity discrimination network strives to distinguish between images output by the image generation network and real MRI images, thereby improving its ability to identify real and generated images.

6. The method for converting computer tomography to magnetic resonance imaging according to claim 5, characterized in that: The expression of the adversarial learning strategy is: Among them, x represents the data distribution p data The real CT image, y represents the image from the data distribution p data The real MRI image, G x→y is the image generation network from CT to MRI, E x→y is the feature extraction network from CT to MRI, D y It is a combination of MRI to CT feature extraction network and authenticity discrimination network. During training, the feature extraction network and the authenticity discrimination network are trained together.

7. The method for converting computer tomography to magnetic resonance imaging according to claim 6, characterized in that: In step 7, the cycle consistency loss is used to ensure that the information of the original image can be retained during the conversion from the CT image to the MRI image and then converted back to the CT image, thereby maintaining consistency in content.

8. The method for converting computer tomography to magnetic resonance imaging according to claim 7, characterized in that: The expression of cycle consistency loss is: Among them, G x→y is the image generation network from CT to MRI, G y→x It is the corresponding inverse generator from MRI to CT.

9. The method for converting computer tomography to magnetic resonance imaging according to claim 8, characterized in that: Comprehensive loss function, used for overall training of the entire network; Combining the above loss terms, the final optimization target of the model is the total loss function, which is the weighted sum of each loss term: L total =λ adv L adv +λ cyc L cyc Among them, λ adv and λ cyc They represent the weight coefficients of adversarial loss and cycle consistency loss, respectively, which are used to balance the contribution of different loss terms to the total loss.

10. The converted MRI image obtained according to any one of the methods of claims 1 to 9, characterized in that: The converted MRI images are used in medical diagnosis and research.

Images