Method for converting brain magnetic resonance image modalities based on deformation correction consistency

Abstract

The application provides a brain magnetic resonance image modality conversion method based on deformation correction consistency, which has the following characteristics: step S1, performing a pretreatment operation on a source modality image to obtain a pretreated source modality image; step S2, constructing a generative adversarial network, training the generative adversarial network according to existing multiple brain magnetic resonance images imaged in a weighted mode A and corresponding brain magnetic resonance images imaged in a weighted mode B to construct a training set, and constructing a modality conversion model according to the trained generative adversarial network; and step S3, inputting the pretreated source modality image into the modality conversion model to obtain a target modality image. In summary, the method can generate a modality conversion image with higher precision.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision and medical imaging, and particularly relates to a brain magnetic resonance imaging modality conversion method based on deformation correction consistency. BACKGROUND

[0002] Cranial examination refers to quantifying and creating visual representations of the anatomical structure, function and pathological characteristics of the brain directly or indirectly through various imaging techniques with different degrees of specificity and invasiveness. Different modalities of cranial examination images can provide diverse and complementary information, which is of great significance for studying human cognitive activities, neurodegenerative diseases and pathology. Magnetic resonance imaging can obtain different weighted magnetic resonance imaging modalities by setting different scanning parameters for specific task scanning, and is widely used in medical image analysis. Different weighted imaging such as T1 weighted, T2 weighted, PD weighted and FLAIR weighted magnetic resonance imaging modalities have their own unique soft tissue contrast, and these multi-modal magnetic resonance images can provide complementary information. Therefore, combining multi-modal images has better disease prediction ability than single modal images, and has significant effect in tasks such as segmenting brain lesion tissues.

[0003] However, it is usually difficult to obtain all multi-modal images of a patient. The acquisition of multi-modal images faces some limitations, including long acquisition time, expensive examination cost, and image damage caused by patient motion during acquisition and a series of factors. The lack and inconsistency of image data is a common problem in neurology research and clinical diagnosis, and obtaining the missing modality image without additional scanning is a clear requirement of current research.

[0004] In recent years, with the continuous development of deep learning, segmentation and classification tasks based on convolutional neural network feature extraction have developed rapidly. And the generation model based on generative adversarial network performs excellently in the task of image domain transformation, so that the image generation and image conversion tasks in the field of natural images can realize efficient automation and diversification. At the same time, some researches also apply the method based on generative adversarial network to the modality conversion of medical images.

[0005] However, the current modal conversion model has strong dependence on image data, and it is difficult to process the influence of spatial misalignment of the same anatomical structure in the paired images on the model accuracy. In reality, due to different patient positioning and patient motion during imaging, the imaging results of paired images often have spatial misalignment, and it is impossible to achieve one-to-one alignment of voxels on the anatomical structure. Although image correction and image registration can alleviate this problem to a certain extent, small spatial misalignment in details is usually unavoidable. In the current medical image modal conversion task, Pix2Pix based on supervised learning and unsupervised conversion based on cycle consistency are the two most commonly used modes, but both modes have defects. When using supervised learning based on Pix2Pix for image modal conversion, the spatial misalignment in the paired images provides incorrect labels for the model, causing errors to propagate in the model and affecting the accuracy of the modal conversion model. Although the unsupervised model based on cycle consistency does not require paired images for training, the model training is not affected by the spatial misalignment of the image pairs. However, the cycle consistency mode may produce multiple different solutions during training, making it difficult to obtain the most accurate optimal solution during model training optimization.

[0006] In summary, the multi-modal images generated by the modal conversion model trained by the prior art still have a large room for improvement. SUMMARY

[0007] The present application is carried out to solve the above problems, and aims to provide a brain magnetic resonance image modal conversion method based on deformation correction consistency.

[0008] The application provides a brain magnetic resonance image modal conversion method based on deformation correction consistency, which is used for processing a source modal image imaged in a weighted mode A to obtain a target modal image imaged in a weighted mode B, and has the following characteristics: step S1, performing a preprocessing operation on the source modal image to obtain a preprocessed source modal image; step S2, constructing a generative adversarial network, training the generative adversarial network according to existing multiple brain magnetic resonance images imaged in the weighted mode A and corresponding brain magnetic resonance images imaged in the weighted mode B to construct a training set, and constructing a modal conversion model according to the trained generative adversarial network; and step S3, inputting the preprocessed source modal image into the modal conversion model to obtain the target modal image, wherein step S2 comprises the following substeps: step S2-1, constructing a generative adversarial network comprising a spatial transformation network R, a backward correction generator G1, a forward correction generator G2, a backward correction discriminator D1 and a forward correction discriminator D2; step S2-2, performing a preprocessing operation on the brain magnetic resonance image imaged in the weighted mode A to obtain a training image, performing a preprocessing operation on the brain magnetic resonance image imaged in the weighted mode B to obtain a real image, and taking all the training images and corresponding real images as a training set; step S2-3, inputting a training image x A into the spatial transformation network R to obtain a deformation field φ; step S2-4, inputting the training image x A into the backward correction generator G1 to obtain a noisy B-weighted image G1(x A ); step S2-5, resampling the deformation field φ and the B-weighted image G1(x A ) to obtain a backward corrected B-weighted image φ(G1(x A )); step S2-6, resampling the training image x A and the deformation field φ to obtain a forward corrected noise-free A-weighted image φ(x A ); step S2-7, inputting the noise-free A-weighted image φ(x A ) into the forward correction generator G2 to obtain a forward corrected B-weighted image G2(φ(x A )); step S2-8, inputting the B-weighted image G2(φ(x A )) into the forward correction discriminator D2 to obtain a first discriminant classification result; step S2-9, inputting the training image x A and the corresponding real image y B into the forward correction discriminator D2 to obtain a second discriminant classification result; step S2-10, inputting the B-weighted image φ(G1(x A )) into the backward correction discriminator D1 to obtain a third discriminant classification result; and step S2-11, inputting the real image y BThe backward correction discriminator D1 is input, and a fourth discriminant classification result is obtained; in step S2-12, the first discriminant classification result, the third discriminant classification result, the B-weighted image G2 (φ(x A ), the B-weighted image φ(G1(x A )) and the real image y B , are used to calculate a loss function L G , and the parameters of the spatial transformation network R, the backward correction generator G1 and the forward correction generator G2 are optimized through back propagation according to the loss function calculation result; in step S2-13, the first discriminant classification result, the second discriminant classification result, the third discriminant classification result and the fourth discriminant classification result are used to calculate a loss function L D , and the parameters of the backward correction discriminator D1 and the forward correction discriminator D2 are optimized through back propagation according to the loss function calculation result; in step S2-14, steps S2-3 to S2-13 are repeatedly executed until all data in the training set have been used for training, and then step S2-15 is entered; in step S2-15, steps S2-3 to S2-14 are repeatedly executed until a training completion condition is reached, and then a trained generative adversarial network is obtained; in step S2-16, a modal conversion model is constructed according to the trained spatial transformation network R and the forward correction generator G2.

[0009] In the brain magnetic resonance image modal conversion method based on deformation correction consistency provided by the application, the spatial transformation network R can further have the following characteristics: the spatial transformation network R comprises 7 convolutional down-sampling modules C1, 1 convolutional layer C2, 3 residual modules C3, 1 convolutional layer C4, 7 jump connection convolutional up-sampling modules C5 and 1 convolutional module C6 arranged in sequence.

[0010] In the brain magnetic resonance image modal conversion method based on deformation correction consistency provided by the application, the backward correction generator G1 and the forward correction generator G2 can further have the following characteristics: the backward correction generator G1 and the forward correction generator G2 have the same network structure, the corresponding network parameters of the backward correction generator G1 and the forward correction generator G2 are different, and the network structure comprises 1 input convolutional module N1, 2 convolutional down-sampling modules N2, 9 residual modules C3, 2 convolutional up-sampling modules N4 and 1 output convolutional module N5.

[0011] In the brain magnetic resonance image modal conversion method based on deformation correction consistency provided by the application, the backward correction discriminator D1 and the forward correction discriminator D2 can further have the following characteristics: the backward correction discriminator D1 and the forward correction discriminator D2 have the same network structure, the corresponding network parameters of the backward correction discriminator D1 and the forward correction discriminator D2 are different, and the network structure comprises 3 convolutional layer down-sampling modules M1, 1 convolutional down-sampling module M2, 1 convolutional module M3 and 1 maximum pooling layer.

[0012] The modality conversion method for cranial magnetic resonance imaging based on deformation correction consistency provided by the present invention may also have the following features: the preprocessing operation includes size adjustment, center cropping and standardization operation, and the standardization operation is Z-Score standardization processing.

[0013] The cranial magnetic resonance imaging modality conversion method based on deformation correction consistency provided by this invention may also have the following feature: wherein, in step S2-12, the loss function L G The formula for expressing it is: L BCorr =λ Corr ×||φ(G1(x A ))-y B ||1,L FCorr =λ Corr ×||G2(φ(x A ))-y B ||1, In the formula L BCorr For backward correction loss, L FCorr For forward correction loss, L Perc In order to perceive loss, For the forward generator loss, For the backward generator loss, λ Corr , λ Perc , λ Adv1 and λ Adv2 All are regularization parameters, θ j Let C be the parameters of each layer in the existing pre-trained network, j be the j-th layer of the pre-trained network, and C be the parameter of each layer. j H j W j Let D2(G2(φ(x) be the dimension of the feature map of the j-th layer in the pre-trained network.) A ))) represents the first discriminant classification result, D1(φ(G1(x) A ))) is the third discriminant classification result.

[0014] The cranial magnetic resonance imaging modality conversion method based on deformation correction consistency provided by this invention may also have the following feature: wherein, in step S2-13, the loss function L D The formula for expressing it is: In the formula The least-squares adversarial loss is used for the backward correction discriminator. For the least-squares adversarial loss of the forward-corrected discriminator, D1(y) B D2(y) represents the fourth discriminant classification result. B () represents the second discrimination classification result.

[0015] The role and effect of invention

[0016] According to the present invention, the modal conversion method for cranial magnetic resonance imaging based on deformation correction consistency has the following advantages: First, spatial misalignment between different modalities of the same cranial magnetic resonance image is corrected by a spatial transformation network R, reducing the impact of spatial misalignment on the accuracy of the converted image. Second, backward correction is constructed by a backward correction generator G1 and a backward correction discriminator D1, and forward correction is constructed by a forward correction generator G2 and a forward correction discriminator D2. Utilizing the unique optimal solutions of both forward and backward corrections, a constrained generative adversarial network is formed, thereby improving the robustness of the network to noise labeling caused by spatial misalignment during the conversion of incompletely aligned paired images. Therefore, the modal conversion method for cranial magnetic resonance imaging based on deformation correction consistency of the present invention can generate modal conversion images with higher accuracy. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the modal conversion method for cranial magnetic resonance imaging based on deformation correction consistency in an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram illustrating the structure and working principle of the spatial transformation network R in an embodiment of the present invention;

[0019] Figure 3 This is a schematic diagram illustrating the working principle of the residual module C3 in an embodiment of the present invention;

[0020] Figure 4 This is a schematic diagram of the network structure of the backward correction generator G1 and the forward correction generator G2 in an embodiment of the present invention;

[0021] Figure 5 This is a schematic diagram of the network structure of the backward correction discriminator D1 and the forward correction discriminator D2 in an embodiment of the present invention;

[0022] Figure 6 This is a schematic diagram illustrating the working principle and training principle of the generative adversarial network in an embodiment of the present invention. Detailed Implementation

[0023] To make the technical means, creative features, objectives and effects of this invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate the modality conversion method of cranial magnetic resonance imaging based on deformation correction consistency.

[0024] In this embodiment, the cranial magnetic resonance imaging modality conversion method based on deformation correction consistency is used to process the source modality image image imaged in weighted mode A to obtain the target modality image imaged in weighted mode B. Weighted mode A and weighted mode B are different weighting modes. In this embodiment, the weighting mode is the method of obtaining a specific image by setting different scanning parameters in magnetic resonance imaging and then scanning a specific task, including T1 weighting, T2 weighting, PD weighting and FLAIR weighting.

[0025] In this embodiment, weighting method A is T1 weighting, and weighting method B is T2 weighting.

[0026] Figure 1 This is a flowchart illustrating the modal conversion method for cranial magnetic resonance imaging based on deformation correction consistency in an embodiment of the present invention.

[0027] like Figure 1 As shown, the modality conversion method for cranial magnetic resonance imaging based on deformation correction consistency in this embodiment includes the following steps:

[0028] Step S1: Perform preprocessing operations on the source modal image to obtain a preprocessed source modal image.

[0029] The preprocessing operations include resizing, center cropping, and standardization. The standardization operation is Z-Score standardization, meaning that for the source modal image x to be measured, the corresponding preprocessed source modal image x' = (xx... min ) / (x max -x min ).

[0030] Step S2: Construct a generative adversarial network (GAN), then train the GAN using a training set constructed from multiple existing cranial MRI images imaged in weighted mode A and corresponding cranial MRI images imaged in weighted mode B, and finally construct a modality transformation model based on the trained GAN.

[0031] Step S2 includes the following sub-steps:

[0032] Step S2-1: Construct a generative adversarial network that includes a spatial transformation network R, a backward correction generator G1, a forward correction generator G2, a backward correction discriminator D1, and a forward correction discriminator D2.

[0033] Figure 2 This is a schematic diagram illustrating the structure and working principle of the spatial transformation network R in an embodiment of the present invention.

[0034] like Figure 2As shown, the spatial transformation network R includes 7 convolutional downsampling modules C1, 1 convolutional layer C2, 3 residual modules C3, 1 convolutional layer C4, 7 skip-connection convolutional upsampling modules C5, and 1 convolutional module C6 arranged sequentially. Its working principle is as follows: the input of the spatial transformation network R is passed sequentially through the 7 convolutional downsampling modules C1, 1 convolutional layer C2, 3 residual modules C3, 1 convolutional layer C4, 7 skip-connection convolutional upsampling modules C5, and 1 convolutional module C6 to obtain the output of the spatial transformation network R. This process includes 6 skip connections, that is, the output of the convolutional downsampling module C1 connected by the dashed line is merged with the output of the same size of the previous module of the corresponding skip-connection convolutional upsampling module C5, and the fused result is used as the input of the skip-connection convolutional upsampling module C5.

[0035] In this embodiment, seven convolutional downsampling modules C1 are used to extract high-dimensional features from the input and perform corresponding feature transformations. Each convolutional downsampling module C1 contains one linear interpolation layer, one convolutional layer, a LeakyReLU activation function with α = 0.2, and a max-pooling layer. The convolutional layer has a kernel size of 3, a padding size of 1, and a stride of 1. The output channels of the seven convolutional downsampling modules C1 are [32, 64, 64, 64, 64, 64, 64], and the feature map dimensions of the seven convolutional downsampling modules C1 are 32×120×120, 64×60×60, 64×30×30, 64×15×15, 64×8×8, 64×4×4, and 64×2×2, respectively.

[0036] In this embodiment, the kernel size of convolutional layer C2 is 1, the padding size is 0, the stride is 1, the input channels are 64, the output channels are 128, the output feature map size is 128×1×1, and the LeakyReLU activation function α=0.2.

[0037] Figure 3 This is a schematic diagram illustrating the working principle of the residual module C3 in an embodiment of the present invention.

[0038] like Figure 3 As shown, in the residual module C3, the input is processed sequentially through reflection filling, the first convolutional layer, the instance normalization function and the ReLU activation function, reflection filling and the second convolutional layer. The processing result is then added to the input and subjected to the instance normalization function and the ReLU activation function again to obtain the output of the residual module C3.

[0039] In this embodiment, in the residual module C3 of the spatial transformation network R, the padding size for both reflections is 1. The kernel size of the first convolutional layer is 3, the padding size is 0, the stride is 1, the input channels are 128, and the output channels are 128. The kernel size of the second convolutional layer is 3, the padding size is 0, the stride is 1, the input channels are 128, and the output channels are 128. The feature map size obtained through the three residual modules C3 is 128×2×2.

[0040] In this embodiment, the convolutional layer C4 has a kernel size of 1, a padding size of 0, a stride of 1, 128 input channels, 64 output channels, and an output feature map size of 64×1×1. It also includes an instance normalization function and a LeakyReLU activation function with α = 0.2.

[0041] In this embodiment, seven skip-connection convolutional upsampling modules C5 are used to transform and decode high-dimensional features and restore them to the original resolution. Each skip-connection convolutional upsampling module C5 includes one cross-connection channel fusion, a convolutional kernel with a kernel size of 3 and a padding size of 1, an instance normalization layer, a LeakyReLU activation function with α = 0.2, and a linear interpolation layer. The output dimensions of the seven skip-connection convolutional upsampling modules C5 are 64×2×2, 64×4×4, 64×8×8, 64×15×15, 64×30×30, 64×60×60, and 32×120×120, respectively.

[0042] In this embodiment, the convolution module C6 includes an output convolutional layer with a kernel size of 3, a zero-padding size of 1, a stride of 1, an input size of 32×120×120, and an output size of 2×240×240.

[0043] Backward correction generator G1 and forward correction generator G2 have the same network structure, but their corresponding network parameters are different.

[0044] Figure 4 This is a schematic diagram of the network structure of the backward correction generator G1 and the forward correction generator G2 in an embodiment of the present invention.

[0045] like Figure 4 As shown, the network structure includes one input convolutional module N1, two convolutional downsampling modules N2, nine residual modules C3, two convolutional upsampling modules N4, and one output convolutional module N5.

[0046] In this embodiment, the input convolution module N1 includes a mirror padding layer of size 3, a convolutional layer, an instance normalization layer of size 64, and a ReLU activation function. The output of this module is a feature map of size 64×240×240. The convolutional layer is configured with the following settings: input dimension 1, output dimension 64, kernel size 7, padding size 0, and sliding window size 1.

[0047] In this embodiment, the first convolutional downsampling module N2 includes a convolutional layer with a kernel size of 3, a padding size of 1, and a sliding window size of 2. The input dimension of this convolutional layer is 64, and the output dimension is 128. Following this convolutional layer is instance normalization with a size of 128, and after instance normalization, a ReLU activation function is applied. The first convolutional downsampling module N2 outputs a 128×120×120 feature map. The second convolutional downsampling module N2 includes a convolutional layer with a kernel size of 3, a padding size of 1, and a sliding window size of 2. The input dimension of this convolutional layer is 128, and the output dimension is 256. Following this convolutional layer is instance normalization with a size of 256, and after instance normalization, a ReLU activation function is applied. The second convolutional downsampling module N2 outputs a 256×60×60 feature map.

[0048] In this embodiment, in each residual module C3 of the network structure of the backward correction generator G1 and the forward correction generator G2, the padding size for both reflections is 1. The kernel size of the first convolutional layer is 3, the padding size is 0, the stride is 1, the input channels are 256, and the output channels are 256. The kernel size of the second convolutional layer is 3, the padding size is 0, the stride is 1, the input channels are 256, and the output channels are 256. The feature map size obtained through the 9 residual modules C3 is 256×60×60.

[0049] In this embodiment, the first convolutional upsampling module N4 includes a transposed convolutional layer with a kernel size of 3, a padding size of 1, a sliding window size of 2, an input dimension of 256, and an output dimension of 128. Following the transposed convolutional layer is an instance normalization function of size 128, followed by a ReLU activation function. The first convolutional upsampling module N4 ultimately outputs a feature map with a size of 128×120×120. The second convolutional upsampling module N4 also includes a transposed convolutional layer with a kernel size of 3, a padding size of 1, a sliding window size of 2, an input dimension of 128, and an output dimension of 64. Following the transposed convolutional layer is an instance normalization function of size 256, followed by a ReLU activation function. The second convolutional upsampling module N4 ultimately outputs a feature map with a size of 64×240×240.

[0050] In this embodiment, the output convolution module N5 includes a mirror padding of size 3, a convolutional layer, and a Tanh activation function. The final output size is 1×240×240. The input dimension of this convolutional layer is 64, the output dimension is 1, the kernel size is 7, the padding size is 0, and the sliding window size is 1.

[0051] The backward correction discriminator D1 and the forward correction discriminator D2 have the same network structure, but their corresponding network parameters are different.

[0052] Figure 5 This is a schematic diagram of the network structure of the backward correction discriminator D1 and the forward correction discriminator D2 in an embodiment of the present invention.

[0053] like Figure 5 As shown, the network structure includes three convolutional downsampling modules M1, one convolutional downsampling module M2, one convolutional module M3, and one max pooling layer.

[0054] In this embodiment, each of the three convolutional layer downsampling modules M1 includes a convolutional layer with a kernel size of 4, a padding size of 1, and a sliding window size of 2, an instance normalization layer, and a LeakyReLU activation function with α = 0.2. The first convolutional layer downsampling module M1 has an input dimension of 1 and an output dimension of 64 for its convolutional layer, the second convolutional layer downsampling module M1 has an input dimension of 64 and an output dimension of 128 for its convolutional layer, and the third convolutional layer downsampling module M1 has an input dimension of 128 and an output dimension of 256 for its convolutional layer.

[0055] In this embodiment, the convolutional downsampling module M2 includes a convolutional layer, an instance normalization layer, and a LeakyReLU activation function with α = 0.2. The convolutional layer has a kernel size of 4, a padding size of 1, a sliding window size of 1, an input dimension of 256, and an output dimension of 512. The instance normalization layer has a dimension of 512.

[0056] In this embodiment, the convolution module M3 includes a convolutional layer. The kernel size of the convolutional layer (3) is 4, the padding size is 0, the sliding window size is 1, the input dimension is 512, and the output dimension is 1.

[0057] Step S2-2: Preprocess the cranial MRI images imaged in weighted mode A to obtain training images, and preprocess the cranial MRI images imaged in weighted mode B to obtain real images. Then, use all training images and corresponding real images as the training set. In this embodiment, the training images are T1 weighted images with a size of 1×240×240, and the real images are T2 weighted images with a size of 1×240×240.

[0058] Figure 6This is a schematic diagram illustrating the working principle and training principle of the generative adversarial network in an embodiment of the present invention.

[0059] like Figure 6 As shown, the working principle of a generative adversarial network is as follows: training images x... A The deformation field φ is obtained by inputting the spatial transformation network R, and the training image x is then used. A Inputting the backward correction generator G1 yields the B-weighted image G1(x) A ), for B-weighted image G1(x A The B-weighted image φ(G1(x) is obtained by resampling the deformation field φ and the image φ(G1(x)). A )), for training image x A Resampling with the deformation field φ yields a noise-free A-weighted image φ(x) A ), and the noise-free A-weighted image φ(x) A Inputting the forward correction generator G2 yields the B-weighted image G2(φ(x)). A )), the B-weighted image G2(φ(x) A )) and real images y B The first and second classification results are obtained by sequentially inputting the forward correction discriminator D2. The B-weighted image φ(G1(x) is then processed. A )) and real images y B The third and fourth classification results are obtained by sequentially inputting the backcorrection discriminator D1.

[0060] The training principle of generative adversarial networks is as follows: based on the B-weighted image φ(G1(x) A )) and real images y B Calculate the backward correction loss L BCorr According to the B-weighted image G2(φ(x) A )) and real images y B Calculate the forward correction loss L FCorr and perceived loss L Perc Based on the forward correction discriminator D2, the B-weighted image G2(φ(x) is processed. A The true / false classification result, i.e., the first classification result, is used to calculate the forward generator loss. Based on the backward correction discriminant D1, the B-weighted image φ(G1(x) is processed. A The true / false classification result, i.e., the third classification result, is used to calculate the backward generator loss. Based on the third discriminant classification result and the backward correction discriminant D1, the true image y is analyzed. B The true / false classification results, i.e., the fourth classification results, are used to calculate the least squares adversarial loss of the backward correction discriminator. And based on the forward correction discriminator D2, the true image y BThe true / false classification results, i.e., the second and first classification results, are used to calculate the least-squares adversarial loss of the forward-corrected discriminator. L Adv1 Right now and L Adv2 Right now and Then, the backward correction loss L is applied. BCorr Forward correction loss L FCorr Perceived loss L Perc Backward generator loss and forward generator loss The parameters of the backward correction generator G1, the forward correction generator G2, and the spatial transformation network R are optimized, and least-squares adversarial loss is applied through the backward correction discriminator. Least-squares adversarial loss of forward-corrected discriminator The parameters of the backward correction discriminator D1 and the forward correction discriminator D2 are optimized to train the entire generative adversarial network.

[0061] Steps S2-3: Select one training image x from the training set. A The spatial transformation network R is input to obtain the deformation field φ. In this embodiment, the deformation field φ is a vector vector that is related to the training image x. A The images are all the same size and are a matrix with a dimension of 2, that is, the size is 2*240*240.

[0062] Steps S2-4: Transfer the training image x A Inputting the backward correction generator G1 yields a noisy B-weighted image G1(x) A ).

[0063] Steps S2-5: Combine the deformation field φ and the B-weighted image G1(x) A Resampling is performed to obtain the back-corrected B-weighted image φ(G1(x)). A )).

[0064] Steps S2-6: Transfer the training image x A Resampling with the deformation field φ yields a forward-corrected noise-free A-weighted image φ(x). A ).

[0065] Steps S2-7: The noise-free A-weighted image φ(x) is processed... A Inputting the forward correction generator G2 yields the forward-corrected B-weighted image G2(φ(x)). A )).

[0066] Steps S2-8: The B-weighted image G2(φ(x) is processed... AInput the forward correction discriminator D2 to obtain the first discriminant classification result.

[0067] Steps S2-9: Transfer the training image x A Corresponding real image y B Input the forward correction discriminator D2 to obtain the second discriminant classification result.

[0068] Step S2-10, the B-weighted image φ(G1(x) is processed. A Input the backward correction discriminator D1 to obtain the third discriminant classification result.

[0069] Step S2-11, transfer the real image y B Input the backward correction discriminator D1 to obtain the fourth discriminant classification result.

[0070] In this embodiment, the first, second, third, and fourth discrimination classification results are all 0 or 1. 0 indicates that the discriminator determines that the image is not a real image, and 1 indicates that the discriminator determines that the image is a real image.

[0071] Step S2-12, based on the first discrimination classification result, the third discrimination classification result, and the B-weighted image G2(φ(x) A B-weighted image φ(G1(x)) A )) and real images y B Calculate the loss function L G Then, based on the loss function calculation results, the parameters of the spatial transformation network R, the backward correction generator G1, and the forward correction generator G2 are optimized through backpropagation.

[0072] Wherein, the loss function L G The formula for expressing it is:

[0073]

[0074] L BCorr =λ Corr ×||φ(G1(x A ))-y B ||1,

[0075] L FCore =λ Corr ×||G2(φ(x A ))-y B ||1,

[0076]

[0077]

[0078]

[0079] In the formula L BCorr For backward correction loss, L FCorr For forward correction loss, L Perc In order to perceive loss, For the forward generator loss, For the backward generator loss, λ Corr , λ Prec , λ Adv1 and λ Adv2 All are regularization parameters, θ j Let C be the parameters of each layer in the existing pre-trained network, j be the j-th layer of the pre-trained network, and C be the parameter of each layer. j H j W j Let D2(G2(φ(x) be the dimension of the feature map of the j-th layer in the pre-trained network.) A ))) represents the first discriminant classification result, D1(φ(G1(x) A ))) represents the third discriminant classification result. In this embodiment, λ Corr :λ Perc :λ Adv1 :λ Adv2 =100:100:1:1.

[0080] Step S2-13: Calculate the loss function L based on the first, second, third, and fourth classification results. D Then, based on the loss function calculation results, the parameters of the backward correction discriminator D1 and the forward correction discriminator D2 are optimized through backpropagation.

[0081] Wherein, the loss function L D The formula for expressing it is:

[0082]

[0083]

[0084]

[0085] In the formula The least-squares adversarial loss is used for the backward correction discriminator. For the least-squares adversarial loss of the forward-corrected discriminator, D1(y) B D2(y) represents the fourth discriminant classification result. B () represents the second discrimination classification result.

[0086] Step S2-14: Repeat steps S2-3 to S2-13 until all data in the training set has been used for training, then proceed to step S2-15.

[0087] Step S2-15: Repeat steps S2-3 to S2-14 until the training completion condition is met, and a trained generative adversarial network is obtained. In this embodiment, the training completion condition is to complete the preset number of iterative training rounds or for the loss function to converge and stabilize.

[0088] Step S2-16: Construct a mode conversion model based on the trained spatial transformation network R and the forward correction generator G2.

[0089] Step S3: Input the preprocessed source modal image into the modal conversion model to obtain the target modal image.

[0090] In this embodiment, to verify the effectiveness of the modality conversion method for cranial magnetic resonance imaging based on deformation correction consistency of the present invention, the modality conversion model of this embodiment is used as the model of the present invention and compared with the Pix2Pix model, CycleGAN model and RegGAN model constructed by the existing Pix2Pix method, CycleGAN method and RegGAN method.

[0091] The T1-weighted and T2-weighted images in the existing BRATS2018 dataset were divided into training and validation sets. The initial models of this invention, Pix2Pix, CycleGAN, and RegGAN were trained on the training set using the same generator-discriminator architecture, resulting in well-trained models. Then, based on 979 T1-weighted images and corresponding T2-weighted images in the validation set, the Normalized Mean Squared Error (NMSE), Peak Signal-to-Noise Ratio (PSNR), and Structural Similarity (SSIM) of each model were calculated. The results of these three metrics for each model are shown in the table below.

[0092] Model name NMSE PSNR SSIM Pix2Pix model 0.067 28.75 0.926 CycleGAN model 0.102 26.78 0.904 RegGAN model 0.081 27.90 0.919 Invention model 0.058 29.34 0.937

[0093] The first column of the table above lists the model names, the second column lists the NMSE (Neural Mode Estimate) results, the third column lists the PSNR (Power Score Ratio) results, and the fourth column lists the SSIM (Short Score) results. For example, the cell in the fifth row and second column indicates that the SSIM of the model in this invention is 0.058 in the validation set. As shown in the table, the model in this invention achieves the best calculation results in all three metrics compared to the other three models. Therefore, the cranial magnetic resonance imaging modality conversion method based on deformation correction consistency in this invention has better image modality conversion performance than existing methods.

[0094] The role and effect of the embodiments

[0095] According to the cranial magnetic resonance imaging modality conversion method based on deformation correction consistency involved in this embodiment, on the one hand, the spatial misalignment between different modalities of the same cranial magnetic resonance image is corrected by a spatial transformation network R, reducing the impact of spatial misalignment on the accuracy of the converted image. On the other hand, backward correction is constructed by a backward correction generator G1 and a backward correction discriminator D1, and forward correction is constructed by a forward correction generator G2 and a forward correction discriminator D2. Utilizing the unique optimal solutions of the forward and backward corrections, a constrained generative adversarial network is formed, thereby improving the robustness of the network to noise labeling caused by spatial misalignment when converting incompletely aligned paired images. In summary, this method can generate modality-converted images with higher accuracy.

[0096] The above embodiments are preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention.

Claims

1. A method for brain magnetic resonance image modality conversion based on deformation correction consistency, for processing a source modality image imaged in a weighted manner A to obtain a target modality image imaged in a weighted manner B, characterized in that, The method comprises the following steps: Step S1, performing a preprocessing operation on the source modality image to obtain a preprocessed source modality image; Step S2, constructing a generative adversarial network, training the generative adversarial network according to an existing plurality of brain magnetic resonance images imaged in a weighted manner A and corresponding brain magnetic resonance images imaged in a weighted manner B, and constructing a modality conversion model according to the trained generative adversarial network; Step S3, inputting the preprocessed source modality image into the modality conversion model to obtain the target modality image, The step S2 comprises the following sub-steps: Step S2-1, constructing the generative adversarial network comprising a spatial transformation network R, a backward correction generator G1, a forward correction generator G2, a backward correction discriminator D1, and a forward correction discriminator D2; Step S2-2, performing the preprocessing operation on the brain magnetic resonance images imaged in the weighted manner A to obtain training images, performing the preprocessing operation on the brain magnetic resonance images imaged in the weighted manner B to obtain real images, and taking all the training images and corresponding real images as the training set; Step S2-3, obtaining a training image x in the training set A inputting the spatial transformation network R to obtain a deformation field φ; Step S2-4, generating a noisy B-weighted image G1(x A inputting the backward correction generator G1, obtaining a noisy B-weighted image G1(x A ); Step S2-5, re-sampling the deformation field φ and the B-weighted image G1(x A ) to obtain the backward corrected B-weighted image φ(G1(x A )). Step S2-6, resample the training image x A and the deformation field φ to obtain a forward corrected noise-free A-weighted image φ(x A ). Step S2-7, inputting the noiseless A-weighted image φ(x A ) into the forward correction generator G2 to obtain a forward-corrected B-weighted image G2(φ(x A )). Step S2-8, input the B-weighted image G2(φ(x A )) into the forward correction discriminator D2 to obtain a first discriminant classification result; Step S2-9, inputting the training image x A corresponding real image y B inputting the forward correction discriminator D2 to obtain a second discriminant classification result; Step S2-10, inputting the B-weighted image φ(G1(x A )) into the backward correction discriminator D1 to obtain a third discriminant classification result; Step S2-11, obtaining the real image y B The backward correction discriminator D1 is inputted to obtain a fourth discrimination classification result. Step S2-12, according to the first discriminant classification result, the third discriminant classification result, the B-weighted image G2(φ(x A )), the B-weighted image φ(G1(x A )) and the real image y B , calculate the loss function L G , and then optimize the parameters of the spatial transformation network R, the backward correction generator G1 and the forward correction generator G2 through back propagation according to the loss function calculation result. Step S2-13, calculating a loss function L according to the first discriminant classification result, the second discriminant classification result, the third discriminant classification result and the fourth discriminant classification result D According to the loss function calculation result, the parameters of the backward correction discriminator D1 and the forward correction discriminator D2 are optimized through back propagation. Step S2-14, repeatedly performing steps S2-3 to S2-13 until all data in the training set have been used for training, and then proceeding to step S2-15; Step S2-15, repeatedly performing steps S2-3 to S2-14 until a training completion condition is reached, and then obtaining the trained generative adversarial network; Step S2-16, constructing the modality conversion model according to the trained spatial transformation network R and the forward correction generator G2.

2. The brain magnetic resonance image modality conversion method based on deformation correction consistency according to claim 1, wherein: wherein The spatial transformation network R comprises 7 convolutional down-sampling modules C1, 1 convolutional layer C2, 3 residual modules C3, 1 convolutional layer C4, 7 jump connection convolutional up-sampling modules C5, and 1 convolutional module C6 arranged in sequence.

3. The brain magnetic resonance image modality conversion method based on deformation correction consistency according to claim 1, wherein: wherein The backward correction generator G1 and the forward correction generator G2 have the same network structure, the corresponding network parameters of the backward correction generator G1 and the forward correction generator G2 are different, and the network structure comprises 1 input convolutional module N1, 2 convolutional down-sampling modules N2, 9 residual modules C3, 2 convolutional up-sampling modules N4, and 1 output convolutional module N5.

4. The brain magnetic resonance image modality conversion method based on deformation correction consistency according to claim 1, wherein: wherein The backward correction discriminator D1 and the forward correction discriminator D2 have the same network structure, the corresponding network parameters of the backward correction discriminator D1 and the forward correction discriminator D2 are different, and the network structure comprises 3 convolutional layer down-sampling modules M1, 1 convolutional down-sampling module M2, 1 convolutional module M3, and 1 maximum pooling layer.

5. The method of claim 1, wherein the pre-processing operation comprises a size adjustment, a center cropping, and a normalization operation. wherein, 6. The method of claim 1, wherein the normalization operation is a Z-Score normalization.

7. The method of claim 1, wherein the registration operation comprises a rigid registration and a non-rigid registration. ​ wherein, In the step S2-12, the loss function L G The expression formula is: L BCorr = λ Corr ×||φ(G1(x A ))-y B ||1, L FCorr = λ Corr ×||G2(φ(x A ))-y B ||1, wherein L BCorr is a backward correction loss, L FCorr is a forward correction loss, L Perc is a perception loss, is a forward generator loss, is a backward generator loss, λ Corr , λ Perc , λ Adv1 , and λ Adv2 are all regularization parameters, θ j are parameters of each layer in the existing pre-trained network, j is the jth layer of the pre-trained network, C j H j W j is the size dimension of the jth layer of the pre-trained network, D2(G2(φ(x A )) is the first discriminant classification result, and D1(φ(G1(x A )) is the third discriminant classification result. ​ wherein In the step S2-13, the loss function L D The expression formula is: In the formula is the least square adversarial loss of the backward correction discriminator, is the least square adversarial loss of the forward correction discriminator, D1(y B ) is the fourth discriminant classification result, D2(y B ) is the second discriminant classification result.

Images