MRI (Magnetic Resonance Imaging) image registration generation method based on Cycle consistency framework

Abstract

The invention discloses an MRI (Magnetic Resonance Imaging) image registration generation method based on a Cycle consistency framework, which comprises the following steps of: inputting a T1 weighted image into an image generation network to obtain a T2 weighted image, inputting the T2 weighted image generated by the image generation network and an original T2 weighted image into an image registration network to obtain an aligned T2 weighted image, and outputting the aligned T2 weighted image to a Cycle consistency framework; and inputting the aligned T2 weighted image into an image generation network to obtain a T1 weighted image, and inputting the T1 weighted image generated by the image generation network and the original T1 weighted image into an image registration network to obtain an aligned T1 weighted image. According to the invention, the defects in the prior art can be improved, and the generation quality of the MRI image is improved.

Description

Technical Field

[0001] This invention relates to the field of magnetic resonance image processing technology, and in particular to an MRI image registration generation method based on the Cycle consistency framework. Background Technology

[0002] Magnetic resonance imaging (MRI) has become one of the core technologies in brain imaging diagnosis because it can provide high-resolution soft tissue contrast images and is widely used in neuroscience, neurosurgery, and the diagnosis and monitoring of various brain diseases. MRI technology provides clinicians with multi-dimensional tissue information through different weighted imaging modalities, such as T1-weighted imaging (T1WI) and T2-weighted imaging (T2WI). T1WI images typically show the anatomical structures of the brain, especially the contrast between white and gray matter, while T2WI can clearly reveal pathological changes such as cerebrospinal fluid, tumors, or edema. Therefore, in neurological imaging evaluation, T1WI and T2WI often need to complement each other to allow doctors to have a comprehensive understanding of the state of lesion areas and normal tissues. Although multimodal imaging provides valuable information for clinical diagnosis, modality loss remains a common problem in practical applications. Furthermore, while multiple modal images can provide a more comprehensive perspective, it is difficult for patients to remain completely still during long scans. Physiological factors, such as breathing, heart rate, and the degree of patient relaxation, often introduce small motion artifacts. These factors cause spatial displacement between different modal images, thus affecting image accuracy and potentially adversely affecting disease diagnosis. Therefore, how to solve these problems, especially in brain MRI images, has become an urgent challenge.

[0003] Pseudo-label learning has been widely applied in medical image processing, especially when dealing with unlabeled images or pixels in the training set. The basic idea is to use the model's predictions of unlabeled data as "pseudo-labels," and then use these pseudo-labels along with labeled data for further model training. However, because pseudo-labels originate from an insufficiently trained model, they often contain noise, especially when the model's predictions are inaccurate. This noise can negatively impact model training, thus limiting its performance.

[0004] Medical image registration is a fundamental task in medical image analysis, aligning moving and stationary images to a common coordinate system by estimating their spatial correspondence. Traditional medical image registration methods rely on pre-defined transformation models (such as rigid, affine, or non-rigid transformations) and similarity measures (such as mutual information and mean squared error) to calculate registration parameters. While these methods are effective, they have limitations: they require manual parameter tuning and are computationally inefficient when dealing with large-scale 3D images or complex deformations.

[0005] Classical image generation methods mainly include algorithms based on generative adversarial networks (GANs), variational autoencoders (VAEs), and autoencoders. However, these algorithms often suffer from various problems in latent space representation. With the development of unsupervised learning in image generation, unsupervised learning methods such as GANs and their variant CycleGAN have been widely applied in various scenarios, especially for image-to-image conversion without labeled data. However, they are only suitable for task-specific generation and cannot be directly applied to T1WI and T2WI generation, or the generated images are not well performed. The reasons are as follows: First, due to the high complexity of medical images, the requirements for effectively constructing feature extraction networks are very high. Second, due to the limitations of T1-T2 image acquisition, T1 and T2 images are unregistered. This makes the generation of transmembrane brain MRI images an unsupervised task, and how to register and set an accurate loss function is very challenging. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide an MRI image registration generation method based on the Cycle consistency framework, which can overcome the shortcomings of the prior art and improve the generation quality of MRI images.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical invention.

[0008] An MRI image registration generation method based on the Cycle consistency framework includes the following steps: inputting a T1-weighted image into an image generation network to obtain a T2-weighted image; inputting the T2-weighted image generated by the image generation network and the original T2-weighted image into an image registration network to obtain an aligned T2-weighted image; inputting the aligned T2-weighted image into an image generation network to obtain a T1-weighted image; and inputting the T1-weighted image generated by the image generation network and the original T1-weighted image into an image registration network to obtain an aligned T1-weighted image.

[0009] Preferably, the image generation network includes a generator and a discriminator.

[0010] Preferably, the generator includes,

[0011] The convolutional layer uses a stride of 2 to reduce the spatial resolution of the image while increasing the number of channels, thus extracting deeper features.

[0012] The normalization layer normalizes the output of the convolutional layer, promotes the effective propagation of features and prevents gradient vanishing, maintaining a stable gradient flow during training.

[0013] The activation layer uses the ReLU activation function to introduce non-linearity and enhance the feature representation ability, and uses the tanh activation function to ensure that the output range is between -1 and 1.

[0014] Transposed convolutional layers with skip connections are used to restore the spatial resolution of an image.

[0015] Preferably, the discriminator includes,

[0016] Four convolutional layers, each using a 4×4 convolutional kernel, generate feature maps of different dimensions through different numbers of output channels; each convolutional layer is followed by the LeakyReLU activation function to introduce non-linearity;

[0017] A fully convolutional layer is used to output a single-channel feature map. After average pooling and flattening operations, the output image is the probability that it is a real image.

[0018] Preferably, the loss function of the image generation network is L = L SR T1-T2 +L SR T2-T1 L SR T1-T2 L is the loss function for the process of generating a T2-weighted image from a T1-weighted image. SR T2-T1 The loss function for generating a T1-weighted image from a T2-weighted image;

[0019] Where α, γ, η, and υ are weighting coefficients;

[0020] Where R(G(X), Y) represents the deformation field, and G(X) represents the generated pseudo-T2 weighted image. This represents the overlay resampling process;

[0021] Where Ω is the local window position centered at (x, y), ε is a constant, cov(X, Y)(G, R) represents the local covariance, and var(G)(X, Y) represents the local variance. μ1 and μ2 represent the local mean of the image, respectively;

[0022]

[0023] Where G is the generator, D is the discriminator, X is the image in domain T1, Y is the image in domain T2, and E is the expectation operator;

[0024] Where S represents the area of ​​the image domain;

[0025]

[0026] Preferably, the image registration network includes,

[0027] The input layer uses multiple convolutional layers for feature extraction, with each convolutional layer using a different number of kernels to generate feature maps with different dimensions; all convolutional layers use LeakyReLU as the activation function to introduce non-linear transformations.

[0028] The downsampling layer extracts higher-dimensional features step by step through convolution, and uses residual blocks for additional feature transformations based on the configuration to enhance the model's expressive power.

[0029] The upsampling layer concatenates the upsampled features with the feature maps in the skip connections, and then further processes them through convolutional layers to gradually restore the spatial resolution of the image.

[0030] The output layer generates a two-dimensional image representing the classification result for each pixel.

[0031] The beneficial effects of the above-mentioned technical invention are as follows: This invention combines image registration with the generation process, significantly improving the application effect of the model in the field of medical imaging. First, we use a cross-membrane generator (CSG) to generate T2WI modal images, which contain the location information of the T1WI modality. Next, we use a joint registration network to extract the deformation field and superimpose it onto the original T2WI image to obtain pseudo-labels with T1WI location information. These pseudo-labels are used to effectively guide the training of the generation network. Finally, by combining the Cycle consistency framework, we enhance the network's ability to learn features of different modalities, while promoting robust learning by suppressing unreliable pseudo-labels through image-level and pixel-level weighting. Our proposed joint registration generation method gradually transforms the unsupervised task into a supervised task, effectively guiding the generation of images across membranes. Under the Cycle consistency framework, through repeated training between different modalities, the generator can effectively learn features of different domains, thereby significantly improving the model's performance. The main contributions of this invention are summarized as follows:

[0032] A joint registration generative network framework is proposed for cross-membrane medical image generation;

[0033] By generating pseudo-labels through a joint registration generative network, an unsupervised task is gradually transformed into a supervised task.

[0034] By designing a recurrent strategy, the training structure was optimized, thereby better guiding the generation of the network;

[0035] Extensive qualitative and quantitative experiments were conducted on T1-T2 images. The results show that this method can effectively generate high-quality T2 modal images from T1 modal images. Attached Figure Description

[0036] Figure 1 This is a flowchart of the present invention.

[0037] Figure 2 This is a structural diagram of the image generation network in this invention.

[0038] Figure 3 This is a comparison of different methods using clinical MRI images.

[0039] Figure 4 This is a graph showing the results of comparing different methods using the BRTAS public dataset. Detailed Implementation

[0040] See Figure 1-2 A method for MRI image registration generation based on the Cycle consistency framework includes the following steps: inputting a T1-weighted image into an image generation network to obtain a T2-weighted image; inputting the T2-weighted image generated by the image generation network and the original T2-weighted image into an image registration network to obtain an aligned T2-weighted image; inputting the aligned T2-weighted image into an image generation network to obtain a T1-weighted image; and inputting the T1-weighted image generated by the image generation network and the original T1-weighted image into an image registration network to obtain an aligned T1-weighted image.

[0041] Image generation networks consist of a generator and a discriminator.

[0042] The generator includes,

[0043] The convolutional layer uses a stride of 2 to reduce the spatial resolution of the image while increasing the number of channels, thus extracting deeper features.

[0044] The normalization layer normalizes the output of the convolutional layer, promotes the effective propagation of features and prevents gradient vanishing, maintaining a stable gradient flow during training.

[0045] The activation layer uses the ReLU activation function to introduce non-linearity and enhance the feature representation ability, and uses the tanh activation function to ensure that the output range is between -1 and 1.

[0046] Transposed convolutional layers with skip connections are used to restore the spatial resolution of an image.

[0047] The discriminator includes,

[0048] Four convolutional layers, each using a 4×4 convolutional kernel, generate feature maps of different dimensions through different numbers of output channels; each convolutional layer is followed by the LeakyReLU activation function to introduce non-linearity;

[0049] A fully convolutional layer is used to output a single-channel feature map. After average pooling and flattening operations, the output image is the probability that it is a real image.

[0050] The loss function of the image generation network is L = L SR T1-T2 +L SR T2-T1 L SB T1-T2 L is the loss function for the process of generating a T2-weighted image from a T1-weighted image. SR T2-T1 The loss function for generating a T1-weighted image from a T2-weighted image;

[0051] Where α, γ, η, and υ are weighting coefficients;

[0052] Where R(G(X), Y) represents the deformation field, and G(X) represents the generated pseudo-T2 weighted image. This represents the overlay resampling process;

[0053] Where Ω is the local window position centered at (x, y), ε is a constant, cov(X, Y)(G, R) represents the local covariance, and var(G)(X, Y) represents the local variance. μ1 and μ2 represent the local mean of the image, respectively;

[0054]

[0055] Where G is the generator, D is the discriminator, X is the image in domain T1, Y is the image in domain T2, and E is the expectation operator;

[0056] Where S represents the area of ​​the image domain;

[0057]

[0058] Image registration networks include,

[0059] The input layer uses multiple convolutional layers for feature extraction, with each convolutional layer using a different number of kernels to generate feature maps with different dimensions; all convolutional layers use LeakyReLU as the activation function to introduce non-linear transformations.

[0060] The downsampling layer extracts higher-dimensional features step by step through convolution, and uses residual blocks for additional feature transformations based on the configuration to enhance the model's expressive power.

[0061] The upsampling layer concatenates the upsampled features with the feature maps in the skip connections, and then further processes them through convolutional layers to gradually restore the spatial resolution of the image.

[0062] The output layer generates a two-dimensional image representing the classification result for each pixel.

[0063] Experimental process

[0064] We used two multi-contrast brain MRI datasets, BRTA-S2021 (Brats 2021) and a T1-T2 dataset of meningiomas collected from hospital clinics.

[0065] 1) BRTA-S2021 Dataset: This dataset contains MRI images of brain tumor patients from multiple institutions, primarily including four modalities: T1-weighted, contrast-enhanced T1-weighted (T1ce), T2-weighted, and FLAIR (fluid attenuation inversion recovery). The dataset covers approximately 125 patients, including different types of brain tumors such as glioblastoma (GBM) and low-grade glioma (LGG). All image data underwent rigorous registration and normalization to ensure that images from different modalities were aligned to the same anatomical template. Each patient's MRI images were divided into training, validation, and test sets: data from 80 patients were used for training, 20 for validation, and 25 for testing. Each patient's image data contains multiple slices, and the image resolution was uniformly adjusted to 1×1×1mm. 3 Furthermore, all images underwent skull dissection during processing to focus on brain tumor analysis. The dataset provides a rich collection of high-quality multimodal MRI images for brain tumor segmentation tasks, aiming to support the research and development of automated brain tumor segmentation techniques.

[0066] 2) Meningioma T1-T2 Dataset: This dataset contains MRI images of meningiomas from 36 subjects, covering both T1-weighted and T2-weighted modalities. The image data in the dataset was collected based on various clinical protocols and scanner devices, ensuring its broad clinical applicability and diversity. Data from 30 subjects was retained for the training set, data from 2 subjects for the validation set, and data from 4 subjects for the test set. The sampling parameters are as follows: T1 Modality: Standard T1-weighted imaging (T1WI) was used, with scan parameters including a slice thickness (ST) of 6.00 mm, a repetition time (RT) of 2074.04 ms, an echo time (ET) of 27.02 ms, and a flip angle (FS) of 1.50°. This helps to clearly display the contrast of brain structures and lesion areas, and is particularly suitable for the boundary identification of meningiomas. T2 modality: T2-weighted imaging (T2WI) was used to assess the structure and abnormalities of soft tissue. Scanning parameters included a slice thickness (ST) of 6.00 mm, a repetition time (RT) of 4000.00 ms, an echo time (ET) of 105.20 ms, and a flip angle (FS) of 1.50°. The T2 modality is particularly effective in visualizing the tumor region of meningiomas, highlighting the signal differences between the tumor and surrounding brain tissue. All image data in this dataset underwent standardization and registration to ensure that images from each modality were aligned to the same anatomical template, facilitating cross-modal data integration and analysis. Furthermore, image resolution and slice thickness were uniformly adjusted to ensure data consistency and accuracy.

[0067] The comparative experiments included pix2pix, CycleGAN, ResViT, and PAMRFuse. All comparative methods underwent hyperparameter optimization using the same cross-validation process.

[0068] 1) pix2pix includes an encoder-decoder structure, where the encoder extracts features from the input image, and the decoder generates the target image based on these features. The model is optimized using a pixel-level loss function (e.g., L2 loss) to make the output image as close as possible in detail to the real target image.

[0069] 2) CycleGAN is a cross-domain image transformation model based on unsupervised learning. It achieves image style / content transformation without the need for paired data through a dual generator (G and F) and dual discriminator architecture combined with Cycle-Consistency Loss.

[0070] 3) ResViT integrates the context sensitivity of visual Transformers, the spatial accuracy of convolution operators, and the realistic generation capabilities of adversarial learning. The core bottleneck structure of this generator consists of the original Aggregated Residual Transformer (ART) module. By connecting residual convolution and Transformer units in parallel, it achieves the organic fusion of local feature refinement and global semantic association. Furthermore, by adopting a weight sharing strategy among ART modules, it significantly reduces the computational complexity by 42%, ultimately realizing multimodal adaptive synthesis.

[0071] 4) PAMRFuse addresses the issues of image misalignment and spatial distortion in current photoacoustic tomography (PAT) and magnetic resonance imaging (MRI). It designs an unsupervised, multi-stage deep learning framework for accurately aligning input PAT-MRI image pairs with a multimodal-to-single-modal registration network and for selecting information-rich features. Ultimately, an end-to-end mutually reinforcing model is obtained, which makes joint optimization of cross-modal image generation and registration possible.

[0072] The method of the present invention was compared with the methods described above.

[0073] Figure 3 The figure shows the experimental results we obtained on a hospital clinical dataset. The final generated image was obtained after processing Source-1 through multiple networks, with Source-2 serving as a control. The boxes in the figure are enlarged to more clearly compare the differences between the results of different methods. Analysis shows that the image quality generated by CycleGAN and Pix2Pix networks is biased, with the generated brain tissue appearing blurry. ResViT performs poorly on smaller datasets, generating blurry lesions and losing some detail information from the original image. These methods all exhibit varying degrees of detail loss during the generation process. Although the PAMRFuse method improves image quality compared to the aforementioned networks, due to the limitations of the network itself, it still fails to effectively capture detailed information in the image, resulting in biased lesion structures. In contrast, the method proposed in this invention not only generates higher quality images but also significantly outperforms other comparative methods in capturing details of medical MRI images, while other methods lose image detail information to varying degrees.

[0074] See Figure 4 The results of this invention not only preserve the tissue and brightness information of the original image well, but also are more effective than other networks in generating lesion parts.

[0075] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this invention, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0076] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for MRI image registration generation based on the Cycle consistency framework, characterized in that... Includes the following steps: The T1-weighted image is input into the image generation network to obtain the T2-weighted image. The T2-weighted image generated by the image generation network and the original T2-weighted image are input into the image registration network to obtain the aligned T2-weighted image. The aligned T2-weighted image is input into the image generation network to obtain the T1-weighted image. The T1-weighted image generated by the image generation network and the original T1-weighted image are input into the image registration network to obtain the aligned T1-weighted image.

2. The MRI image registration generation method based on the Cycle consistency framework according to claim 1, characterized in that: The image generation network includes a generator and a discriminator.

3. The MRI image registration generation method based on the Cycle consistency framework according to claim 2, characterized in that: The generator includes, The convolutional layer uses a stride of 2 to reduce the spatial resolution of the image while increasing the number of channels, thus extracting deeper features. The normalization layer normalizes the output of the convolutional layer, promotes the effective propagation of features and prevents gradient vanishing, maintaining a stable gradient flow during training. The activation layer uses the ReLU activation function to introduce non-linearity and enhance the feature representation ability, and uses the tanh activation function to ensure that the output range is between -1 and 1. Transposed convolutional layers with skip connections are used to restore the spatial resolution of an image.

4. The MRI image registration generation method based on the Cycle consistency framework according to claim 3, characterized in that: The discriminator includes, Four convolutional layers, each using a 4×4 convolutional kernel, generate feature maps of different dimensions through different numbers of output channels; each convolutional layer is followed by the LeakyReLU activation function to introduce non-linearity; A fully convolutional layer is used to output a single-channel feature map. After average pooling and flattening operations, the output image is the probability that it is a real image.

5. The MRI image registration generation method based on the Cycle consistency framework according to claim 1, characterized in that: The loss function of the image generation network is L = L SR T1-T2 +L SR T2-T1 L SR T1-T2 L is the loss function for the process of generating a T2-weighted image from a T1-weighted image. SR T2-T1 The loss function for generating a T1-weighted image from a T2-weighted image; Where α, γ, η, and υ are weighting coefficients; Where R(G(X), Y) represents the deformation field, and G(X) represents the generated pseudo-T2 weighted image. This represents the overlay resampling process; Where Ω is the local window position centered at (x, y), ε is a constant, cov(X, Y)(G, R) represents the local covariance, and var(G)(X, Y) represents the local variance. μ1 and μ2 represent the local mean of the image, respectively; Where G is the generator, D is the discriminator, X is the image in domain T1, Y is the image in domain T2, and E is the expectation operator; Where S represents the area of ​​the image domain; 6. The MRI image registration generation method based on the Cycle consistency framework according to claim 1, characterized in that: The image registration network includes, The input layer uses multiple convolutional layers for feature extraction, with each convolutional layer using a different number of kernels to generate feature maps with different dimensions; all convolutional layers use LeakyReLU as the activation function to introduce non-linear transformations. The downsampling layer extracts higher-dimensional features step by step through convolution, and uses residual blocks for additional feature transformations based on the configuration to enhance the model's expressive power. The upsampling layer concatenates the upsampled features with the feature maps in the skip connections, and then further processes them through convolutional layers to gradually restore the spatial resolution of the image. The output layer generates a two-dimensional image representing the classification result for each pixel.

Images