Two-stage magnetic resonance image super-resolution method based on high-quality codebook prior

Abstract

This invention relates to a two-stage super-resolution method for MRI images using high-quality codebook prior information. The method includes: a) data preprocessing, undersampling the fully sampled K-space data; b) designing a one-stage codebook network, pre-trained using ground truth images to obtain high-quality codebook information; d) designing a two-stage codebook-prior super-resolution network model that integrates prior information; e) supervised training of the model using ground truth images; and f) selecting the optimal model based on a validation set, inputting test data into the model to obtain the super-resolution MRI image. Compared with existing technologies, this invention introduces high-quality codebook prior information during the super-resolution process, effectively solving the problem of poor restoration of details and texture structure in super-resolution images. Better restoration of detailed texture information aids in clinical diagnosis and has promising application prospects.

Description

Technical Field

[0001] This invention relates to the field of magnetic resonance image processing technology, and to a two-stage magnetic resonance image super-resolution method based on high-quality codebook priors. Background Technology

[0002] Magnetic resonance imaging (MRI) is a non-invasive medical imaging technique that plays an irreplaceable role in clinical diagnosis due to its superior soft tissue resolution and multimodal imaging capabilities. However, the acquisition of high-resolution MRI images is often limited by hardware performance and scan time, especially in scenarios requiring rapid imaging or high spatial resolution, presenting a series of challenges. Prolonged scans not only increase patient discomfort but can also introduce artifacts due to patient movement, further degrading image quality. These issues limit the application of MRI technology in certain complex scenarios, such as the accurate diagnosis of brain diseases and the early detection of small lesions.

[0003] To address this bottleneck, super-resolution (SR) technology has become an important means of solving the problem of insufficient resolution in MRI images. As a post-processing technique, super-resolution methods aim to recover high-resolution (HR) images by compensating for information in low-resolution (LR) images. However, the task of super-resolution of MRI images presents the following unique challenges:

[0004] 1. Information Missing Problem: Low-resolution MRI images suffer from a severe lack of high-frequency components in the frequency domain (K-space), yet these high-frequency components are crucial for determining image details and texture. This lack of information prevents traditional interpolation algorithms from recovering fine structures.

[0005] 2. Uncertainty in reconstruction: MRI super-resolution is an ill-posed problem, meaning that a single low-resolution input may correspond to multiple high-resolution outputs, which makes the reconstruction results susceptible to model assumptions and parameter settings.

[0006] 3. Multimodal complexity: MRI data often contains multiple modalities (such as T1, T2, etc.), and the structural information and texture characteristics between different modalities vary greatly, which places higher demands on the generalization ability of the model.

[0007] Traditional super-resolution methods, such as compressed sensing (CS)-based reconstruction methods, recover undersampled K-space data by leveraging the sparsity of signals. These methods typically combine optimization algorithms (such as alternating least squares and gradient descent) to iteratively compute high-resolution images. However, compressed sensing methods suffer from drawbacks such as high computational complexity, time-consuming iterative optimization, sensitivity to hyperparameter settings, insufficient robustness, and difficulty in effectively capturing complex texture information and image details.

[0008] In recent years, with the rapid development of deep learning technology, neural network-based super-resolution methods have become a research hotspot. These methods, by constructing convolutional neural networks (CNNs), generative adversarial networks (GANs), or Transformer structures, can directly learn the mapping relationship between low-resolution and high-resolution images. For example, the U-Net network effectively preserves spatial details of images through skip connections, the ResNet network improves training efficiency through residual learning, and the Transformer further introduces a global attention mechanism to capture long-range dependencies in images. However, existing deep learning methods in MRI super-resolution mainly focus on establishing a mapping between low-resolution and high-resolution images. Because low-resolution images contain less high-frequency information, texture details are difficult to reconstruct.

[0009] To overcome the aforementioned problems, we introduce a codebook-based prior learning method. This method extracts high-quality prior information from high-resolution images, stores it by constructing a codebook, and utilizes it as guiding information during the reconstruction process. The codebook method was initially applied to natural image generation and restoration tasks, demonstrating significant performance improvements. Similar structural details (such as the boundaries between gray and white matter in the brain, gyri, etc.) are ubiquitous in MRI images, and these texture features can serve as powerful prior information to improve the reconstruction results of MRI super-resolution. Summary of the Invention

[0010] The purpose of this invention is to propose a two-stage super-resolution method for magnetic resonance images based on high-quality codebook priors. This method addresses the shortcomings of existing technologies and solves the problem of poor texture detail recovery, thus better restoring the texture details of magnetic resonance images to facilitate clinical diagnosis by doctors.

[0011] The specific technical solution for achieving the objective of this invention is as follows:

[0012] A magnetic resonance image super-resolution method based on high-quality codebook priors includes the following specific steps:

[0013] Step 1: Perform data preprocessing on the fully sampled K-space data in the training set using a mask to obtain the low-resolution K-space data required for the input during the training phase;

[0014] Step 2: Design a one-stage codebook network model and train the model with high-quality images to obtain codebook modules with high-quality information;

[0015] Step 3: Design the codebook information fusion module and construct the final two-stage codebook prior network (CPNet) as the image super-resolution model;

[0016] Step 4: Use a supervised training strategy to train the model using labeled ground truth images;

[0017] Step 5: Select the best model based on the results of the validation set, and input the test data into the model to obtain the super-resolution magnetic resonance image.

[0018] Furthermore, step 1 specifically includes:

[0019] By applying a mask to the complete K-space data center region, low-resolution undersampled data in K-space is obtained. After normalization, low-resolution and high-resolution image pairs are obtained. The high-resolution images are used as input to the codebook network model and ground truth supervision, while the low-resolution images are used as input to the super-resolution network model.

[0020] Furthermore, step 2 specifically includes:

[0021] A one-stage codebook network model is constructed using an encoder-decoder structure, placing the codebook module between the encoder and decoder. The encoder extracts the features... New features are obtained after matching and replacement by the codebook module. The module is represented as:

[0022]

[0023] in, These are the features obtained after an image is encoded by an encoder. These are features obtained after codebook matching and replacement. The codebook information is saved. Features are encoded by the encoder. With codebook After nearest neighbor matching is performed on the features in the codebook, the corresponding features in the codebook are used to replace them. From the features in, new features are obtained. Subsequently The image is sent to the decoder to obtain a new magnetic resonance image.

[0024] The nearest neighbor matching operation used in codebook matching is non-differentiable. To avoid affecting model training, the decoder's gradient is directly passed to the encoder during gradient backpropagation. During the training of the codebook network model, a codebook loss function is constructed to constrain the model, ensuring the codebook module remembers useful information. The loss function is expressed as:

[0025]

[0026] in, It is the model input. It is the model output. It is the gradient stopping operator. It was set to 0.25 in the experiment.

[0027] Furthermore, step 3 specifically includes:

[0028] We freeze the gradients of the codebook and decoder modules to construct a two-stage codebook prior network as the super-resolution model. The codebook contains high-quality texture information learned from high-resolution images, while the features encoded by the encoder contain information outside the codebook. Therefore, a reasonable structure should be designed to integrate the codebook prior information into the encoder's encoded features. Using an attention mechanism, we designed a cross-channel attention fusion module (CCAFM) for this fusion process. The fused features are then fed into the decoder to obtain the super-resolution image.

[0029] A cross-channel fusion module is designed using an attention mechanism to integrate high-quality prior information obtained from codebook matching into the features encoded by the encoder, and finally feed it into the decoder to obtain the super-resolution image.

[0030] To enhance the learning ability of the network, a gated convolutional Transformer layer (GCTB) is designed as the base layer of the super-resolution network model.

[0031] Furthermore, step 4 specifically includes:

[0032] Determine the loss function, optimizer, and learning rate, initialize the model parameters, and iterate and train using the processed dataset under the supervision of ground truth images.

[0033] Furthermore, step 5 specifically includes:

[0034] After the model converges, the model weights are saved, and the model with the best performance is selected based on the validation set. During inference, the model weights are loaded, a low-resolution image is input, and a high-resolution magnetic resonance image after super-resolution is output.

[0035] After the above steps, good results are achieved in both visual and quantitative metrics. Specifically, on the IXI dataset T2 modality, the PSNR reaches 32.26 and SSIM reaches 0.9277 with a 4x super-resolution, and the PSNR reaches 39.69 and SSIM reaches 0.9806 with a 2x super-resolution. For the BraTS dataset T1 modality, the PSNR reaches 37.84 and SSIM reaches 0.9656 with a 4x super-resolution, and the PSNR reaches 44.58 and SSIM reaches 0.9901 with a 2x super-resolution. Therefore, the method proposed in this invention achieves excellent super-resolution performance by introducing a codebook learning model. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the process of the present invention;

[0037] Figure 2This is a schematic diagram of the overall architecture of the network proposed by our institute;

[0038] Figure 3 This is a schematic diagram illustrating the detailed texture restoration of an embodiment of the present invention.

[0039] Figure 4 This is a schematic diagram illustrating the errors in an embodiment of the present invention.

[0040] Figure 5 This is a schematic diagram of codebook prior ablation in this invention. Detailed Implementation

[0041] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0042] See Figure 1 The present invention provides a magnetic resonance image super-resolution method based on high-quality codebook priors, comprising the following steps:

[0043] Step 1: Perform data preprocessing on the fully sampled K-space data to obtain the required low-resolution K-space data for input;

[0044] Step 1 specifically includes: applying a mask to the complete K-space data center region to obtain K-space low-resolution undersampled data, and after normalization processing, obtaining low-resolution and high-resolution image pairs. The high-resolution images are used as input to the codebook network model and ground truth supervision, while the low-resolution images are used as input to the super-resolution network model.

[0045] Step 2: Design a codebook network model and train the model with high-quality images to obtain codebook modules with high-quality information;

[0046] Step 2 specifically includes: constructing a codebook network model using an encoder-decoder structure, placing the codebook module between the encoder and decoder, and extracting features from the encoder. New features are obtained after matching and replacement by the codebook module. The module is represented as:

[0047]

[0048] in, These are the features obtained after an image is encoded by an encoder. These are features obtained after codebook matching and replacement. The codebook information is saved. Features are encoded by the encoder. With codebook After nearest neighbor matching is performed on the features in the codebook, the corresponding features in the codebook are used to replace them. From the features in, new features are obtained. Subsequently The image is sent to the decoder to obtain a new magnetic resonance image.

[0049] The nearest neighbor matching operation used in codebook matching is non-differentiable. To avoid affecting model training, the decoder's gradient is directly passed to the encoder during gradient backpropagation. During the training of the codebook network model, a codebook loss function is constructed to constrain the model, ensuring the codebook module remembers useful information. The loss function is expressed as:

[0050]

[0051] in, It is the model input. It is the model output. It is the gradient stopping operator. This is a hyperparameter.

[0052] Step 3: Design the codebook information fusion module to construct the final magnetic resonance image super-resolution model;

[0053] Step 3 specifically includes: freezing the gradients of the codebook module and the decoder module, and constructing the super-resolution network model. The codebook contains high-quality texture information learned from high-resolution images, while the features encoded by the encoder contain information outside the codebook. A reasonable structure should be designed to integrate the prior information of the codebook into the features encoded by the encoder. Using an attention mechanism, we designed a cross-channel attention fusion module for this fusion process. The fused features are then fed into the decoder to obtain the super-resolution image.

[0054] A cross-channel fusion module is designed using an attention mechanism to integrate high-quality prior information obtained from codebook matching into the features encoded by the encoder, and finally feed it into the decoder to obtain the super-resolution image.

[0055] To enhance the network's learning ability, leveraging the complementary features of convolution and Transformer in feature extraction, a gated mechanism is applied to design a convolutional Transformer layer as the foundation layer of the super-resolution network model. The module is represented as follows:

[0056]

[0057] in, This represents the feedforward neural network module in the Transformer. It is a gating mechanism implemented through dot product. It is a multi-head self-attention operation in Transformer. It is a convolution operation.

[0058] Compared to traditional Transformer layers, introducing convolutional branches next to the attention module can significantly improve the network's performance in the model.

[0059] Step 4: Use a supervised training strategy to train the model using labeled ground truth images;

[0060] Step 4 specifically includes: determining the use The norm is used as the loss function, the Adam optimizer is selected, and the optimizer parameters are... Set to 0.9, parameter Set to 0.99, initial learning rate set to The learning rate is halved at 200k, 250k, 270k, and 290k iterations as the number of network iterations increases.

[0061] Step 5: Select the best model based on the results of the validation set, and input the test data into the model to obtain the super-resolution magnetic resonance image;

[0062] This example uses two common evaluation metrics, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM), to quantify model performance. Higher metric values ​​indicate better network reconstruction. After model training converges, the best model is saved, its weights are loaded into the model, undersampled K data is input, and the reconstructed magnetic resonance image is output.

[0063] Example

[0064] Step 1: Perform data preprocessing on the fully sampled K-space data to obtain the required low-resolution K-space data for input;

[0065] The datasets used in this embodiment are the IXI T2 modal dataset and the BraTS T1 modal dataset. The original data is in nii.gz format. By applying a mask to the complete K-space data center region, low-resolution undersampled data in K-space is obtained. After normalization, low-resolution and high-resolution png format image pairs are obtained. The high-resolution images are used as input to the codebook network model and ground truth supervision, while the low-resolution images are used as input to the super-resolution network model.

[0066] Step 2: Design a one-stage codebook network model and train the model with high-quality images to obtain codebook modules with high-quality information;

[0067] See appendix Figure 2 In this embodiment, an encoder-decoder structure is used to construct a one-stage codebook network model. The codebook module is placed between the encoder and the decoder. The features extracted by the encoder... New features are obtained after matching and replacement by the codebook module. The module is represented as:

[0068]

[0069] in, These are the features obtained after an image is encoded by an encoder. These are features obtained after codebook matching and replacement. The codebook information is saved. Features are encoded by the encoder. With codebook After nearest neighbor matching is performed on the features in the codebook, the corresponding features in the codebook are used to replace them. From the features in, new features are obtained. Subsequently The image is sent to the decoder to obtain a new magnetic resonance image.

[0070] The nearest neighbor matching operation used in codebook matching is non-differentiable. To avoid affecting model training, the decoder's gradient is directly passed to the encoder during gradient backpropagation. During the training of the codebook network model, a codebook loss function is constructed to constrain the model, ensuring the codebook module remembers useful information. The loss function is expressed as:

[0071]

[0072] in, It is the model input. It is the model output. It is the gradient stopping operator. In this embodiment, it is set to 0.25.

[0073] In this embodiment, the high-resolution data of IXI and BraTS obtained in step one are input into a one-stage network, and the output of the one-stage codebook network is constrained by the high-resolution data, so that the codebook module learns texture information that can be used for image restoration.

[0074] Step 3: Design the codebook information fusion module to construct the final magnetic resonance image super-resolution model;

[0075] See appendix Figure 2 In this embodiment, the gradients of the codebook module and the decoder module are frozen to construct a two-stage super-resolution network model. The codebook contains high-quality texture information learned from high-resolution images in the first stage, while the features encoded by the encoder contain information outside the codebook. A reasonable structure should be designed to integrate the prior information of the codebook into the features encoded by the encoder. In this embodiment, an attention mechanism is used to design a cross-channel attention fusion module (CCAFM) for this fusion process. The fused features are then fed into the decoder to obtain the super-resolution image.

[0076] A cross-channel fusion module is designed using an attention mechanism to integrate high-quality prior information obtained from codebook matching into the features encoded by the encoder, and finally feed it into the decoder to obtain the super-resolution image.

[0077] To enhance the network's learning ability, this embodiment leverages the complementary characteristics of convolution and Transformer in feature extraction, employing a gating mechanism to design a gated convolutional Transformer layer (GCTB) as the foundational layer of the super-resolution network model. The module is represented as follows:

[0078]

[0079] in, This represents the feedforward neural network module in the Transformer. It is a gating mechanism implemented through dot product. It is a multi-head self-attention operation in Transformer. It is a convolution operation.

[0080] Compared to traditional Transformer layers, introducing convolutional branches next to the attention module can significantly improve the network's performance in the model.

[0081] Step 4: Use a supervised training strategy to train the model using labeled ground truth images;

[0082] Step 4 specifically includes: determining the use The norm is used as the loss function, the Adam optimizer is selected, and the optimizer parameters are... Set to 0.9, parameter Set to 0.99, initial learning rate set to The learning rate is halved at 200k, 250k, 270k, and 290k iterations as the number of network iterations increases.

[0083] Step 5: Select the best model based on the results of the validation set, and input the test data into the model to obtain the super-resolution magnetic resonance image;

[0084] This embodiment uses two common evaluation metrics, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM), to quantify model performance. The higher the evaluation metric value, the better the network recovery effect. After the model training converges, the best model is saved, the model weights are loaded into the model, undersampled K data is input, and the reconstructed magnetic resonance image is output.

[0085] The neural network constructed using this invention was compared with bicubic interpolation, RCAN, CSN, SwinIR, MHCA, and HAT. It outperformed other models in both PSNR and SSIM. The evaluation metrics for each network are detailed in Table 1 below:

[0086]

[0087] See appendix Figure 3The codebook prior network constructed in this invention can recover better detailed texture information by leveraging codebook priors.

[0088] See appendix Figure 4 The model constructed in this invention has a smaller error in the super-resolution results.

[0089] This invention demonstrates the effectiveness of codebook prior, cross-channel attention fusion module (CCAFM), and gated convolutional Transformer module (GCTB) on a 4x super-resolution task using the BraTS T1 modality dataset through quantitative ablation experiments. The experimental results are shown in Table 2.

[0090] As shown in Table 2, removing the codebook prior module significantly reduced model performance, validating the effectiveness of the codebook prior. In the second ablation experiment, this embodiment compared the direct concat operation with the cross-channel attention fusion module, demonstrating that CCAFM improved the metrics. The third ablation experiment validated the effectiveness of the network's base layer; it was observed that the base layer with gating and convolution exhibited stronger learning capabilities.

[0091] See appendix Figure 5 This invention verifies the effectiveness of codebook priors in restoring texture details through qualitative comparison.

[0092] The above embodiments provide a detailed description of the two-stage magnetic resonance image super-resolution method based on high-quality codebook priors proposed in this invention. For researchers in the art, based on the ideas of this invention, there will be changes in the specific implementation methods and application scope. For example, the one-stage codebook network model can be replaced with other, better encoders to extract features, and future improvements to the network and model can be considered to obtain better results. Therefore, the content of this specification should not be construed as limiting the invention.

[0093] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of the inventive concept are included in this invention and are protected by the appended claims.

Claims

1. A two-stage magnetic resonance image super-resolution method based on high-quality codebook priors, characterized in that, The specific steps include the following: Step 1: Perform data preprocessing on the fully sampled K-space data in the training set using a mask to obtain the low-resolution K-space data required for the input during the training phase; Step 2: Design a one-stage codebook network model and pre-train the model with high-resolution images to enable the codebook to remember high-quality information; Step 3: Design the codebook information fusion module and construct the final two-stage codebook prior network as the image super-resolution model; Step 4: Use a supervised training strategy to train the model using labeled ground truth images; Step 5: Select the best model based on the results of the validation set, and input the test data into the model to obtain the super-resolution magnetic resonance image.

2. The two-stage magnetic resonance image super-resolution method based on high-quality codebook prior as described in claim 1, characterized in that, Step 1 specifically includes: By applying a mask to the complete K-space data center region, low-resolution undersampled data in K-space is obtained. After normalization, low-resolution and high-resolution image pairs are obtained. The high-resolution images are used as input to the codebook network model and ground truth supervision, while the low-resolution images are used as input to the super-resolution network model.

3. The two-stage magnetic resonance image super-resolution method based on high-quality codebook prior as described in claim 1, characterized in that, Step 2 specifically includes: A codebook network model is constructed using an encoder-decoder structure, placing the codebook module between the encoder and decoder. The encoder extracts the features... New features are obtained after matching and replacement by the codebook module. The module is represented as: in, These are the features obtained after an image is encoded by an encoder. These are features obtained after codebook matching and replacement. The codebook information is saved. Features are encoded by the encoder. With codebook After nearest neighbor matching is performed on the features in the codebook, the corresponding features in the codebook are used to replace them. From the features in, new features are obtained. Subsequently The image is sent to the decoder to obtain a new magnetic resonance image. The nearest neighbor matching operation used in codebook matching is non-differentiable. To avoid affecting model training, the decoder's gradient is directly passed to the encoder during gradient backpropagation. During the training of the codebook network model, a codebook loss function is constructed to constrain the model, ensuring the codebook module remembers useful information. The loss function is expressed as: in, It is the model input. It is the model output. It is the gradient stopping operator. It was set to 0.25 in the experiment.

4. The two-stage magnetic resonance image super-resolution method based on high-quality codebook prior as described in claim 1, characterized in that, Step 3 specifically includes: We freeze the gradients of the codebook and decoder modules to construct a two-stage codebook prior network as the super-resolution model. The codebook contains high-quality texture information learned from high-resolution images, while the features encoded by the encoder contain information outside the codebook. Therefore, a reasonable structure should be designed to integrate the codebook prior information into the encoder's encoded features. Using an attention mechanism, we designed a cross-channel attention fusion module for this fusion process. The fused features are then fed into the decoder to obtain the super-resolution image. A cross-channel fusion module is designed using an attention mechanism to integrate high-quality prior information obtained from codebook matching into the features encoded by the encoder, and finally feed it into the decoder to obtain the super-resolution image. To enhance the learning ability of the network, a convolutional Transformer layer was designed as the base layer of the super-resolution network model.

5. The two-stage magnetic resonance image super-resolution method based on high-quality codebook prior as described in claim 1, characterized in that, Step 4 specifically includes: Determine the loss function, optimizer, and learning rate, initialize the model parameters, and iterate and train using the processed dataset under the supervision of ground truth images.

6. The two-stage magnetic resonance image super-resolution method based on high-quality codebook prior as described in claim 1, characterized in that, Step 5 specifically includes: After the model converges, the model weights are saved, and the model with the best performance is selected based on the validation set. During inference, the model weights are loaded, a low-resolution image is input, and a high-resolution magnetic resonance image after super-resolution is output.

Images