Hybrid domain multi-contrast MRI (Magnetic Resonance Imaging) super-division method and system based on variational network

Abstract

The invention belongs to the technical field of medical image processing, and particularly discloses a hybrid domain multi-contrast MRI super-division method and system based on a variational network, and the method comprises the steps: obtaining and preprocessing a high-resolution auxiliary image, a high-resolution target image and a mask, and obtaining a multi-channel high-resolution auxiliary image, a low-resolution target image and a mask; the mixed domain multi-contrast variational network iteratively updates the low-resolution target image of each channel, and weights and fuses the super-resolution target images of all channels to obtain a final super-resolution target image; measuring a loss function to train an MRI super-resolution model; and inputting the actual low-resolution MRI image into the MRI super-resolution model to obtain an actual super-resolution image. According to the method, good interpretability based on a model method can be kept, the credibility of clinical practice is improved, the image reconstruction effect is effectively improved by means of the powerful feature expression ability of the deep neural network, and more accurate pathological information is provided for precise medical treatment.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and particularly to a hybrid-domain multi-contrast MRI super-resolution method and system based on a variational network. Background Art

[0002] With the continuous development of medical imaging technology, various medical imaging devices have become important tools for clinical diagnosis. Among them, magnetic resonance imaging (MRI) has been widely used in medical research and clinical applications due to its non-invasive nature, high soft tissue contrast, and multi-modal characteristics. In particular, MRI images of different modalities can provide complementary information to highlight specific tissue characteristics, thus assisting in accurate clinical diagnosis and surgical planning. However, due to limitations in scanning time, signal-to-noise ratio (SNR), and hardware conditions, high-resolution (HR) MRI imaging usually faces many challenges, often accompanied by problems such as image blurring and detail loss, which bring inconvenience to doctors' diagnosis and subsequent treatment. To address these issues, medical image super-resolution technology has emerged. Its core goal is to enhance the spatial resolution while maintaining or enhancing the structural and texture information of the image. By means of an efficient super-resolution reconstruction method, more details can be restored from low-resolution images, thereby obtaining high-quality MRI images, which can further improve the accuracy of clinical judgment and provide strong support for precision medicine.

[0003] The research on medical image super-resolution began with traditional interpolation methods, such as bilinear interpolation and bicubic interpolation, which magnify images through simple mathematical interpolation. However, due to the lack of effective ability to restore high-frequency details, such methods often result in image blurring and are difficult to significantly improve the actual resolution. With the rapid development of deep learning, the research on medical image super-resolution has gradually shifted to solutions based on deep learning models. Methods such as convolutional neural networks (CNNs), generative adversarial networks (GANs), and transformer models (Transformers) learn mapping relationships from a large number of low-resolution images through end-to-end training to effectively reconstruct high-resolution images. These methods perform well in restoring edge information and high-frequency details, significantly improving the quality of super-resolution reconstruction and providing more accurate image data support for medical image analysis and clinical applications.

[0004] Deep learning has made remarkable progress in the task of medical image super-resolution due to its powerful feature representation ability. It is mainly divided into two methods: single-contrast super-resolution and multi-contrast super-resolution. The single-contrast super-resolution method only relies on a single-modal MRI image for reconstruction and does not fully utilize the auxiliary information of other modalities. Therefore, when restoring high-resolution images, there is often a loss of some texture details. Further analysis shows that compared with pixel-level information, different-modal MRI images have higher consistency in texture features. This is because although images of different modalities are acquired under different scanning parameters, resulting in differences in contrast, they all reflect the same anatomical structure and thus have similar texture features. Compared with the single-contrast super-resolution method, the multi-contrast method aggregates complementary information from reference images of different modalities through a designed fusion strategy to improve the quality of the target image. However, existing methods still have certain limitations. First, most existing methods only perform feature extraction and fusion in the image domain, ignoring the original data information in the k-space. However, each data point in the k-space carries global frequency information, which is crucial for the consistency of image spectral features and the restoration of high-frequency details. Second, current fusion strategies, whether based on simple connection or feature matching, usually lack physical interpretability and are difficult to accurately utilize the common features and complementary information of each modal image.

[0005] In summary, there is an urgent need to provide an MRI super-resolution method that can fully utilize the prior information in the k-space domain, has strong interpretability, can effectively model the potential relationship between different-modal images, and thus significantly improve the image quality. Summary of the Invention

[0006] In view of this, the present invention proposes a hybrid-domain multi-contrast MRI super-resolution method and system based on a variational network to solve the technical defects existing in the prior art. The method and system can fully utilize the k-space frequency domain information, explicitly model the potential relationship between different-modal images, improve the quality of super-resolution MRI images, enhance their credibility in processing medical images in clinical practice, further improve the doctor's ability to diagnose diseases, and provide better treatment options for patients.

[0007] In a first aspect, an embodiment of the present application provides a hybrid-domain multi-contrast MRI super-resolution method based on a variational network, including: S1, obtaining a high-resolution auxiliary image, a high-resolution target image, and a mask; S2, preprocessing the high-resolution auxiliary image, the high-resolution target image, and the mask to obtain a multi-channel high-resolution auxiliary image, a low-resolution target image, and a mask; S3. The multi-contrast variational network in the hybrid domain iteratively updates each channel of the low-resolution target image, and performs weighted fusion on the super-resolution target images of all channels obtained to obtain the final super-resolution target image; S4. Measure the loss function between the final super-resolution target image and the high-resolution target image, and train the multi-contrast variational network in the hybrid domain into an MRI super-resolution model; S5. Input the actual low-resolution MRI image into the MRI super-resolution model to obtain the actual super-resolution image.

[0008] In a possible implementation, the preprocessing is specifically: Generate a low-resolution target image based on the high-resolution target image and the mask; Through the channel expansion strategy, copy the high-resolution auxiliary image, the low-resolution target image, and the mask along the channel dimension, and expand the high-resolution auxiliary image, the low-resolution target image, and the mask from a single channel to a multi-channel representation.

[0009] In a possible implementation, S3 is specifically: Initialize the multi-contrast variational network in the hybrid domain; Perform iterative calculations on each channel of the low-resolution target image and the auxiliary variable to obtain the super-resolution target images of all channels; Perform weighted fusion on the super-resolution target images of all channels to obtain the final super-resolution target image.

[0010] In a possible implementation, initializing the multi-contrast variational network in the hybrid domain is specifically: Use the denoising network U-Net to initialize the low-resolution target image and the high-resolution auxiliary image to obtain the initial low-resolution target image and the denoised high-resolution auxiliary image ; Obtain the undersampled k-space data by performing Fourier transform on the low-resolution target image ; Extract the texture information by applying the Laplace operator to the high-resolution auxiliary image G ; Set the initial auxiliary variable to .

[0011] In a possible implementation, the calculation formula for performing iterative calculations on each channel of the low-resolution target image and the auxiliary variable is: , , where, , respectively represent the reconstructed images of the -th and -th iterations; represents the inverse Fourier transform; represents the k-space data corresponding to the reconstructed image during the iterative process; , , , , and all represent the trade-off parameters of the -th iteration; represents the mask; represents the undersampled k-space data; , , , respectively represent the convolutional dictionary layers; and respectively represent the transposed convolutions of the convolutional dictionary layers and ; represents the denoised high-resolution auxiliary image; G represents the texture information; and respectively represent the auxiliary variables of the -th and -th iterations; represents the U-Net network.

[0012] In a possible implementation, the calculation formula for weighted fusion of the super-resolution target images of all channels is: , where, represents the super-resolution target image; represents the total number of channels; represents the channel number; represents the learnable weight of the -th channel; represents the super-resolution target image of the -th channel.

[0013] In a second aspect, the embodiments of the present application provide a hybrid-domain multi-contrast MRI super-resolution system based on a variational network, including: A data acquisition module, configured to acquire a high-resolution auxiliary image, a high-resolution target image, and a mask; A preprocessing module, configured to preprocess the high-resolution auxiliary image, the high-resolution target image, and the mask to obtain a multi-channel high-resolution auxiliary image, a low-resolution target image, and a mask; A hybrid-domain multi-contrast variational network module is used to iteratively update each channel of the low-resolution target image, and perform weighted fusion on the super-resolution target images of all channels obtained, to obtain the final super-resolution target image; A deep supervision module is used to measure the loss function between the final super-resolution target image and the high-resolution target image, and train the hybrid-domain multi-contrast variational network into an MRI super-resolution model; An image generation module is used to input the actual low-resolution MRI image into the MRI super-resolution model to obtain the actual super-resolution image.

[0014] In a possible implementation, the hybrid-domain multi-contrast variational network module includes an initialization module, an iterative update module, and a reconstruction module; The iterative update module includes T iterative update sub-modules connected in series in sequence, and each iterative update sub-module includes a target image update module and an auxiliary variable update module; The target image update module includes a numerical fidelity layer and a structure texture refinement layer connected in parallel.

[0015] In a possible implementation, the numerical fidelity layer includes a Fourier transform module, a first formula calculation module, and a Fourier inverse transform module, where, The Fourier transform module is used to perform Fourier transform on the low-resolution target image and the reconstructed image to obtain undersampled k-space data and iteratively updated k-space data; The first formula calculation module is used for the iterative expansion operation connection in the k-space domain of; The Fourier inverse transform module is used to inverse transform the iteratively expanded k-space domain data back to the image domain to ensure that the reconstructed image is consistent with the target modality in spatial structure.

[0016] In a possible implementation, the structure texture refinement layer includes a convolutional dictionary module and a second formula calculation module, where, The convolutional dictionary module is used to implement the transformation matrix in the iterative expansion in the image domain; The second formula calculation module includes an auxiliary image calculation module, a texture calculation module, an auxiliary variable calculation module, and an integration module; the auxiliary image calculation module is used to denoise the high-resolution auxiliary image related iterative expansion in the image domain; the texture calculation module is used for iterative expansion in the image domain related to texture information; the auxiliary variable calculation module is used for iterative expansion in the image domain related to auxiliary variables; the integration module is used to integrate the calculation results of the auxiliary image calculation module, the texture calculation module, and the auxiliary variable calculation module to complete the iterative expansion in the image domain.

[0017] The present invention proposes a mixed-domain multi-contrast MRI super-resolution method and system based on a variational network, and constructs a mixed-domain multi-contrast variational network to solve the problem of medical image super-resolution reconstruction. In the model building stage, the present invention explicitly models the relationship matrix between images of different modalities, making full use of the complementary information between images of different modalities; in the model solving stage, the present invention adopts an optimization method based on a semi-quadratic splitting algorithm to map the optimization process of the variational model to the iterative expansion of a deep neural network. In the neural network optimization stage, the present invention designs channel expansion and mixed-domain optimization strategies, combining the advantages of the image domain and the k-space domain, and effectively improving the image reconstruction effect. Compared with the prior art, the present invention has achieved the following beneficial effects: (1) This paper proposes a model-driven hybrid domain multi-contrast variational network to solve the problem of medical image super-resolution reconstruction. This method not only maintains the good interpretability of the model, increases the credibility required for processing medical images in clinical practice, but also improves the accuracy of the super-resolution process by leveraging the powerful feature expression ability of deep neural networks.

[0018] (2) The present invention designs a hybrid domain optimization strategy that combines the advantages of the image domain and the k-space domain, effectively improving the image reconstruction effect. In the k-space domain, the data fidelity layer imposes consistency constraints to ensure global consistency and reduce reconstruction errors; while in the image domain, the structure and texture refinement layer optimizes spatial features, reduces artifacts, and enhances image structure and texture details.

[0019] (3) In the structural texture refinement layer of the image domain, the present invention uses auxiliary images of different modalities to provide stable macroscopic structural guidance to ensure overall anatomical consistency; and uses the extracted texture information to provide rich detail compensation capabilities to finely reconstruct local areas. With the synergistic effect of the two, the model can take into account both structural accuracy and detail authenticity, thereby significantly improving the overall reconstruction quality of the image.

[0020] (4) This paper introduces texture loss based on traditional reconstruction loss and constructs a new hybrid texture loss function to supervise the reconstruction of image content and edge details. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0022] Figure 1 A schematic diagram of a flow chart of a hybrid domain multi-contrast MRI super-resolution method based on a variational network provided in an embodiment of the present invention; Figure 2 The structural block diagram of the hybrid-domain multi-contrast MRI super-resolution system based on the variational network provided by the embodiment of the present invention; Figure 3 The network structure diagram of the hybrid-domain multi-contrast variational network module provided by the embodiment of the present application; Figure 4 The network structure diagrams of each module in the iterative update module provided by the embodiment of the present invention; wherein, Fig. (a) is the network structure diagram of the numerical fidelity layer, Fig. (b) is the network structure diagram of the structural texture refinement layer, and Fig. (c) is the network structure diagram of the auxiliary variable update module; Figure 5 The comparison diagrams of the high-resolution target image, high-resolution auxiliary image, low-resolution target image, and super-resolution target image provided by the embodiment of the present invention.

[0023] Figure 6 The structural schematic diagram of an electronic device provided by the embodiment of the present invention. Detailed implementation manners

[0024] To better understand the technical solution of the present application, the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0025] It should be clear that the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the scope of protection of the present application.

[0026] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms of "a", "the", and "said" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0027] It should be understood that the term " / and / " used herein is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " herein generally represents an "or" relationship between the associated objects before and after.

[0028] See Figure 1 , which is the flowchart of the hybrid-domain multi-contrast MRI super-resolution method based on the variational network provided by the embodiment of the present invention. As Figure 1 shown, the method includes: Step S1, obtaining a high-resolution auxiliary image, a high-resolution target image, and a mask.

[0029] Specifically, the high-resolution auxiliary image and the high-resolution target image are different modality images of the same part of the body.

[0030] Preferably, the high-resolution target image is a T2 modality image (long acquisition time); the high-resolution auxiliary image is a T1 modality image (short acquisition time), providing complementary information during the process of the target image transitioning from low resolution to high resolution.

[0031] Specifically, the mask is a central downsampling mask, generated and obtained through program code.

[0032] Step S2: Preprocess the high-resolution auxiliary image, the high-resolution target image, and the mask to obtain a multi-channel high-resolution auxiliary image, a low-resolution target image, and a mask. Specifically: Step S21: Generate a low-resolution target image based on the high-resolution target image and the mask. Specifically: Slice the 3D NII format image of the high-resolution target image to obtain 2D MRI images; Multiply the 2D MRI images by the mask in the k-space for downsampling, and then transform back to the image domain to obtain the low-resolution target image.

[0033] Step S22: Through a channel expansion strategy, duplicate the high-resolution auxiliary image, the low-resolution target image, and the mask along the channel dimension, expanding the high-resolution auxiliary image, the low-resolution target image, and the mask from single-channel to multi-channel representation.

[0034] It should be particularly noted that to overcome the limitations of the single-channel representation of MRI images, this embodiment adopts a channel expansion strategy to enhance the feature expression ability and reconstruction quality of the model. Specifically, duplicate the input high-resolution auxiliary image, low-resolution target image, and their corresponding masks along the channel dimension, expanding them from single-channel to multi-channel representation. This can not only provide richer feature information during network training but also utilize the complementarity of different channels in subsequent iterations to more comprehensively restore structural and texture details.

[0035] Step S3: The hybrid-domain multi-contrast variational network iteratively updates each channel of the low-resolution target image and performs weighted fusion on the obtained super-resolution target images of all channels to obtain the final super-resolution target image. Specifically: Step S31: Initialize the hybrid-domain multi-contrast variational network. Specifically Use the denoising network U-Net to initialize the low-resolution target image and the high-resolution auxiliary image to obtain an initial low-resolution target image and a denoised high-resolution auxiliary image ; The low-resolution target image is Fourier-transformed to obtain undersampled k-space data for numerical fidelity constraint. ; The texture information is extracted by applying the Laplacian operator to the high-resolution auxiliary image. G ; The initial auxiliary variable is set to for guiding the learning of prior information during the optimization process.

[0036] It should be noted that in the subsequent iterative optimization process, only the low-resolution target image and the auxiliary variable participate in the iterative update, while other variables remain fixed to reduce the computational overhead and improve the optimization stability.

[0037] Step S32: Perform T rounds of iterative calculations on the low-resolution target image and the auxiliary variable for each channel to obtain the super-resolution target images for all channels. The iterative calculation formula for any channel is: , , where , represent the reconstructed images at the -th and -th iterations respectively, and the reconstructed image obtained in the last iteration is the super-resolution target image for this channel; represents the inverse Fourier transform; represents the k-space data corresponding to the reconstructed image during the iterative process, is obtained by Fourier-transforming ; , , , , and all represent the trade-off parameters at the -th iteration, and each parameter is learnable, , , , , and can be set according to actual needs. In this embodiment, , , , , and are initially set to 1.0; represents the mask; represents the undersampled k-space data; , , , respectively represent the convolutional dictionary layer; and respectively represent the convolutional dictionary layer and transpose convolution; represents the denoised high-resolution auxiliary image; G represents the texture information; and respectively represent the -th and -th iteration auxiliary variables; represents the U-Net network.

[0038] Furthermore, the U-Net network consists of three encoder modules, three decoder modules and skip connections. The encoder module contains multiple convolutional layers and activation function (ReLU) layers, and extracts multi-scale features of the image by downsampling layer by layer. A skip connection mechanism is introduced between the encoder and the decoder to fuse the shallow low-level features with the deep high-level features, so as to effectively retain the detailed information of the image while improving the reconstruction performance. The decoder part first performs an upsampling operation, and splices the corresponding encoder features through skip connections, and then gradually restores the high-resolution target image through multiple convolutional layers and ReLU layers.

[0039] Preferably, in this embodiment, the low-resolution target image and the auxiliary variable are respectively subjected to 6 rounds of iterative calculations, that is, the iterative update module of the hybrid-domain multi-contrast variational network includes 6 iterative update sub-modules, and the iterative process is as follows: the undersampled k-space data , texture information G , denoised high-resolution auxiliary image , initial low-resolution target image and initial auxiliary variable are input into the first iterative update sub-module to obtain the reconstructed image of the first iteration and the auxiliary variable of the first iteration; the reconstructed image of the first iteration, the auxiliary variable of the first iteration and other unchanged input variables are input into the second iterative update sub-module to obtain the reconstructed image of the second iteration and the auxiliary variable of the second iteration; and so on until 6 iterations are completed.

[0040] It should be specifically noted that iterative calculation, as a key component in this embodiment, gradually optimizes the super-resolution reconstruction result of the low-resolution target image through multiple iterations. In each iteration, two core update processes are mainly involved: the iteration of the low-resolution target image and the iteration of the auxiliary variable. Specifically, this stage first receives the inputs from the initialization stage and the previous stage, including the reconstructed image and the auxiliary variable after the previous iteration, as well as the input variables that remain unchanged: the undersampled k-space data for numerical fidelity constraint, the high-resolution auxiliary image, and the texture information extracted from the high-resolution auxiliary image. In this stage, the reconstructed image is first processed by the numerical fidelity layer (DFL) and the structure texture refinement layer (STRL). Among them, DFL imposes global consistency constraints based on the undersampled k-space data to ensure that the reconstruction result conforms to the physical prior, while STRL utilizes the high-resolution auxiliary image and texture information to enhance the structural and texture expression of the target image, thereby generating a new reconstructed image after iteration. At the same time, the auxiliary variable is updated according to the change of the reconstructed image to generate a new auxiliary variable, which provides additional prior information for the subsequent optimization process. Finally, the updated reconstructed image and auxiliary variable are taken as outputs and passed to the next stage to continuously optimize the quality of super-resolution reconstruction. The reconstructed image obtained from the last iterative calculation is the super-resolution target image of this channel.

[0041] In the iterative calculation formula of the low-resolution target image in this embodiment, the iteration of the low-resolution target image is divided into two parts to give full play to the respective advantages of the k-space and the image domain: one part is the formula of the data fidelity layer (DFL) , which is mapped to the k-space through Fourier transform , and is iteratively updated in the k-space domain. By making full use of the global frequency information carried by the original k-space data, since the k-space data directly reflects the frequency components obtained during the MRI image sampling process, updating in the k-space can more effectively maintain the data consistency, thereby maintaining data reliability and physical feasibility and reducing the reconstruction error. After the update, the image is converted back to the image domain through inverse Fourier transform to ensure that the reconstructed image is consistent with the target modality in the spatial structure for further optimization in the image domain. The second part is the formula of the structure texture refinement layer (STRL) . In this embodiment, it is iteratively updated in the image domain. By introducing the auxiliary image as a structural prior, the model's ability to restore the overall structure of the target image is significantly improved. At the same time, the texture information is used to enhance the recovery of high-frequency features, strengthen the local detail expression and texture recovery ability, and further replace the matrix transformation operation with convolution-based sparse representation, that is .

[0042] Among them, 、 、 、 respectively represent the convolutional dictionary layer with a window size of 3×3, and respectively represent the convolutional dictionary layer and its transposed convolution. In this way, the k-space consistency ensures physical authenticity and global constraints, the auxiliary image provides structural guidance, and the texture information supplements high-frequency details. The three form a global-structure-texture collaborative mechanism, jointly promoting the model to perform high-quality restoration of undersampled images from multiple perspectives, so as to achieve medical image reconstruction results with more structural consistency and perceptual quality.

[0043] In the iteration of the auxiliary variable in this embodiment, this embodiment transforms the gradient operator of the prior information with respect to the auxiliary variable into the prior learning task of the deep neural network, regards it as the denoising process of the low-resolution target image, and automatically learns the prior distribution through the neural network, and adaptively adjusts the update strategy of the auxiliary variable during the optimization process to improve the reconstruction quality and enhance the generalization ability of the model.

[0044] Step S33, perform weighted fusion on the super-resolution target images of all channels to obtain the final super-resolution target image. The calculation formula is: , where, represents the super-resolution target image; represents the total number of channels; represents the channel number; represents the learnable weight of the th channel; represents the super-resolution target image of the

[0045] After this embodiment performs T rounds of iteration, a learnable channel weighted fusion mechanism is adopted to integrate the multi-channel super-resolution target images generated in the initialization stage to synthesize the super-resolution target image. Specifically, each channel represents different intermediate reconstruction results, which are gradually optimized in different iteration stages and capture multi-level information of the image in terms of structure and texture, etc. To fully integrate these reconstruction results, this embodiment introduces a channel-wise weighted fusion operation, that is, assigns a learnable weight to each channel , and perform a linear combination at each spatial position. During implementation, the weighted fusion operation is completed by a 1×1 convolutional layer, whose convolutional kernel linearly weights each intermediate result along the channel direction, thereby improving efficiency and trainability while maintaining the expressive ability. Through the above fusion mechanism, the model can adaptively synthesize the feature extraction results of the final stage, enhancing the detail performance and structural consistency of the target image. In addition, this method can also effectively suppress the noise or bias that may be introduced by single reconstruction, making the finally generated super-resolution target image more accurate and stable, thus improving the overall reconstruction quality.

[0046] Step S4, measure the loss function between the final super-resolution target image and the high-resolution target image to train the hybrid-domain multi-contrast variational network into an MRI super-resolution model.

[0047] Specifically, calculate the loss function between the final super-resolution target image and the high-resolution target image. If the loss function is acceptable, then the hybrid-domain multi-contrast variational network at this time is the MRI super-resolution model; if the loss function is unacceptable, then repeat step S3 to retrain the hybrid-domain multi-contrast variational network until the loss function is acceptable, and the hybrid-domain multi-contrast variational network at this time is the MRI super-resolution model.

[0048] The loss function The calculation formula of is: , where, represents the image reconstruction loss; represents the texture loss; and respectively represent weight coefficients, and the default settings are , .

[0049] It should be noted that, in order to improve the structural clarity and texture fidelity of the final super-resolution target image, in this embodiment, a texture loss is introduced on the basis of the traditional reconstruction loss, and a hybrid texture loss function is constructed to jointly optimize the content and edge details of the image. Specifically, the loss function includes two parts. One is the pixel-level L1 reconstruction loss, which is used to ensure the overall consistency between the final super-resolution target image and the high-resolution auxiliary image. The other is the texture loss extracted based on the Laplacian filter, which is used to enhance the reconstruction ability of high-frequency details such as edges and textures. The texture loss is achieved by calculating the L1 difference between the final super-resolution target image and the high-resolution target image in the texture domain, effectively guiding the network to focus on the accuracy of the structural region. This scheme can effectively enhance the texture and edge clarity of the image while maintaining the overall reconstruction accuracy, especially having a significant improvement effect on the restoration of tissue boundaries and lesion details in medical images.

[0050] Step S5: Input the actual low-resolution MRI image into the MRI super-resolution model to obtain the actual super-resolution image.

[0051] Based on the above embodiments, this embodiment further provides a hybrid-domain multi-contrast MRI super-resolution system based on a variational network. Refer to Figure 2 , which is the structural block diagram of the hybrid-domain multi-contrast MRI super-resolution system provided by the embodiment of the present invention. As Figure 2 shown, the system includes: A data acquisition module, configured to acquire a high-resolution auxiliary image, a high-resolution target image, and a mask; A preprocessing module, configured to preprocess the high-resolution auxiliary image, the high-resolution target image, and the mask to obtain a multi-channel high-resolution auxiliary image, a low-resolution target image, and a mask; A hybrid-domain multi-contrast variational network module, configured to iteratively update each channel of the low-resolution target image, and perform weighted fusion on the obtained super-resolution target images of all channels to obtain the final super-resolution target image; A deep supervision module, configured to measure the loss function between the final super-resolution target image and the high-resolution target image, and train the hybrid-domain multi-contrast variational network into an MRI super-resolution model; An image generation module, configured to input the actual low-resolution MRI image into the MRI super-resolution model to obtain the actual super-resolution image.

[0052] Refer to Figure 3 , which is the network structure diagram of the hybrid-domain multi-contrast variational network module provided by the embodiment of the present application. As Figure 3 shown, the hybrid-domain multi-contrast variational network module includes an initialization module, an iterative update module, and a reconstruction module, where An initialization module for initializing the hybrid-domain multi-contrast variational network; An iterative update module for iteratively updating each channel of the low-resolution target image to obtain a multi-channel super-resolution target image; A reconstruction module for weighted fusion of the multi-channel super-resolution target image to obtain the final super-resolution target image.

[0053] Furthermore, the iterative update module includes T iterative update sub-modules connected in series. Each iterative update sub-module includes a target image update module and an auxiliary variable update module. The target image update module includes a numerically fidelity layer (DFL) and a structure texture refinement layer (STRL) connected in parallel; the outputs of the numerically fidelity layer and the structure texture refinement layer of the previous iterative update sub-module are superimposed and fused as the input of the next iterative update sub-module; the outputs of the numerically fidelity layer and the structure texture refinement layer of the last iterative update sub-module are superimposed and fused as the input of the reconstruction module.

[0054] See Figure 4 , which is the network structure diagram of each module in the iterative update module provided by the embodiment of the present invention; among them, Figure (a) is the network structure diagram of the numerically fidelity layer, Figure (b) is the network structure diagram of the structure texture refinement layer, and Figure (c) is the network structure diagram of the auxiliary variable update module. As Figure 4 shown, the numerically fidelity layer is used for iterative expansion in the k-space domain, making full use of the global frequency information carried by the original data in the k-space, and performing iterative update in the k-space domain. Since the k-space data directly reflects the frequency components obtained during the MRI image sampling process, updating in the k-space can more effectively maintain data consistency and reduce reconstruction errors.

[0055] The numerically fidelity layer includes a Fourier transform module, a first formula calculation module, and a Fourier inverse transform module. Among them, The Fourier transform module is used for performing Fourier transform on the low-resolution target image and the reconstructed image to obtain undersampled k-space data and iteratively updated k-space data.

[0056] The first formula calculation module is used for iterative expansion in the k-space domain operation connection.

[0057] The Fourier inverse transform module is used for inverse-transforming the iteratively expanded k-space domain data back to the image domain to ensure that the reconstructed image and the target modality are consistent in spatial structure for further optimization in the image domain.

[0058] The output feature of the numerically fidelity layer (DFL) is: .

[0059] The structural texture refinement layer is used for iterative unfolding in the image domain, explicitly utilizing the information of the high-resolution auxiliary image and the similar texture features between different modality images to enhance the overall structure and local details of the reconstructed image.

[0060] The structural texture refinement layer includes a convolutional dictionary module and a second formula calculation module, where, The convolutional dictionary module is used to implement the transformation matrix in the iterative unfolding in the image domain. The convolutional dictionary module includes 4 convolutional dictionary layers 、 、 、 and 2 corresponding transposed convolutional dictionary layers and .

[0061] The convolutional dictionary layer is used to implement the transformation matrix in the iterative unfolding in the image domain A , and includes two convolutional layers and a ReLU activation layer, with a convolutional kernel size of 3×3. The convolutional dictionary layer is used to implement the transformation matrix in the iterative unfolding in the image domain B , and includes two convolutional layers and a ReLU activation layer, with a convolutional kernel size of 3×3. The convolutional dictionary layer is used to implement the transformation matrix in the iterative unfolding in the image domain C , and includes two convolutional layers and a ReLU activation layer, with a convolutional kernel size of 3×3. The convolutional dictionary layer is used to implement the transformation matrix in the iterative unfolding in the image domain D , and includes two convolutional layers and a ReLU activation layer, with a convolutional kernel size of 3×3.

[0062] The transposed convolutional dictionary layer is used to implement the transposed matrix in the iterative unfolding in the image domain , and includes two transposed convolutional layers and a ReLU activation layer, with a convolutional kernel size of 3×3. The transposed convolutional dictionary layer is used to implement the transformation matrix in the iterative unfolding in the image domain , and includes two transposed convolutional layers and a ReLU activation layer, with a convolutional kernel size of 3×3.

[0063] The second formula calculation module is used for the arithmetic connection of the iterative unfolding in the image domain .

[0064] The second formula calculation module includes an auxiliary image calculation module, a texture calculation module, an auxiliary variable calculation module, and an integration module. The auxiliary image calculation module is used to denoise the high-resolution auxiliary image related to the iterative unfolding in the image domain, that is, The texture calculation module is used for iterative expansion in the image domain related to texture information, that is The auxiliary variable calculation module is used for iterative expansion in the image domain related to auxiliary variables, that is The integration module is used to integrate the calculation results of the auxiliary image calculation module, the texture calculation module, and the auxiliary variable calculation module, that is , and complete the iterative expansion in the image domain.

[0065] The output features of the structure texture refinement layer (STRL) are: .

[0066] The auxiliary variable update module includes a prior information module and a third formula calculation module.

[0067] The prior information module is used to automatically learn the prior distribution through the neural network U-Net.

[0068] The third formula calculation module is used for the iterative expansion of the auxiliary variable W . .

[0069] The output features of the auxiliary variable update module are: .

[0070] See Figure 5 , which is a comparison chart of the high-resolution target image, high-resolution auxiliary image, low-resolution target image, and super-resolution target image provided by the embodiment of the present invention. As Figure 5 shown, it can be clearly seen that the super-resolution target image generated by the method and system provided by the embodiment of the present disclosure is very similar to the high-resolution target image, proving the effectiveness of the method and system provided by the embodiment of the present disclosure.

[0071] Corresponding to the above embodiment, the embodiment of the present invention also provides an electronic device.

[0072] See Figure 6 , which is a schematic structural diagram of an electronic device provided by the embodiment of the present invention. As Figure 6 shown, the electronic device 600 may include: a processor 601, a memory 602, and a communication unit 603. These components communicate through one or more buses. Those skilled in the art can understand that the structure of the electronic device shown in the figure does not constitute a limitation on the embodiment of the present invention. It can be a bus structure, a star structure, and may also include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0073] Among them, the communication unit 603 is used to establish a communication channel so that the electronic device can communicate with other devices.

[0074] The processor 601 is the control center of the electronic device, connecting various parts of the entire electronic device through various interfaces and circuits. By running or executing software programs and / or modules stored in the memory 602, and by invoking data stored in the memory, it performs various functions of the electronic device and / or processes data. The processor may be composed of an integrated circuit (IC), for example, it may be composed of a single packaged IC, or it may be composed of multiple packaged ICs with the same or different functions connected together. For example, the processor 601 may only include a central processing unit (CPU). In the embodiment of the present invention, the CPU may be a single computing core or may include multiple computing cores.

[0075] The memory 602 is used to store the execution instructions of the processor 601. The memory 602 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0076] When the execution instructions in the memory 602 are executed by the processor 601, the electronic device 600 is enabled to execute some or all of the steps in the above method embodiments.

[0077] Corresponding to the above embodiments, the embodiment of the present invention also provides a computer-readable storage medium. The computer-readable storage medium can store a program, and when the program runs, it can control the device where the computer-readable storage medium is located to execute some or all of the steps in the above method embodiments. Specifically, the computer-readable storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), etc.

[0078] Corresponding to the above embodiments, the embodiment of the present invention also provides a computer program product. The computer program product contains executable instructions, and when the executable instructions are executed on a computer, the computer is enabled to execute some or all of the steps in the above method embodiments.

[0079] In the embodiments of the present invention, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the association relationship of associated objects and indicates that there can be three relationships. For example, A and / or B can represent the cases of A existing alone, A and B existing simultaneously, and B existing alone. Here, A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one of the following" and its similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, and c can be single or multiple.

[0080] Those of ordinary skill in the art can realize that the various units and algorithm steps described in the embodiments disclosed herein can be implemented by a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0081] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0082] In several embodiments provided by the present invention, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0083] The above is only the specific implementation manner of the present invention. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention and should be covered by the protection scope of the present invention. The protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A variational network-based multi-contrast MRI super-resolution method in the hybrid domain, characterized in that, Including: S1. Obtain a high-resolution auxiliary image, a high-resolution target image, and a mask; S2. Preprocess the high-resolution auxiliary image, the high-resolution target image, and the mask to obtain a multi-channel high-resolution auxiliary image, a low-resolution target image, and a mask; S3. The hybrid-domain multi-contrast variational network iteratively updates each channel of the low-resolution target image, and performs weighted fusion on the obtained super-resolution target images of all channels to obtain the final super-resolution target image; S4. Measure the loss function between the final super-resolution target image and the high-resolution target image, and train the hybrid-domain multi-contrast variational network into an MRI super-resolution model; S5. Input the actual low-resolution MRI image into the MRI super-resolution model to obtain the actual super-resolution image.

2. The super-resolution method for hybrid-domain multi-contrast MRI based on a variational network according to claim 1, wherein The preprocessing specifically is: Generate a low-resolution target image based on the high-resolution target image and the mask; Through a channel expansion strategy, copy the high-resolution auxiliary image, the low-resolution target image, and the mask along the channel dimension, and expand the high-resolution auxiliary image, the low-resolution target image, and the mask from a single channel to a multi-channel representation.

3. The super-resolution method for multi-contrast MRI in the hybrid domain based on the variational network according to claim 1, wherein S3 specifically is: Initialize the hybrid-domain multi-contrast variational network; Perform iterative calculations on each channel of the low-resolution target image and the auxiliary variable to obtain the super-resolution target images of all channels; Perform weighted fusion on the super-resolution target images of all channels to obtain the final super-resolution target image.

4. The super-resolution method for hybrid-domain multi-contrast MRI based on variational network according to claim 3, wherein, Initializing the hybrid-domain multi-contrast variational network specifically is: Initialize the low-resolution target image and the high-resolution auxiliary image using the denoising network U-Net to obtain an initial low-resolution target image and a denoised high-resolution auxiliary image ; The low-resolution target image is Fourier-transformed to obtain undersampled k-space data ; The texture information is extracted by applying the Laplace operator to the high-resolution auxiliary image G ; Set the initial auxiliary variable to .

5. The super-resolution method for hybrid-domain multi-contrast MRI based on a variational network according to claim 3, wherein The calculation formula for performing iterative calculations on each channel of the low-resolution target image and the auxiliary variable is: ， ， Among them, , respectively represent the reconstructed images of the th and th iterations; represents the inverse Fourier transform; represents the k-space data corresponding to the reconstructed image during the iterative process; , , , , and all represent the trade-off parameters of the th iteration; represents the mask; represents the undersampled k-space data; , , , respectively represent the convolutional dictionary layers; and respectively represent the transposed convolutions of the convolutional dictionary layers and ; represents the denoised high-resolution auxiliary image; G represents the texture information; and respectively represent the auxiliary variables of the th and th iterations; represents the U-Net network.

6. The method for super-resolution of multi-contrast MRI in a hybrid domain based on a variational network according to claim 3, wherein The calculation formula for performing weighted fusion on the super-resolution target images of all channels is: ， Among them, represents the super-resolution target image; represents the total number of channels; represents the channel number; represents the learnable weight of the th channel; represents the super-resolution target image of the th channel.

7. A variational network-based hybrid-domain multi-contrast MRI super-resolution system, characterized in that, Including: A data acquisition module for obtaining a high-resolution auxiliary image, a high-resolution target image, and a mask; A preprocessing module for preprocessing the high-resolution auxiliary image, the high-resolution target image, and the mask to obtain a multi-channel high-resolution auxiliary image, a low-resolution target image, and a mask; A hybrid-domain multi-contrast variational network module for iteratively updating each channel of the low-resolution target image and performing weighted fusion on the obtained super-resolution target images of all channels to obtain the final super-resolution target image; A deep supervision module for measuring the loss function between the final super-resolution target image and the high-resolution target image and training the hybrid-domain multi-contrast variational network into an MRI super-resolution model; An image generation module for inputting the actual low-resolution MRI image into the MRI super-resolution model to obtain the actual super-resolution image.

8. The super-resolution system for hybrid-domain multi-contrast MRI based on a variational network according to claim 7, wherein The hybrid-domain multi-contrast variational network module includes an initialization module, an iterative update module, and a reconstruction module; The iterative update module includes T iterative update sub-modules connected in series, and each iterative update sub-module includes a target image update module and an auxiliary variable update module; The target image update module includes a numerical fidelity layer and a structure texture refinement layer connected in parallel.

9. The super-resolution system for hybrid-domain multi-contrast MRI based on a variational network according to claim 8, wherein The numerical fidelity layer includes a Fourier transform module, a first formula calculation module, and a Fourier inverse transform module, where A Fourier transform module, which is used to perform Fourier transform on a low-resolution target image and a reconstructed image to obtain undersampled k-space data and iteratively updated k-space data; The first formula calculation module is used for the operation connection of the iterative expansion in the k-space domain of the operation connection; An inverse Fourier transform module, which is used to inverse-transform the iteratively expanded k-space domain data back to the image domain to ensure that the reconstructed image is consistent with the target modality in terms of spatial structure.

10. The super-resolution system for hybrid-domain multi-contrast MRI based on a variational network according to claim 8, wherein, The structural texture refinement layer includes a convolutional dictionary module and a second formula calculation module, where The convolutional dictionary module is used to implement the transformation matrix in the iterative expansion in the image domain; The second formula calculation module includes an auxiliary image calculation module, a texture calculation module, an auxiliary variable calculation module, and an integration module; the auxiliary image calculation module is used to denoise the high-resolution auxiliary image related iterative expansion in the image domain; the texture calculation module is used for iterative expansion in the image domain related to texture information; the auxiliary variable calculation module is used for iterative expansion in the image domain related to auxiliary variables; the integration module is used to integrate the calculation results of the auxiliary image calculation module, the texture calculation module, and the auxiliary variable calculation module to complete the iterative expansion in the image domain.

Images