Multi-contrast nuclear magnetic resonance reconstruction method and device based on multi-domain learning, terminal and medium

Abstract

The application provides a multi-contrast nuclear magnetic resonance reconstruction method and device based on multi-field learning, a terminal and a medium. A full-sampling auxiliary contrast nuclear magnetic resonance image is obtained and input into a pre-trained cross-modal generation network based on an image domain and a frequency domain. A full-sampling auxiliary contrast nuclear magnetic resonance image with an offset is output as a gold standard. The weights of the cross-modal generation network based on the image domain and the frequency domain are fixed, and the full-sampling auxiliary contrast nuclear magnetic resonance image with the offset output by each cross-modal generation network is input into a registration network of a corresponding domain together with an original undersampling auxiliary contrast nuclear magnetic resonance image. An aligned image of each domain aligned with the original undersampling auxiliary contrast nuclear magnetic resonance image is output. The aligned image of each domain and the original undersampling auxiliary contrast nuclear magnetic resonance image are commonly input into a reconstruction network of a corresponding domain, and a nuclear magnetic resonance reconstruction image is output.

Description

Technical Field

[0001] This application relates to the field of nuclear magnetic resonance reconstruction technology, and in particular to multi-contrast nuclear magnetic resonance reconstruction methods, devices, terminals and media based on multi-domain learning. Background Technology

[0002] Magnetic resonance imaging (MRI), a non-invasive and radiation-free medical imaging technique, plays a vital role in clinical diagnosis. However, the sharp noise generated during MRI scans and the confined space of the scanning instrument can easily lead to a poor patient experience. Furthermore, the long scan time per session results in high costs. Therefore, various techniques are being developed to accelerate sampling, including designing reconstruction models and optimizing undersampling trajectories to improve reconstruction performance as much as possible even with high-magnification undersampling. However, due to the difficulty of the task and the fact that existing methods do not fully utilize the various types of information in the data, this paper proposes a framework that simultaneously utilizes information from the image and frequency domains, enabling already acquired contrast images to better assist in the acquisition of another contrast image.

[0003] MRI reconstruction has been extensively studied to date. From traditional reconstruction models based on compressed sensing to parallel imaging, which is now mature and used in scanners from major manufacturers, and even to the rapidly developing deep learning-based reconstruction methods in recent years, deep learning reconstruction methods have played a significant role in multi-contrast MRI reconstruction. For example, Xiang et al. used simple stacking of multi-contrast images as model input for reconstruction; Dar et al. utilized high-frequency information from acquired contrast images to supplement undersampled contrast; Xuan et al. employed a strategy of cross-modal generation and registration of acquired contrast images to better assist multi-contrast MRI reconstruction. Furthermore, the fact that raw MRI data can be obtained from k-space allows information from both the frequency and image domains to be utilized simultaneously. For instance, Feng et al. used complex networks to process the low-frequency and high-frequency regions of k-space signals in MRI separately; Zhou et al. improved reconstruction results through iterative optimization targeting k-space and image. However, there is currently no similar work that utilizes multiple types of data to improve the performance of multi-contrast reconstruction methods. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a multi-contrast nuclear magnetic resonance reconstruction method, apparatus, terminal and medium based on multi-domain learning, so as to solve the problem of poor performance of multi-contrast nuclear magnetic resonance reconstruction based on multi-domain learning.

[0005] To achieve the above and other related objectives, a first aspect of this application provides a multi-contrast NMR reconstruction method based on multi-domain learning, comprising: acquiring NMR images with full-sampling assisted contrast and inputting them into pre-trained cross-modal generation networks based on the image domain and frequency domain, respectively; using the NMR images with full-sampling assisted contrast as the gold standard, and outputting offset NMR images with full-sampling assisted contrast respectively; fixing the weights of the cross-modal generation networks based on the image domain and frequency domain, and inputting the offset NMR images with full-sampling assisted contrast output by each cross-modal generation network, together with the original undersampled NMR images with assisted contrast, into a registration network of the corresponding domain, and outputting aligned images of each domain that are aligned with the original undersampled NMR images with assisted contrast; inputting the aligned images of each domain and the original undersampled NMR images with assisted contrast into a reconstruction network of the corresponding domain, and outputting reconstructed NMR images respectively.

[0006] In some embodiments of the first aspect of this application, the cross-modal generation network, registration network, and reconstruction network can all be obtained by training the U-Net network using training data. The training process includes: acquiring data based on various MRI data scanned by an MRI scanning device; the various MRI data include actual full-sampling multi-contrast MRI data; preprocessing the acquired data, the preprocessing process including converting 3D image data into 2D image data, normalizing the data, extracting images usable for full-sampling data and breaking them down into an image dataset, dividing the image dataset into a training set and a test set according to a preset ratio; inputting the training set into the U-Net network, training the model according to a corresponding set objective function and an optimization algorithm; testing the trained model based on the test set, and determining the model performance by comparing the test results with a preset index of the gold standard, thereby obtaining the corresponding cross-modal generation network, registration network, or reconstruction network.

[0007] In some embodiments of the first aspect of this application, the objective function corresponding to the cross-modal generation network includes: in, It is the loss function of the cross-modal generative network corresponding to the image domain; It is the loss function of the cross-modal generator network corresponding to the frequency domain; It is the loss function of the cross-modal generative network after inverse Fourier transform to the image domain; α is the loss function of the cross-modal generative network after Fourier transform to the frequency domain; α is the weight of the frequency domain branch relative to the image domain branch; β is the weight of the dual-domain consistency relative to the original training branch.

[0008] In some embodiments of the first aspect of this application, the objective function corresponding to the registration network includes: in, It is the loss function of the registration network corresponding to the image domain; It is the loss function of the registration network corresponding to the frequency domain; It is the loss function of the registration network after inverse Fourier transform to the image domain; α is the loss function of the registration network after Fourier transform to the frequency domain; α is the weight of the frequency domain branch relative to the image domain branch; β is the weight of the dual-domain consistency relative to the original training branch.

[0009] In some embodiments of the first aspect of this application, the objective function corresponding to the reconstruction network includes: in, It is the loss function of the reconstruction network corresponding to the image domain; It is the loss function of the reconstruction network corresponding to the frequency domain; It is the loss function for reconstructing the network to the image domain via inverse Fourier transform; α is the loss function of the reconstructed network after Fourier transform to the frequency domain; α is the weight of the frequency domain branch relative to the image domain branch; β is the weight of the dual-domain consistency relative to the original training branch.

[0010] In some embodiments of the first aspect of this application, based on the symmetry between the image domain and the frequency domain, the output images of the cross-modal generation network, registration network and reconstruction network in one domain can be constrained by Fourier transform / inverse Fourier transform of the output images of the cross-modal generation network, registration network and reconstruction network in the other domain.

[0011] To achieve the above and other related objectives, a second aspect of this application provides a multi-contrast NMR reconstruction device based on multi-domain learning, comprising: a cross-modal generation module, configured to acquire NMR images with full-sampling assisted contrast and input them into pre-trained cross-modal generation networks based on the image domain and frequency domain, using the NMR images with full-sampling assisted contrast as the gold standard, and output offset NMR images with full-sampling assisted contrast respectively; a registration module, configured to fix the weights of the cross-modal generation networks based on the image domain and frequency domain, and input the offset NMR images with full-sampling assisted contrast output by each cross-modal generation network, together with the original undersampled NMR images with assisted contrast, into a registration network of the corresponding domain, and output aligned images of each domain that are aligned with the original undersampled NMR images with assisted contrast; and a reconstruction module, configured to input the aligned images of each domain and the original undersampled NMR images with assisted contrast into a reconstruction network of the corresponding domain, and output reconstructed NMR images respectively.

[0012] In some embodiments of the second aspect of this application, a model training module is further included, used to train the U-Net network using training data to obtain the cross-modal generation network, registration network, and reconstruction network; the training process of the model training module includes: acquiring data based on various MRI data scanned by an MRI scanning device; the various MRI data include actual full-sampling multi-contrast MRI data; preprocessing the acquired data, the preprocessing process including converting 3D image data into 2D image data and performing normalization processing, then extracting images usable for full-sampling data and breaking them into image datasets, dividing the image datasets into training sets and test sets according to a preset ratio; inputting the training set into the U-Net network, and training the model according to a corresponding set objective function and based on an optimization algorithm; testing the trained model based on the test set, and determining the model performance by comparing the test results with a preset index of the gold standard, thereby obtaining the corresponding cross-modal generation network, registration network, or reconstruction network.

[0013] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-contrast nuclear magnetic resonance reconstruction method based on multi-domain learning.

[0014] To achieve the above and other related objectives, a fourth aspect of this application provides an electronic terminal, comprising: a processor and a memory; the memory for storing a computer program, and the processor for executing the computer program stored in the memory, so that the terminal executes the multi-contrast nuclear magnetic resonance reconstruction method based on multi-domain learning.

[0015] As described above, the multi-contrast nuclear magnetic resonance reconstruction method, apparatus, terminal, and medium based on multi-domain learning of this application have the following beneficial effects:

[0016] (1) This invention provides a fusion strategy based on cross-modal generation and registration to better extract information from the auxiliary contrast image that can be used for reconstruction of the auxiliary contrast image by minimizing the differences between the auxiliary contrast image and the assisted contrast image. In contrast to existing methods, which simply stack the original multi-contrast images along the channel dimension as input to the reconstruction network, no further operations are performed to allow the full-color contrast image to better play its complementary auxiliary role. The technical solution of this invention, through a cross-modal generation and registration network, achieves the following: First, a full-sampled image corresponding to the undersampled contrast is generated from the full-sampled contrast; second, the full-sampled image and the undersampled image are aligned to eliminate image misalignment caused by human motion during multi-contrast acquisition. The image after this transformation can better reflect the information of the undersampled contrast contained in the full-sampled contrast image, thereby better assisting reconstruction.

[0017] (2) This invention proposes a dual-domain (frequency domain and image domain) framework for synergistic optimization of the domain networks, thereby better utilizing the raw signals acquired by MRI and can be applied in conjunction with the proposed fusion strategy to further improve MRI reconstruction performance. Existing implementation methods mainly fall into two categories: one is iterative optimization of the two domains (e.g., image domain optimization followed by conversion to the frequency domain and then optimization in the frequency domain, repeating this process); the other is optimization of the two domains separately followed by a simple stacking fusion similar to that used in multi-contrast image reconstruction. While current strategies allow the optimization problem to be performed simultaneously across multiple domains, the synergistic effect between the models in the two domains has not been optimally utilized. Therefore, this technique amplifies the synergistic effect between domains through inter-domain consistency constraints output from intermediate steps, and this synergistic framework can be applied simultaneously on top of other dual-domain optimization methods, further improving performance. Attached Figure Description

[0018] Figure 1 The diagram shown is a flowchart of a multi-contrast nuclear magnetic resonance reconstruction method based on multi-domain learning, according to an embodiment of this application.

[0019] Figure 2 The flowchart shown is a multi-contrast nuclear magnetic resonance reconstruction method based on multi-domain learning according to an embodiment of this application.

[0020] Figure 3 The diagram shown is a structural schematic of an electronic terminal according to an embodiment of this application.

[0021] Figure 4 The diagram shown is a schematic diagram of a multi-contrast nuclear magnetic resonance reconstruction device based on multi-domain learning in one embodiment of this application. Detailed Implementation

[0022] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0023] As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of the stated feature, operation, element, component, item, kind, and / or group, but do not preclude the presence, occurrence, or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms “or” and “and / or” as used herein are interpreted as inclusive, or mean any one or any combination thereof. Thus, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” Exceptions to this definition occur only when combinations of elements, functions, or operations are inherently mutually exclusive in some manner.

[0024] To address the problems in the aforementioned background technology, this invention provides a multi-contrast MRI reconstruction method, apparatus, terminal, and medium based on multi-domain learning. Building upon multi-contrast MRI reconstruction and dual-domain single-contrast MRI reconstruction, this invention further improves and integrates the main ideas of these two methods, thereby maximizing the auxiliary effect of full-sampling contrast. Considering the theoretical feasibility of utilizing more information to improve reconstruction results, this invention proposes a method that can simultaneously utilize multi-domain and multi-contrast information to enhance reconstruction performance. Meanwhile, to make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments of this invention are further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention.

[0025] Before providing a further detailed description of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:

[0026] <1> MRI (Magnetic Resonance Imaging): Magnetic resonance imaging is achieved by using an external high-frequency magnetic field to generate signals from the energy radiated from substances within the body to the surrounding environment. The imaging process is similar to image reconstruction and CT, except that MRI does not rely on external radiation, absorption, or reflection, nor on the gamma radiation of radioactive substances within the body. Instead, it uses the interaction between an external magnetic field and an object to create an image. High-energy magnetic fields are harmless to the human body.

[0027] <2> RC (Reference Contrast): The reference contrast is used to assist in the reconstruction of images with undersampled contrast.

[0028] <3> TC (Target Contrast): The target contrast, relative to the reference contrast of full sampling, is the auxiliary contrast.

[0029] <4> U-Net network: A semantic segmentation network originally designed to solve problems in biomedical images. Due to its excellent performance, it has been widely used in various aspects of semantic segmentation, such as satellite image segmentation or industrial defect detection. The U-Net network has a symmetrical network structure resembling the letter U, with the left half for feature extraction and the right half for upsampling, forming an Encoder-Decoder structure.

[0030] This invention provides a multi-contrast NMR reconstruction method based on multi-domain learning, a system for implementing the multi-contrast NMR reconstruction method based on multi-domain learning, and a storage medium storing an executable program for implementing the multi-contrast NMR reconstruction method based on multi-domain learning. Regarding the implementation of the multi-contrast NMR reconstruction method based on multi-domain learning, this invention will describe exemplary implementation scenarios.

[0031] like Figure 1 The diagram illustrates a flowchart of a multi-contrast NMR reconstruction method based on multi-domain learning, as described in an embodiment of the present invention. The multi-contrast NMR reconstruction method based on multi-domain learning in this embodiment mainly includes the following steps:

[0032] Step S11: Obtain the NMR image with full sampling assisted contrast and input it into the pre-trained cross-modal generation network based on the image domain and frequency domain respectively. Using the NMR image with full sampling assisted contrast as the gold standard, output the offset NMR image with full sampling assisted contrast respectively.

[0033] It should be noted that, for ease of understanding by those skilled in the art, in this embodiment, the NMR image with full-sampling assisted contrast is named RC image; the NMR image with full-sampling assisted contrast is named full-sampling TC image; and the offset NMR image with full-sampling assisted contrast is named TC. S The original undersampled NMR image with enhanced contrast is named undersampled TC. U ; will be compared with the undersampled TC U The aligned image is named TC SR Those skilled in the art will understand that the full names or abbreviations appearing in the following description actually have the same meaning.

[0034] In this embodiment, the offset NMR image refers to the NMR image with assisted contrast that is fully sampled, meaning the image output by the cross-modal generation network is offset relative to the NMR image with assisted contrast. Specifically, the input data to the cross-modal generation network is RC image data, and the output is TC image data. S Image data, TC S The image data has an offset relative to the TC image data.

[0035] It should be understood that the gold standard refers to the most reliable method for diagnosing diseases currently recognized in clinical medicine. The purpose of using the gold standard is to accurately distinguish whether a subject is a patient with a certain disease. For example, commonly used gold standards include, but are not limited to, biopsy, surgical findings, microbial culture, autopsy, special examinations, and imaging diagnosis. Full sampling and oversampling are techniques used in machine learning to handle imbalanced classification problems. If the ratio of positive to negative samples in the training set is severely imbalanced (e.g., there are 1000 positive samples and 100,000 negative samples, a ratio as high as 1:100, which is severely imbalanced), full sampling (or oversampling) repeatedly selects samples from the smaller class to make the severely imbalanced sample ratio more balanced; undersampling randomly selects samples from the larger class to make the severely imbalanced sample ratio more balanced.

[0036] In this embodiment, "cross-modal" refers to spanning multiple modalities. A modality refers to the form in which data exists, broadly encompassing file formats such as text, audio, images, and video. Some data may exist in different forms but all describe the same thing or event. Therefore, the types of data input into the cross-modal generation network can be images, videos, text, audio, etc. Compared to the single-modal approach used in traditional modeling, the technical solution in this embodiment can accommodate more raw data and achieve better modeling results. In this embodiment, "modality" specifically refers to the contrast of MRI images; MRI images with different contrasts contain different information about the same object.

[0037] In some examples of this embodiment, during the training of the cross-modal generative network, a corresponding first composite objective function is constructed based on the loss function in the image domain and the loss function in the frequency domain; wherein, the expression of the first composite objective function is as follows:

[0038]

[0039] in, It is the loss function of the cross-modal generative network corresponding to the image domain; It is the loss function of the cross-modal generator network corresponding to the frequency domain; It is the loss function of the cross-modal generative network after inverse Fourier transform to the image domain; α is the loss function of the cross-modal generative network after Fourier transform to the frequency domain; α is the weight of the frequency domain branch relative to the image domain branch; β is the weight of the dual-domain consistency relative to the original training branch; (e.g., α = 0.1, β = 0.5).

[0040] In this embodiment, the cross-modal generation network can be obtained by training the U-Net network using training data. The training process includes the following:

[0041] First, data acquisition is performed: several sets of actual full-sampling single-contrast (e.g., T1w) MRI data (which may be 3D data) from an MRI scanner and corresponding full-sampling or under-sampling single-contrast (e.g., T2w) MRI data (for training and inference).

[0042] Next, data preprocessing is performed: first, the 3D images in the acquired image data are cropped into 2D images in the image domain (along any direction), and then all acquired image data are normalized; then, pairs of usable images from the acquired data are extracted and broken down to form an image dataset; the image dataset is divided into a training set and a test set according to a preset ratio (this preset ratio can be set based on experience, such as 7:3, or it can be set randomly).

[0043] Finally, the objective function (as in Formula 1 above) and optimization algorithm are set. The training set is input into the U-Net network and the model is trained according to the objective function and optimization algorithm. After training, the model is tested using the test set, and the performance of the model is judged based on the metrics (such as PSNR peak signal-to-noise ratio) between the model and the gold standard, so as to obtain a cross-modal generative network that can be used in this embodiment.

[0044] It should be noted that the optimization algorithm is used to find a set of parameters that optimizes the objective function given the objective function. The optimization algorithms in this embodiment include, but are not limited to, SGD (stochastic gradient descent), BGD (batch gradient descent), MBGD (Mini-batch gradient descent), Adam (Adaptive Moment Estimation) algorithm, etc. Since these algorithms are already existing, they will not be described in detail.

[0045] Step S12: Fix the weights of the cross-modal generation network based on the image domain and frequency domain, and input the NMR image with full sampling and auxiliary contrast offset output by each of the cross-modal generation networks, together with the original NMR image with undersampled and auxiliary contrast, into the registration network of the corresponding domain, and output the aligned image of each domain that is aligned with the original NMR image with undersampled and auxiliary contrast.

[0046] For example, using a cross-modal generation network A1 as the cross-modal generation network in the image domain and a cross-modal generation network A2 as the cross-modal generation network in the frequency domain, the weights of cross-modal generation networks A1 and A2 can be fixed. These weights are obtained by training the networks according to the loss function corresponding to Formula 1. Cross-modal generation network A1 outputs image B1, and cross-modal generation network A2 outputs image B2. Image B1 and the original undersampled NMR image C with auxiliary contrast are input together into the image domain registration network D1, and the output is the aligned image E1 in the image domain. Image B2 and the original undersampled NMR image C with auxiliary contrast are input together into the frequency domain registration network D2, and the output is the aligned image E2 in the frequency domain.

[0047] In some examples of this embodiment, during the training of the registration network, a corresponding second composite objective function is constructed based on the loss function in the image domain and the loss function in the frequency domain; wherein, the expression of the second composite objective function is as follows:

[0048]

[0049] in, It is the loss function of the registration network corresponding to the image domain; It is the loss function of the registration network corresponding to the frequency domain; It is the loss function of the registration network after inverse Fourier transform to the image domain; α is the loss function of the registration network after Fourier transform to the frequency domain; α is the weight of the frequency domain branch relative to the image domain branch; β is the weight of the dual-domain consistency relative to the original training branch; (e.g., α = 0.1, β = 0.5).

[0050] In this embodiment, the registration network can be obtained by training the U-Net network using training data. The training process includes the following:

[0051] The objective function (as described in Formula 2 above) and optimization algorithm are set. The required data (images B1 / B2 generated in the steps corresponding to cross-modal generation networks A1 / A2 in Formula 1) and the original undersampled MRI image C with auxiliary contrast are input into the U-Net network. The model is then trained according to the objective function and optimization algorithm. After training, the model is tested using the test set. The performance of the model is judged based on the metrics (e.g., peak signal-to-noise ratio PSNR) between the model and the gold standard to obtain a registration network that can be used in this embodiment.

[0052] It should be noted that the optimization algorithm is used to find a set of parameters that optimizes the objective function given the objective function. The optimization algorithms in this embodiment include, but are not limited to, SGD (stochastic gradient descent), BGD (batch gradient descent), MBGD (Mini-batch gradient descent), Adam (Adaptive Moment Estimation) algorithm, etc. Since these algorithms are already existing, they will not be described in detail.

[0053] Step S13: Input the aligned images of each domain and the original undersampled NMR images with auxiliary contrast into the corresponding domain reconstruction network to output the NMR reconstructed images respectively.

[0054] For example, the aligned image E1 in the image domain and the original undersampled NMR image C with auxiliary contrast are input into the reconstruction network F1 in the image domain, and the output is the NMR reconstructed image G1 in the image domain; the aligned image E2 in the frequency domain and the original undersampled NMR image C with auxiliary contrast are input into the reconstruction network F2 in the frequency domain, and the output is the NMR reconstructed image G2 in the frequency domain.

[0055] In some examples of this embodiment, during the training of the reconstruction network, a corresponding third composite objective function is constructed based on the loss function in the image domain and the loss function in the frequency domain; wherein, the expression of the third composite objective function is as follows:

[0056]

[0057] in, It is the loss function of the reconstruction network corresponding to the image domain; It is the loss function of the reconstruction network corresponding to the frequency domain; It is the loss function for reconstructing the network to the image domain via inverse Fourier transform; α is the loss function of the reconstructed network after Fourier transform to the frequency domain; α is the weight of the frequency domain branch relative to the image domain branch; β is the weight of the dual-domain consistency relative to the original training branch; (e.g., α = 0.1, β = 0.5).

[0058] In this embodiment, the reconstructed network can be obtained by training the U-Net network using training data. The training process includes the following:

[0059] The objective function (as described in Formula 2 above) and optimization algorithm are set. The required data (aligned images E1 / E2 and the original undersampled NMR image C with auxiliary contrast) are input into the U-net network and the model is trained according to the objective function and optimization algorithm. After training, the model is tested using the test set, and the performance of the model is judged based on the indicators (such as PSNR peak signal-to-noise ratio) between it and the gold standard, so as to obtain a reconstruction network that can be used in this embodiment.

[0060] It should be noted that the optimization algorithm is used to find a set of parameters that optimizes the objective function given the objective function. The optimization algorithms in this embodiment include, but are not limited to, SGD (stochastic gradient descent), BGD (batch gradient descent), MBGD (Mini-batch gradient descent), Adam (Adaptive Moment Estimation) algorithm, etc. Since these algorithms are already existing, they will not be described in detail.

[0061] In some examples of this embodiment, after all the above-mentioned artificial intelligence neural networks have been trained, the weight data of each network can be stored for use in multi-contrast MRI reconstruction tasks when the gold standard is unknown.

[0062] In a preferred embodiment of this example, based on the symmetry between the image domain and the frequency domain, the output images of the cross-modal generation network, registration network, and reconstruction network in one domain can be similarly constrained by the output images of the cross-modal generation network, registration network, and reconstruction network in the other domain after undergoing Fourier transform / inverse Fourier transform. That is, for the output image of the neural network in each stage, the similarity constraint can be applied by the output of another branch after undergoing Fourier transform / inverse Fourier transform, thereby improving the quality of image reconstruction. For example, the image-domain cross-modal generation network A1 outputs image B1, and the frequency-domain cross-modal generation network A2 outputs image B2. Due to the symmetry between the image and frequency domains, the image obtained after the Fourier transform of image B1 should theoretically be image B2. Therefore, the output of the cross-modal generation network A2 can be constrained by the Fourier transform result of image B1. Conversely, the image obtained after the inverse Fourier transform of image B2 should theoretically be image B1. Therefore, the output of the cross-modal generation network A1 can also be constrained by the inverse Fourier transform result of image B2. Based on the same principle, the outputs of the registration and reconstruction networks in the image and frequency domains can also be constrained by symmetry, which will not be elaborated here.

[0063] To facilitate understanding by those skilled in the art, the following is combined with Figure 2 The flowchart of the multi-contrast NMR reconstruction method based on multi-domain learning further illustrates this point:

[0064] In this flowchart, the upper part represents the image domain, and the lower part represents the frequency domain. The RC image represents a full-sampling assisted contrast NMR image, and TC... U The image represents the original undersampled NMR image with enhanced contrast, TC SR Image representation with the undersampled TC U Image-aligned image, TC S The image represents an undersampled MRI image with auxiliary contrast that is offset relative to the TC image.

[0065] The RC image is first input into the image domain and frequency domain cross-modal generation network respectively to generate the corresponding TC. S Image. The TC generated at this time. S Because the image may have an offset relative to the TC image, it needs to undergo registration processing to align it with the undersampled TC image. U Image alignment, therefore the TC of each field is... S Images and TC U The images are input together into the registration network of the corresponding domain to output the TC of the image domain respectively. SR TC in image and frequency domain SR Images. TCs in each domain SR Images compared with undersampled TCU The image is input into the reconstruction network of the corresponding domain and outputs the reconstructed TC image.

[0066] Given raw MRI data from both full-sampling resonant radiography (RC) and under-sampling transcranial Doppler (TC), the trained framework can be used to reconstruct the under-sampling TC data. First, the given full-sampling RC data is used as input to a cross-modal generation network to generate its corresponding TC contrast data. This data, along with the under-sampling TC data, is then used as input to a registration network to align with the under-sampling TC data. Finally, the aligned MRI data generated from the RC data is used as input to a reconstruction network to obtain the reconstructed data.

[0067] Considering the current universality of multi-contrast MRI acquisition and the dual-domain nature of MRI data, and given the lack of existing methods that can simultaneously utilize both types of information for MRI reconstruction, this technical solution effectively improves MRI reconstruction performance, resulting in better image quality for high-magnification undersampled images, thus better serving clinical use and diagnosis. Furthermore, better image quality also means the ability to further increase MRI acquisition magnification, shorten acquisition time, and allow medical institutions to accommodate more patients daily, reducing patient imaging costs while improving the patient imaging experience.

[0068] The multi-contrast NMR reconstruction method based on multi-domain learning provided in this invention can be implemented on the terminal side or the server side. For the hardware structure of the multi-contrast NMR reconstruction terminal based on multi-domain learning, please refer to [link to relevant documentation]. Figure 3 This is a schematic diagram of an optional hardware structure of a multi-contrast nuclear magnetic resonance reconstruction terminal 300 based on multi-domain learning provided in an embodiment of the present invention. The terminal 300 can be a mobile phone, computer device, tablet device, personal digital processing device, factory back-end processing device, etc. The multi-contrast nuclear magnetic resonance reconstruction terminal 300 based on multi-domain learning includes: at least one processor 301, a memory 302, at least one network interface 304, and a user interface 306. The various components in the device are coupled together through a bus system 305. It is understood that the bus system 305 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 305 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 3 The general will label all buses as bus systems.

[0069] The user interface 306 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0070] It is understood that memory 302 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0071] In this embodiment of the invention, the memory 302 is used to store various types of data to support the operation of the multi-domain learning-based multi-contrast NMR reconstruction terminal 300. Examples of this data include: any executable program for operation on the multi-domain learning-based multi-contrast NMR reconstruction terminal 300, such as the operating system 3021 and application program 3022; the operating system 3021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 3022 may contain various applications, such as media players, browsers, etc., for implementing various application services. The implementation of the multi-domain learning-based multi-contrast NMR reconstruction method provided in this embodiment of the invention can be included in the application program 3022.

[0072] The methods disclosed in the above embodiments of the present invention can be applied to processor 301, or implemented by processor 301. Processor 301 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 301 or by instructions in the form of software. The processor 301 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 301 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 301 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0073] In an exemplary embodiment, the multi-contrast NMR reconstruction terminal 300 based on multi-domain learning can be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to perform the aforementioned method.

[0074] like Figure 4 The diagram illustrates the structure of a multi-contrast nuclear magnetic resonance reconstruction device based on multi-domain learning, according to an embodiment of the present invention. In this embodiment, the multi-contrast nuclear magnetic resonance reconstruction device 400 based on multi-domain learning includes a cross-modal generation module 401, a registration module 402, and a reconstruction module 403.

[0075] The cross-modal generation module 401 is used to acquire NMR images with full sampling assisted contrast and input them into the pre-trained cross-modal generation network based on the image domain and frequency domain, respectively. The NMR images with full sampling assisted contrast are used as the gold standard, and the offset NMR images with full sampling assisted contrast are output respectively.

[0076] The registration module 402 is used to fix the weights of the cross-modal generation network based on the image domain and frequency domain, and input the full-sampling-assisted contrast NMR image output by each of the cross-modal generation networks, together with the original undersampled-assisted contrast NMR image, into the registration network of the corresponding domain, and output the aligned image of each domain that is aligned with the original undersampled-assisted contrast NMR image.

[0077] The reconstruction module 403 is used to input the aligned images of each domain and the original undersampled NMR images with assisted contrast into the reconstruction network of the corresponding domain, so as to output the NMR reconstructed images respectively.

[0078] In some examples, the device 400 further includes a model training module 404, which is used to train the U-Net network using training data to obtain the cross-modal generation network, registration network, and reconstruction network. The training process of the model training module includes: acquiring data based on various MRI data scanned by an MRI scanning device; the various MRI data include actual full-sampling multi-contrast MRI data; preprocessing the acquired data, including converting 3D image data into 2D image data, normalizing the data, extracting images usable for full-sampling data, and breaking them down into an image dataset; dividing the image dataset into a training set and a test set according to a preset ratio; inputting the training set into the U-Net network, and training the model according to a corresponding set objective function and an optimization algorithm; testing the trained model based on the test set, and determining the model performance by comparing the test results with a preset index of the gold standard, thereby obtaining the corresponding cross-modal generation network, registration network, or reconstruction network.

[0079] In some examples, the objective function set by the cross-modal generation module 401 includes:

[0080] in, It is the loss function of the cross-modal generative network corresponding to the image domain; It is the loss function of the cross-modal generator network corresponding to the frequency domain; It is the loss function of the cross-modal generative network after inverse Fourier transform to the image domain; α is the loss function of the cross-modal generative network after Fourier transform to the frequency domain; α is the weight of the frequency domain branch relative to the image domain branch; β is the weight of the dual-domain consistency relative to the original training branch.

[0081] In some examples, the registration module 402 corresponds to the set objective function including:

[0082] in, It is the loss function of the registration network corresponding to the image domain; It is the loss function of the registration network corresponding to the frequency domain; It is the loss function of the registration network after inverse Fourier transform to the image domain; α is the loss function of the registration network after Fourier transform to the frequency domain; α is the weight of the frequency domain branch relative to the image domain branch; β is the weight of the dual-domain consistency relative to the original training branch.

[0083] In some examples, the objective function set for the reconstruction module 403 includes:

[0084] in, It is the loss function of the reconstruction network corresponding to the image domain; It is the loss function of the reconstruction network corresponding to the frequency domain; It is the loss function for reconstructing the network to the image domain via inverse Fourier transform; α is the loss function of the reconstructed network after Fourier transform to the frequency domain; α is the weight of the frequency domain branch relative to the image domain branch; β is the weight of the dual-domain consistency relative to the original training branch.

[0085] It should be noted that the multi-contrast NMR reconstruction device based on multi-domain learning provided in the above embodiments is only illustrated by the division of the above-described program modules when performing multi-contrast NMR reconstruction based on multi-domain learning. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the multi-contrast NMR reconstruction device based on multi-domain learning provided in the above embodiments and the multi-contrast NMR reconstruction method embodiments based on multi-domain learning belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0086] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented using computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0087] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-contrast nuclear magnetic resonance reconstruction method based on multi-domain learning.

[0088] In the embodiments provided in this application, the computer-readable and writable storage medium may include read-only memory, random access memory, EEPROM, CD-ROM or other optical disc storage devices, disk storage devices or other magnetic storage devices, flash memory, USB flash drive, portable hard drive, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Additionally, any connection may be appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it should be understood that computer-readable and writable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are intended for non-transient, tangible storage media. The disks and optical discs used in the application include compact discs (CDs), laser discs, optical discs, digital multifunction discs (DVDs), floppy disks, and Blu-ray discs, where disks typically copy data magnetically, while optical discs use lasers to copy data optically.

[0089] In summary, this application provides a method, apparatus, terminal, and medium for multi-contrast NMR reconstruction based on multi-domain learning. This invention offers a method to improve the efficiency of multi-contrast NMR reconstruction based on multi-domain learning. Therefore, this application effectively overcomes the various shortcomings of the prior art and has high industrial applicability.

[0090] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A multi-contrast nuclear magnetic resonance reconstruction method based on multi-domain learning, characterized in that, include: Full-sampling assisted contrast NMR images are acquired and input into pre-trained cross-modal generation networks based on image and frequency domains respectively. Using the full-sampling assisted contrast NMR images as the gold standard, offset full-sampling assisted contrast NMR images are output respectively. The weights of the cross-modal generation network based on the image domain and frequency domain are fixed, and the NMR images with full sampling and auxiliary contrast output by each of the cross-modal generation networks are input together with the original NMR images with undersampling and auxiliary contrast into the corresponding domain registration network, and the alignment images of each domain are output respectively, which are aligned with the original NMR images with undersampling and auxiliary contrast. The aligned images of each domain and the original undersampled NMR images with auxiliary contrast are input into the reconstruction network of the corresponding domain to output the NMR reconstructed images respectively. in, When training the cross-modal generation network, its corresponding objective functions include: a loss function in the image domain, a loss function in the frequency domain, a loss function of the cross-modal generation network in the image domain after inverse Fourier transform, and a loss function of the cross-modal generation network in the frequency domain after Fourier transform. When training the registration network, its corresponding objective functions include: a loss function in the image domain, a loss function in the frequency domain, a loss function of the registration network in the image domain after inverse Fourier transform, and a loss function of the registration network in the frequency domain after Fourier transform. When training the reconstruction network, its corresponding objective functions include: a loss function in the image domain, a loss function in the frequency domain, a loss function of the reconstruction network in the image domain after inverse Fourier transform, and a loss function of the reconstruction network in the frequency domain after Fourier transform. Furthermore, based on the symmetry between the image domain and the frequency domain, the output images of the cross-modal generation network, registration network, and reconstruction network in one domain can be constrained by Fourier transform / inverse Fourier transform of the output images of the cross-modal generation network, registration network, and reconstruction network in the other domain.

2. The multi-contrast nuclear magnetic resonance reconstruction method based on multi-domain learning according to claim 1, characterized in that, The cross-modal generation network, registration network, and reconstruction network can all be obtained by training the U-Net network using training data. The training process includes: Data acquisition is performed based on various MRI data obtained from MRI scanning equipment; these MRI data include actual full-import multi-contrast MRI data. The collected data is preprocessed. The preprocessing process includes converting 3D image data into 2D image data and then normalizing it. Then, images usable from the full collection data are extracted and shuffled to form an image dataset. The image dataset is divided into a training set and a test set according to a preset ratio. After the training set is input into the U-Net network, the model is trained according to the corresponding set objective function and based on the optimization algorithm. The trained model is tested based on the test set, and the model performance is determined by the preset index between the test results and the gold standard, so as to obtain the corresponding cross-modal generation network, registration network or reconstruction network.

3. The multi-contrast nuclear magnetic resonance reconstruction method based on multi-domain learning according to claim 2, characterized in that, The objective function set for the cross-modal generation network includes: ； in, It is the loss function of the cross-modal generative network corresponding to the image domain; It is the loss function of the cross-modal generator network corresponding to the frequency domain; It is the loss function of the cross-modal generative network after inverse Fourier transform to the image domain; It is the loss function of the cross-modal generator network after Fourier transform to the frequency domain; It is the weight of the frequency domain branch relative to the image domain branch; It is the weight of the bi-domain consistency relative to the original training branch.

4. The multi-contrast NMR reconstruction method based on multi-domain learning according to claim 2, characterized in that, The objective function set for the registration network includes: ； in, It is the loss function of the registration network corresponding to the image domain; It is the loss function of the registration network corresponding to the frequency domain; It is the loss function of the registration network after inverse Fourier transform to the image domain; It is the loss function of the registration network after Fourier transform to the frequency domain; It is the weight of the frequency domain branch relative to the image domain branch; It is the weight of the bi-domain consistency relative to the original training branch.

5. The multi-contrast nuclear magnetic resonance reconstruction method based on multi-domain learning according to claim 2, characterized in that, The objective function set for the reconstructed network includes: ； in, It is the loss function of the reconstruction network corresponding to the image domain; It is the loss function of the reconstruction network corresponding to the frequency domain; It is the loss function for reconstructing the network to the image domain via inverse Fourier transform; It is the loss function for reconstructing the network into the frequency domain after Fourier transform; It is the weight of the frequency domain branch relative to the image domain branch; It is the weight of the bi-domain consistency relative to the original training branch.

6. A multi-contrast nuclear magnetic resonance reconstruction device based on multi-domain learning, characterized in that, include: The cross-modal generation module is used to acquire NMR images with full sampling assisted contrast and input them into the pre-trained cross-modal generation network based on the image domain and frequency domain respectively. The NMR images with full sampling assisted contrast are used as the gold standard and the offset NMR images with full sampling assisted contrast are output respectively. The registration module is used to fix the weights of the cross-modal generation network based on the image domain and frequency domain, and input the full-sampling-assisted contrast NMR image output by each of the cross-modal generation networks, together with the original undersampled-assisted contrast NMR image, into the registration network of the corresponding domain, and output the aligned image of each domain that is aligned with the original undersampled-assisted contrast NMR image. The reconstruction module is used to input the aligned images of each domain and the original undersampled NMR images with auxiliary contrast into the reconstruction network of the corresponding domain, so as to output the NMR reconstructed images respectively. in, When training the cross-modal generation network, its corresponding objective functions include: a loss function in the image domain, a loss function in the frequency domain, a loss function of the cross-modal generation network in the image domain after inverse Fourier transform, and a loss function of the cross-modal generation network in the frequency domain after Fourier transform. When training the registration network, its corresponding objective functions include: a loss function in the image domain, a loss function in the frequency domain, a loss function of the registration network in the image domain after inverse Fourier transform, and a loss function of the registration network in the frequency domain after Fourier transform. When training the reconstruction network, its corresponding objective functions include: a loss function in the image domain, a loss function in the frequency domain, a loss function of the reconstruction network in the image domain after inverse Fourier transform, and a loss function of the reconstruction network in the frequency domain after Fourier transform. Furthermore, based on the symmetry between the image domain and the frequency domain, the output images of the cross-modal generation network, registration network, and reconstruction network in one domain can be constrained by Fourier transform / inverse Fourier transform of the output images of the cross-modal generation network, registration network, and reconstruction network in the other domain.

7. The multi-contrast nuclear magnetic resonance reconstruction device based on multi-domain learning according to claim 6, characterized in that, It also includes a model training module, used to train the U-Net network using training data to obtain the cross-modal generation network, registration network, and reconstruction network; the training process of the model training module includes: Data acquisition is performed based on various MRI data obtained from MRI scanning equipment; these MRI data include actual full-import multi-contrast MRI data. The collected data is preprocessed. The preprocessing process includes converting 3D image data into 2D image data and then normalizing it. Then, images usable from the full collection data are extracted and shuffled to form an image dataset. The image dataset is divided into a training set and a test set according to a preset ratio. After the training set is input into the U-Net network, the model is trained according to the corresponding set objective function and based on the optimization algorithm. The trained model is tested based on the test set, and the model performance is determined by the preset index between the test results and the gold standard, so as to obtain the corresponding cross-modal generation network, registration network or reconstruction network.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-contrast nuclear magnetic resonance reconstruction method based on multi-domain learning as described in any one of claims 1 to 5.

9. An electronic terminal, characterized in that, include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory to cause the terminal to perform the multi-contrast nuclear magnetic resonance reconstruction method based on multi-domain learning as described in any one of claims 1 to 5.

Images