Method and apparatus for synthesizing magnetic resonance weighted images using variational autoencoders

The variational autoencoder method addresses limitations in existing synthesis techniques by training with multi-contrast data to generate accurate and diverse magnetic resonance weighted images, overcoming errors in parameter measurement and data reliance.

JP2024543755AInactive Publication Date: 2024-11-26ZHEJIANG LAB
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023542019
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2022-11-04
Filing Date
2023-03-09
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing methods for synthesizing magnetic resonance weighted images using magnetic resonance signal formulas and deep learning techniques face limitations in accuracy and applicability due to errors in measuring magnetic resonance quantitative tissue parameters and reliance on specific contrast data in training, limiting the synthesis of different contrasts.

Method used

A method and apparatus using a variational autoencoder to synthesize magnetic resonance weighted images by constructing a pre-trained model with encoder and decoder structures, training it with multi-contrast real images and quantitative parameter maps, and decoupling the input from the training label to learn contrast information across individuals, enabling the generation of magnetic resonance weighted images of arbitrary contrasts.

Benefits of technology

The variational autoencoder model achieves more accurate synthesis of magnetic resonance weighted images by learning characteristics of actual images and generating images with continuous contrast information, allowing for the production of images not present in training data, enhancing applicability and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024543755000001_ABST
    Figure 2024543755000001_ABST
Patent Text Reader

Abstract

The present application discloses a method and apparatus for synthesizing a magnetic resonance weighted image by a variational autoencoder, which includes the steps of: S1: obtaining multi-contrast real magnetic resonance weighted images and magnetic resonance quantitative parameter maps using a magnetic resonance scanner; S2: constructing a magnetic resonance weighted image; S3: building a pre-trained variational autoencoder model having an encoder and decoder structure; S4: obtaining a variational autoencoder model; and S5: synthesizing a second magnetic resonance weighted image by the variational autoencoder model for the magnetic resonance weighted image and the magnetic resonance quantitative parameter map. The present invention uses the variational autoencoder model to train magnetic resonance weighted images of multiple contrasts, which can obtain a continuous distribution of approximate contrast information, so that the variational autoencoder model according to the present invention can be reconstructed to obtain a magnetic resonance weighted image that does not exist in the training data.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] This application claims the benefit of priority from a Chinese patent application for invention patent application No. 202211375033.5, entitled "Method and apparatus for synthesizing magnetic resonance weighted images using variational autoencoder", filed with the State Intellectual Property Office of China on November 4, 2022, the entire contents of which are incorporated herein by reference.

[0002] The present invention relates to the field of medical image processing, and in particular to a method and apparatus for synthesizing magnetic resonance weighted images using a variational autoencoder. [Background technology]

[0003] Magnetic resonance imaging (MRI) is a non-invasive, non-ionizing radiation-free medical imaging method that is widely used in scientific research and clinical practice.

[0004] Magnetic resonance imaging relies on the polarization of protons under a strong electric field and magnetic field. When protons are excited to a resonant state by a radio frequency pulse, they gradually return to an equilibrium state, which is called the proton relaxation process. The magnetic resonance signal is the electromagnetic signal generated during the relaxation process. The magnetic resonance signal can be expressed as a longitudinal relaxation parameter T according to the difference in the parameters of the acquisition sequence. 1 The contrast weighting and lateral relaxation parameter T 2 The contrast weighting of the contrast weighting of the proton density PD, the contrast weighting of the proton density PD, etc. are shown. Therefore, the magnetic resonance imaging can obtain different contrast weighting images by changing the parameters of the acquisition sequence, and the different contrast weighting images can reflect different tissue characteristics. Therefore, in the actual clinical examination process, a number of magnetic resonance weighting images with different contrasts are usually acquired, and the acquisition process makes the magnetic resonance examination take a lot of time, which brings a heavy pressure on medical resources.

[0005] In recent years, the quantitative magnetic resonance imaging method, which has made rapid development, has provided new ideas for solving the above problems. The quantitative magnetic resonance imaging method collects magnetic resonance quantitative parameter maps of tissues, and the collected magnetic resonance quantitative parameter maps can be used to describe the quantitative properties of tissues. By setting appropriate collection parameters according to the magnetic resonance signal formula, the corresponding magnetic resonance signals can be synthesized using the magnetic resonance quantitative parameters, and in principle, a magnetic resonance weighted image of any contrast can be obtained. However, the magnetic resonance weighted image obtained by the method of synthesis using the magnetic resonance signal formula has certain limitations compared with the magnetic resonance weighted image actually collected, due to errors in the measurement of the magnetic resonance quantitative tissue parameters. In addition, as research has revealed, the T synthesized by the magnetic resonance quantitative parameter map is difficult to obtain. 2 FLAIR imaging cannot achieve complete cerebrospinal fluid suppression.

[0006] The deep learning method is expected to solve the problems that arise in the synthesis process according to the above formula, and in recent research, the synthesis of magnetic resonance weighted images has been realized by using an adversarial generative network. The collected magnetic resonance quantitative parameter map is used as the input to the generator, and the actually collected magnetic resonance weighted image is used as the label in the generator training process, and adversarial training is performed according to the discriminator to achieve a good synthesis effect. However, the above method also has certain limitations. When the deep learning method is used, it is limited to the contrast of the actually collected magnetic resonance weighted image in the training data, and only the magnetic resonance weighted image of the existing contrast in the training data can be synthesized, and the application scope of the magnetic resonance quantitative parameter map in the synthesis of magnetic resonance weighted images of different contrasts is greatly limited. Summary of the Invention [Problem to be solved by the invention]

[0007] Therefore, in order to solve the above technical problems, a method and apparatus for synthesizing magnetic resonance weighted images using a variational autoencoder is proposed. [Means for solving the problem]

[0008] In order to solve the above technical problems, the present invention provides a method and apparatus for synthesizing magnetic resonance weighted images using a variational autoencoder.

[0009] The technical solutions used in the present invention are as follows: 1. A method for synthesis of magnetic resonance weighted images using a variational autoencoder, comprising: Step S1 of acquiring multi-contrast real magnetic resonance weighted images and magnetic resonance quantitative parameter maps using a magnetic resonance scanner; a step S2 of synthesizing a first magnetic resonance weighted image based on the corresponding quantitative value in the magnetic resonance quantitative parameter map, a repetition time assumed when synthesizing image signals, an echo time assumed when synthesizing image signals, and / or an inversion time assumed when synthesizing image signals, and constructing the first magnetic resonance weighted image and the real magnetic resonance weighted image as a magnetic resonance weighted image; Step S3 of constructing a pre-trained variational autoencoder model having an encoder and decoder structure; S4: constructing a training set using the magnetic resonance weighted images and the magnetic resonance quantitative parameter maps, training the pre-trained variational autoencoder model, and updating parameters of the pre-trained variational autoencoder model to obtain a variational autoencoder model; and S5 synthesizing a second magnetic resonance weighted image using the variational autoencoder model for the magnetic resonance weighted image and the magnetic resonance quantitative parameter map.

[0010] Furthermore, the real magnetic resonance weighted image and the magnetic resonance quantitative parameter map in step S1 are generated by a magnetic resonance scanner executing a predetermined scan sequence.

[0011] Furthermore, the magnetic resonance quantitative parameter map is T 1 Quantitative Map, T 2 It consists of a quantitative map and a proton density quantitative map.

[0012] Furthermore, the real magnetic resonance weighted image has at least T 1 Weighted normal image, T 2 Weighted normal image, proton density weighted image, T 1 Weighted Flair images and / or T 2 Contains either a weighted Flair image.

[0013] Furthermore, step S3 specifically includes the following steps: Step S31 of constructing an encoder using several 3D convolutional layers, each of which is followed by a coding activation layer and a pooling layer; Step S32 of constructing a decoder using an encoding layer and a decoding layer, the encoding layer being composed of a plurality of transposed convolution layers, the decoding layer being composed of a plurality of convolution layers, each of the convolution layers being followed by a decoding activation layer; and step S33 of connecting the encoder and decoder using a fully connected layer to obtain a pre-trained variational autoencoder model.

[0014] Furthermore, step S4 specifically includes the following steps: Step S41: registering the real magnetic resonance weighted image to the first magnetic resonance weighted image by linear and non-linear registration methods to obtain a registered real magnetic resonance image; Step S42: Matching the resolution of the registered real magnetic resonance image, the first magnetic resonance weighted image, and the magnetic resonance quantitative parameter map by linear interpolation to obtain a training set; Step S43: inputting the registered real MR image and / or the first MR weighted image into an encoder in the pre-trained variational autoencoder model, outputting a mean value and a variance assuming a multivariate Gaussian distribution after convolution, and performing a sampling operation on the mean value and the variance to obtain hidden layer variables that characterize contrast encoding; a step S44 of connecting the encoder to a coding layer of a decoder in the pre-trained variational autoencoder model via a fully connected layer; Step S45, in which the hidden layer variables are passed through a transposed convolutional layer in the encoding layer to recover the hidden layer variables into a contrast encoding knowledge matrix of the same size as the magnetic resonance quantitative parameter map; a step S46 of combining the contrast encoding knowledge matrix with the magnetic resonance quantitative parameter maps in the training set to obtain a matrix; a step S47 of obtaining a second magnetic resonance weighted image of the corresponding contrast, the matrix being output by a decoding layer of the decoder, and calculating a loss function from the real magnetic resonance weighted image of the corresponding contrast in the training set; The method includes a sub-step of repeating steps S41 to S47, setting a predetermined learning degree, performing reverse gradient propagation according to a loss function, updating the parameters of the pre-trained variational autoencoder model until the loss function no longer decreases, and completing the training, and obtaining a variational autoencoder model, with the sub-step of step S48.

[0015] Furthermore, the combination method in step S46 includes splicing the contrast encoding knowledge matrix to the magnetic resonance quantitative parameter maps in the training set, or splicing the contrast encoding knowledge matrix to the magnetic resonance quantitative parameter maps in the training set through several 3D convolutional layers, or adding the contrast encoding knowledge matrix to the magnetic resonance quantitative parameter maps in the training set.

[0016] Furthermore, the real magnetic resonance weighted image of the corresponding contrast in the training set for calculating the loss function in step S47 has the same contrast as the input by the real magnetic resonance image and / or the first magnetic resonance weighted image in step S43, and is the same individual as the individual of the magnetic resonance quantitative parameter map in the training set in step S46.

[0017] Furthermore, the training loss function of the pre-trained variational autoencoder model in step S4 is JPEG2024543755000002.jpg20170, where σ and μ are the mean and variance of the normal distribution of the hidden layer variables output from the encoder, μ' is the output result of the decoder, and x i is the second magnetic resonance weighted image of the corresponding contrast, i is the input sample, j is the input sample for extracting contrast encoding information, and n, d are the amount of samples input when calculating the loss function once.

[0018] The present invention further provides an apparatus for synthesizing a magnetic resonance weighted image using a variational autoencoder, comprising a memory and one or more processors, the memory storing executable code, which, when executed by the one or more processors, is used to realize a method for synthesizing a magnetic resonance weighted image using a variational autoencoder as described in any one of the above embodiments.

[0019] The present invention further provides a computer-readable storage medium having a program stored thereon, the program realizing the method for synthesis of magnetic resonance weighted images by a variational autoencoder according to any one of the above embodiments when executed by a processor. Effect of the Invention

[0020] Beneficial effects of the present invention include: 1. The method of synthesizing a magnetic resonance weighted image by using a magnetic resonance signal formula of the present invention can synthesize corresponding magnetic resonance signals by using magnetic resonance quantitative parameters by setting appropriate collection parameters. However, the magnetic resonance weighted image obtained by the method of synthesizing using a magnetic resonance signal formula has certain limitations compared with the magnetic resonance weighted image actually collected, because there is an error in the measurement of the magnetic resonance quantitative tissue parameters. The present invention generates a synthetic magnetic resonance weighted image by using a deep learning method, and uses a deep learning model to learn the features of the magnetic resonance weighted image actually collected, so that a synthetic magnetic resonance weighted image that is more consistent with the magnetic resonance weighted image actually collected can be obtained.

[0021] 2. Currently, other deep learning methods are limited to the contrast of the magnetic resonance weighted images actually collected in the training data, and can only synthesize magnetic resonance weighted images of existing contrasts in the training data, which greatly limits the scope of application of the magnetic resonance quantitative parameter map in synthesizing magnetic resonance weighted images of different contrasts.The present invention uses a variational autoencoder model to train magnetic resonance weighted images of multiple contrasts to obtain a continuous distribution of approximate contrast information, so that the variational autoencoder model of the present invention can be reconstructed to obtain magnetic resonance weighted images that do not exist in the training data.

[0022] 3. In the training of the conditional variational autoencoder model, the magnetic resonance weighting image input from the encoder and the real magnetic resonance weighting image, which is the training label of the decoder, are decoupled at the individual level, and the encoder of the variational autoencoder model is made to learn contrast information that is not related to the individual. The decoupling in the above training process can realize the use of the magnetic resonance weighting image of any individual to extract low-dimensional contrast encoding information, so that in the actual application process, the magnetic resonance weighting image of a single individual can be used to generate a large amount of synthetic magnetic resonance images of target contrast.

[0023] 4. Variational autoencoder is a common data generation model. The encoder of the variational autoencoder can map the input high-dimensional data into a simple multivariate Gaussian distribution. By sampling in this distribution, the corresponding hidden layer variable can be obtained, which can reflect certain low-dimensional characteristics of the input high-dimensional data, and the value of this hidden layer variable corresponds to the above Gaussian distribution. Based on the above characteristics of the variational autoencoder, the decoder of the variational autoencoder can map the contrast information of the corresponding magnetic resonance weighted image into a multivariate Gaussian distribution. By sampling in this distribution, the corresponding hidden layer variable can be obtained, which reflects the contrast information of the high-dimensional magnetic resonance weighted image. Based on this contrast information, combined with the magnetic resonance quantitative parameter map of the individual, the decoder of the variational autoencoder can be used to realize the synthesis and reconstruction of the magnetic resonance weighted image of the corresponding contrast. Since the magnetic resonance weighted images of the same contrast of different individuals are consistent in low-dimensional contrast information, the magnetic resonance weighted images of different individuals can be input to the variational autoencoder, and the corresponding contrast information can be obtained by further sampling. By training magnetic resonance weighted images of multiple contrasts, a continuous distribution of approximate contrast information can be obtained, from which a variational autoencoder model can be reconstructed to obtain magnetic resonance weighted images not present in the training data. The present invention uses a conditional variational autoencoder model to condition the variational autoencoder on the quantitative magnetic resonance images of an individual, and further controls the variational autoencoder to accurately generate a composite magnetic resonance weighted image of the individual. [Brief description of the drawings]

[0024] [Figure 1] 1 is a flow chart of a method for synthesizing magnetic resonance weighted images using a variational autoencoder of the present invention. [Diagram 2]FIG. 1 is a diagram illustrating the configuration of a model of a conditional variational autoencoder used in an embodiment. [Diagram 3] FIG. 1 is a schematic diagram of a magnetic resonance weighted image synthesis device using a variational autoencoder according to the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] The following description of at least one exemplary embodiment is merely illustrative in nature and does not limit the present invention and its application or use in any way. Based on the embodiments of the present invention, all other embodiments that a person skilled in the art can obtain without creative ingenuity are within the scope of protection of the present invention.

[0026] As shown in FIG. 1, the method for synthesizing a magnetic resonance weighted image by a variational autoencoder includes the following steps: in step S1, a magnetic resonance scanner is used to obtain a multi-contrast real magnetic resonance weighted image and a magnetic resonance quantitative parameter map; the real magnetic resonance weighted image and the magnetic resonance quantitative parameter map are generated by a magnetic resonance scanner performing a predetermined scan sequence; The magnetic resonance quantitative parameter map is T 1 Quantitative Map, T 2 It consists of a quantitative map and a proton density quantitative map, The real magnetic resonance weighted image is at least T 1 Weighted normal image, T 2 Weighted normal image, proton density weighted image, T 1 Weighted Flair images and / or T 2 Contains either a weighted Flair image.

[0027] In step S2, a first magnetic resonance weighted image is synthesized based on the corresponding quantitative value in the magnetic resonance quantitative parameter map, the repetition time assumed when synthesizing image signals, the echo time assumed when synthesizing image signals, and / or the inversion time assumed when synthesizing image signals, and the first magnetic resonance weighted image and the real magnetic resonance weighted image are configured as a magnetic resonance weighted image; In step S3, a pre-trained variational autoencoder model having an encoder and decoder structure is constructed; In step S31, an encoder is constructed using several 3D convolutional layers, each of which is followed by a coding activation layer and a pooling layer; In step S32, a decoder is constructed using an encoding layer and a decoding layer, the encoding layer being composed of a plurality of transposed convolution layers, the decoding layer being composed of a plurality of convolution layers, each of which is followed by a decoding activation layer; In step S33, the encoder and decoder are connected using a fully connected layer to obtain a pre-trained variational autoencoder model.

[0028] In step S4, the magnetic resonance weighted image and the magnetic resonance quantitative parameter map are used to construct a training set, train the pre-trained variational autoencoder model, and update parameters of the pre-trained variational autoencoder model to obtain a variational autoencoder model; In step S41, the real magnetic resonance weighted image is registered to the first magnetic resonance weighted image by a method of linear registration and nonlinear registration to obtain a registered real magnetic resonance image; In step S42, the resolution of the registered real magnetic resonance image, the first magnetic resonance weighted image, and the magnetic resonance quantitative parameter map is matched by linear interpolation to obtain a training set; In step S43, the registered real magnetic resonance image and / or the first magnetic resonance weighted image are input to an encoder in the pre-trained variational autoencoder model, and mean values ​​and variances assuming a multivariate Gaussian distribution after convolution are output, and a sampling operation is performed on the mean values ​​and variances to obtain hidden layer variables that characterize contrast encoding; In step S44, the encoder is connected to a coding layer of a decoder in the pre-trained variational autoencoder model via a fully connected layer; In step S45, the hidden layer variables are passed through a transposed convolutional layer in the encoding layer to recover the hidden layer variables into a contrast encoding knowledge matrix having the same size as the magnetic resonance quantitative parameter map; In step S46, the contrast encoding knowledge matrix is ​​combined with the magnetic resonance quantitative parameter maps in the training set to obtain a matrix; The combination method includes splicing the contrast encoding knowledge matrix to the magnetic resonance quantitative parameter maps in the training set, or splicing the contrast encoding knowledge matrix to the magnetic resonance quantitative parameter maps in the training set through several 3D convolution layers, or adding the contrast encoding knowledge matrix to the magnetic resonance quantitative parameter maps in the training set; In step S47, the matrix is ​​output by a decoding layer of the decoder to obtain a second magnetic resonance weighted image of the corresponding contrast, and a loss function is calculated from the real magnetic resonance weighted image of the corresponding contrast in the training set; The real magnetic resonance weighted image of the corresponding contrast in the training set for calculating the loss function in step S47 has the same contrast as the input by the real magnetic resonance image and / or the first magnetic resonance weighted image in step S43, and is the same individual as the individual of the magnetic resonance quantitative parameter map in the training set in step S46.

[0029] In step S48, steps S41 to S47 are repeated, a predetermined learning degree is set, and reverse gradient propagation is performed according to the loss function. The parameters of the pre-trained variational autoencoder model are updated until the loss function does not decrease, and the training is completed to obtain a variational autoencoder model. The training loss function of the pre-trained variational autoencoder model is JPEG2024543755000003.jpg22170, where σ and μ are the mean and variance of the normal distribution of the hidden layer variables output from the encoder, μ' is the output result of the decoder, and X i is the second magnetic resonance weighted image of the corresponding contrast, i is the input sample, j is the input sample for extracting contrast encoding information, and n, d distribution are the amount of samples input when calculating the loss once.

[0030] In step S5, a second magnetic resonance weighted image is synthesized using the variational autoencoder model for the magnetic resonance weighted image and the magnetic resonance quantitative parameter map.

[0031] As shown in FIG. 2, in an embodiment, a method for synthesizing multi-contrast magnetic resonance weighted images using a conditional variational autoencoder includes the following steps: In step S1, a magnetic resonance scanner is used to obtain multi-contrast real magnetic resonance weighted images and magnetic resonance quantitative parameter maps; the real magnetic resonance weighted image and the magnetic resonance quantitative parameter map are generated by a magnetic resonance scanner performing a predetermined scan sequence; The magnetic resonance quantitative parameter map is T 1 Quantitative Map, T 2 It consists of a quantitative map and a proton density quantitative map, The real magnetic resonance weighted image is at least T 1 Weighted normal image, T 2 Weighted normal image, proton density weighted image, T 1 Weighted Flair images and / or T 2Contains either a weighted Flair image.

[0032] The acquisition of the magnetic resonance quantitative parameter map and the real magnetic resonance weighted image using the magnetic resonance scanner is realized by the magnetic resonance scanner performing a specific scan sequence. Here, the magnetic resonance quantitative parameter map can be acquired using multiple scan sequences, for example, T 1 When acquiring quantitative maps, a multiple inversion time inversion recovery sequence such as the MP2RAGE sequence can be used, and the corresponding T 1 Quantitative maps can be calculated and 2 When collecting quantitative maps, spin echo sequences of multiple echo times can be collected, and the corresponding T can be calculated from the correspondence between the signal values ​​in the collected real magnetic resonance weighted images and the collection parameters (echo times). 2A quantitative map can be calculated, and multiple quantitative magnetic resonance parameter maps can be obtained in a single scan by new quantitative magnetic resonance imaging sequences including MDME (Multiple Dynamic Multiple Echo) sequence and Magnetic Resonance Fingerprinting (MRF) sequence, and multiple quantitative magnetic resonance parameter maps can be obtained simultaneously by a specific reconstruction method of the corresponding sequence, which will not be described further. In this embodiment, the magnetic resonance quantitative parameter map is obtained by the magnetic resonance fingerprint imaging MRF sequence, and as for the method according to the present invention, the specific form of obtaining the magnetic resonance quantitative parameter map does not affect all subsequent steps of the method according to the present invention, so this is merely one specific example of the present invention, and does not limit the present invention to selecting other methods to obtain the magnetic resonance quantitative parameter map in other embodiments. The real magnetic resonance weighted image can be obtained using a specific scan sequence and scan parameters, and when a different scan sequence is selected or different scan parameters are set, a real magnetic resonance weighted image with different contrast can be obtained. In this embodiment, the repetition time, echo time and inversion time are controlled to obtain real magnetic resonance weighted images with different contrasts, and the number of contrasts of the collected real magnetic resonance weighted images is more than 5, so as to ensure the effect of subsequent training and to achieve efficiency. The magnetic resonance quantitative parameter map and the real magnetic resonance weighted images obtained in this embodiment are for the same individual, and the number of individuals is more than 10.

[0033] In step S2, a first magnetic resonance weighted image is synthesized based on the corresponding quantitative value in the magnetic resonance quantitative parameter map, the repetition time assumed when synthesizing image signals, the echo time assumed when synthesizing image signals, and / or the inversion time assumed when synthesizing image signals, and the first magnetic resonance weighted image and the real magnetic resonance weighted image are configured as a magnetic resonance weighted image; The image is T 1 Weighted normal image, T2 In the case of a weighted normal image and a proton density weighted image, the formula 1 for synthesizing the first magnetic resonance weighted image is: JPEG2024543755000004.jpg15170, where T 1 , T 2 , PD is T 1 Quantitative Map, T 2 where TR is the repetition time assumed when the image signal is synthesized, and TE is the echo time assumed when the image signal is synthesized. Select appropriate TR and TE parameters to determine the contrast at T 1 It corresponds to a weighted normal image.

[0034] The image is T 1 Weighted Flair image, T 2 In the case of other images including a single inversion pulse sequence, such as a weighted Flair image, Equation 2 for synthesizing the first magnetic resonance weighted image is: JPEG2024543755000005.jpg14170, where T 1 , T 2 , PD is T 1 Quantitative Map, T 2 are the corresponding quantitative values ​​in the quantitative map and proton density quantitative map, TR is the repetition time assumed when the image signals are synthesized, TE is the echo time assumed when the image signals are synthesized, and TI is the inversion time assumed when the image signals are synthesized.

[0035] In step S3, a pre-trained variational autoencoder model having an encoder and decoder structure is constructed; In step S31, an encoder is constructed using several 3D convolutional layers, each of which is followed by a coding activation layer and a pooling layer; The activation function of the encoding activation layer is a "relu" function, and the pooling function of the pooling layer is max pooling; In step S32, construct a decoder using an encoding layer and a decoding layer, the encoding layer being composed of a plurality of transposed convolution layers, the decoding layer being composed of a plurality of convolution layers, each of which is followed by a decoding activation layer; the activation function of the decoding activation layer is a "relu" function, In step S33, the encoder and decoder are connected using a fully connected layer to obtain a pre-trained variational autoencoder model.

[0036] In step S4, the magnetic resonance weighted image and the magnetic resonance quantitative parameter map are used to construct a training set, train the pre-trained variational autoencoder model, and update parameters of the pre-trained variational autoencoder model to obtain a variational autoencoder model; Suppose there is one low-dimensional contrast information z in a high-dimensional magnetic resonance weighted image, and the low-dimensional contrast information can be approximately represented by a simple multivariate Gaussian distribution. We have JPEG2024543755000006.jpg6170, where I denotes the identity matrix and z is a multidimensional random variable following a standard multivariate Gaussian distribution.

[0037] JPEG2024543755000007.jpg23170JPEG2024543755000008.jpg32170

[0038] JPEG2024543755000009.jpg8170JPEG2024543755000010.jpg13170 is obtained, The training loss function of the pre-trained variational autoencoder model is JPEG2024543755000011.jpg24170, where σ and μ are the mean and variance of the normal distribution of the hidden layer variables output from the encoder, μ' is the output result of the decoder, and x iis the second magnetic resonance weighted image of the corresponding contrast, i is the input sample, j is the input sample for extracting contrast encoding information, and n, d distribution are the amount of samples input when calculating the loss once.

[0039] In step S41, the real magnetic resonance weighted image is registered to the first magnetic resonance weighted image by a method of linear registration and nonlinear registration to obtain a registered real magnetic resonance image; In step S42, the resolution of the registered real magnetic resonance image, the first magnetic resonance weighted image, and the magnetic resonance quantitative parameter map is matched by linear interpolation to obtain a training set; In step S43, the registered real magnetic resonance image and / or the first magnetic resonance weighted image are input to an encoder in the pre-trained variational autoencoder model, and mean values ​​and variances assuming a multivariate Gaussian distribution after convolution are output, and a sampling operation is performed on the mean values ​​and variances to obtain hidden layer variables z characterizing contrast encoding; Here, the sampling formula is: JPEG2024543755000012.jpg7170, where σ and μ are the mean and variance of the normal distribution of the hidden layer variables output from the encoder, and λ corresponds to the standard normal distribution.

[0040] The training set is used as data for model training. During the model training process, an individual is randomly selected, and the registered real magnetic resonance image and / or the first magnetic resonance weighted image of a certain contrast are input to the encoder.

[0041] When selecting the first magnetic resonance weighted image, it is necessary to synthesize the first magnetic resonance weighted image with the pre-processed magnetic resonance quantitative parameter map. In particular, when used as input to the encoder in the training process, the contrast of the first magnetic resonance weighted image should match the acquired real magnetic resonance weighted image, i.e., the contrast of this first magnetic resonance weighted image should match one of the contrasts of the acquired real magnetic resonance weighted images.

[0042] In step S44, the encoder is connected to a coding layer of a decoder in the pre-trained variational autoencoder model via a fully connected layer; In step S45, the hidden layer variables are passed through a transposed convolutional layer in the encoding layer to recover the hidden layer variables into a contrast encoding knowledge matrix M having the same size as the magnetic resonance quantitative parameter map; In step S46, the contrast encoding knowledge matrix M is combined with the magnetic resonance quantitative parameter maps in the training set to obtain a matrix F; The combination method includes splicing the contrast encoding knowledge matrix M to the magnetic resonance quantitative parameter maps in the training set, or splicing the contrast encoding knowledge matrix M to the magnetic resonance quantitative parameter maps in the training set through several 3D convolutional layers, or adding the contrast encoding knowledge matrix M to the magnetic resonance quantitative parameter maps in the training set.

[0043] Specifically, the contrast coding knowledge matrix M and T 1 Quantitative Map, T 2 The quantitative map and the magnetic resonance quantitative parameter map, including the proton density quantitative map, are combined in a matrix splicing manner to obtain the matrix F.

[0044] In step S47, the matrix is ​​output by a decoding layer of the decoder to obtain a second magnetic resonance weighted image of the corresponding contrast, and a loss function is calculated from the real magnetic resonance weighted image of the corresponding contrast in the training set; In step S48, steps S41 to S47 are repeated, a predetermined learning degree is set, and reverse gradient propagation is performed according to the loss function. The parameters of the pre-trained variational autoencoder model are updated until the loss function does not decrease, and the training is completed to obtain a variational autoencoder model. The model is backpropagated based on the loss function to update the model parameters. In the implementation, the Adam optimizer is used during model training, and the corresponding learning rate is 0.0001.

[0045] In step S5, a second magnetic resonance weighted image is synthesized using the variational autoencoder model for the magnetic resonance weighted image and the magnetic resonance quantitative parameter map.

[0046] The trained conditional variational autoencoder model is loaded, and the magnetic resonance weighted image and the magnetic resonance quantitative parameter map are selected as inputs to the encoder. During the training of the pre-trained variational autoencoder model, the real magnetic resonance weighted image and the first magnetic resonance image are used as training data, so in this step, the real magnetic resonance weighted image may be selected as input to the encoder, or the first magnetic resonance weighted image may be selected as input to the encoder. The individual selected here is not related to the second magnetic resonance weighted image finally output from the model, so the target magnetic resonance weighted image of any one individual is selected. Due to the characteristics of model training, the target magnetic resonance weighted image selected here may be a contrast type that did not appear in the training data set. Therefore, according to the actual application needs, different types of magnetic resonance weighted data are selected as inputs to extract hidden layer variables, and here, the first magnetic resonance weighted image data of the contrast type that did not appear in the training data set is taken as an example to be input to the model. First, construct the first magnetic resonance weighted image data of the contrast type that did not appear in the training data set, select the appropriate synthesis parameters, and select the magnetic resonance signal synthesis formula 1 or the magnetic resonance signal synthesis formula 2 to synthesize the first magnetic resonance weighted image data. The synthesized data is input to the encoder of the loaded conditional variational autoencoder model, and the mean and variance of the posterior normal distribution of the hidden layer variables are output, and the hidden layer variable z is obtained by sampling with the sampling formula.

[0047] The trained decoder is used to synthesize a second magnetic resonance weighted image of corresponding contrast based on the extracted hidden layer variables and the magnetic resonance quantitative parameter map.

[0048] Load the trained variational autoencoder model. Select the extracted hidden layer variables and the magnetic resonance quantitative parameter map of an individual. Here, the individual is selected, that is, it is determined that the second magnetic resonance weighted image of this individual is output from the conditional variational autoencoder model.

[0049] Corresponding to the embodiment of the method for synthesizing multi-contrast magnetic resonance weighted images by a conditional variational autoencoder described above, the present invention further provides an embodiment of an apparatus for synthesizing multi-contrast magnetic resonance weighted images by a conditional variational autoencoder.

[0050] As shown in FIG. 3, the device for synthesizing multi-contrast magnetic resonance weighted images using a conditional variational autoencoder provided in an embodiment of the present invention includes a memory and one or more processors, and the memory stores executable code, which is used to realize the method for synthesizing multi-contrast magnetic resonance weighted images using a variational autoencoder of the above embodiment when the one or more processors execute the executable code.

[0051] The embodiment of the apparatus for synthesizing multi-contrast magnetic resonance weighted images by a conditional variational autoencoder of the present invention can be applied to any device with data processing capabilities, and the arbitrary device with data processing capabilities may be, for example, a computer or other device or equipment. The device embodiment may be realized by software, hardware, or a combination of hardware and software. Taking the implementation by software as an example, as an apparatus in a logical sense, it is formed by loading corresponding computer program instructions in a non-volatile memory into an internal memory and executing them through a processor of the arbitrary device with data processing capabilities in which it is located. In terms of hardware, as shown in FIG. 3, it is a hardware configuration diagram of an arbitrary device with data processing capabilities in which the apparatus for synthesizing multi-contrast magnetic resonance weighted images by a conditional variational autoencoder of the present invention is located. In addition to the processor, internal memory, network interface, and non-volatile memory shown in FIG. 3, in the embodiment, the arbitrary device with data processing capabilities in which the apparatus is located may usually further include other hardware depending on the actual function of the arbitrary device with data processing capabilities, which will not be described further.

[0052] For details of the process of realizing the functions and actions of each unit in the above device, please refer to the process of realizing the corresponding steps in the above method, and no further description will be given here.

[0053] The device embodiments basically correspond to the method embodiments, so that the relevant contents can be referred to the description of the method embodiments. The device embodiments described above are merely schematic, and the units described herein as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed among multiple network units. According to actual needs, some or all of the modules can be selected to achieve the purpose of the technical solution of the present invention. Those skilled in the art can understand and implement it without creative ingenuity.

[0054] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, realizes the method for synthesis of multi-contrast magnetic resonance weighted images using a variational autoencoder of the above embodiment.

[0055] The computer-readable storage medium may be an internal storage unit of any device capable of processing data described in any of the above embodiments, such as a hard disk or an internal memory. The computer-readable storage medium may be an external storage device of any device capable of processing data, such as a plug-in hard disk, a Smart Media (registered trademark) Card (SMC), an SD card, a flash memory card, etc., disposed in the device. Furthermore, the computer-readable storage medium may include an internal storage unit of any device capable of processing data, and may also include an external storage device. The computer-readable storage medium may be used to store the computer program and other programs and data required for the device capable of processing data, and may also be used to temporarily store output data or data to be output.

[0056] The above is merely a preferred embodiment of the present invention, and is not intended to limit the present invention. Those skilled in the art can make various modifications and changes to the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. 1. A method for synthesis of magnetic resonance weighted images using a variational autoencoder, comprising: Step S1 of acquiring multi-contrast real magnetic resonance weighted images and magnetic resonance quantitative parameter maps using a magnetic resonance scanner; a step S2 of synthesizing a first magnetic resonance weighted image based on the corresponding quantitative value in the magnetic resonance quantitative parameter map, a repetition time assumed when synthesizing image signals, an echo time assumed when synthesizing image signals, and / or an inversion time assumed when synthesizing image signals, and constructing the first magnetic resonance weighted image and the real magnetic resonance weighted image as a magnetic resonance weighted image; Step S3 of constructing a pre-trained variational autoencoder model having an encoder and decoder structure; S4: constructing a training set using the magnetic resonance weighted images and the magnetic resonance quantitative parameter maps, training the pre-trained variational autoencoder model, and updating parameters of the pre-trained variational autoencoder model to obtain a variational autoencoder model; and (S5) synthesizing a second magnetic resonance weighted image using the variational autoencoder model for the magnetic resonance weighted image and the magnetic resonance quantitative parameter map.

2. The method for synthesizing a magnetic resonance weighted image using a variational autoencoder according to claim 1, wherein the real magnetic resonance weighted image and the magnetic resonance quantitative parameter map in step S1 are generated by a magnetic resonance scanner executing a predetermined scan sequence.

3. The magnetic resonance quantitative parameter map is T 1 Quantitative map, T 2 2. The method for synthesizing a magnetic resonance weighted image using a variational autoencoder according to claim 1, further comprising: a quantitative map; and a proton density quantitative map.

4. The real magnetic resonance weighted image is at least T 1 Weighted normal image, T 2 Weighted normal image, proton density weighted image, T 1 Weighted Flair image and / or T 2 2. The method of claim 1, further comprising the steps of: synthesizing a weighted magnetic resonance image using a variational autoencoder;

5. Specifically, step S3 is Step S31 of constructing an encoder using several 3D convolutional layers, each of which is followed by a coding activation layer and a pooling layer; Step S32 of constructing a decoder using an encoding layer and a decoding layer, the encoding layer being composed of a plurality of transposed convolution layers, the decoding layer being composed of a plurality of convolution layers, each of the convolution layers being followed by a decoding activation layer; 2. The method for synthesizing magnetic resonance weighted images using a variational autoencoder according to claim 1, further comprising the sub-step of: connecting the encoder and the decoder using a fully connected layer to obtain a pre-trained variational autoencoder model S33.

6. Specifically, step S4 is a step S41 of registering the real magnetic resonance weighted image to the first magnetic resonance weighted image by linear and non-linear registration methods to obtain a registered real magnetic resonance image; A step S42 of matching the resolution of the registered real magnetic resonance image, the first magnetic resonance weighted image, and the magnetic resonance quantitative parameter map by linear interpolation to obtain a training set; Step S43: inputting the registered real MR image and / or the first MR weighted image into an encoder in the pre-trained variational autoencoder model, outputting a mean value and a variance assuming a multivariate Gaussian distribution after convolution, and performing a sampling operation on the mean value and the variance to obtain hidden layer variables that characterize contrast encoding; A step S44 of connecting the encoder to a coding layer of a decoder in the pre-trained variational autoencoder model via a fully connected layer; Step S45, in which the hidden layer variables are passed through a transposed convolutional layer in the encoding layer to recover the hidden layer variables into a contrast encoding knowledge matrix of the same size as the magnetic resonance quantitative parameter map; a step S46 of combining the contrast encoding knowledge matrix with the magnetic resonance quantitative parameter maps in the training set to obtain a matrix; a step S47 in which the matrix is ​​output by a decoding layer of the decoder to obtain a second magnetic resonance weighted image of the corresponding contrast, and a loss function is calculated from the real magnetic resonance weighted image of the corresponding contrast in the training set; The method for synthesizing a magnetic resonance weighted image using a variational autoencoder according to claim 1, further comprising the sub-steps of: repeating steps S41 to S47, setting a predetermined learning degree, performing inverse gradient propagation according to a loss function, updating parameters of the pre-trained variational autoencoder model until the loss function does not decrease, completing training, and obtaining a variational autoencoder model.

7. The method for synthesizing magnetic resonance weighted images using a variational autoencoder according to claim 6, characterized in that the combination method in step S46 includes splicing the contrast encoding knowledge matrix to the magnetic resonance quantitative parameter maps in the training set, or splicing the contrast encoding knowledge matrix to the magnetic resonance quantitative parameter maps in the training set via several three-dimensional convolutional layers, or adding the contrast encoding knowledge matrix to the magnetic resonance quantitative parameter maps in the training set.

8. The method for synthesizing a magnetic resonance weighted image using a variational autoencoder according to claim 6, characterized in that the real magnetic resonance weighted image of the corresponding contrast in the training set for calculating the loss function in step S47 has the same contrast as the input by the real magnetic resonance image and / or the first magnetic resonance weighted image in step S43, and is the same individual as the individual of the magnetic resonance quantitative parameter map in the training set in step S46.

9. The training loss function of the pre-trained variational autoencoder model in step S4 is where σ and μ are the mean and variance of the normal distribution of the hidden layer variables output from the encoder, μ′ is the output result of the decoder, and X i is a second magnetic resonance weighted image of the corresponding contrast, i is an input sample, j is an input sample for extracting contrast encoding information, and n, d distribution are the amount of samples input when calculating loss once. The method of synthesizing a magnetic resonance weighted image using a variational autoencoder according to claim 1,

10. An apparatus for synthesizing a magnetic resonance weighted image using a variational autoencoder, comprising a memory and one or more processors, the memory storing executable code, the apparatus being used to realize the method for synthesizing a magnetic resonance weighted image using a variational autoencoder according to any one of claims 1 to 9, when the one or more processors execute the executable code.

11. A computer-readable storage medium having a program stored thereon, the program being capable of implementing the method for synthesis of magnetic resonance weighted images by variational autoencoder according to any one of claims 1 to 9 when executed by a processor.

Citation Information

Patent Citations

  • Medical image diagnosis support apparatus, and magnetic resonance imaging apparatus

    JP2019180637A