A multimodal medical image encoding and generation method based on generative adversarial networks

Abstract

The present invention discloses a multimodal medical image encoding and generation method based on a generative adversarial network, comprising the following steps: preprocessing data to obtain a training data set, simulating a scenario in which a clinical MRI sequence is missing, constructing a generative adversarial neural network, and training the generative adversarial neural network; the present invention realizes feature extraction of multiple MRI sequences at the same time by training a network at one time, and compared with the existing model, the quality of the translated image is higher, and the ability to flexibly translate the input MRI image into any required MRI sequence can be achieved, and the feature encoding of MRI images of various modalities and the mutual generation between MRI multimodal images can be solved, and an unsupervised training method is adopted, and the generative network can still be trained in the case of missing training data modality, which is more in line with clinical scenarios and can realize image generation between any multimodalities.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multimodal medical image encoding and generation method based on a generative adversarial network. Background Art

[0002] Magnetic resonance imaging (MRI) can obtain a variety of different sequences (such as T1-weighted, T2-weighted, T1 contrast-enhanced (T1c) and T2 fluid-attenuated inversion recovery (T2FLAIR)), each of which can provide different tissue contrast views and spatial resolutions. The combination of sequences provides more complementary information, enabling doctors to make more accurate diagnoses and treatments. In many cases, specific sequences are most effective for diagnosing specific diseases. For example, for the diagnosis of glioblastoma, T1 and T2 FLAIR sequences can clearly show the edema area of ​​the tumor, and the T1c sequence can clearly show the enhanced area around the tumor.

[0003] However, due to differences in acquisition protocols between institutions, time constraints, and the presence of image artifacts, it is often difficult to obtain enough MRI sequences for patients, resulting in frequent missing sequences in MRI examinations. In addition, MRI sequences produced by different centers vary greatly due to differences in technicians, machines, patient conditions, and other factors, which limits the comparison and research of MRI sequences across centers.

[0004] The above problems not only interfere with doctors' diagnosis, but also hinder the analysis of many downstream data, which usually assume the existence of a specific set of pulse sequences to perform their tasks. Although brain tumor segmentation based on deep learning and with clear clinical needs has made great progress, it is still a major challenge to convert the most advanced computational methods into routine clinical tools. Because most methods rely on inputting specific sequences into the model, the model may fail when some sequences are missing. In addition, sparse MRI sequences from a single center are usually too small to train the model, and the differences in MRI sequences across centers make it difficult for the model to work on local data even if it can be trained on a large amount of public data.

[0005] Among the current solutions to missing MRI sequences and cross-center differences, it is difficult to modify the existing analysis methods and model structures to handle the missing sequences, and this may lead to a decrease in model performance. In addition, due to changes in regional anatomical structures, rescanning to obtain the missing sequences is time-consuming, and there is a lack of consistency between the previous and subsequent scans. Therefore, the present invention proposes a multimodal medical image encoding and generation method based on a generative adversarial network to solve the problems existing in the prior art. Summary of the invention

[0006] In view of the above problems, the purpose of the present invention is to propose a multimodal medical image encoding and generation method based on a generative adversarial network. This method realizes the feature extraction of multiple MRI sequences at the same time by training a network at one time. It has the ability to flexibly translate the input MRI image into any required MRI sequence, and can solve the feature encoding of MRI images of various modalities and the mutual generation of MRI multimodal images. It adopts an unsupervised training method, and can still train the generation network in the absence of training data modality.

[0007] In order to achieve the purpose of the present invention, the present invention is implemented by the following technical scheme: a multimodal medical image encoding and generation method based on a generative adversarial network, comprising the following steps:

[0008] Step 1: First collect MRI scan data of different patients from different institutions, and then use a linear interpolator to resample all MRI scan data to 1mm 3 The training dataset is obtained by isotropic resolution, stripping the skull, and co-registering with a single anatomical template using a rigid registration model with mutual information similarity metric;

[0009] Step 2: Divide the training data set into four groups, and each group has only one MRI sequence, and use this as a basis to simulate the MRI sequence missing scenario that conforms to clinical reality;

[0010] Step 3: Based on the simulated MRI sequence missing scenario that conforms to clinical reality, a generative adversarial neural network suitable for multimodal image generation is designed. Specifically, let X and Y be the set of MRI images and MRI sequences respectively. Given an image x∈X and a sequence y∈Y, train a generator G to generate images of sequence y corresponding to image x through the generator G, then design a style encoder E to learn the style code s representing the specific sequence y, and train the generator G to reflect the style code s, and then design a discriminator D to determine whether image x is a real image of the target sequence y or a fake image G(x, s) generated by the generator G;

[0011] Step 4: Train the generative adversarial neural network, and accept pictures of different modalities from different patients in each round of training, one of which is used as the source sequence and the other as the target sequence. The source sequence and the target sequence are randomly selected in each round of training to enable the generative adversarial neural network to realize mutual conversion between arbitrary sequences. First, the style features of the target image sequence are extracted through the style encoder E and represented by the style code s. Then, the generator G uses the style code s as a reference to convert the pictures of the source sequence to the pictures of the target sequence. Then, the discriminator D is used to determine whether the generated sequence belongs to the target sequence.

[0012] Further improvements are as follows: in step 1, the training data set is divided into two cohorts, namely glioblastoma / high-grade glioma and low-grade glioma, the patient's MRI scan data contains four pulse sequences, namely T1, T2, T1ce and FLAIR, clinicians and board-certified radiologists manually annotate all MRI scan data, and each tumor is segmented into enhanced tumor, peritumoral edema, and necrotic and non-enhanced tumor core.

[0013] A further improvement is that in step three, the generative adversarial neural network realizes conversion between arbitrary sequences so that any existing sequence generates any missing sequence, and the generative adversarial neural network accepts different modalities from different patients in a single training process.

[0014] A further improvement is that in step three, the generator G converts the input image x into an output image G(x, s) reflecting the style code s, and the output image G(x, s) of the style code s is provided by the mapping network F or the style encoder E, where s is designed to represent the style of y of a specific sequence.

[0015] A further improvement is that in step three, the style encoder E is given an image x and its target sequence y, and the style encoder E extracts the style code s=Ey(x) of x, where Ey(·) represents the output of E corresponding to domain y, and the style encoder E uses different reference images to generate different style codes.

[0016] A further improvement is that in step three, the discriminator D is a multi-task discriminator and is composed of multiple groups of output branches. The discriminator D determines whether the image x is a real image of the target sequence y or a false image G(x, s) generated by the generator G through learning of each group of branches.

[0017] Further improvements are as follows: in step 4, the loss function in the training process includes feature preservation loss, style encoding loss and adversarial loss, wherein the feature preservation loss is specifically: the generator G restores the generated target sequence image to the image of the source sequence with reference to the style code s of the source sequence, and the restored image is consistent with the real image of the source sequence; the style encoding loss is specifically: the style encoder E extracts the style code s' of the generated image, and the extracted style code s' is similar to the style code s extracted from the target sequence image; the adversarial loss is specifically: the discriminator D determines whether the generated image is input into the target sequence, then fixes the parameters of the generator G and the style encoder E, and uses the adversarial loss to update the parameters of the discriminator D.

[0018] The beneficial effects of the present invention are as follows: the present invention uses an improved generative adversarial neural network to perform simultaneous conversion between multiple MRI sequence images. The generative adversarial neural network framework structure allows for one-time training using multi-sequence and multi-center MRI sequences within a single network, that is, by training one network at a time, feature extraction of multiple MRI sequences can be achieved simultaneously. Compared with existing models, the quality of the translated images is higher, and the ability to flexibly translate the input MRI images into any required MRI sequences can be achieved. The feature encoding of MRI images of various modalities and the mutual generation of MRI multimodal images can be solved. In addition, an unsupervised training method is adopted, and the generative network can still be trained in the absence of training data modalities, which is more in line with clinical scenarios and can achieve image generation between any multimodalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0020] Figure 1 is a schematic diagram of a method flow in an embodiment of the present invention;

[0021] Figure 2 is a schematic diagram of a training data set in an embodiment of the present invention;

[0022] Figure 3 Schematic diagram of a generative adversarial neural network framework in an embodiment of the present invention;

[0023] Figure 4 is a schematic diagram of a round of training process in an embodiment of the present invention;

[0024] Figure 5 Schematic diagram of the loss function in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] See also Figure 1 , 2, 3, 4, 5, this embodiment provides a multimodal medical image encoding and generation method based on a generative adversarial network, comprising the following steps:

[0027] Step 1: First, collect magnetic resonance (MR) scan data of different patients from different institutions, and then use a linear interpolator to resample all MR scan data to 1mm 3 The training dataset is obtained by isotropic resolution, stripping the skull, and co-registering with a single anatomical template using a rigid registration model with mutual information similarity metric;

[0028] The training dataset consists of a total of 335 patient MR scans from multiple different institutions, divided into two cohorts: glioblastoma / high-grade glioma (GBM / HGG) (259 cases) and low-grade glioma (LGG) (76 cases). The patient scans contain four pulse sequences: T1, T2, T1c, and T2FLAIR, such as Figure 2 As shown;

[0029] Clinicians and board-certified radiologists manually annotated all scans, and each tumor was segmented into the enhancing tumor, the peritumoral edema, and the necrotic and non-enhancing tumor core.

[0030] Step 2: Divide the training data set into four groups, and each group has only one magnetic resonance imaging (MRI) sequence, and based on this, simulate the MRI sequence missing scenario that conforms to clinical practice. The most extreme case of this scenario is that each patient has only one MRI sequence (in clinical practice, patients may have more MRI sequences. This embodiment only uses this most difficult example to demonstrate the effectiveness of the method of the present invention);

[0031] Step 3: Based on the simulated MRI sequence missing scenario that meets clinical reality, a generative adversarial neural network (GAN) suitable for multimodal image generation is designed. Its framework is as follows: Figure 3 As shown in FIG. 1 , specifically, let X and Y be the set of MRI images and MRI sequences respectively. Given an image x∈X and a sequence y∈Y, train a generator G to generate images of sequence y corresponding to image x, then design a style encoder E to learn the style code s representing a specific sequence y, and train the generator G to reflect the style code s, and then design a discriminator D to determine whether image x is a real image of the target sequence y or a fake image G(x, s) generated by the generator G;

[0032] The generative adversarial neural network has the following characteristics:

[0033] Multimodality: It is assumed that it is not known which sequence a patient may have lost in the clinic. For example, in the scenario simulated in this embodiment, all four sequences may be lost. Therefore, the generative adversarial neural network should achieve conversion between arbitrary sequences so that any existing sequence can generate any lost sequence.

[0034] Unsupervised: Since the training data sequence is missing and cannot provide supervision information, the training process needs to accept different modalities from different patients instead of paired different modalities of the same patient.

[0035] The generator G converts the input image x into an output image G(x, s) reflecting the style code s, the output image G(x, s) of the style code s is provided by the mapping network F or the style encoder E, where s is designed to represent the style of y of a specific sequence and allows the generator G to synthesize MRI images of all sequences;

[0036] The style encoder E is given an image x and its target sequence y. The style encoder E extracts a style code s=Ey(x) of x, where Ey(·) represents the output of E corresponding to the domain y. Similar to the mapping network F, the style encoder E benefits from a multi-task learning setting. The style encoder E can use different reference images to produce different style codes. The generator allows G to synthesize an output image that reflects the style s of the reference image x.

[0037] The discriminator D is a multi-task discriminator and consists of multiple groups of output branches. The discriminator D learns through each group of branches to determine whether the image x is a real image of the target sequence y or a fake image G(x, s) generated by the generator G;

[0038] Step 4: Train the generative adversarial neural network. The training process of the generative adversarial neural network is unsupervised, that is, each round of training accepts pictures of different modalities from different patients, one of which is used as the source sequence and the other as the target sequence (such as Figure 3 As shown in the figure, the arrow pointing to GAN represents the source sequence, and the arrow from GAN represents the target sequence). Each round of training randomly selects the source sequence and the target sequence, so that the generative adversarial neural network can realize the mutual conversion between arbitrary sequences. First, the style encoder E extracts the style features of the target image sequence and represents it with a style code s. Then, the generator G uses the style code s as a reference to convert the source sequence image to the target sequence image. Then, the discriminator D determines whether the generated sequence belongs to the target sequence.

[0039] like Figure 4As shown, the training process is demonstrated by taking one training as an example. In this round of training, T1 is regarded as the source sequence, T1ce is regarded as the target sequence, the style encoder E extracts the features of the T1ce sequence from patient 2, the generator G uses the features as a reference to generate the images of the T1ce sequence from the images of the T1 sequence of patient 1, and the discriminator D determines whether the generated T1ce image is consistent with the extracted T1ce features;

[0040] like Figure 5 As shown, the loss function in the training process includes feature preservation loss, style encoding loss and adversarial loss. The feature preservation loss is specifically: the generator G restores the generated target sequence image to the source sequence image with reference to the style code s of the source sequence, and the restored image is consistent with the real image of the source sequence; the style encoding loss is specifically: the style encoder E extracts the style code s' of the generated image, and the extracted style code s' is similar to the style code s extracted from the target sequence image; the adversarial loss is specifically: the discriminator D determines whether the generated image is input into the target sequence, then fixes the parameters of the generator G and the style encoder E, and uses the adversarial loss to update the parameters of the discriminator D.

[0041] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal medical image encoding and generation method based on a generative adversarial network, characterized in that: The following steps are involved: Step 1: First collect MRI scan data of different patients from different institutions, and then use a linear interpolator to resample all MRI scan data to 1mm 3 The training dataset is obtained by isotropic resolution, stripping the skull, and co-registering with a single anatomical template using a rigid registration model with mutual information similarity metric; Step 2: Divide the training data set into four groups, and each group has only one MRI sequence, and use this as a basis to simulate the MRI sequence missing scenario that conforms to clinical reality; Step 3: Based on the simulated MRI sequence missing scenario that conforms to clinical reality, a generative adversarial neural network suitable for multimodal image generation is designed. Specifically, let X and Y be the set of MRI images and MRI sequences respectively. Given an image x∈X and a sequence y∈Y, train a generator G to generate images of sequence y corresponding to image x through the generator G, then design a style encoder E to learn the style code s representing the specific sequence y, and train the generator G to reflect the style code s, and then design a discriminator D to determine whether image x is a real image of the target sequence y or a fake image G(x, s) generated by the generator G; Step 4: Train the generative adversarial neural network, and accept pictures of different modalities from different patients in each round of training, one of which is used as the source sequence and the other as the target sequence. The source sequence and the target sequence are randomly selected in each round of training to enable the generative adversarial neural network to realize mutual conversion between arbitrary sequences. First, the style features of the target image sequence are extracted through the style encoder E and represented by the style code s. Then, the generator G uses the style code s as a reference to convert the pictures of the source sequence to the pictures of the target sequence. Then, the discriminator D is used to determine whether the generated sequence belongs to the target sequence.

2. The method for encoding and generating multimodal medical images based on a generative adversarial network according to claim 1, characterized in that: In the step 1, the training data set is divided into two cohorts, namely glioblastoma / high-grade glioma and low-grade glioma. The patients' MRI scan data contain four pulse sequences, namely T1, T2, T1ce and FLAIR. Clinicians and board-certified radiologists manually annotate all MRI scan data, and each tumor is segmented into enhanced tumor, peritumoral edema, and necrotic and non-enhanced tumor core.

3. The multimodal medical image encoding and generation method based on a generative adversarial network according to claim 1, characterized in that: In step three, the generative adversarial neural network realizes conversion between arbitrary sequences so that any existing sequence generates any missing sequence, and the generative adversarial neural network accepts different modalities from different patients in a single training process.

4. The multimodal medical image encoding and generation method based on a generative adversarial network according to claim 1, characterized in that: In step three, the generator G converts the input image x into an output image G(x, s) reflecting the style code s, and the output image G(x, s) of the style code s is provided by the mapping network F or the style encoder E, where s is designed to represent the style of y of a specific sequence.

5. The multimodal medical image encoding and generation method based on a generative adversarial network according to claim 1, characterized in that: In step three, the style encoder E is given an image x and its target sequence y, and the style encoder E extracts the style code s=Ey(x) of x, where Ey(·) represents the output of E corresponding to domain y, and the style encoder E uses different reference images to generate different style codes.

6. The method for encoding and generating multimodal medical images based on a generative adversarial network according to claim 1, characterized in that: In the step three, the discriminator D is a multi-task discriminator and is composed of multiple groups of output branches. The discriminator D determines whether the image x is a real image of the target sequence y or a fake image G(x, s) generated by the generator G through learning of each group of branches.

7. The method for encoding and generating multimodal medical images based on a generative adversarial network according to claim 1, characterized in that: In the step 4, the loss function in the training process includes feature preservation loss, style encoding loss and adversarial loss. The feature preservation loss is specifically: the generator G restores the generated target sequence image to the image of the source sequence with reference to the style code s of the source sequence, and the restored image is consistent with the real image of the source sequence; the style encoding loss is specifically: the style encoder E extracts the style code s' of the generated image, and the extracted style code s' is similar to the style code s extracted from the target sequence image; the adversarial loss is specifically: the discriminator D determines whether the generated image is input into the target sequence, then fixes the parameters of the generator G and the style encoder E, and uses the adversarial loss to update the parameters of the discriminator D.

Images