Semi-supervised multimodal MRI image synthesis method based on coarse-to-fine learning

Abstract

The present invention provides a semi-supervised multimodal magnetic resonance imaging synthesis method based on coarse-to-fine learning. The method comprises: using a multimodal data set of all cases to construct a majority of unpaired data and a small portion of paired data, constructing an MRI synthesis model including a generative network and an enhancement network, using all the data to train the generative network, using a small portion of paired data to train the enhancement network, inputting the image of the source modality into the trained generative network, using the learned cross-modal distribution mapping relationship to map the image of the source modality into a coarse synthetic image of the corresponding target modality, and inputting it into the trained enhancement network, the enhancement network performs fine enhancement on the coarse synthetic image of the target modality, and obtains a cross-modal synthetic MR image of the target modality. The method of the present invention can be applied to cross-modal medical image synthesis, using the patient's existing source modality images to synthesize target modality images, providing assistance to assist doctors in disease diagnosis.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a semi-supervised multimodal magnetic resonance image synthesis method based on coarse-to-fine learning. Background Art

[0002] In recent years, with the rapid development of digital information in areas such as imaging, healthcare, sensor networks, and multi-sensory devices, the generation and collection of multimodal data has become increasingly convenient and accessible. Multimodal data refers to information from multiple sources or forms for the same semantic object, effectively describing it from different perspectives. For example, when we browse a webpage, the description of an object may include images, text, and hyperlinks. Compared to unimodal data, multimodal data contains more information. Comprehensively considering the information contained in multimodal data can provide a deeper understanding of the object, and therefore, multimodal data is frequently used in the medical field. For example, T1-weighted brain images in MRI (Nuclear Magnetic Resonance Imaging) clearly distinguish between gray matter and white matter, while T2-weighted images distinguish between fluid and cortical tissue. MRI also has the advantage of being radiation-free, making it more acceptable to patients than CT (Computed Tomography) imaging, which emits higher levels of radiation. Therefore, it is widely used in clinical practice to aid in disease diagnosis.

[0003] However, in practice, due to allergies to certain contrast agents, incorrect machine settings, excessive noise levels, and scanning costs, some patients' MRI scans may lack certain modalities or be completely unusable, adversely impacting clinical diagnosis and treatment. Furthermore, in these situations, many downstream analysis models that require complete multimodal data often fail to achieve the desired results.

[0004] A simple solution to this problem is to discard the missing modality data and only use the remaining complete multimodal data, but this will result in a lot of waste of useful information. Therefore, studying how to synthesize missing or damaged modality images from other successfully acquired modality images has potential application value.

[0005] To address these challenges, cross-modal medical image synthesis has attracted significant attention from researchers. Deep neural networks, with their powerful learning capabilities, have demonstrated exceptional performance across a wide range of computer vision tasks, including medicine. In recent years, deep learning has been widely applied to cross-modal medical image synthesis, achieving significant success. Important applications include CT-to-PET synthesis, MRI-to-CT synthesis, CT-to-MRI synthesis, and the synthesis of retinal vascular maps into color fundus retinal images. Many effective methods are also available for multimodal MRI synthesis.

[0006] Currently, one existing method for cross-modal medical image synthesis based on deep learning includes the use of supervised learning models. Supervised learning models utilize paired multimodal data to enable a deep network to learn a nonlinear mapping from source modality images to target modality images. However, this method has the following drawbacks: due to various real-world conditions, paired multimodal data is difficult to obtain, and datasets often contain a large proportion of unpaired data. Directly discarding this unpaired data would result in a significant waste of useful information. Furthermore, using only a small amount of paired data for training would prevent supervised learning models that require a large amount of paired data from achieving the desired results. Summary of the Invention

[0007] The embodiments of the present invention provide a semi-supervised multimodal MRI image synthesis method based on coarse-to-fine learning to overcome the problems of the prior art.

[0008] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions.

[0009] A semi-supervised multimodal magnetic resonance imaging synthesis method based on coarse-to-fine learning, comprising:

[0010] The multimodal data sets of all cases are used to construct a large number of unpaired data and a small number of paired data. The large number of unpaired data is used to simulate the large number of modality-missing data in reality, and the small number of paired data is used to simulate the small number of complete multimodal data in reality.

[0011] Constructing a semi-supervised multimodal MRI synthesis model including a generative network and an enhancement network, training the generative network using the entire data to obtain a trained generative network, and training the enhancement network using the small amount of paired data to obtain a trained enhancement network;

[0012] The image of the source modality is input into the trained generative network, and the cross-modal distribution mapping relationship learned by the generative network is used to map the image of the source modality into a coarse synthetic image of the corresponding target modality. The coarse synthetic image of the target modality is input into the trained enhancement network, and the enhancement network performs fine enhancement on the coarse synthetic image of the target modality to obtain a cross-modal synthetic MR image of the target modality.

[0013] Preferably, the method of constructing a majority of unpaired data and a minority of paired data using the multimodal dataset of all cases includes:

[0014] Obtain T1 and T2 modality data for all cases, and linearly normalize the volume data of each modality of each case to the range of [-1, 1];

[0015] The size of the volume data of each modality of each case was cropped from 155*240*240 to 155*192*168;

[0016] When constructing the training data, the complete paired multimodal data of 100% of the cases, 50% of the cases, 20% of the cases, 10% of the cases, and only one case are retained. At the same time, for the remaining cases, the multimodal data alignment within these cases is destroyed, and the volume data of one modality of each case is combined with the volume data of another modality of another case to form unpaired data;

[0017] Each pair of volume data is divided into 155 slices, and the slices are input into the network for training one by one to adapt to the slice-by-slice image synthesis method.

[0018] Preferably, the construction includes an MRI synthesis model of a generative network and an enhancement network, and the generative network is trained using all the data to obtain a trained generative network, including:

[0019] The CycleGAN network is used as the generative network of the MRI synthesis model. The generative network includes a generator and a discriminator of two modalities. The generator uses source domain data to generate data similar to the data in the target domain, and the discriminator distinguishes the generated target domain data from the real data. The loss function of the generative network includes adversarial loss and cycle-consistent loss. The generative network is trained using all the data. The parameters of the generative network are optimized by iteratively reducing the adversarial loss and cycle-consistent loss during the training process. The generative network is used to learn the cross-modal distribution mapping relationship between different modalities from all unpaired and paired data:

[0020] Assume that the two generators G and F in the generative network CycleGAN learn the distribution mapping from A mode to B mode and the distribution mapping from B mode to A mode respectively, and the corresponding discriminators are DA and D B , responsible for identifying the authenticity of the corresponding two modal data, let the T1 mode of the MR image be mode A, and let the T2 mode be mode B;

[0021] For the A modal and B-modal data b∈χ B , adversarial loss is used to train the generator and corresponding discriminator of the two mapping directions. The adversarial loss of mapping from A modality to B modality is expressed as:

[0022]

[0023] The adversarial loss for the mapping from B-modality to A-modality direction is expressed as:

[0024]

[0025] The cycle consistency loss is expressed as:

[0026]

[0027] The overall loss function is:

[0028]

[0029] The optimization objective function is:

[0030]

[0031] G, F are the generators corresponding to A and B modes in the generative network, respectively. A and D B They are the corresponding discriminators. argminmax means: update G and D B , F and D A The network parameters make D B and D A Make the loss function as large as possible while G and F make the loss function as small as possible.

[0032] Preferably, the step of inputting the coarse synthetic image of the target modality into a trained enhancement network, wherein the enhancement network performs fine enhancement on the coarse synthetic image of the target modality to obtain a cross-modality synthetic MR image of the target modality, comprises:

[0033] For paired data from modality A and modality B The two generators G and F in the generative network are used to preliminarily synthesize the reference image G(a P ) and F(b P), and then use two denoising autoencoders to perform fine enhancement of the coarse synthesized image in two directions; the impact of different numbers of residual blocks on the synthesis performance in the enhancement network is tested, and the number of residual blocks in the enhancement network structure is determined by comprehensively considering the generation results and computational overhead.

[0034] It can be seen from the technical solutions provided by the above-mentioned embodiments of the present invention that the method of the embodiments of the present invention can be applied to cross-modal medical image synthesis, using the patient's existing source modality images to synthesize target modality images, thereby assisting doctors in diagnosing diseases.

[0035] Additional aspects and advantages of the invention will be set forth in the description which follows, will be obvious from the description, or may be learned by practice of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 A schematic diagram of problem definition for an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of the overall framework of a semi-supervised multimodal MRI image synthesis method based on coarse-to-fine learning provided by an embodiment of the present invention;

[0039] Figure 3 A processing flow chart of a semi-supervised multimodal MRI image synthesis method based on coarse-to-fine learning provided by an embodiment of the present invention;

[0040] Figure 4 Schematic diagram of a cycle-consistent adversarial network (CycleGAN) according to an embodiment of the present invention;

[0041] Figure 5 An enhanced network structure diagram provided by an embodiment of the present invention;

[0042] Figure 6 This is a T1 modal synthesis performance experimental diagram provided by an embodiment of the present invention;

[0043] Figure 7 This is a T2 modal synthesis performance experimental diagram provided by an embodiment of the present invention;

[0044] Figure 8 A qualitative comparison diagram of the enhanced network effects of different numbers of residual blocks provided by an embodiment of the present invention;

[0045] Figure 9 A quantitative comparison diagram of the enhanced network effects of different numbers of residual blocks provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limiting the present invention.

[0047] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the description of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or couplings. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0048] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention pertains. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and, unless defined as such herein, will not be interpreted in an idealized or overly formal sense.

[0049] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0050] Adversarial learning technology enables deep networks to learn data distributions of different modalities and cross-modal distribution mapping relationships from massive unpaired data. Unsupervised learning models with this feature can reduce the requirements for the quality of training datasets to a certain extent.

[0051] As attached Figure 1As shown, compared with the existing cross-modality medical image synthesis technology, the present invention aims to better cross-modality synthesize MR (Magnetic Resonance) images in the presence of a large amount of modality-missing data and a small amount of complete multimodal data.

[0052] Figure 2 This is a schematic diagram of the overall framework of a semi-supervised multimodal MRI image synthesis method based on coarse-to-fine learning in an embodiment of the present invention. Therefore, the present invention first constructs a large amount of unpaired data and a small amount of paired data from an existing data set, and then uses a semi-supervised multimodal MRI synthesis model including a generation network and an enhancement network to learn the cross-modal distribution mapping relationship between different modalities from all unpaired data and paired data through a coarse-to-fine learning strategy, and learns the detailed differences between multiple modality images in the same case from a small amount of paired data. In the present invention, semi-supervised learning refers to training using a large amount of unpaired data and a small amount of paired data, unsupervised learning refers to using only unpaired data, and supervised learning refers to using only paired data, wherein paired data refers to MR images of two modalities from the same patient, and unpaired data refers to MR images of two modalities from different patients. In the model proposed in the present invention, the generative network trained by unsupervised adversarial learning learns multimodal data distribution information from all data (including all unpaired data and paired data), and at the same time preliminarily synthesizes the target modality reference image, and trains the enhancement network in a supervised learning manner so that it learns sample-level image detail information that only exists in the paired data, so as to enhance the rough synthesis result of the generative network. During model training, these two parts are divided into two stages: the first stage trains the generative network, and the second stage trains the enhancement network. Through the combined use of unpaired data and paired data, the model of the embodiment of the present invention can achieve the advantages of semi-supervised learning. After the model training is completed, in the inference stage, that is, the test stage, it can use any given input modality image to synthesize the corresponding target modality image.

[0053] The processing flow of a semi-supervised multimodal MRI image synthesis method based on coarse-to-fine learning provided by an embodiment of the present invention is as follows: Figure 3 As shown, the processing steps include the following:

[0054] Step S10: Use the multimodal data sets of all cases to construct a majority of unpaired data and a minority of paired data, use the majority of unpaired data to simulate a majority of data with modality missing in reality, and use the minority of paired data to simulate a minority of complete multimodal data in reality.

[0055] Step S20: construct a semi-supervised multimodal MRI synthesis model including a generator network and an enhancement network, use the majority of the unpaired data to train the generator network to obtain a trained generator network, and use the minority of paired data to train the enhancement network to obtain a trained enhancement network.

[0056] The MRI synthesis model proposed in this paper is an architecture that includes a generative network and an augmentation network. The training process of this network architecture is divided into two stages: the first stage uses unpaired data to train the generative network. CycleGAN serves as the generative network in this model, which includes two modal generators and two discriminators, corresponding to the two modal data. The loss functions used in this process include adversarial loss and cycle consistency loss. By reducing the loss during the training process, the parameters of the generative network are iteratively optimized. The second stage uses a small amount of paired data to train the augmentation network (also consisting of two parts, corresponding to the two modal data). This process uses the MAE loss to iteratively optimize the parameters of the augmentation network.

[0057] Cross-modal distribution mapping: The network optimizes network parameters by iteratively reducing the adversarial loss function and the cycle-consistent loss function. This set of network parameters can then be viewed as a mapping function. Depending on the specific task, the network input and output vary, and the learned mapping relationship also varies. The coarse composite image is the output of the source modality image passing through the generator of the corresponding modality in the generative network (data passing through the network undergoes a series of calculations based on the network parameters).

[0058] The denoising autoencoder is selected as the enhancement network, and the enhanced network is trained using paired data in the training set.

[0059] When both training stages are completed and the parameters of the generation network and the enhancement network are updated, a trained MRI synthesis model is obtained.

[0060] Step S30: Input the image of the source modality into the trained generation network, use the learned cross-modal distribution mapping relationship to map the image of the source modality into a coarse synthetic image of the corresponding target modality, input the coarse synthetic image of the target modality into the trained enhancement network, and the enhancement network performs fine enhancement on the coarse synthetic image of the target modality to obtain a cross-modal synthetic MR image of the target modality.

[0061] For a trained MRI synthesis model, assuming that modality A is the source modality and modality B is the target modality, the present invention inputs the source modality image into the A-modality generator of the generative network. Using the learned domain mapping relationship (A->B), the A-modality image is mapped to the corresponding B-modality coarse synthetic image. This B-modality coarse synthetic image is then input (this input is automatically completed within the model, without the need for human intervention) into the B-modality enhancement network. The B-modality enhancement network then refines the B-modality coarse synthetic image to produce a clinically effective cross-modality synthetic MR image of B-modality as the final output.

[0062] If the source modality is B-modality, the situation is similar: the B-modality image is input into the B-modality generator of the generative network. The learned domain mapping relationship (B->A) is used to map the B-modality image to the corresponding A-modality coarse synthetic image, which is then refined and enhanced by the A-modality enhancement network. The final output is the A-modality synthetic image.

[0063] For multimodal data, there is a cross-modal distribution mapping relationship between different modalities, and there are detailed differences between multiple modality images of the same patient. However, in reality, it is very difficult to obtain complete multimodal data due to various reasons, and there is often a large amount of unpaired data in the dataset.

[0064] The present invention aims to utilize a large amount of missing multimodal data and a small amount of complete multimodal data in real conditions to better synthesize medical images across modalities. Therefore, it is necessary to construct unpaired data and paired data during model training. The operation process is as follows:

[0065] 1: To facilitate neural network training and improve convergence speed, for all cases’ T1 and T2 modal data, we linearly normalize the volume data of each case to the range of [-1, 1].

[0066] In order to adapt to the two-downsampling-upsampling structure of the enhanced network and to reduce the influence of the background area in the image on the model learning, the size of each volume data is cropped from 155*240*240 to 155*192*168. In order to verify the effectiveness of the present invention in different situations, when constructing training data using the multimodal dataset, 100% of the cases, 50% of the cases, 20% of the cases, 10% of the cases and only one case of complete paired multimodal data are retained. The constructed paired data sets are recorded as In the multimodal MRI dataset BRATS2015, each case has complete, paired multimodal imaging data. When constructing the training set, we considered different experimental conditions. For example, the "20% cases" mentioned here means retaining complete, paired multimodal images for 20% of the cases in the multimodal dataset, while shuffling and reassembling the multimodal images of the remaining 80% to ensure that the imaging data for each case is unpaired. In other cases, 100% cases is equivalent to using all case data from the original multimodal dataset directly, while 50% cases means retaining paired multimodal images for 50% of the cases in the original dataset, and shuffling the pairings for the remaining 50% to construct unpaired data. The remaining cases are similar.

[0067] At the same time, for the remaining cases, we destroyed the multimodal data registration within these cases and combined the volume data of one modality of each case with the volume data of another modality of another case to form a large amount of unpaired data. The constructed unpaired data set is recorded as For example, for two patients A and B, the A-modality volume data of patient A is combined with the B-modality volume data of patient B to form a set of unpaired data.

[0068] Therefore, to simulate different real-world scenarios, the training data was constructed using paired data from the registered multimodal volume data of five different patient proportions in the training dataset BRATS2015: 100%, 50%, 20%, 10%, and one patient. The availability of paired data for only one patient represents the most extreme case under real-world conditions. For the remaining patients, we simultaneously destroyed the modality registration relationships within these patients and combined the volume data of one modality for each patient with the volume data of another modality from another patient, generating a large amount of unpaired data. Although the present invention uses T1 and T2 modality data from the BRATS2015 dataset, this architecture is still applicable to data from other modalities.

[0069] Since this system uses a slice-by-slice method to synthesize images, each pair of volume data (including all paired data and unpaired data) is divided into 155 slices, and then input into the network slice by slice for training.

[0070] 2: Cross-modal distribution mapping learning

[0071] In terms of multimodal data, if each modality is regarded as a domain, then each domain has its corresponding data distribution. Therefore, to achieve cross-modal synthesis, it is necessary to find the mapping relationship between the distributions of each domain and map the data in the source domain to the target domain through the mapping relationship. The adversarial learning introduced by the generative adversarial network enables the network to learn the distribution mapping of the data domain. The generative adversarial network consists of a generator and a discriminator, where the generator is responsible for trying to use the source domain data to generate data similar to the data in the target domain, while the discriminator aims to distinguish the generated target domain data from the real data. Through adversarial learning, the model can learn the domain distribution mapping without the need for multimodal paired data, which can maximize the use of a large amount of unpaired data in the dataset, thereby avoiding the situation where many supervised learning models perform poorly due to the limited amount of multimodal paired training data.

[0072] Figure 4 A schematic diagram of a Cycle-Consistent Adversarial Network (CycleGAN) in an embodiment of the present invention. In this invention, a Cycle-Consistent Adversarial Network (CycleGAN) is used as a generative network. The generative network learns bidirectional mapping relationships between domain distributions from multimodal data, and during its training process, it utilizes both adversarial loss and cycle-consistent loss to improve mapping accuracy. After training, the mapping relationship learned by the Cycle-Consistent Adversarial Network can be used to map a given source modality image to a target modality, thereby achieving a coarse synthesis of a reference image.

[0073] Since CycleGAN itself cannot exploit the spatial correlation in paired data, it has the same effect when inputting unpaired data as paired data. Therefore, to better utilize the existing data and fully explore the domain distribution information in the data, the following uses all unpaired and paired data to train the generative network to learn the cross-modal data domain mapping relationship and use the learned mapping to initially synthesize the target reference image. The operation process is as follows:

[0074] As attached Figure 4 As shown, it is assumed that the two generators G and F in the generative network CycleGAN learn the distribution mapping from A mode to B mode and the distribution mapping from B mode to A mode respectively, and the corresponding discriminators are D A and D B , responsible for identifying the authenticity of the corresponding two modal data. In the present invention, the T1 mode of the MR image is called the A mode, and the T2 mode is called the B mode.

[0075] For data from modality A and modality B (whether paired or unpaired) and b∈χ B , we use adversarial loss to train the generator and corresponding discriminator of the two mapping directions. The adversarial loss for mapping from A modality to B modality is expressed as:

[0076]

[0077] in, Table mathematical expectation, Indicates sampling of samples a, D from the data distribution of mode A B (G(a)) indicates that the real sample data a of modality A is input into the modality A generator G to obtain the generated image sample of modality B, and then the generated image is input into the corresponding discriminator to obtain the score of the discriminator on the sample; Indicates sampling of samples b from the data distribution of mode B, D B (b) represents the input of real sample data b of modality B into the corresponding discriminator to obtain the discriminator's score for the sample. The adversarial loss for the mapping from modality A to modality B (i.e., learned by the generator G) is formed by summing up all sample scores and taking the average.

[0078] The adversarial loss for the mapping from B-modality to A-modality direction is expressed as:

[0079]

[0080] The cycle consistency loss is expressed as:

[0081]

[0082] F(G(a)) represents the process of passing sample a, sampled from the real data distribution of modality A, through generator G (which learns the mapping from modality A to modality B), to generate a synthetic sample of modality B. This synthetic sample is then fed into generator F (which learns the mapping from modality B to modality A), to generate a synthetic sample of modality A. This cyclic process ensures that the modal mappings learned by G and F become one-to-one.

[0083] The overall loss function of the first stage is:

[0084]

[0085] The optimization objective function is:

[0086]

[0087] G, F are the generators corresponding to A and B modes in the generative network, respectively. A and D B They are the corresponding discriminators. argminmax means: update G and D B , F and D A The network parameters are set to minimize the overall loss (the smaller the overall loss, the smaller the loss of each part. Figure 4From the perspective of the discriminator D, the adversarial loss of the mapping from A mode to B mode is reduced. B It's impossible to tell whether the received data is real data from modality B or fake data synthesized by G from modality A. When the generator reaches a level that can fool the discriminator, we can assume that the generated image output by the generator is very close to the real image. By reducing the cycle consistency loss, we ensure that the cross-modal mapping determined by the generator parameters is consistent with a one-to-one mapping relationship. At the same time, the use of pixel-level loss also improves the quality of the synthesized images.

[0088] 3. Refined image enhancement

[0089] Because different structural organizations have different visual effects in images of different modalities, it is difficult to directly synthesize clinically meaningful target modality images using source modality images solely through domain mapping learned by a cycle-consistent adversarial network. Therefore, the coarse images synthesized by the cycle-consistent adversarial network can be regarded as real images with some noise added. To eliminate noise, improve image quality, and make the synthesized images more valuable for clinical applications, an enhancement network is introduced. In this invention, a denoising autoencoder is used as the enhancement network. Compared to autoencoders that simply retain information from the original input data without ensuring a useful feature representation, the denoising autoencoder can solve this problem by reconstructing the noisy input data. The corrupted input image and the true uncorrupted output image force the denoising autoencoder to learn important image features while ignoring unimportant ones, and the use of a bottleneck intermediate layer structure gives it strong generalization ability. Considering this characteristic, to ensure spatial correlation between the synthesized target modality images and the true target modality images, we use a small amount of paired data from the dataset to train the denoising autoencoder in a supervised learning manner to enhance the coarse synthesized images of the cycle-consistent adversarial network.

[0090] In order to verify the effectiveness of the semi-supervised learning feature of the present invention, the complete paired multimodal data of 100% of the cases, 50% of the cases, 20% of the cases, 10% of the cases, and only one case were retained when constructing the training data. At the same time, for the remaining cases, we destroyed the multimodal data alignment within these cases to form a large amount of unpaired data. Figure 6 and attached Figure 7 The quantitative performance (PSNR) comparison of the semi-supervised learning model proposed in this invention and the unsupervised learning model CycleGAN in five cases is shown.

[0091] Since different tissues have different visual effects in images of different modalities, it is difficult to directly synthesize clinically meaningful target modality images from source modality images using only the domain mapping learned by CycleGAN. Therefore, in order to ensure the spatial correlation between the synthesized target modality images and the real images, we use a small amount of paired data in the dataset to train the denoising autoencoder in a supervised learning manner to enhance the coarse synthesized images of CycleGAN. The network structure of a denoising autoencoder provided by an embodiment of the present invention is shown in the attached figure. Figure 5 shown.

[0092] For paired data from modality A and modality B We use the two generators G and F in the generative network to synthesize the reference image G(a P ) and F(b P ), and then use two denoising autoencoders to perform fine enhancement of the coarse composite image in two directions. The enhancement network, i.e., the denoising autoencoder, is trained using the pixel-level loss - mean absolute error (MAE). The loss function can be expressed as:

[0093]

[0094] Among them, h and w represent the height and width of the image respectively. MAE loss is the mathematical expectation of the L1 norm of the pixel-level deviation between the synthetic image and the corresponding real image in two directions. The training goal of the enhancement network DAE is to make the pixels of the enhanced image as consistent as possible with the real image.

[0095] In order to balance network performance and computational overhead, we tested the effects of different numbers of residual blocks (0, 1, 2, 4, 6) on the synthesis performance in the enhanced network. The results are shown in the attached figure. Figure 8 and attached Figure 9 As shown. Figure 8 and Figure 9 It can be seen from the figure that when the number of residual blocks increases, the visual effect of the synthesis task in both directions is generally improved, but the quantitative index decreases; on the contrary, when the number of residual blocks decreases, the visual effect of the synthesis task in both directions is generally reduced, but the quantitative index increases. Therefore, in order to comprehensively consider the generation results and computational overhead, the enhanced network structure of the embodiment of the present invention is finally determined to include 4 residual blocks (as shown in the attached figure). Figure 4 shown).

[0096] According to the technical solution introduced above, the present invention can be easily applied to cross-modal medical image synthesis, using the patient's existing source modality images to synthesize target modality images, thereby assisting doctors in disease diagnosis.

[0097] In summary, the method according to the embodiment of the present invention can be applied to cross-modality medical image synthesis, using the patient's existing source modality images to synthesize target modality images, thereby assisting doctors in diagnosing diseases.

[0098] The present invention proposes a deep learning architecture that can better perform cross-modal image synthesis when there is a large amount of unpaired data and a small amount of paired data in the dataset. It improves the current supervised learning model in the field of cross-modal MRI synthesis (training requires a large amount of paired data) which suffers from insufficient available paired training data in actual situations. It can also improve the unsupervised learning model (training does not require data pairing) which suffers from a lack of supervised sample-level information.

[0099] Those skilled in the art will appreciate that the accompanying drawings are merely schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0100] From the above description of the embodiments, it can be seen that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus the necessary general-purpose hardware platform. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0101] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without making any creative efforts.

[0102] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A semi-supervised multimodal MRI image synthesis method based on coarse-to-fine learning, characterized in that: include: The multimodal data sets of all cases are used to construct a large number of unpaired data and a small number of paired data. The large number of unpaired data is used to simulate the large number of modality-missing data in reality, and the small number of paired data is used to simulate the small number of complete multimodal data in reality. Constructing a semi-supervised multimodal MRI synthesis model including a generative network and an enhancement network, training the generative network using all the data to obtain a trained generative network, and training the enhancement network using the small amount of paired data to obtain a trained enhancement network; Inputting the source modality image into a trained generative network, mapping the source modality image into a corresponding coarse synthetic image of the target modality using the cross-modal distribution mapping relationship learned by the generative network, inputting the coarse synthetic image of the target modality into a trained enhancement network, and performing fine enhancement on the coarse synthetic image of the target modality by the enhancement network to obtain a cross-modal synthetic MR image of the target modality; The structure includes an MRI synthesis model of a generative network and an enhancement network, and the generative network is trained using all the data to obtain a trained generative network, including: The CycleGAN network is used as the generative network of the MRI synthesis model. The generative network includes a generator and a discriminator of two modalities. The generator uses source domain data to generate data similar to the data in the target domain, and the discriminator distinguishes the generated target domain data from the real data. The loss function of the generative network includes adversarial loss and cycle-consistent loss. The generative network is trained using all the data. The parameters of the generative network are optimized by iteratively reducing the adversarial loss and cycle-consistent loss during the training process. The generative network is used to learn the cross-modal distribution mapping relationship between different modalities from all unpaired and paired data: The two generators G and F in the generative network CycleGAN learn the distribution mapping from A mode to B mode and the distribution mapping from B mode to A mode respectively, and the corresponding discriminators are D A and D B , responsible for identifying the authenticity of the corresponding two modal data, let the T1 mode of the MR image be mode A, and let the T2 mode be mode B; For the A modal and B-mode data The adversarial loss is used to train the generator and the corresponding discriminator in the two mapping directions. The adversarial loss for mapping from A-modality to B-modality is expressed as: The adversarial loss for the mapping from B-modality to A-modality direction is expressed as: The cycle consistency loss is expressed as The overall loss function is: The optimization objective function is: G, F are the generators corresponding to A and B modes in the generative network, respectively. A and D B They are the corresponding discriminators, argminmax means: update G and D B , F and D A The network parameters make D B and D A Make the loss function as large as possible while G and F make the loss function as small as possible; The step of inputting the coarse synthetic image of the target modality into a trained enhancement network, wherein the enhancement network performs fine enhancement on the coarse synthetic image of the target modality to obtain a cross-modality synthetic MR image of the target modality, includes: For paired data from modality A and modality B The two generators G and F in the generative network are used to preliminarily synthesize the reference image G(a P ) and F(b P ), and then use two denoising autoencoders to perform fine enhancement of the coarse synthesized image in two directions; the impact of different numbers of residual blocks on the synthesis performance in the enhancement network is tested, and the number of residual blocks in the enhancement network structure is determined by comprehensively considering the generation results and computational overhead.

2. The method according to claim 1, characterized in that The multimodal dataset of all cases is used to construct a large portion of unpaired data and a small portion of paired data, including: Obtain T1 and T2 modality data for all cases, and linearly normalize the volume data of each modality of each case to the range of [-1, 1]; The size of the volume data of each modality of each case was cropped from 155*240*240 to 155*192*168; When constructing the training data, the complete paired multimodal data of 100% of the cases, 50% of the cases, 20% of the cases, 10% of the cases, and only one case are retained. At the same time, for the remaining cases, the multimodal data alignment within these cases is destroyed, and the volume data of one modality of each case is combined with the volume data of another modality of another case to form unpaired data; Each pair of volume data is divided into 155 slices, and the slices are input into the network for training one by one to adapt to the slice-by-slice image synthesis method.

Images