An unsupervised multimodal brain image translation method guided by tumor perception

Abstract

This invention discloses an unsupervised multimodal brain image translation method guided by tumor perception, comprising the following steps: using a U-Net-based neural network to construct a brain tumor image translation model suitable for translating between any two MRI modalities; the model comprises global and local branches for extracting and fusing feature information from the overall source image and the corresponding tumor image to generate an overall image and tumor image of the target modality, alleviating distortion issues in the generated image in the tumor region; and designing a teacher-student network architecture in which the student network learns the teacher network's knowledge and tumor perception capabilities through distillation, generating more realistic target modality images without relying on tumor label information. This method can generate high-quality target modality images using unpaired single-modality brain images, and does not require tumor label information during the testing phase, making it more consistent with real-world medical scenarios.

Description

Technical Field

[0001] This invention belongs to the field of computer vision in deep learning, involves the cross-application of computers and medicine, and realizes unsupervised multimodal brain image translation in medical images. Background Art

[0002] Multimodal medical images are widely used in various scenarios, including clinical testing. Brain tumors appear differently in different modalities. Therefore, in clinical diagnosis and treatment, doctors can obtain more comprehensive information by analyzing images from different modalities, leading to more accurate diagnoses. However, due to various practical factors such as time and cost, paired multimodal brain tumor images are often difficult to obtain. Therefore, research on cross-modal translation of brain tumor images has attracted widespread attention. Generating other modalities of a known image through image translation technology can assist in clinical diagnosis and medical treatment, while also reducing testing costs for patients. Furthermore, the generated images can be used for further medical research. Therefore, utilizing current advanced artificial intelligence technologies to automatically translate multimodal brain tumor images is of great significance.

[0003] Due to the complexity of the hierarchical structure of brain tumors and the differences in their morphology under different modalities, existing methods often produce deformed or distorted results when translating images of brain tumors. This is because these methods fail to focus the translation on brain tumors, but only perform a global translation of the image of the entire brain. Therefore, the translation results are often unsatisfactory in terms of the location where the tumor appears. A method that uses labels to assist translation (TarGAN: Target-Aware Generative Adversarial Networks for Multi-modality Medical Image Translation) has been proposed, but its performance on brain tumor datasets is poor, and it does not conform to the situation where image labels are often missing in actual application scenarios. Summary of the Invention

[0004] In order to alleviate the local distortion problem in brain tumor image translation in the existing technology, the present invention proposes an unsupervised multimodal brain image translation method based on tumor perception guidance. It uses unpaired brain image training samples and their tumor labels to enable the model to focus on the tumor area, improve the translation effect in the tumor area, and through distillation, enable the student network to learn the teacher network's ability to perceive tumors without the need for tumor labels, and finally achieve the ability to translate the input single-modal brain image into a more accurate target modality image during the testing phase.

[0005] The purpose of the present invention is achieved by at least one of the following technical solutions.

[0006] An unsupervised multimodal brain image translation method based on tumor perception guidance includes the following steps:

[0007] S1. Collect single-modality brain tumor images and their corresponding tumor labels as a training dataset;

[0008] S2. Build a teacher network model based on a generative adversarial network. The model consists of a generator and a pair of discriminators. The generator is divided into a global branch and a local branch, and the discriminator is divided into a global discriminator and a local discriminator.

[0009] S3. Given an overall image of any modality and its corresponding tumor label, first obtain the tumor image by calculating the matrix dot product between the two. Then, the overall image and the local image are input into the global branch and the local branch of the teacher network generator respectively to obtain the target modality image corresponding to the two images.

[0010] S4, construct a loss function for the teacher network model and train it;

[0011] S5. Construct a student network with the same structure as the teacher network. Given an overall image of any modality, it is simultaneously input into the global and local branches of the student network. After translation, the overall image and tumor image of the target modality are obtained. By distilling the knowledge of the teacher network, the student network is given the ability to perceive tumors similar to the teacher network.

[0012] S6. Use the loss function of the teacher network and the distillation loss function as constraints for the student network for training;

[0013] S7. In view of the fact that most brain tumor images in reality are unimodal and lack labels, a student network is used for testing to translate the unimodal source images into target modality images.

[0014] Furthermore, in step S2, the generator includes two U-Net networks and an MFB feature fusion module, each U-Net network contains a pair of encoders and decoders, wherein the encoder includes four groups of downsampling networks, and the decoder includes four groups of upsampling networks. Each group of downsampling networks includes two convolutional layers and a maximum pooling layer with a convolution kernel size of 2 and a stride of 2, wherein the convolution layer uses a convolution kernel size of 3×3, a padding of 1, and a stride of 1, followed by an instance normalization layer and a Leakly ReLU activation function; each group of upsampling networks includes two convolutional layers and a bilinear sampling layer; the two encoders respectively encode features of the overall image and the tumor image, and input the obtained global features and local features into the feature fusion module for fusion, and then input the obtained features into their respective corresponding decoders to realize the translation of the overall image and the local image, wherein the encoder-feature fusion module-decoder used for the overall image translation is called the global branch, and the encoder-feature fusion module-decoder used for the tumor image translation is called the local branch.

[0015] Furthermore, in step S2, the discriminator includes a global discriminator and a local discriminator, both of which adopt the PatchGAN architecture and consist of eight groups of convolutional layers, of which the first six groups of convolutional layers use convolutions with a kernel size of 4×4, a padding of 1, and a stride of 2, followed by an instance normalization layer and a Leakly ReLU activation function; the seventh group of convolutional layers uses convolutions with a kernel size of 3×3, a padding of 1, and a stride of 1; the eighth group of convolutional layers uses convolutions with a kernel size of 2×2, a padding of 0, and a stride of 1, wherein the global discriminator is used to identify the overall image, and the local discriminator is used to identify the tumor image.

[0016] Furthermore, in step S4, the teacher network uses adversarial loss and domain classification loss as the common loss function of the generator, global discriminator and local discriminator. In addition, the generator is also constrained by local consistency loss and reconstruction loss.

[0017] Furthermore, the teacher network adopts an adversarial loss function with gradient constraints. Through adversarial training, the generator can generate more realistic images to confuse the discriminator. The loss function is as follows:

[0018]

[0019]

[0020] in, and Represent the adversarial losses of the global branch and the local branch, I source and T source Represent the overall image and tumor image of the source modality, I target and T targetCorresponding to the overall image and tumor image of the target modality, and Represent the true probabilities of the overall image and tumor image respectively, Represents a random interpolation between the generated overall image and the corresponding real image, which is used to perform gradient penalty and optimize the training of the generative adversarial network. represents a random interpolation between the generated tumor image and the corresponding real image, is the gradient operator, λ gp is the coefficient of the gradient penalty.

[0021] Furthermore, the discriminator determines the modality category to which the image belongs. During the training phase of the discriminator, the discriminator needs to be as prepared as possible to determine whether the input source image is real. The corresponding domain classification loss function is as follows:

[0022]

[0023]

[0024] in, and Represent the domain classification losses of the global discriminator and the local discriminator for the real whole image and tumor image, respectively, s represents the source modality encoding, I source and T source represent the whole image and tumor image of the source modality, respectively. and represents the probability that the discriminator considers the input true whole image and tumor image to be the source modality. Similarly, during the training phase of the generator, the generator needs to generate images that are sufficient to confuse the discriminator as much as possible. The corresponding domain classification loss function is as follows:

[0025]

[0026]

[0027]

[0028] in, and Represent the domain classification losses of the global discriminator and the local discriminator for the overall image and tumor image generated by the generator, respectively, t represents the target modality encoding, and Represents the probability that the discriminator considers the generated overall image and tumor image to be the target modality.

[0029] Furthermore, in order to alleviate the distortion of brain image translation in the tumor area, a local consistency loss is used to constrain the similarity between the overall image generated by the generator and the tumor image. Its form is as follows:

[0030] L local =E(||T target -T′ target ||1)+E(||T source -T′ source ||1)

[0031] Among them, T source is the source modality tumor image input to the local branch of the teacher network, T target is the target modality tumor image generated by the local branch, T′ target is the tumor image obtained by dot multiplication of the target modality overall image generated by the global branch and the binary tumor label, T′ source Is a local branch T target Reconstructed source modality tumor image.

[0032] Furthermore, in order to ensure that the input source image only undergoes a modal style transformation during the entire translation process and avoid the loss of content information, the generator is constrained using reconstruction loss. The function is as follows:

[0033] L rec =E(||I source -I′ source ||1)

[0034] Among them, I′ source Represents the reconstructed overall image of the source modality.

[0035] Furthermore, the final loss functions of the generator and discriminator of the teacher network are as follows:

[0036]

[0037]

[0038] Among them, λ1 is the weight coefficient, represents the final loss function of the global discriminator and the local discriminator in the teacher network, Represents the loss function of the generator.

[0039] Furthermore, in step S6, the student network learns the knowledge of the teacher network through distillation, and the distillation loss function is as follows:

[0040]

[0041] Among them, F teacher and Fstudent Represents the features obtained by the teacher network and the student network in the feature fusion layer, and The target modality image finally generated by the teacher network and the student network is constrained at both the feature and image levels, enabling the student network to better learn the capabilities of the teacher network. The final loss function of the student network is as follows:

[0042]

[0043] Among them, λ2 is the weight coefficient of distillation learning loss.

[0044] Compared with the existing technology, the advantages of the present invention are:

[0045] The present invention effectively alleviates the tumor area distortion phenomenon that occurs during the translation process of brain tumor images, generates more accurate brain images, and assists in better treatment.

[0046] The present invention uses unpaired samples for training, and through distillation, the model does not need to rely on tumor labels during the testing phase, which is more in line with actual application scenarios.

[0047] The present invention has a simple structure and does not rely on additional segmentation networks or target detection networks. It can realize the perception of brain tumor information and enable the model to better focus on the tumor area. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 4 is a model diagram of an unsupervised multimodal brain image translation method based on tumor perception guidance in an embodiment of the present invention;

[0049] Figure 2 This is the structural diagram of the MFB feature fusion module;

[0050] Figure 3 Flowchart of teacher network training in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The solution of the present invention is further described below with reference to the accompanying drawings and specific embodiments.

[0052] Example 1

[0053] The application object of this embodiment is multimodal MRI brain images. MRI is divided into different modalities according to different imaging methods, such as Flair, T1, T1ce, T2, etc.

[0054] Unsupervised multimodal brain image translation methods based on tumor perception guidance, such as Figure 1 As shown, the following steps are included:

[0055] S1. Collect single-modality brain tumor images and their corresponding tumor labels as a training dataset.

[0056] This embodiment uses the BRATS2020 dataset, which contains MRI brain images in four modalities: Flair, T1, T1ce, and T2. Tumor labels are divided into three types according to the region: Whole Tumor, Tumor Core, and Enhancing Tumor. The training set contains 180 cases, the validation set contains 3 cases, and the test set contains 159 cases, each case containing 155 two-dimensional images. All brain images are sampled from different institutions by professional equipment, and tumor labels are annotated by medical experts to ensure the accuracy of the data.

[0057] S2. Construct a teacher network based on the generative adversarial network. The model consists of a generator and a pair of discriminators. The generator is divided into a global branch and a local branch, and the discriminator is divided into a global discriminator and a local discriminator, such as Figure 1 As shown in the figure. The generator consists of two U-Net networks and an MFB feature fusion module. Each U-Net network contains a pair of encoders and decoders, where the encoder contains four groups of downsampling networks and the decoder contains four groups of upsampling networks. Each group of downsampling networks consists of two convolutional layers and a maximum pooling layer with a convolution kernel size of 2 and a stride of 2. The convolution layer uses a convolution kernel size of 3×3, a padding of 1, and a stride of 1, followed by an instance normalization layer and a Leakly ReLU activation function. Each group of upsampling networks consists of two convolutional layers and a bilinear sampling layer. The convolution layer has the same settings as above except for the number of channels.

[0058] The two sets of encoders encode the features of the whole image and the tumor image respectively, and input the obtained global features and local features into the feature fusion module for fusion, and then input the obtained features into their respective decoders to achieve the translation of the whole image and the local image. The structure of the feature fusion module is as follows: Figure 2 As shown in the figure, the two input features are concatenated after three calculations: element-by-element addition, element-by-element multiplication, and element-by-element maximum. The final fused feature is then passed through a convolutional layer. The encoder-feature fusion module-decoder used for overall image translation is called the global branch, while the encoder-feature fusion module-decoder used for tumor image translation is called the local branch.

[0059] The discriminator uses the PatchGAN architecture, consisting of eight convolutional layers. The first six layers use a 4×4 kernel, 1 padding, and a stride of 2, followed by an instance normalization layer and a Leakly ReLU activation function. The seventh layer uses a 3×3 kernel, 1 padding, and a stride of 1, and the eighth layer uses a 2×2 kernel, 0 padding, and a stride of 1.

[0060] S3: Given a whole image of any modality and its corresponding tumor label, three different tumor labels, Whole Tumor, TumorCore, and Enhancing Tumor, are used as guidance. First, the tumor image is obtained by calculating the matrix dot product of the two. The whole image and local image are then input into the global and local branches of the teacher network generator, respectively, to obtain the target modality images corresponding to the two images. The generator translates the whole image and tumor image from the source modality to the target modality. The discriminator is used to identify the authenticity of the image and the modality to which it belongs. The global discriminator is used to identify the whole image, and the local discriminator is used to identify the tumor image.

[0061] S4. Construct a loss function for the teacher network and train it, such as Figure 3 As shown in Figure 2, the teacher network uses adversarial loss and domain classification loss as the common loss function for the generator, global discriminator, and local discriminator. In addition, the generator is constrained by local consistency loss and reconstruction loss.

[0062] In order to make the adversarial learning training more stable, the teacher network adopts an adversarial loss function with gradient constraints, so that the generator can generate more realistic images to confuse the discriminator. The loss function is as follows:

[0063]

[0064]

[0065] in, and Represent the adversarial losses of the global branch and the local branch respectively. source and T source Represent the overall image and tumor image of the source modality, I target and T target Corresponding to the whole image and tumor image of the target modality. and represent the true probabilities of the overall image and tumor image, respectively. Represents a random interpolation between the generated overall image and the corresponding real image, which is used to perform gradient penalty and optimize the training of the generative adversarial network. represents a random interpolation between the generated tumor image and the corresponding real image. is the gradient operator, λ gp is the coefficient of the gradient penalty.

[0066] The discriminator needs to determine the modality category to which the image belongs. During the training phase of the discriminator, the discriminator needs to be as prepared as possible to determine whether the input source image is real. The corresponding domain classification loss function is as follows:

[0067]

[0068]

[0069] in, and Represents the domain classification loss of the global discriminator and the local discriminator for the real whole image and tumor image respectively. s represents the source modality encoding, I source and T source represent the whole image and tumor image of the source modality, respectively. and represents the probability that the discriminator considers the input true whole image and tumor image to be the source modality. Similarly, during the training phase of the generator, the generator needs to generate images that are as confusing as possible to the discriminator. The corresponding domain classification loss function is as follows:

[0070]

[0071]

[0072]

[0073] in, and Represents the domain classification loss of the global discriminator and the local discriminator for the overall image and tumor image generated by the generator, respectively. t represents the target modality encoding, and Represents the probability that the discriminator considers the generated overall image and tumor image to be the target modality.

[0074] In order to alleviate the distortion of brain image translation in the tumor area, local consistency loss is used to constrain the similarity between the overall image generated by the generator and the tumor image. Its form is as follows:

[0075] L local =E(||T target -T′ target ||1)+E(||T source -T′ source ||1)

[0076] Among them, T source is the source modality tumor image input to the local branch of the teacher network, T target is the target modality tumor image generated by the local branch, T′ target is the tumor image obtained by dot multiplication of the target modality overall image generated by the global branch and the binary tumor label, T′ source Is a local branch T target Reconstructed source modality tumor image.

[0077] In order to ensure that the input source image only undergoes a modal style transformation during the entire translation process and avoid the loss of content information, the generator is constrained using reconstruction loss. The function is as follows:

[0078] L rec =E(||I source -I′ source ||1)

[0079] Among them, I′ source Represents the reconstructed overall image of the source modality.

[0080] The final loss functions of the generator and discriminator of the teacher network are as follows:

[0081]

[0082]

[0083] Among them, λ1 is the weight coefficient, represents the final loss function of the global discriminator and the local discriminator in the teacher network,

[0084] Represents the loss function of the generator.

[0085] S5. Build a student network that maintains the same structure as the teacher network, such as Figure 1 As shown in the figure, unlike the training of the teacher network, the local branches of the student network do not use tumor images as input. Given a global image of any modality, it is simultaneously input into the global and local branches of the student network. After translation, the global image and tumor image of the target modality are obtained. By distilling knowledge from the teacher network, the student network acquires similar automatic tumor perception capabilities as the teacher network.

[0086] S6. Use the teacher network's loss function and the distillation loss function as constraints for the student network. The student network learns the teacher network's knowledge through distillation. The distillation loss function is as follows:

[0087]

[0088] Among them, F teacher and F student Represents the features obtained by the teacher network and the student network in the feature fusion layer, and The target modality image finally generated by the teacher network and the student network is constrained at both the feature and image levels, so that the student network can better learn the capabilities of the teacher network.

[0089] The final loss function of the student network is as follows:

[0090]

[0091] Among them, λ2 is the weight coefficient of distillation learning loss.

[0092] S7. Tested on the BRATS2020 dataset, the input unimodal source image is translated into a target modality image. This embodiment uses SSIM, PSNR, and LPIPS as evaluation indicators of image translation quality, and compares with StarGAN, DRIT++, TarGAN, REMIC, and ResViT models. SSIM (structural similarity index) is used to measure the brightness, contrast, and structural similarity between two images, PSNR (peak signal-to-noise ratio) is used to measure the degree of image distortion, and LPIPS (learned perceptual image block similarity) is used to represent the perceptual similarity of two images, and the results are more in line with the actual perception of the human eye. The larger the SSIM and index values, the more similar the generated image is to the real image. The larger the PSNR indicator value, the less distorted the generated image. The smaller the LPIPS indicator value, the closer the generated image is to the real image. The experimental results are as follows:

[0093] Table 1 Quality evaluation of translation results of different models

[0094]

[0095]

[0096] The results in Table 1 show that compared with other models, the tumor images generated by this embodiment are more realistic and clear.

[0097] Example 2

[0098] To verify that the tumor labels used in the local branch are beneficial to improving the translation quality of the model, this example replaces the tumor labels used in the local branch with all-zero labels and random labels. While keeping other settings unchanged, training and testing are performed on the same data samples. The results are shown below:

[0099] Table 2 Quality evaluation of ablation experiment results

[0100]

[0101] Using all zero labels is equivalent to having no valid input to the local branch. The resulting image quality is lower than using tumor labels to assist in generation. Using random labels interferes with image translation, further degrading the quality. This example demonstrates that the input to the local branch can influence the model's final generation performance. Using meaningful tumor labels as an aid can effectively guide the model to generate higher-quality images.

[0102] Example 3

[0103] To verify the practical value of the images generated by the model, this example inputs the images translated by different models into the downstream segmentation model and uses the DSC indicator to evaluate the segmentation effect. The results are shown below:

[0104] Table 3 Quality evaluation of results of different models in downstream segmentation

[0105]

[0106]

[0107] The larger the DSC index, the more accurate the segmentation result. The above results show that the image generated by this embodiment is more realistic and has higher practical value.

[0108] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes based on the technical solutions and inventive concepts of the present invention within the scope disclosed by the present invention, which fall within the scope of protection of the present invention.

Claims

1. An unsupervised multimodal brain image translation method based on tumor perception guidance, characterized in that: The following steps are involved: S1. Collect single-modality brain tumor images and their corresponding tumor labels as a training dataset; S2. Build a teacher network model based on a generative adversarial network. The model consists of a generator and a pair of discriminators. The generator is divided into a global branch and a local branch, and the discriminator is divided into a global discriminator and a local discriminator. S3. Given an overall image of any modality and its corresponding tumor label, first obtain the tumor image by calculating the matrix dot product between the two. Then, the overall image and the local image are input into the global branch and the local branch of the teacher network generator respectively to obtain the target modality image corresponding to the two images. S4, construct a loss function for the teacher network model and train it; S5. Construct a student network with the same structure as the teacher network. Given an overall image of any modality, it is simultaneously input into the global and local branches of the student network. After translation, the overall image and tumor image of the target modality are obtained. By distilling the knowledge of the teacher network, the student network is given the ability to perceive tumors similar to the teacher network. S6. Use the loss function of the teacher network and the distillation loss function as constraints for the student network for training; S7. In view of the fact that most brain tumor images in reality are unimodal and lack labels, a student network is used for testing to translate the unimodal source images into target modality images.

2. The unsupervised multimodal brain image translation method based on tumor perception guidance according to claim 1, characterized in that: In step S2, the generator includes two U-Net networks and an MFB feature fusion module. Each U-Net network contains a pair of encoders and decoders, where the encoder includes four groups of downsampling networks and the decoder includes four groups of upsampling networks. Each group of downsampling networks includes two convolutional layers and a maximum pooling layer with a convolution kernel size of 2 and a stride of 2. The convolution layer uses a convolution kernel size of 3×3, a padding of 1, and a stride of 1, followed by an instance normalization layer and a Leakly ReLU activation function; each group of upsampling networks includes two convolutional layers and a bilinear sampling layer; the two encoders encode features of the overall image and the tumor image respectively, and input the obtained global features and local features into the feature fusion module for fusion, and then input the obtained features into their respective decoders to achieve translation of the overall image and the local image. The encoder-feature fusion module-decoder used for overall image translation is called the global branch, and the encoder-feature fusion module-decoder used for tumor image translation is called the local branch.

3. The unsupervised multimodal brain image translation method based on tumor perception guidance according to claim 1, characterized in that: In step S2, the discriminator includes a global discriminator and a local discriminator, both of which adopt the PatchGAN architecture and consist of eight groups of convolutional layers. The first six groups of convolutional layers use convolutions with a kernel size of 4×4, a padding of 1, and a stride of 2, followed by an instance normalization layer and a Leakly ReLU activation function; the seventh group of convolutional layers uses convolutions with a kernel size of 3×3, a padding of 1, and a stride of 1; the eighth group of convolutional layers uses convolutions with a kernel size of 2×2, a padding of 0, and a stride of 1. The global discriminator is used to identify the overall image, and the local discriminator is used to identify the tumor image.

4. The unsupervised multimodal brain image translation method based on tumor perception guidance according to claim 1, characterized in that: In step S4, the teacher network uses adversarial loss and domain classification loss as the common loss function of the generator, global discriminator and local discriminator. In addition, the generator is also constrained by local consistency loss and reconstruction loss.

5. The unsupervised multimodal brain image translation method based on tumor perception guidance according to claim 4, characterized in that: The teacher network uses an adversarial loss function with gradient constraints. Through adversarial training, the generator can generate more realistic images to confuse the discriminator. The loss function is as follows: in, and Represent the adversarial losses of the global branch and the local branch, I source and T source Represent the overall image and tumor image of the source modality, I target and T target Corresponding to the overall image and tumor image of the target modality, and Respectively represent the true probability of judging the overall image and the tumor image, Represents a random interpolation between the generated overall image and the corresponding real image, which is used to perform gradient penalty and optimize the training of the generative adversarial network. represents a random interpolation between the generated tumor image and the corresponding real image, is the gradient operator, λ gp is the coefficient of the gradient penalty.

6. The tumor-aware guided unsupervised multimodal brain image translation method according to claim 4, characterized in that: The discriminator determines the modality classification to which the image belongs. During the training phase of the discriminator, the discriminator needs to be as prepared as possible to determine whether the input source image is real. The corresponding domain classification loss function is as follows: in, and Represent the domain classification losses of the global discriminator and the local discriminator for the real whole image and tumor image, respectively, s represents the source modality encoding, I source and T source represent the whole image and tumor image of the source modality, respectively. and represents the probability that the discriminator considers the input true whole image and tumor image to be the source modality. Similarly, during the training phase of the generator, the generator needs to generate images that are sufficient to confuse the discriminator as much as possible. The corresponding domain classification loss function is as follows: in, and Represent the domain classification losses of the global discriminator and the local discriminator for the overall image and tumor image generated by the generator, respectively, t represents the target modality encoding, and Represents the probability that the discriminator considers the generated overall image and tumor image to be the target modality, I target and T target Corresponding to the whole image and tumor image of the target modality.

7. The unsupervised multimodal brain image translation method based on tumor perception guidance according to claim 4, characterized in that: In order to alleviate the distortion of brain image translation in the tumor area, local consistency loss is used to constrain the similarity between the overall image generated by the generator and the tumor image. Its form is as follows: L local =E(||T target -T′ target ||1)+E(‖T source -T′ source ‖1) Among them, T source is the source modality tumor image input to the local branch of the teacher network, T target is the target modality tumor image generated by the local branch, T′ target is the tumor image obtained by dot multiplication of the target modality overall image generated by the global branch and the binary tumor label, T′ source Is a local branch T target Reconstructed source modality tumor image.

8. The unsupervised multimodal brain image translation method based on tumor perception guidance according to claim 4, characterized in that: In order to ensure that the input source image only undergoes a modal style transformation during the entire translation process and avoid the loss of content information, the generator is constrained using reconstruction loss. The function is as follows: THE rec =E(‖I source -THE' source ‖1) Among them, I′ source Represents the reconstructed source modality overall image, I source An overall image representing the source modality.

9. The unsupervised multimodal brain image translation method based on tumor perception guidance according to claim 4, characterized in that: The final loss functions of the generator and discriminator of the teacher network are as follows: Among them, λ1 is the weight coefficient, represents the final loss function of the global discriminator and the local discriminator in the teacher network, Represents the loss function of the generator.

10. The unsupervised multimodal brain image translation method based on tumor perception guidance according to claim 1, characterized in that: In step S6, the student network learns the knowledge of the teacher network through distillation. The distillation loss function is as follows: Among them, F teacher and F student Represents the features obtained by the teacher network and the student network in the feature fusion layer, and The target modality image finally generated by the teacher network and the student network is constrained at both the feature and image levels so that the student network can better learn the capabilities of the teacher network. The final loss function of the student network is as follows: Among them, λ2 is the weight coefficient of distillation learning loss.

Images