A multi-modal medical image conversion method and system based on knowledge distillation and adversarial attack

Abstract

The application discloses a kind of based on knowledge distillation and multi-modal medical image conversion method and system of antagonism, comprising: preparing image dataset;Conversion network model is constructed, including antagonism module, generator network, knowledge distillation module;Original modality image is input into antagonism module to obtain antagonistic sample, and original modality image is together as the input of generator, and generator output intermediate conversion result;In pre-training stage, knowledge distillation module is teacher network, and the whole network architecture is pre-trained, and the pre-trained teacher model is obtained;In formal training stage, knowledge distillation module is student network, and the pre-trained teacher model is transferred to student network by knowledge distillation, and the whole network architecture is formally trained, and student model is obtained;The original modality image to be converted is input to generator to obtain intermediate conversion result, and the intermediate conversion result is input into student model, and target modality image is output.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of medical image processing, and relates to a multi-modal medical image conversion method and system based on deep learning. In particular, it relates to a multi-modal medical image conversion method based on knowledge distillation and adversarial attack technology, which constructs a conversion network to achieve high conversion accuracy and strong anti-robustness. BACKGROUND

[0002] In clinical medicine, a single modality image cannot provide enough information to meet the needs of complex diagnosis. Multi-modal medical images can provide detailed information of the diagnosis site from multiple angles, and doctors can combine and analyze the information of different modalities. Multi-modal medical image conversion converts an image of one modality to another modality. With the development and help of this task, patients do not need to undergo multiple time-consuming image scans, which helps to reduce the cost and pressure of image acquisition, and also provides doctors with multiple modal images for reference for disease treatment.

[0003] In recent years, various end-to-end methods in deep learning have been used to solve image conversion problems. Among them, the related methods based on generative adversarial network (GAN) [document 1] have achieved remarkable results in this problem. MUNIT [document 2] decouples the content and style of the image, completes the conversion by exchanging style information and retains the content information, which can realize multi-modal image conversion and improve the detection accuracy of brain tumors [document 3]. UNIT [document 4] uses a shared latent space to learn the common representation between two modalities, which is used to realize the conversion of T1 to T2 modality MRI images [document 5]. NiceGAN [document 6] uses a novel regularization method to independently standardize the feature channels, which is used to realize the conversion of brain MRI to CT images [document 7].

[0004] However, these methods still have some shortcomings. They apply general GAN to the medical image field, although they achieve the purpose of modality conversion, but the results are still slightly rough for the medical field. Because medical image diagnosis is a field that requires high-precision results, the image must have the correct human tissue contour and boundary. Therefore, designing a suitable model for multi-modal medical image conversion becomes a meaningful research direction. Images from different devices, or different parameter settings of the same medical device, are usually contaminated by different degrees and types of noise [document 8]. Considering this, medical image conversion can be rephrased as a noise adaptive task, and the present application also obtains inspiration from it.

[0005] The noise-based method has made good progress, and a recent study [Reference 9] uses GAN to calculate the noise transfer matrix and distinguishes between artificial labeling and real labeling in medical images, considering artificial labeling as noisy training data, and the input image after denoising and enhancement improves the performance of downstream tasks. Inspired by this study, the present invention regards the difference between the source domain image and the target domain image as a certain noise distribution, adds an additional network to assist training after the generator, and calculates the loss of both to adaptively convert the inter-modal conversion rule. Similar to the work of the present invention are the works of Kong [Reference 10] and Chen [Reference 11] et al., the former adds a Unet [Reference 12] network after the generator to improve the quality of the generated medical images, and the latter introduces pixel loss to replace the adversarial loss to improve the conversion and registration accuracy of medical images. However, there is still room for improvement in these studies. Although they focus on the medical image field, they do not fully utilize the learned modal correspondence relationship in the actual test phase.

[0006] REFERENCES

[0007] [Reference 1]: GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks [J]. Communications of the ACM, 2020, 63(11): 139-144.

[0008] [Reference 2]: HUANG X, LIU M-Y, BELONGIE S, et al. Multimodal unsupervised image-to-image translation [A]. Proceedings of the European conference on computer vision (ECCV) [C], 2018: 172-189.

[0009] [Reference 3]: HAN C, RUNDO L, ARAKI R, et al. Combining noise-to-image and image-to-image GANs: Brain MR image augmentation for tumor detection [J]. Ieee Access, 2019, 7: 156966-156977.

[0010] [Document 4]: LIU M-Y, BREUEL T, KAUTZ J. Unsupervised image-to-image translation networks [A]. Proceedings of the 31st International Conference on Neural Information Processing Systems [C], 2017: 700-708.

[0011] [Document 5]: WELANDER P, KARLSSON S, EKLUND A. Generative adversarial networks for image-to-image translation on multi-contrast mr images - a comparison of cyclegan and unit [J]. arXiv preprint arXiv:1806.07777, 2018.

[0012] [Document 6]: CHEN R, HUANG W, HUANG B, et al. Reusing discriminators for encoding: Towards unsupervised image-to-image translation [A]. Proceedings of the IEEE / CVF conference on computer vision and pattern recognition [C], 2020: 8168-8177.

[0013] [Document 7]: WANG J, WU Q, POURPANAH F. DC-cycleGAN: Bidirectional CT-to-MR synthesis from unpaired data [J]. arXiv preprint arXiv:2211.01293, 2022.

[0014] [Ref. 8]: ZAITSEV M, MACLAREN J, HERBST M. Motion artifacts in MRI: A complex problem with many partial solutions [J]. Journal of Magnetic Resonance Imaging, 2015, 42(4): 887-901.

[0015] [Ref. 9]: ZHANG L, TANNO R, XU M-C, et al. Disentangling human error from ground truth in segmentation of medical images [J]. Advances in Neural Information Processing Systems, 2020, 33: 15750-15762.

[0016] [Ref. 10]: KONG L, LIAN C, HUANG D, et al. Breaking the dilemma of medical image-to-image translation [J]. Advances in Neural Information Processing Systems, 2021, 34: 1964-1978.

[0017] [Ref. 11]: CHEN Z, WEI J, LI R. Unsupervised Multi-Modal Medical Image Registration via Discriminator-Free Image-to-Image Translation [A]. [C], 2022.

[0018] [Document 12]: RONNEBERGER O, FISCHER P, BROX T. U-net: Convolutional networks for biomedical image segmentation [A]. Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 [C], 2015: 234-241. SUMMARY

[0019] In order to solve the above-mentioned technical problems, the present application proposes a multi-modal medical image conversion method based on knowledge distillation and adversarial attack. The focus is to introduce the teacher-student module in knowledge distillation, and the additional network mentioned in the previous text is regarded as a pre-trained teacher model, which is used to guide the training process of the student model. In the actual application and test stage, the student model is connected to the generator to further improve the quality of the converted image, which will get more accurate results than using a single generator. This method greatly improves the accuracy and robustness of multi-modal medical image conversion.

[0020] To achieve this purpose, the technical solution adopted by the present application is: a multi-modal medical image conversion method based on knowledge distillation and adversarial attack, the specific implementation includes the following steps:

[0021] Step 1, construct a medical image dataset including the original modality and the target modality;

[0022] Step 2, construct a conversion network model based on deep neural network, which includes an adversarial attack module, a generator network, and a knowledge distillation module;

[0023] Step 3, input the original modality image into the adversarial attack module to get the adversarial sample, and input the adversarial sample and the original modality image together as the input set of the generator network, and the generator network outputs the intermediate conversion result;

[0024] Step 4, in the pre-training stage, the knowledge distillation module is the teacher network, its input is composed of two parts of the intermediate conversion result of step 3 and the target modality image, and outputs the teacher deformation field, and inputs the teacher deformation field and the intermediate conversion result into the spatial deformation network, and outputs the target modality image converted by deformation calculation, after the pre-training stage, the pre-trained teacher network is obtained;

[0025] Step 5, in the formal training stage, the knowledge distillation module is the student network, the input of which only contains the intermediate conversion result of step 3, and the teacher network transmits knowledge to the student network through knowledge distillation, the student network outputs the student deformation field, and the student deformation field and the intermediate conversion result are input into the spatial deformation network to output the target modality image converted through deformation calculation, and after the formal training stage, the trained student network is obtained;

[0026] Step 6, input the original modality image to be converted into the generator network to obtain the intermediate conversion result, input the intermediate conversion result into the student network, and output the converted image, which is the target modality image.

[0027] Further, the specific implementation of step 3 includes the following sub-steps:

[0028] Step 3.1, construct a black box attacker A in the adversarial attack module, the input of the attacker is the original modality image I, and the output is the adversarial sample I adv ;

[0029] I adv = I + A(I, z),

[0030] Where z represents Gaussian noise, the purpose of generating the adversarial sample I adv is to interfere with the generator network G;

[0031] Step 3.2, define the generator network as G, input the original modality image I and the adversarial sample I adv together as the input of G, and the intermediate conversion result output by the generator G is collectively referred to as G(I).

[0032] Further, the specific implementation of step 4 includes the following sub-steps:

[0033] Step 4.1, in the pre-training stage, the knowledge distillation module is the teacher network, define the teacher network as R t , R t is composed of a Unet network and a spatial deformation network STN;

[0034] Step 4.2, input the intermediate conversion result G(I) and the target modality image Y into the Unet of R t , the Unet network will calculate the teacher deformation field φ, I represents the input original modality image, and G is the generator network;

[0035] Step 4.3, input the teacher deformation field φ and the intermediate conversion result G(I) into the STN of R t , and calculate the target modality image through deformation;

[0036] Step 4.4: Iteratively train the transformation network model on the training set. After training, the trained teacher network is obtained.

[0037] Furthermore, the overall loss function used in step 4.4 when training the transformation network model is as follows:

[0038] The symbol ° represents the deformation calculation performed by the STN network, R. t After being connected to the generator network G to improve the quality of the generated images, a teacher refinement loss exists. Calculation:

[0039]

[0040] E in the above formula I,Y This represents the mathematical expectation calculation based on I and Y; to ensure R t Robustness, introducing a teacher's adversarial loss The teacher adversarial loss is expressed as follows: This encourages attackers to generate perturbations that are more meaningful to input I based on the current training feedback in real time.

[0041]

[0042] In the above formula Indicates based on I adv The mathematical expectation of Y is calculated, where ε is the relaxation factor and is a constant;

[0043] With teachers detailing the losses The losses from confrontation with teachers The pre-training of the teacher network involves the following optimization objective function, which also considers other loss terms and corresponding weight coefficients λ, μ, θ, δ, δ:

[0044]

[0045] in This ensures that adversarial examples can share the target image Y with the clean input during training. The loss is calculated using the L1 norm of both the clean and disturbed inputs.

[0046]

[0047] During the pre-training process of the teacher network, the discriminator D judges the generated results obtained by the generator network G. The adversarial process between the generator G and the discriminator D generates an adversarial loss term. Calculation:

[0048]

[0049] Among them, E Y and EI These represent the mathematical expectation calculation based on Y and the mathematical expectation calculation based on I, respectively. To avoid The smoothing loss term set for excessive distortion limits R. t The model produces non-smooth deformations, meaning it encourages similar deformations between adjacent pixels to avoid abrupt changes. Formally, a regularization term is set to smooth the deformations and find R0. t The minimum of the first-order gradient, therefore the smoothing loss term is defined as:

[0050]

[0051] To minimize the teacher network R t Total loss function With the goal of iterative optimization using data from the training set, once the training termination condition is met, the teacher model at this point is saved as the pre-trained teacher network, and then step 5 is executed.

[0052] Furthermore, the specific implementation of step 5 includes the following sub-steps:

[0053] Step 5.1, in the formal training phase, the knowledge distillation module is the student network, defined as R. s R s It also consists of a Unet network and a spatial deformation network STN;

[0054] Step 5.2: The pre-trained teacher network continuously transmits knowledge to the student network through knowledge distillation, guiding the output of the student network to move closer to the output of the teacher network;

[0055] Step 5.3, use the intermediate transformation result G(i) as R s The input to Unet is used to calculate the student deformation field φ, where i represents the original modal image of the input, and G is the generator network.

[0056] Step 5.4, use the student deformation field φ and the intermediate transformation result G(i) as R s The target modality image is obtained by taking the STN input as input and performing deformation calculations.

[0057] Step 5.5: Iteratively train the transformation network model on the training set. After training, the trained student network is obtained.

[0058] Furthermore, the overall loss function used in step 5.5 when training the transformation network model is as follows;

[0059] In the knowledge distillation process of the student network, the loss function consists of four parts, namely the student refinement loss. Teacher-student soft loss Attention loss And student adversarial loss The overall loss function and weight coefficients when training the student module are as follows:

[0060]

[0061] Wherein, α, β, μ are weight coefficients, And Allowing the student network to simulate the output of the teacher network, Ensuring that the attention feature maps in the teacher and student networks are as close as possible, because the student needs to focus on the key information of image conversion as the teacher does, and the student adversarial loss Design concept and teacher adversarial loss Same;

[0062] Student refinement loss The gap between the predicted output R s of the student network and the true label Y is evaluated:

[0063]

[0064] In the above formula, E I,Y represents the mathematical expectation calculation based on I and Y, and the symbol represents the deformation calculation performed by the STN network;

[0065] Student adversarial loss Same design concept as teacher adversarial loss In the training of the student model, an attacker is still introduced to improve the robustness of the model, and there is a student adversarial loss as follows:

[0066]

[0067] ε is a relaxation factor, which is a constant, and I adv is an adversarial sample;

[0068] Teacher-student soft loss Is calculated based on soft targets, because the teacher's soft target has more information than the hard target, and the temperature factor temp is used to control the importance of the soft target. This loss evaluates the gap between the teacher and the student output:

[0069]

[0070] Wherein KL represents the KL divergence loss between the output soft targets of the teacher network and the student network, and softmax represents the normalization exponential function;

[0071] Attention loss The attention feature maps in the teacher and student networks are made as similar as possible. The attention loss function is expressed as follows:

[0072]

[0073] In the formula, τ represents the set of indices of the feature maps to be passed attention, j represents the j-th feature map to be passed attention, and F represents the mapping function, which transforms the input 3D feature map into a 2D attention map. and These represent feature graphs of the teacher network and the student network, respectively.

[0074] To minimize the teacher network R s Total loss function With the goal of iterative optimization using data from the training set, once the training termination condition is met, the student network at this point is saved, and then step 6 is executed.

[0075] Furthermore, the specific formula for calculating the mapping function F is as follows:

[0076]

[0077] Among them U i ∈R H×W Let be the i-th channel of the feature map, C be the total number of channels, and |·| denotes the calculation of the absolute value of the matrix. The above formula means that an attention map is obtained by aggregating the features of different channels.

[0078] This invention also provides a multimodal medical image conversion system based on knowledge distillation and adversarial attacks, comprising the following modules:

[0079] The dataset construction module is used to build medical image datasets that include the original modality and the target modality;

[0080] The transformation network model building module is used to build a deep neural network-based transformation network model, which includes an adversarial attack module, a generator network, and a knowledge distillation module.

[0081] The adversarial example generation module is used to input the original modality image into the adversarial attack module to obtain adversarial examples. The adversarial examples and the original modality image are used together as the input set of the generator network, and the generator network outputs intermediate transformation results.

[0082] In the pre-training module, the knowledge distillation module is the teacher network. Its input consists of two parts: intermediate transformation results and target modal image. It outputs the teacher deformation field. This teacher deformation field and intermediate transformation results are input together into the spatial deformation network. Through deformation calculation, the transformed target modal image is output. After the pre-training phase, the pre-trained teacher network is obtained.

[0083] The formal training module, in the formal training stage, the knowledge distillation module is a student network, the input of which only contains the intermediate conversion result, while the teacher network transmits knowledge to the student network through knowledge distillation, and the student network outputs a student deformation field. The student deformation field and the intermediate conversion result are input into the spatial deformation network together, and the target modality image converted through deformation calculation is output. After the formal training stage ends, the trained student network is obtained;

[0084] The prediction module is used for inputting the original modality image to be converted into the generator network to obtain the intermediate conversion result, and inputting the intermediate conversion result into the student network to output the converted image, which is the target modality image.

[0085] Further, the specific implementation of the pre-training module includes the following sub-steps:

[0086] Step 4.1, in the pre-training stage, the knowledge distillation module is a teacher network, and the teacher network is defined as R t , R t is composed of a Unet network and a spatial deformation network STN;

[0087] Step 4.2, the intermediate conversion result G(I) and the target modality image Y are input into the Unet network as R t , the Unet network will calculate the teacher deformation field φ, I represents the input original modality image, and G is the generator network;

[0088] Step 4.3, the teacher deformation field φ and the intermediate conversion result G(I) are input into the STN as R t , and the target modality image is obtained through deformation calculation;

[0089] Step 4.4, the conversion network model iteration training is performed in the training set, and after the training ends, the trained teacher network is obtained.

[0090] Further, the specific implementation of the formal training module includes the following sub-steps:

[0091] Step 5.1, in the formal training stage, the knowledge distillation module is a student network, and the student network is defined as R s , R s is also composed of a Unet network and a spatial deformation network STN;

[0092] Step 5.2, the pre-trained teacher network transmits knowledge to the student network through the knowledge distillation method, and guides the output of the student network to approach the output of the teacher network;

[0093] Step 5.3, the intermediate conversion result G(I) is input into the student network as R sThe input of the Unet, the Unet network calculates a student deformation field φ, I represents the input original modality image, and G is a generator network;

[0094] Step 5.4, the student deformation field φ and the intermediate conversion result G(I) are taken as R s The input of the STN, the image of the target modality is calculated by deformation;

[0095] Step 5.5, the training of the conversion network model iteration is carried out in the training set, and after the training is completed, the trained student network is obtained.

[0096] Compared with the existing multi-modal medical image conversion method, the present application mainly has the following innovations and advantages:

[0097] (1) The present application proposes a robust multi-modal medical image conversion method based on knowledge distillation and adversarial attack, which includes an adversarial attacker, a generator and a knowledge distillation module, and can be theoretically applied to any generator.

[0098] (2) The knowledge distillation module is based on the idea of knowledge distillation and is composed of a teacher and a student network with different inputs but similar structures, which can improve the accuracy of image conversion in the actual test and application stage.

[0099] (3) The black box attacker based on adversarial attack uses the real-time feedback of the conversion network to add meaningful interference to the training data to improve the robustness of the model. BRIEF DESCRIPTION OF DRAWINGS

[0100] Figure 1 is a general introduction to the medical image conversion task in the embodiment of the present application;

[0101] Figure 2 is a general structure diagram of the conversion network model in the embodiment of the present application;

[0102] Figure 3 is a knowledge distillation teacher network pre-training flowchart in the embodiment of the present application;

[0103] Figure 4 is a knowledge distillation student network formal training flowchart in the embodiment of the present application;

[0104] Figure 5 is a test stage flowchart in the embodiment of the present application;

[0105] Figure 6 is a flowchart of the embodiment of the present application. DETAILED DESCRIPTION

[0106] For the convenience of those skilled in the art to understand and implement the present application, the present application is further described in detail below in combination with the drawings and implementation examples. It should be understood that the implementation examples described herein are only for illustration and explanation of the present application, and are not intended to limit the present application.

[0107] The present application is based on a robust medical image conversion method based on knowledge distillation and adversarial attack, aiming to reduce the cost and burden of image acquisition, and to provide better medical decision reference for doctor diagnosis. From the perspective of improving conversion accuracy, a knowledge distillation module is proposed, and from the perspective of improving model robustness, an adversarial attacker is introduced for auxiliary training.

[0108] See Figure 1 , the present application provides an overview of the image conversion task, and the knowledge distillation module R is the focus of the present application. The figure contains two networks G and R. G is a generator network that realizes cross-modal conversion. R is a knowledge distillation module that further refines and improves the preliminary conversion results obtained by the generator. In the pre-training phase, it is a teacher network, and in the formal training and testing phase, it is a student network. For general conversion methods based on generative adversarial networks, the final conversion result is mainly obtained by means of the generator network G. These methods use the trained generator for conversion, but the results are still slightly rough for medical images. Because medical images have characteristics different from natural images, the anatomical features and medical information of the image must still be guaranteed after conversion, not just the appearance change. The preliminary results obtained by using the generator network G alone usually have a large error with the target modality image, which can be reflected from the error map in Figure 1 . The error image is calculated from the generated image and the target image. The brighter the color, the greater the error between the two. In the image conversion task, it is naturally desirable to have a smaller error, which means that the generated result is closer to the target image. As can be seen from the error map in Figure 1 , the refined and improved results should be better than the preliminary conversion results before improvement.

[0109] As Figure 6 shown, the present application provides a robust medical image conversion method based on knowledge distillation and adversarial attack, comprising the following steps:

[0110] Step 1: Construct a medical image dataset including the original modality and the target modality;

[0111] Figure 1 The left input part shows the medical image display diagram of the original modality and the target modality in the prior art, modality A is the original modality image, and modality B is the target modality image.

[0112] Step 2: Construct a conversion network model based on a deep neural network, which includes three parts in the overall architecture of the method, namely the adversarial attack module, the generator network, and the knowledge distillation module.

[0113] See Figure 2 , the overall structure diagram of the conversion network. The network mainly consists of three parts, namely the adversarial attack module on the left, the generator in the middle, and the knowledge distillation module on the right. The knowledge distillation module includes a teacher network in the pre-training stage and a student network in the formal training stage. Given an input I, the attacker A adds interference and generates an adversarial sample I adu . The original clean input I and the disturbed sample I adv are input into the generator and get G(I) respectively. In the pre-training stage of the knowledge distillation module, the teacher network receives G(I) and the image Y of the target modality as input, calculates the noise distribution through the network to get the deformation field, and then inputs the deformation field and G(I) into the spatial transformation network STN to refine and improve G(I) to get better results. In the formal training stage of the knowledge distillation module, the teacher network transmits knowledge to the student network through knowledge distillation. In the test stage, the student network only receives G(I) and improves the result.

[0114] The adversarial attack module is a black-box attacker network, which aims to disturb the input image and generate an adversarial sample. The adversarial sample and the original modality image are used as training data for the subsequent model, and the process of the attacker and the conversion model against each other can improve the robustness of the conversion model.

[0115] The generator network, i.e. the generator in the generative adversarial network, takes the image to be converted as input and outputs the intermediate conversion result. In theory, this generator network can be replaced by any existing generator model.

[0116] The knowledge distillation module is based on knowledge distillation. It includes a teacher network and a student network, which improve the image quality of the intermediate conversion result in the pre-training stage and the formal training stage respectively, aiming to obtain better final conversion images.

[0117] Step 3: Input the original modality image into the adversarial attack module to obtain the adversarial sample, and input the adversarial sample and the original modality image together into the generator as the input set of the generator to generate the intermediate conversion result. The specific implementation of step 3 includes the following sub-steps:

[0118] Step 3.1: Construct a black-box attacker A in the adversarial attack module, and the input of the attacker is the original modality image I, and the output is the adversarial sample I adv ;

[0119] The black-box attacker A is used to generate the adversarial sample I advThe purpose of the adversarial sample is to reflect the weakness of the conversion network, and the conversion network needs to improve its robustness according to the real-time feedback of the loss function in the training process, so as to counteract the interference of the adversarial sample on the model training. Adding an adversarial interference to each input I can obtain I adv Based on this purpose, the present application uses a conditional generator as an attacker A to generate an adversarial sample I adv :

[0120] I adv = I + A(I, z),

[0121] Where z represents Gaussian noise, and the purpose of generating an adversarial sample I adv is to interfere with the generator G. The feedback of the conversion model (target model) in the training process enables the attacker A to learn and adjust its attack force independently, and the mutual interference mechanism of the attacker and the conversion network can improve the robustness of the conversion model.

[0122] Step 3.2: define the generator network as G, and input the original modality image I and the adversarial sample I adv together as the input of G, and the intermediate conversion result output by the generator G is collectively referred to as G(I).

[0123] Step 4: see Figure 3 , the pre-training process of the teacher network of the present application. In the pre-training stage, the knowledge distillation module is the teacher network, the input of which is composed of the intermediate conversion result of step 3 and the target modality image, and the output is the teacher deformation field. Input the teacher deformation field and the intermediate conversion result into the spatial deformation network, and output the target modality image obtained by deformation calculation. After the pre-training stage, the pre-trained teacher network is obtained. The specific implementation of step 4 includes the following sub-steps:

[0124] Step 4.1: in the pre-training stage, the knowledge distillation module is the teacher network, and the teacher network is defined as R t . R t is composed of a Unet network and a spatial deformation network STN;

[0125] The teacher network R t improves the generated picture G(I) from the generator G, and the main function of R t is to learn the transformation correspondence between images, which is composed of a Unet network and a spatial deformation network STN. Essentially, it is to learn the conversion relationship from G(I) to the target modality image Y, and to improve the quality of the generated image by adjusting the intermediate result G(I) to be closer to the target image Y. Here, the transformation applied is a kind of nonlinear dense deformation registration, which can learn the non-uniform mapping relationship between two images, and the non-uniform mapping relationship is represented as a deformation field φ;

[0126] Step 4.2: Take the intermediate conversion result G(I) and the target modality image Y as inputs of R t The input of Unet, the Unet network will calculate the teacher morph field φ;

[0127] In the image-to-image conversion task of the present application, define (I, Y) as the input modality image and the target modality image respectively, then Space. However, in fact, the absolute true label Y is unknown, because there are errors and artifacts in image acquisition, so the label that can be used is a noisy label The present application regards the difference between Y and as a noise distribution and satisfies This can be simplified as the offset error between the two modality images. Here the representation of the morph field is φ and satisfies Because the offset of a two-dimensional image involves data on two coordinate axes, the number of channels of φ is equal to 2. R t Take G(I) and Y as two inputs, calculate the morph field φ = R t (G(I), Y), φ calculates how to deform G(I) to map to Y. In the present application, φ is a two-dimensional matrix calculated by Unet, which describes the deformation of each pixel in the image G(I), which will be used as the input of the STN network in the next step;

[0128] Step 4.3: Take the teacher morph field φ and the intermediate conversion result G(I) as inputs of R t The input of STN, calculate the target modality image through deformation;

[0129] Next, use the STN network to resample the generator's result G(I) according to the morph field φ to obtain the final result. Use the symbol ° to represent the deformation calculation of the STN network, R t Connected to the generator G to improve the quality of the generated picture, here there is a teacher refinement loss The calculation is as follows:

[0130]

[0131] E I,Y in the above formula represents the mathematical expectation calculation based on I and Y. In order to ensure the robustness of R t , combined with the adversarial attack introduced earlier, a teacher adversarial loss is also introduced here to encourage attackers to generate perturbations based on the current training to make more sense to the input I. The teacher adversarial loss can be represented as:

[0132]

[0133] In the above formula represents the mathematical expectation calculation based on I adv When the generator G can learn the features of the adversarial samples and correctly process the interference, the attacker A must generate more difficult adversarial samples. This game not only reduces the risk of overfitting of the generator, but also improves the robustness of the overall conversion model. The introduction of the relaxation factor ε = 0.1 limits the loss value to change within a controllable range, and also avoids excessive attacks.

[0134] Step 4.4: Train the conversion network model iteration in the training set, and after the training is completed, the trained teacher network is obtained.

[0135] With the teacher refinement loss and the adversarial loss of the teacher When pre-training the teacher network, the following optimization objective function is considered, in which some other loss terms and corresponding weights are also considered:

[0136]

[0137] Among them ensure that the adversarial samples can share the target image γ with the clean input during training, which is calculated by the L1 norm loss of the clean input and the disturbed input:

[0138]

[0139] In the pre-training process of the teacher network, the discriminator D judges the generated results of the generator G, and the adversarial process of the generator G and the discriminator D will generate an adversarial loss term The calculation is as follows:

[0140]

[0141] Among them, E Y and E I respectively represent the mathematical expectation calculation based on Y and the mathematical expectation calculation based on I;

[0142] is a smoothing loss term set to avoid excessive distortion and deformation, which limits R t model to produce non-smooth deformation, that is, to encourage adjacent pixels to have similar deformation without being abrupt. In form, a regularization term for smoothing deformation is set to find the minimum of the first order gradient of R t , so the smoothing loss term is defined as:

[0143]

[0144] To minimize the teacher network R t Total loss function To achieve the goal, iterative optimization is performed using data from the training set. Once the training termination condition is met, the teacher model at this point is saved as the pre-trained teacher network, and then step 5 is executed.

[0145] Step 5: Please see Figure 4 This outlines the formal training process for the student network in this application. During the formal training phase, the knowledge distillation module serves as the student network, its input consisting only of the intermediate transformation results from step 3. Simultaneously, the teacher network transmits knowledge to the student network through knowledge distillation, and the student network outputs a student deformation field. This student deformation field, along with the intermediate transformation results, is input into a spatial deformation network, which calculates the transformation and outputs the target modal image. After the formal training phase, the trained student network is obtained. Step 5 specifically includes the following sub-steps:

[0146] Step 5.1: In the formal training phase, the knowledge distillation module is the student network, defined as R. s R s It also consists of a Unet network and a spatial deformation network STN;

[0147] In the actual testing or application phase, the target image Y is unknown and therefore cannot be used as input to the student network. Since the pre-trained teacher network is merely an ideal transformation network available only during the training phase, this invention introduces a knowledge distillation strategy to ensure the teacher's knowledge is fully utilized during the testing phase. The goal is to distill a student network that can improve and refine G(I) during the testing phase without requiring the target modality Y as input. Specifically, attention feature maps are used as the knowledge transfer carrier, allowing the student network to learn the teacher's knowledge through distillation during the formal training phase. The teacher and student networks have similar structures, both consisting of a Unet network and an STN network. The difference between the teacher and student lies in their inputs; the student input only contains the generator's result G(I) and does not include the target image Y. Therefore, during the testing phase, the student network can be connected to the generator to further improve the quality of the transformed image. Table 1 shows the specific structures of the Unet in the teacher and student networks in this example. Teacher Network R t And student network R s All are based on registration networks; the registration network in this study consists of Unet and STN. R t and R s The Unet part has a similar structure, only the number of input channels is different, because R t The input compared to R sThe target image Y is added, as shown in Table 1. The rest of the network is the same as the input channel number. Down1 to Down4 are down-sampling layers, each of which includes two consecutive convolution layers and a max-pooling layer. Up1 to Up4 are up-sampling layers, each of which includes a de-convolution layer, a skip connection (merge operation in Table 1) and two convolution layers. Bridge is a connection layer between down-sampling and up-sampling, which is composed of two consecutive convolution layers. The Output output layer is completed by a convolution operation.

[0148] Table 1

[0149]

[0150] Step 5.2: The pre-trained teacher network continuously transmits knowledge to the student network through knowledge distillation, guiding the output of the student network to approach the output of the teacher network.

[0151] In order to focus on the key areas of the image and improve the efficiency of knowledge transmission, this study uses attention feature maps as the carrier of knowledge transmission between the teacher and the student. In the knowledge distillation process of the student network, the loss function is composed of four parts. Respectively, the student refinement loss Teacher-student soft loss Attention loss And student adversarial loss The overall loss function and weight coefficient of the student module during training are as follows:

[0152]

[0153] And Allow the student network to simulate the output of the teacher network. Ensure that the attention feature maps in the teacher and student networks are as close as possible, because the student needs to focus on the key information of image conversion as the teacher. Student adversarial loss The design concept is the same as the teacher adversarial loss .

[0154] Step 5.3: Take the intermediate conversion result G(I) as the input of Unet, and the Unet network will calculate the student deformation field φ; s

[0155] R s Only accepts G(I) as input, calculates the deformation field φ = R s (G(I)), φ calculates how to deform and map G(I) to Y.

[0156] Step 5.4: Take the student deformation field φ and the intermediate conversion result G(I) as the input of R​s The target modality image is obtained by taking the STN input as input and performing deformation calculations.

[0157] Students detail the losses The student network R was evaluated. s The difference between the predicted output and the true label Y:

[0158]

[0159] Students fight against losses Fighting against teachers for losses The design philosophy is the same; attackers are still introduced during the training of the student model to improve its robustness. The student adversarial loss is as follows:

[0160]

[0161] Step 5.5: Iteratively train the transformation network model on the training set. After training, the trained student network is obtained.

[0162] Teacher-student soft loss The calculation is based on soft objectives because the teacher's soft objectives contain more information than hard objectives. A temperature factor (temp) is used to control the importance of the soft objectives, and the loss assesses the gap between the teacher's and student's output.

[0163]

[0164] Where KL represents the KL divergence loss between the output soft objectives of the teacher network and the student network, and softmax represents the normalized exponential function.

[0165] Attention loss This ensures that the attention feature maps in the teacher and student networks are as similar as possible. The activation layers of hidden neurons in a neural network can represent hypotheses about the importance of specific regions in the input image. An attention loss is set to force the student network to mimic the teacher network's attention map, thereby transferring the semantic information learned by the teacher network to the student network. This invention selects the feature maps after each downsampling and upsampling operation in UNet, and then aggregates the features from different channels to obtain a spatial attention feature map. This operation can be represented as a mapping function F:R. C×H×W →R H×W It converts the input 3D feature map into a 2D attention map. The spatial attention calculation formula used in this invention is as follows:

[0166]

[0167] Among them U i ∈R H×Wis the i-th channel of the feature map, C is the total number of channels, and |·| represents the absolute value calculation of the matrix. In this way, the feature maps of different channels can be compressed into an attention map. The feature maps of the teacher network and the student network are respectively denoted as and The attention loss function is represented as follows:

[0168]

[0169] In the formula, τ represents the index set of the feature maps that want to pass attention, and j represents the j-th passed feature map.

[0170] The total loss function of the teacher network R s is minimized, and the data in the training set is iteratively optimized. After reaching the training termination condition, the student network at this time is saved, and then step 6 is performed.

[0171] Step 6: see Figure 5 for the process of the test phase of the present application. The original modality image to be converted is input into the generator to obtain an intermediate conversion result, and the intermediate conversion result is input into the student network to output a converted image, which is the target modality image sought. The specific implementation of step 6 includes the following sub-steps:

[0172] Step 6.1: input the original modality image to be converted into the generator to obtain an intermediate conversion result G(I);

[0173] Step 6.2: input the intermediate conversion result G(I) into the student network R s , and output a converted image, which is the target modality image sought.

[0174] So far, the adversarial attack module in the overall network architecture of the present application, the teacher network and the student network in the knowledge distillation module, as well as the complete implementation example and steps have been introduced. The training method adopted by the present application is offline knowledge distillation, so the complete training process is divided into two stages. The first stage is the pre-training process of the teacher network (at this time there is no student network), and the input of the teacher contains the intermediate result G(I) output by the generator and the target image Y. The second stage is the formal training process of the student network, and the input of the student is only G(I), while the teacher transmits knowledge based on the attention map to the student to improve the conversion accuracy of the student network. After completing the training of the two stages, the test phase first uses the generator to obtain the converted intermediate result G(I), and then inputs it into the student network as the input of the student network, and obtains the final output result through the improvement of the student network. It is worth noting that the adversarial attack module plays a role in improving the robustness of the model in the training of the teacher network and the student network.

[0175] ​The knowledge distillation of the present application is different from the design of most existing distillation architectures, which usually design the teacher and student networks as networks with different complexities. However, the teacher network and the student network in the present application have similar structures but different inputs. The purpose of this is to make full use of the prior knowledge from the target image during the training of the teacher model, so that the student can still learn the knowledge of the teacher through knowledge distillation without providing the target image during the training of the student. Therefore, in the test or application stage, the student model can further improve the image quality of the intermediate result generated by the generator to obtain better final conversion results.

[0176] In specific implementation, the method proposed in the technical solution of the present application can be automatically run by a person skilled in the art using computer software technology, and the system device of the method, such as a computer readable storage medium storing the corresponding computer program of the technical solution of the present application and a computer device including running the corresponding computer program, should also be within the protection scope of the present application.

[0177] The present application also provides a multi-modal medical image conversion system based on knowledge distillation and adversarial attack, comprising the following modules:

[0178] A data set construction module for constructing a medical image data set including an original modality and a target modality;

[0179] A conversion network model construction module for constructing a conversion network model based on a deep neural network, the conversion network model comprising an adversarial attack module, a generator network, and a knowledge distillation module;

[0180] An adversarial sample generation module for inputting the original modality image into the adversarial attack module to obtain an adversarial sample, and inputting the adversarial sample and the original modality image together as an input set of the generator network, and the generator network outputs an intermediate conversion result;

[0181] A pre-training module, in the pre-training stage, the knowledge distillation module is a teacher network, the input of which is composed of two parts of the intermediate conversion result and the target modality image, and outputs a teacher deformation field, and the teacher deformation field and the intermediate conversion result are input into a spatial deformation network to output the target modality image converted through deformation calculation, and after the pre-training stage, a pre-trained teacher network is obtained;

[0182] An official training module, in the official training stage, the knowledge distillation module is a student network, the input of which only contains the intermediate conversion result, and at the same time the teacher network transmits knowledge to the student network through knowledge distillation, and the student network outputs a student deformation field, and the student deformation field and the intermediate conversion result are input into a spatial deformation network to output the target modality image converted through deformation calculation, and after the official training stage, a trained student network is obtained;

[0183] The prediction module is configured to input the original modality image to be converted into the generator network to obtain an intermediate conversion result, input the intermediate conversion result into the student network, and output a converted image, which is the target modality image.

[0184] The specific implementation of each module and the corresponding steps will not be described.

[0185] It should be emphasized that the above-described embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them. Although the present application has described a feasible implementation, it is still possible to modify or equivalently replace certain technical features of the specific implementation of the present application without deviating from the spirit of the technical solutions of the present application, within the understanding of those skilled in the art. These modified or equivalently replaced implementations should be included in the scope of the technical solutions claimed by the present application.

Claims

1. A multimodal medical image conversion method based on knowledge distillation and adversarial attacks, characterized in that, Includes the following steps: Step 1: Construct a medical image dataset including the original modality and the target modality; Step 2: Construct a deep neural network-based transformation network model, which includes an adversarial attack module, a generator network, and a knowledge distillation module. Step 3: Input the original modality image into the adversarial attack module to obtain adversarial samples. Use the adversarial samples and the original modality image together as the input set of the generator network. The generator network outputs the intermediate transformation result. Step 4: In the pre-training phase, the knowledge distillation module is the teacher network. Its input consists of the intermediate transformation result from step 3 and the target modality image. It outputs the teacher deformation field. This teacher deformation field and the intermediate transformation result are input into the spatial deformation network. The target modality image is output through deformation calculation. After the pre-training phase, the pre-trained teacher network is obtained. Step 4 includes the following sub-steps: Step 4.1, in the pre-training phase, the knowledge distillation module is the teacher network, defined as... , It consists of a Unet network and a spatial deformation network STN; Step 4.2, convert the intermediate results and target modal image As The Unet network takes the input from the teacher as input and calculates the teacher's deformation field. , This represents the original modal image of the input. Generator networks; Step 4.3, the teacher deformation field and intermediate conversion results As The target modality image is obtained by taking the STN input as input and performing deformation calculations. Step 4.4: Iteratively train the transformation network model on the training set. After training, the trained teacher network is obtained. Step 5: In the formal training phase, the knowledge distillation module is the student network. Its input only includes the intermediate transformation results from step 3. At the same time, the teacher network transmits knowledge to the student network through knowledge distillation. The student network outputs the student deformation field. This student deformation field and the intermediate transformation results are input into the spatial deformation network. The target modal image obtained by deformation calculation is output. After the formal training phase is completed, the trained student network is obtained. Step 5 includes the following sub-steps: Step 5.1, in the formal training phase, the knowledge distillation module is the student network, defined as follows: , It also consists of a Unet network and a spatial deformation network STN; Step 5.2: The pre-trained teacher network continuously transmits knowledge to the student network through knowledge distillation, guiding the output of the student network to move closer to the output of the teacher network; Step 5.3, convert the intermediate results As The Unet network calculates the student's deformation field from the input of the Unet network. , This represents the original modal image of the input. Generator networks; Step 5.4, the student deformation field and intermediate conversion results As The target modality image is obtained by taking the STN input as input and performing deformation calculations. Step 5.5: Iteratively train the transformation network model on the training set. After training, the trained student network is obtained. Step 6: Input the original modal image to be converted into the generator network to obtain the intermediate conversion result, input the intermediate conversion result into the student network, and output the converted image, which is the target modal image.

2. The robust medical image conversion method based on knowledge distillation and adversarial attacks as described in claim 1, characterized in that: Step 3 includes the following sub-steps: Step 3.1: Construct a black-box attacker in the adversarial attack module. The attacker's input is the original modality image. The output is an adversarial example. ; in Representing Gaussian noise, generating adversarial examples The purpose is to interfere with the generator network G; Step 3.2, define the generator network as follows: The original modal image and adversarial examples Together as The input, and the generator The intermediate conversion results of the output are collectively referred to as .

3. The robust medical image conversion method based on knowledge distillation and adversarial attacks as described in claim 1, characterized in that: The overall loss function used in step 4.4 when training the transformation network model is as follows: Use symbols To represent the deformation calculations performed by the STN network, Connected to the generator network Next, we improved the quality of the generated images, where there is a teacher refinement loss. Calculation: In the above formula Indicates based on and Calculate the mathematical expectation; in order to ensure Robustness, introducing a teacher's adversarial loss Attackers are encouraged to generate responses to the input based on real-time feedback from current training. A more meaningful perturbation, the teacher's resistance loss, is represented as: In the above formula Indicates based on and Calculation of mathematical expectation, Let be the relaxation factor, and be a constant. With teachers detailing the losses The losses from confrontation with teachers The pre-training of the teacher network involves the following optimization objective function, which also considers other loss terms and their corresponding weight coefficients. , : in This ensures that adversarial examples can share the target image with clean input during training. It is the L1 norm loss calculated from the clean input and the distorted input: During the pre-training process of the teacher network, the discriminator For generator networks The generated results are judged by the generator. and discriminator The confrontation process will generate confrontation loss terms. Calculation: in, and They respectively represent based on And calculation of mathematical expectation based on I; To avoid The smoothing loss term set for excessive distortion and deformation limits... The model produces non-smooth deformations, meaning it encourages similar deformations between adjacent pixels to avoid abrupt changes. Formally, a regularization term is set to smooth the deformations and find... The minimum of the first-order gradient, therefore the smoothing loss term is defined as: To minimize the teacher network Total loss function With the goal of iterative optimization using data from the training set, once the training termination condition is met, the teacher model at this point is saved as the pre-trained teacher network, and then step 5 is executed.

4. The robust medical image conversion method based on knowledge distillation and adversarial attacks as described in claim 1, characterized in that: The overall loss function used in step 5.5 when training the transformation network model is as follows; In the knowledge distillation process of the student network, the loss function consists of four parts, namely the student refinement loss. Teacher-student soft loss attention loss Fighting losses with students The overall loss function and weight coefficients for student module training are as follows: in, These are the weighting coefficients. and Allows student networks to simulate the output of the teacher network. This ensures that the attention feature maps in the teacher and student networks are as similar as possible, because students need to focus on the same key information about image transformations as the teacher, and students experience adversarial loss. Design philosophy and teacher conflict loss same; Students detail the losses Student networks were evaluated Predicted output and true label The difference between them: In the above formula Indicates based on and Mathematical expectation calculation, symbol This indicates the deformation calculation performed by the STN network; Students fight against losses Fighting against teachers for losses The design philosophy is the same, and attackers are still introduced into the training of the student model to improve the robustness of the model. The student adversarial loss is as follows: Let be the relaxation factor, and be a constant. For adversarial examples; Teacher-student soft loss It is based on soft-objective calculations because teachers' soft objectives contain more information than hard objectives, including the temperature factor. Used to control for the importance of soft objectives, this loss assesses the gap between teacher and student outputs: in This represents the soft target of the outputs of the teacher network and the student network. Divergence loss, where softmax represents the normalized exponential function; Attention loss The attention feature maps in the teacher and student networks are made as similar as possible. The attention loss function is expressed as follows: In the formula Let F represent the set of indices of the feature maps to be passed attention, where j represents the j-th feature map to be passed; F represents the mapping function that transforms the input 3D feature map into a 2D attention map. and These represent feature graphs of the teacher network and the student network, respectively. To minimize the teacher network Total loss function With the goal of iterative optimization using data from the training set, once the training termination condition is met, the student network at this point is saved, and then step 6 is executed.

5. The robust medical image conversion method based on knowledge distillation and adversarial attacks as described in claim 4, characterized in that: The specific formula for calculating the mapping function F is as follows: in Let be the i-th channel of the feature map, and C be the total number of channels. The formula above represents the calculation of the absolute value of the matrix, indicating that an attention map is obtained by aggregating features from different channels.

6. A multimodal medical image conversion system based on knowledge distillation and adversarial attacks, characterized in that, Includes the following modules: The dataset construction module is used to build medical image datasets that include the original modality and the target modality; The transformation network model building module is used to build a deep neural network-based transformation network model, which includes an adversarial attack module, a generator network, and a knowledge distillation module. The adversarial example generation module is used to input the original modality image into the adversarial attack module to obtain adversarial examples. The adversarial examples and the original modality image are used together as the input set of the generator network, and the generator network outputs intermediate transformation results. In the pre-training module, the knowledge distillation module is the teacher network. Its input consists of two parts: intermediate transformation results and target modal image. It outputs the teacher deformation field. This teacher deformation field and intermediate transformation results are input together into the spatial deformation network. Through deformation calculation, the transformed target modal image is output. After the pre-training phase, the pre-trained teacher network is obtained. The specific implementation of the pre-training module includes the following sub-steps: Step 4.1, in the pre-training phase, the knowledge distillation module is the teacher network, defined as... , It consists of a Unet network and a spatial deformation network STN; Step 4.2, convert the intermediate results and target modal image As The Unet network takes the input from the teacher as input and calculates the teacher's deformation field. , This represents the original modal image of the input. Generator networks; Step 4.3, the teacher deformation field and intermediate conversion results As The target modality image is obtained by taking the STN input as input and performing deformation calculations. Step 4.4: Iteratively train the transformation network model on the training set. After training, the trained teacher network is obtained. In the formal training module, the knowledge distillation module is the student network, whose input only includes intermediate transformation results. At the same time, the teacher network transmits knowledge to the student network through knowledge distillation. The student network outputs the student deformation field, which is then input into the spatial deformation network along with the intermediate transformation results. Through deformation calculation, the target modal image obtained by transformation is output. After the formal training phase is completed, the trained student network is obtained. The specific implementation of the formal training module includes the following sub-steps: Step 5.1, in the formal training phase, the knowledge distillation module is the student network, defined as follows: , It also consists of a Unet network and a spatial deformation network STN; Step 5.2: The pre-trained teacher network continuously transmits knowledge to the student network through knowledge distillation, guiding the output of the student network to move closer to the output of the teacher network; Step 5.3, convert the intermediate results As The Unet network calculates the student's deformation field from the input of the Unet network. , This represents the original modal image of the input. Generator networks; Step 5.4, the student deformation field and intermediate conversion results As The target modality image is obtained by taking the STN input as input and performing deformation calculations. Step 5.5: Iteratively train the transformation network model on the training set. After training, the trained student network is obtained. The prediction module is used to input the original modality image to be converted into the generator network to obtain intermediate conversion results, input the intermediate conversion results into the student network, and output the converted image, which is the target modality image.

Images