Brain MRI mode conversion method based on GAN

Abstract

The invention provides a brain MRI (Magnetic Resonance Imaging) mode conversion method based on a GAN (Generalized Area Network), and belongs to a frontier technology in the technical field of medical image processing. According to the mode conversion method, a new generator and discriminator structure is provided, the optimization of a loss function is carried out by depending on the structure, a new loss value calculated by the optimized loss function is used for further training the generator and the discriminator, and the generation effect of the generator is continuously optimized by repeating the steps. Compared with a generator and discriminator structure of a common Cycle-GAN network, the method comprises the following steps: firstly, adding an attention mechanism and a self-adaptive instance normalization structure into the generator to improve image generation details; secondly, optimizing a loss function, in order to make full use of intermediate generated data and improve generalization and robustness of the whole structure, proposing to use a double-circulation consistency module, a same-domain control module and an edge consistency module to calculate a loss value together, and then using the loss function to further train a generator and a discriminator; by using the method provided by the invention, the quality of the image generated by the generator can be obviously improved.

Description

Technical Field

[0001] The present invention relates to a GAN-based brain MRI modality conversion method, which belongs to the cutting-edge technology in the field of medical image processing technology. Background Art

[0002] Glioma is the most common primary brain tumor in the human body and has a high mortality rate. In clinical practice, magnetic resonance imaging (MRI) uses magnetic fields and radio waves to generate anatomical images in a non-invasive manner. There are four common MRI modalities: T1 images, T1 enhanced images (Tlce), T2 images, and fluid attenuated inversion recovery sequence images (FLAIR). Images of different modalities provide additional information for clinical diagnosis.

[0003] However, due to the high cost, limited scanning time and random image damage, it is challenging to obtain complete paired multimodal images. In recent years, cross-modal medical image synthesis of MRI images with the help of neural networks has gradually attracted the attention of researchers.

[0004] Among them, the cycle-consistent adversarial generative network (Cycle-GAN) based on adversarial training using a generator and a discriminator is a deep learning model for unsupervised image conversion. By using cycle consistency loss, the reconstructed image is closer to the original target. In addition, Cycle-GAN also introduces adversarial loss. Through adversarial training, the generator continuously improves the quality of the generated image, making it closer and closer to the real target domain image.

[0005] However, due to the lack of full utilization of intermediate generated information and the use of a single normalization method, some detailed information will be lost in some complex image conversion tasks, and there will be unstable generation effects and lack of generalization. In order to solve the above problems, this method makes the following optimizations based on the overall structure of the generator and discriminator of Cycle-GAN and its loss function. First, in order to enable the generator of Cycle-GAN to more flexibly control the amount of change in shape and texture, this method proposes to add an adaptive normalization module to the generator structure, and by using the attention feature map, the ratio between the two normalization functions is adaptively controlled, so that the generator network can more flexibly control the amount of change in shape and texture. Secondly, in order to make full use of the intermediate generated data and perform both same-domain control and edge loss control at the same time, this method proposes to use a dual cycle consistency module, a same-domain control module, and an edge consistency module to optimize the loss function, and use the calculated loss value to further train the generator and discriminator, thereby further improving the quality of the image generated by the generator. Summary of the invention

[0006] Purpose of the invention: In view of the shortcomings of the prior art, the present invention proposes a GAN-based brain MRI modality conversion method suitable for brain tumor images. This method can improve the stability and robustness of the overall structure of the generator and discriminator, while improving the quality of the images generated by the generator.

[0007] Technical solution: A GAN-based brain MRI modality conversion method. The innovation of this method lies in the following two steps:

[0008] (1) Optimization of the overall structure of the generator and discriminator. Based on the Cycle-GAN generator and discriminator structure, by adding an adaptive normalization module and an attention module to the generator, the generator can generate high-quality images with more details and textures;

[0009] (2) Loss function optimization. In order to improve the robustness of the generator and make full use of the intermediate generated images, based on the loss function of Cycle-GAN under the single-cycle consistency, it is proposed to add a double-cycle consistency module to calculate the loss value; in order to improve the generator's sensitivity to edges so that the generated images retain more edge details, it is further proposed to add an edge consistency module to calculate the loss value; in order to ensure the domain correctness of the generator's image synthesis, it is proposed to add a same-domain control module to calculate the loss value.

[0010] Preferably, in step (1), an adaptive normalization module and an attention module are added to the generator, which specifically comprises the following steps:

[0011] (11) Based on the generator and discriminator structure of the Cycle-GAN network, an attention mechanism and an adaptive normalization module are added to the generator. In the generator network structure, the feature map output from the encoder will be put into the AuxiliaryClassifier auxiliary classifier module to obtain an attentive feature map;

[0012] (12) Then, the attention feature map is divided into two paths, one of which passes through a 1×1 convolution and activation function layer to obtain a1...a n The feature map is passed through the fully connected layer to obtain the α and β of the adaptive normalization module in the decoder. The other way is used as the input of the subsequent decoder, and together with the previously generated α and β, it enters the adaptive residual module, and finally goes through three layers of upsampling to obtain the result map.

[0013] The residual module in the adaptive normalization module here uses the AdapLIN function, which can help the model flexibly control the amount of change in shape and texture by adaptively selecting the appropriate ratio between instance normalization (IN) and layer normalization (LN). The specific equation is as follows:

[0014] AdapLIN(a,γ,β)=γ((0.5+θ)·α I +(0.5-θ)·a L )+β,

[0015]

[0016] θ←(θ-τΔθ)

[0017] Among them, γ and β are calculated by the fully connected layer of the attention feature map, μ I and μ L are the average values ​​of instance channel and layer channel, σ I and σ L are the standard deviations of instance channels and layer channels, τ is the learning rate, τΔρ is the parameter update vector and the gradient is determined by the optimizer, and the value of θ is artificially constrained to the interval [-0.5, 0.5]. The generator can adjust the size of θ, such as adjusting θ closer to 0.5 to match the relatively important modules of IN and adjusting θ closer to -0.5 to match the relatively important modules of LN.

[0018] Preferably, in step (2), the loss function optimization relies on the overall structure of the generator and discriminator optimized in step (1) to perform image generation and discrimination. The loss function optimization is to add additional loss value calculation on the basis of the loss value calculation of the single cycle consistency of Cycle-GAN, including the loss value calculated by the double cycle consistency module, the loss value calculated by the same domain control module and the loss value calculated by the edge consistency module, and specifically includes the following steps:

[0019] (21) In order to make full use of the intermediate generated data, it is proposed to add a double-loop consistency module to calculate the loss value: In the single-loop consistency training of the original Cycle-GAN, the source image A is passed through the generator G A to B Generate B' and pass B' as input through G B to A Generate the image A' of the first layer of cycle consistency training. The calculation of the double cycle consistency loss function of this method requires taking A' generated by the first round of cycle consistency training as the input of the second round of cycle consistency training. A to B Generate B", then calculate L dcc , the specific equation is as follows:

[0020] L dcc =E x～X [||G X→Y (G Y→X (G X→Y (x)))-G X→Y (x)||1]+E y～Y [||G Y→X (C X→Y (G Y→X (y)))-G Y→X (y)||1]

[0021] Among them, G X→Y and G Y→X , representing the generator from A to B and the generator from B to A, respectively, setting two fields Data A and Data B , let X represent Data A , Y stands for Data B , and x i ∈X,y i ∈Y. Use x~X and y~Y to represent the data distribution of A and B. L dcc The number of times the target generator is used is increased by comparing the target images generated twice to enhance the conversion effect of the target generator, that is, to make G X→Y (G Y→X (G X→Y (x)))≈G X→Y (x), G Y→X (G X→Y (G Y→X (y)))≈G Y→X (y).

[0022] (22) In order to ensure the domain correctness of the generated image, it is proposed to add a same-domain control module to calculate the loss value: the source image A is passed through G B to A Generate A', and pass the intermediate result A' generated by (21) through G B to A Generate A'*. In order to ensure the domain correctness of image synthesis, use the source images A, A' and A'* to calculate L dc , the specific equation is as follows:

[0023] L dc =E x～X [||G Y→X (x)-x||1]+E x～X [||G Y→X (G Y→X (G X→Y (x)))-x||1]+E y～Y [||G X→Y (y)-y||1]+E y～Y[||G X→Y (G X→Y (G Y→X (y)))-y||1]

[0024] Where L dc To ensure G Y→X (x)≈x, and G Y→X (G Y→X (G X→Y (x)))≈x(33)The fused features are input into the softmax classifier to obtain the final label.

[0025] (23) In order to make the generated image retain more edge information, it is proposed to add an edge consistency module to calculate the loss value: In order to compare whether the edge generation results of A and A′ are consistent, an edge consistency loss function L is proposed ep Through experimental comparison of multiple edge algorithms, we finally selected the Sobel algorithm, which is more suitable for brain MRI images, to generate edge maps. Edge consistency loss function L ep The specific equation is as follows:

[0026] L ep =E x～X [||S(G Y→X (G X→Y (x)))-S(x)||1]+E y～Y [||S(G X→Y (G Y→X (y)))-S(y)||1]

[0027] Where S is the Sobel edge detector, S(x) and S(y) represent the real edge maps of the input image, and S(G Y→X (G X→Y (x))) and S(G X→Y (G Y→X (y))) is the edge map of the generated image

[0028] The beneficial effects of the present invention are as follows: First, the present invention introduces an adaptive normalization module and an attention mechanism into the generator, and uses the attention feature map to adaptively control the ratio between the two normalization functions, so that the generator network can more flexibly control the change in shape and texture. Secondly, based on the single-cycle consistency loss function structure of Cycle-GAN, the present invention adds a double-cycle consistency loss value, a same-domain control loss value, and an edge consistency loss value, which makes full use of the intermediate generation information and improves the same-domain correctness while improving the generator's sensitivity to edge information. Experiments show that both the quantitative evaluation results and the qualitative evaluation results of the present invention have achieved excellent performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the flow chart of the brain MRI modality conversion method of the present invention;

[0030] Figure 2 This is a schematic diagram of the network structure of the improved generator in this method;

[0031] Figure 3 This is a schematic diagram of the network structure of the improved discriminator in this method;

[0032] Figure 4 Schematic diagram of the adaptive normalization module in the present invention;

[0033] Figure 5 Comparison between the single-cycle schematic diagram of the original Cycle-GAN and the double-cycle schematic diagram of the present invention;

[0034] Figure 6 This is a schematic diagram of a domain control module in the present invention;

[0035] Figure 7 It is a schematic diagram of the edge retention module in the present invention;

[0036] Figure 8 This is a diagram of the experimental results of the present invention. DETAILED DESCRIPTION

[0037] The present invention provides a brain MRI modality conversion method based on GAN, which is suitable for modality conversion of brain MRI image data. The main flow chart of the method is shown in Figure (1). The main steps of the method include (1) optimizing the overall network structure of the generator and discriminator of Cycle-GAN; (2) optimizing the subsequent loss function based on the optimized generator and discriminator structure. The details are as follows:

[0038] Step (1) The optimization of the overall network structure of the generator and discriminator of Cycle-GAN mainly focuses on the generator. The specific structure of the optimized generator is as follows: Figure 2 As shown, the method specifically comprises the following steps:

[0039] The Multimodal Brain Tumor Segmentation Challenge 2020 (BraTS2020) dataset used in step (11) is a dataset related to an international competition for brain tumor segmentation. It consists of multi-parameter MRI scans (mpMRI) of 369 patients with diffuse glioma, including four modes: T1, T1c, T2 and FLAIR. BraTS2020 promotes the development and improvement of brain tumor segmentation algorithms and improves the accuracy of brain tumor diagnosis and treatment by providing a standard dataset and competition platform.

[0040] Step (12) uses the preprocessed 256*256 dataset as the input image and generates the encoded 64*64 feature map E1...E n ,The Encoder module consists of three downsampling layers and six RestNet residual modules.

[0041] Step (13) Then, put the encoded feature map into the Auxiliary Classifier module to obtain the feature map w1E1 with attention . ...w1E n .

[0042] Step (14) divides the attention feature map into two paths, one of which passes through a 1×1 convolution and activation function layer to obtain feature maps a1...a n , the feature map passes through the fully connected layer to obtain the α and B parameters of the adaptive normalization (AdaptiveRestNet) module in the decoder, and the other way is used as the input of the subsequent Decoder decoder, together with the previously generated α and β, to enter the residual module of the improved adaptive normalization module.

[0043] The specific structure diagram of the residual module of the adaptive normalization module is as follows Figure 4 As shown, the generator G X→Y By encoder E G_X→Y and decoder D G_Y→X And auxiliary classifier C G_X Composition, let z∈{X,Y}, represents z is a sample from the source domain and the target domain, where C G_X represents the probability that z comes from X, E k represents the kth activation map of the encoder, and represents the value at (i,j), C G_X The weight w of the k-th feature map of the source domain is learned by using global average pooling and global maximum pooling k ,Right now Using weight w k , we can further calculate the attention feature map a(z)=w*E(z)={w k *E k (z)|1≤k≤n}, where n is the number of encoder feature maps. Different from AdaIN and LN, AdapLIN integrates a certain proportion of LN on the basis of AdaIN. The specific equation is as follows:

[0044]

[0045] Among them, γ and β are calculated by the fully connected layer of the attention feature map, μ I and μL are the average values ​​of instance channel and layer channel, σ I and σ L are the standard deviations of instance channels and layer channels, τ is the learning rate, TΔp is the parameter update vector and the gradient is determined by the optimizer, while the value of θ is artificially constrained to the interval [-0.5, 0.5]. The generator can adjust the size of θ, such as adjusting θ closer to 0.5 to match the relatively important modules of IN and adjusting θ closer to -0.5 to match the relatively important modules of LN.

[0046] Step (15) finally passes through two upsampling layers with a kernel of 3*3 and the last upsampling layer with a kernel of 7*7, thereby obtaining a 256*256 result image.

[0047] Step (2) relies on the optimized generator and discriminator structures to optimize the subsequent loss function. The specific execution steps are as follows:

[0048] Step (21) Intermediate data generation and discrimination of the dual-loop consistency module, and calculation of the dual-loop consistency loss value: Perform inter-modal conversion without paired input and output data, L cyc It is a consistency algorithm sought by Cycle-GAN. Figure 5 As shown, taking A generating B as an example, L cyc The quality of the generated A image is indeed guaranteed, but only using L cyc There is no constraint controlling the generator G from A to B throughout the loop A to B The generation effect is not good, and the intermediate result B′ is not fully utilized. Therefore, this method introduces the double-cycle consistency loss (Double-Cycle Consistency Loss, L dcc ) as the consistency constraint of the target generator. First, the source graph A is passed through the generator G A to B Generate B' and pass B' as input through G B to A Generate the image A' of the first layer of cycle consistency training, and use A' generated by the first round of cycle consistency training as the input of the second round of cycle consistency training. A to B Generate B". Then use the intermediate generated data and the source data to perform L cyc and L dcc The loss value is calculated for subsequent optimization of the generator and discriminator. cyc is the single-cycle consistency loss function of Cycle-GAN, and the specific equation is shown in Equation 2:

[0049] L cyc =E x～X [||G Y→X (G X→Y(x))-x||1]+E y～Y [||G X→Y (G Y→X (y))-y||1] (Formula 2)

[0050] Where X represents Data A , Y stands for Data B , and x i ∈X,y i ∈Y. Use x~X and y~Y to represent the data distribution of A and B.

[0051] L dcc is a double-loop consistency loss function, which increases the number of times the target generator is used once more, generates the target image again on the basis of a single loop, and then enhances the conversion effect of the target generator by comparing the target images generated twice, that is, making G X→Y (G Y→X (G X→Y (x)))≈G X→Y (x), G Y→X (G X→Y (G Y→X (y)))≈G Y→X (y), the specific equation is shown in equation 3:

[0052] L dcc =E x～X [||G X→Y (G Y→X (G X→Y (x)))-G X→Y (x)||1]+E y～Y [||G Y→X (G X→Y (G Y→X (y)))-G Y→X (y)||1] (Formula 3)

[0053] In addition, in order to better constrain and train the discriminator, it is necessary to introduce the adversarial loss L of Cycle-GAN adv_G and L adv_D To conduct adversarial training of the entire model. adv_G The purpose is that the two generators should make it impossible for the discriminator to accurately recognize the synthesized image. For example, the generator G X→Y The expectation is that D Y (G X→Y (x))≈1, D Y (G X→Y (G Y→X (G X→Y (x)))≈1. On the contrary, L adv_DThe purpose is that the two discriminators can accurately identify which is the real image and which is the synthetic image.

[0054] Step (22) Generate and judge the intermediate data of the same domain control module, and calculate the loss function: Figure 6 As shown, A' generated in the above steps is obtained by G B to A Generate A'*, use source graphs A, A, and A'* together to calculate L dc .L dc This is to prevent images that are already in the target domain from being mistakenly converted to another domain, thereby ensuring the domain correctness of image synthesis and improving the stability of the generator. The loss function of the same-domain control module can be divided into two parts: the same-domain control loss (L dc ) and the same domain control adversarial loss (L adv_dc ). Where L dc is used to optimize the generator, and L adv_dc It is used to optimize the discriminator. The specific equation is as follows:

[0055]

[0056] Step (23) Edge consistency module intermediate data generation and loss function calculation: For brain MRI images, edge information is crucial for lesion analysis and downstream segmentation and classification tasks. Therefore, this study proposes an edge preserving module, which uses an edge preserving loss function (L ep ) optimizes the generator by comparing the real edge map of the original modality image with the edge map of the generated image to improve the edge preservation ability of the network model. Figure 7 As shown, use A and A' to perform L ep Loss calculation. ep What is compared is whether the edge generation results of A and A′ are consistent. The specific equation is as follows:

[0057] L ep =E x～X [||S(G Y→X (G X→Y (x)))-S(x)||1]+E y～Y [||S(G X→Y (G Y→X (y)))-S(y)||1](Formula 8)

[0058] Where S is the Sobel edge detector, S(x) and S(y) represent the real edge maps of the input image, and S(G Y→X (G X→Y (x))) and S(G X→Y (G Y→X(y))) is the edge map of the generated image, L ep The purpose is to expect the edge distribution of the generated image to be as close as possible to the edge distribution of the original image

[0059] The experimental verification is as follows:

[0060] The dataset used in this example is the Multimodal Brain Tumor Segmentation Challenge 2020 (BraTS2020) dataset [4] to verify the effectiveness of the proposed EPDC-GAN. The BraTS2020 dataset consists of multi-parameter MRI scans (mpMRI) of 369 patients with diffuse glioma, using four modes: T1, Tlc, T2, and FLAIR. Using the volume of 2D axial plane slices as input, each 2D axial plane slice (240×240) is cropped from the center of the image to 200×200 and further resized to 256×256. Then, the original intensity values ​​are linearly scaled to [0, 1]. 290 cases were randomly selected as the training set, and the remaining 79 cases were used as the test set.

[0061] This example uses a 64-bit Ubuntu 18.04 operating system, an RTXA4000 graphics card with 16G video memory, Python 3.8, and PyTorch 1.7.0 software platform. The data set is divided into a training set, a validation set, and a test set in a ratio of 8:1:1. Adam is used as the optimizer method, and the learning rate is 0.001.

[0062] This experiment uses three widely used evaluation indicators, namely Peak Signal-to-Noise Ratio (PSNR), Normalized Mean Square Error (NMSE) and Structural Similarity Index (SSIM), to quantitatively evaluate the comprehensive performance of different models.

[0063]

[0064] The definition of PSNR is shown in Equation 9, where y(x) is the true value image of the missing modality, G(x) is the generated missing modality image, and N is the product of the image height and width. The higher the PSNR value, the better the performance of the model. The definition of NMSE is shown in Equation 10. The lower the NMSE value, the better the performance of the model. The definition of SSIM is shown in Equation 11, μ y(x) , μ G(x) , σ y(x) and σ G(x) They represent the mean and error of y(x) and G(x) respectively. The constants c1 and c2 with values ​​greater than 0 are to avoid the denominator being 0. The higher the SSIM value, the better the performance of the model. The following are the qualitative experimental results verified on the Brats2020 dataset. Figure 8 The comparison results with the other three models are shown in the following table.

[0065] Table 1 Results of using T1 to generate T2 images on the BRATS 2020 dataset

[0066]

[0067] Table 2 Results of using T2 to generate T1 images on the BRATS 2020 dataset

[0068]

[0069] Among them, EPDC-GAN is the model proposed by this method. It can be seen from the chart data that the method proposed by this patent is superior to other models. Compared with non-attention Cycle-GAN, SynDiff also performs better from the analysis of experimental results. However, SynDiff still has the fuzziness caused by the smooth transition often seen in diffusion models, and MT-NET has made the network more sensitive to edges due to edge perception pre-training. Compared with MT-NET, although EPDC-GAN has only slightly improved in various indicators of quantitative analysis, it also has a slight advantage in overall stability. Therefore, after analysis, it is concluded that EPDC-GAN fully uses the intermediate generated A' as the input of another cycle. Compared with A, the imaging effect of A' will be reduced, which is equivalent to adding noise to the input data of the generator, enhancing the data, and improving the robustness of the generator. Therefore, compared with several other methods, the generated picture quality is higher and the generated effect is better, which proves the effectiveness of the method proposed in this patent.

[0070] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A GAN-based brain MRI modality conversion method, characterized in that Optimization of the overall structure of the generator and discriminator and optimization of the loss function: (1) Optimization of the overall structure of the generator and discriminator. Based on the generator and discriminator structure of Cycle-GAN, in order to enable the generator to generate high-quality images with more details and textures, this method proposes to add an adaptive normalization module and an attention module to the generator structure; (2) Loss function optimization. The optimization of the loss function depends on the optimized generator and discriminator structures in (1) for image generation. In order to improve the robustness of the generator and make full use of the intermediate generated images, based on the single-cycle consistency loss value calculation of Cycle-GAN, this method proposes to add the loss value calculated by the double-cycle consistency module; in addition, in order to improve the generator's sensitivity to edges so that the generated images retain more edge details, this method further proposes to add the loss value calculated by the edge consistency module; finally, in order to ensure the domain correctness of the generator image synthesis, this method further proposes to add the loss value calculated by the same domain control module.

2. A GAN-based brain MRI modality conversion method as claimed in claim 1, characterized in that In step (1), the optimization of the overall structure of the generator and the discriminator includes the following steps: (11) Adding attention mechanism to the generator structure: For the preprocessed dataset, the downsampling layer and residual module in the encoder are first used to generate feature maps E1...E n ; Then the encoded feature map is put into the AuxiliaryClassifier auxiliary classifier module to obtain the feature map with attention a1...a n ; Pass the feature map through the fully connected layer to obtain α and β required for the adaptive normalization module in the decoder. (12) Add an adaptive normalization module to the generator structure: Input the attention feature map into the decoder with the adaptive normalization module, pass through three upsampling layers, and output the result map. The adaptive normalization module here uses the AdapLIN function. Generator G X→Y By encoder E G_X→Y and decoder D G_Y→X And auxiliary classifier C G_X Composition, let z∈{X,Y}, represents z is a sample from the source domain and the target domain, where C G_X represents the probability that z comes from X, E k represents the kth activation map of the encoder, and represents the value at (i, j), C G_X The weight w of the k-th feature map of the source domain is learned by using global average pooling and global maximum pooling k ,Right now Using weight w k , we can further calculate the attention feature map a(z)=w*E(z)={w k *E k (z)|1≤k≤n}, where n is the number of encoder feature maps. The AdapLIN in the residual block is a fusion of a certain proportion of LN on the basis of AdaIN. The specific equation is as follows: Among them, γ and β are calculated by the fully connected layer of the attention feature map, μ I and μ L are the average values ​​of instance channel and layer channel, σ I and σ L are the standard deviations of instance channels and layer channels respectively, τ is the learning rate, TΔρ is the parameter update vector and the gradient is determined by the optimizer, while the value of θ is artificially constrained to the interval [-0.5, 0.5].

3. A method for brain MRI modality conversion based on GAN as claimed in claim 1, characterized in that In step (2), optimization is performed based on the single-cycle consistency loss function of Cycle-GAN, which specifically includes the following steps: (21) In order to make full use of the intermediate generated data and improve the stability and robustness of the generator structure, a loss function of the double cycle consistency module is added to calculate the overall loss value: the first round refers to the single cycle consistency training of Cycle-GAN, and the A' generated by the first round of cycle consistency training is used as the input of the second round of cycle consistency training. A to B Generate B", and then use B' and B" to perform additional cycle consistency loss calculation. The specific loss function is as follows: Among them G X→Y and G Y→X , representing the generator from A to B and the generator from B to A, respectively, setting two fields Data A and Data B , let X represent Data A , Y stands for Data B , and x i ∈X,y i ∈Y. Use x~X and y~Y to represent the data distribution of A and B; (22) In order to ensure the domain correctness of image synthesis, the loss function of the same domain control module is added to calculate the loss value: First, the source image A is passed through G B to A Generate A', and pass the intermediate result A' generated by (21) through G B to A Generate A'*, then use the source images A, A' and A'* to calculate L dc , the specific equation is as follows: (23) In order to reduce the difference between the edge information of the generated image and the source image, the loss function of the edge consistency module is added to calculate the loss value: the edge consistency module uses the Sobel algorithm to generate the edge map. The edge consistency loss function L ep The specific equation is as follows: Where S is the Sobel edge detector, S(x) and S(y) represent the real edge maps of the input image, and S(G Y→X (G X→Y (x))) and S(G X→Y (G Y→X (y))) is the edge map of the generated image.

Images