A PET and MRI image fusion method based on a three-discriminator generative adversarial network

Abstract

The application discloses a PET and MRI image fusion method based on a three-discriminator generative adversarial network, which comprises the following steps: firstly, performing a transformation operation on a PET image from an RGB color space to a YCbCr color space to obtain three component images of the PET image, namely Y, Cb and Cr; secondly, performing a difference operation on the Y component of the PET image and an MRI image, and then performing a connection operation on the obtained difference image and the Y component of the PET image and the MRI image to input the difference image into a generator network; thirdly, inputting an image generated by the generator and an actual image into three discriminators respectively, and continuously improving the generation ability of the generator and the discrimination ability of the three discriminators through a loop adversarial, so as to finally reach a balanced state of the two types of networks, thereby enabling the generator to comprehensively input image information and generate a fusion image containing both spatial structure information of an original MRI image and functional information of an original PET image.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a PET and MRI image fusion method based on a three-discriminator generative adversarial network. BACKGROUND

[0002] With the continuous development of modern sensor and imaging technology, various medical imaging devices have also emerged. Different medical imaging devices acquire human information through various imaging strategies, thereby producing medical images of multiple modalities. Existing multi-modal medical images can be generally divided into anatomical images and functional images. Magnetic resonance imaging (MRI) as an anatomical image can provide soft tissue information of the human body, which is manifested as an image with rich texture details and dense structure information. Positron emission tomography (PET) as a functional image can reflect tumor and human cell metabolism and other human information. In clinical applications, the use of a single modality of medical image often cannot obtain the required human information completely, which requires the use of multiple modalities of medical images in clinical applications. However, simultaneous observation of images of different modalities will greatly consume time. By fusing MRI and PET images through certain technical means, a fused image can simultaneously reflect the structure and functional information of the human body in MRI and PET images, so as to better provide objective human reference information in clinical applications, thereby helping clinicians to make accurate diagnoses.

[0003] Different fusion methods and strategies have been proposed by researchers in view of the characteristics of MRI and PET images. Among them, the fusion method based on deep learning is currently a popular multi-modal medical image fusion method. Using deep learning method for image fusion can to some extent avoid the design of complex fusion rules required in traditional image fusion, thereby reducing the computational complexity of image fusion. In the field of medical image fusion based on deep learning, the method based on generative adversarial network (GAN) is commonly used. The GAN model mainly includes two parts: generator and discriminator. The generator aims to generate an image with PET image tissue information and additional MRI image structure information; the discriminator aims to force the generated image to have more structural detail information, so that the fused image has both the texture details of the MRI image and the clear tissue information of the PET image. The fusion method based on GAN is usually trained unsupervisedly through the establishment of an adversarial game between the generator and the discriminator, thereby continuously improving the effect of the fused image.

[0004] In practice, the existing fusion strategy based on GAN has the following shortcomings:

[0005] 1. In the single discriminator GAN model, the discriminator only judges the fusion image and the single source image, and there is a limitation that the generated image tends to be biased towards the single image with network training.

[0006] 2. In the double discriminator GAN model, the double discriminator structure is used to distinguish different input images, thereby balancing the generated image and eliminating the bias of the source image with the training of the fusion image; however, the encoder-decoder network used in the generator is prone to insufficient utilization of shallow features in the source image. SUMMARY

[0007] The purpose of the present application is to overcome the defects of the prior art and provide a PET and MRI image fusion method based on a three-discriminator generative adversarial network to solve the problems raised in the background art.

[0008] To achieve the above purpose, the present application provides the following technical scheme: a PET and MRI image fusion method based on a three-discriminator generative adversarial network, the specific steps are as follows:

[0009] S1: YCbCr color space conversion;

[0010] The PET image is converted from the RGB color space to the YCbCr color space, and the PET image after color space conversion contains three image components, namely Y, Cb and Cr image components, wherein the Y image component contains brightness information that can reflect the overall structure and brightness change of the PET image, and the Cb and Cr image components represent the concentration offset components of blue and red in the PET image, respectively.

[0011] S2: Y component image fusion;

[0012] The Y component of the PET image and the MRI image are subjected to a difference operation, and the resulting difference image is subjected to a conjunction operation with the Y component of the PET image and the MRI image and then input into the proposed network model to obtain a fused Y component image.

[0013] S3: RGB image reconstruction;

[0014] The Y component image generated by the generator and the Cb and Cr components of the corresponding source PET image are subjected to inverse transformation from the YCbCr color space to the RGB color space, thereby obtaining the final fusion image.

[0015] As a preferred technical scheme of the present application, in the YCbCr color space conversion of S1, the PET image I PPerform the transformation from RGB color space to YCbCr color space and obtain I P The Y, Cb, Cr three-channel component image I P-Y , I P-Cb and I P-Cr The transformation process of PET images from RGB color space to YCbCr color space is as follows:

[0016]

[0017] As a preferred technical solution of the present invention, the fusion of the Y component image of S2 is used to convert the MRI image I M and PET image Y component I P-Y Perform the difference and take the absolute value to obtain the difference image I D , and then the MRI image I M , PET image Y component I P-Y , differential image I D After "connection", it is input into the generator network;

[0018] The generator extracts the feature information of the input image and performs feature selection, and then fuses it to generate I F-Y ,This step requires completing the training of the generator-discriminator network before fusion.

[0019] As a preferred technical solution of the present invention, the training process of the generator-discriminator network is:

[0020] First, I M , I P-Y and the calculated I D The image is input to the generator network, which uses three different branches to extract image features. The three branches are called gradient branch, contrast branch and difference branch based on the characteristics of the extracted image feature information.

[0021] Secondly, the I generated by the generator network F-Y Respectively with I M , I P-Y , I D Input to the corresponding discriminator D M 、D P-Y With D D Identify in

[0022] Finally, based on the output results of the generator and discriminator, the networks of the generator and discriminator are adversarially trained and alternately updated through the loss function until the network balance is reached. At this time, the trained generator network is the generator we need for PET and MRI image fusion.

[0023] As a preferred technical solution of the present application, the loss function comprises a weight of the loss function, a generator loss function and a discriminator loss function.

[0024] As a preferred technical solution of the present application, the weight of the loss function balances the information among the input three types of images by using a weight-based method, specifically, the Y component of the PET image, the MRI image and the difference image are respectively subjected to information entropy measurement, saliency measurement and gradient measurement, then the weight parameters are generated according to the measurement results, in addition, the enhancement and attenuation parameters are set according to the information performance characteristics of the PET and MRI images to balance the loss functions of different types.

[0025] The information entropy measurement is to measure how much information the source image contains by calculating the information entropy of the source image, which is used as a reference for the weight of the loss function, specifically:

[0026]

[0027] Where I En is the information entropy of the image I; l is the number of gray levels, which is set to 256; p l is the probability of the corresponding gray level.

[0028] The saliency measurement measures how much highlight information the source image contains, which is used as a reference for the weight, specifically:

[0029]

[0030] Where I Sal is the saliency measurement of the image I; Sign(·) is the sign function; τ is the threshold for screening the salient region; H, W are the height and width of the image I; I ij is the gray value of the pixel point at the coordinate (i, j) in the image I.

[0031] The gradient measurement is measured by using the Laplacian operator, and the higher the measurement value is, the more texture details the image contains, which is used as a measurement for the loss weight, specifically:

[0032]

[0033] Where I Grad is the gradient measurement of the image I. is the Laplacian operator.

[0034] The weight of the total loss function is defined as:

[0035] η I =λ1I En +λ2I Sal +λ3I Grad

[0036]

[0037] wherein η I is the overall metric of the corresponding image I, λ1, λ2, λ3 are proportional parameters for controlling different metrics; similarly, are the overall metrics of the Y component of the MRI image, the Y component of the PET image and the difference image respectively; ω1, ω2, ω3 are loss parameter weights of the Y component of the MRI image, the Y component of the PET image and the difference image respectively.

[0038] As a preferred technical solution of the present application, the generator loss function adopts a hybrid loss, that is, the network is constrained by combining the loss of the image surface information with the loss of the deep semantic information mined by the network; the loss function of the generator is defined as follows:

[0039] L G = L adv + αL ssim + βL grad + γL per

[0040] wherein L G is the overall loss of the generator, L adv is the adversarial loss, L ssim is the structural similarity loss, L grad is the gradient loss, L per is the perceptual loss; α, β, γ are proportional parameters for adjusting different losses respectively.

[0041] As a preferred technical solution of the present application, the adversarial loss is a deep constraint imposed by the discriminator on the image generated by the generator, which is composed of three parts, respectively from three discriminators, and the generator needs to ensure that the image generated by it can achieve a balance in the information retention of the three types of images in the training process, when the value of the adversarial loss is low, it indicates that the generated image can simultaneously retain the original information of the three types of input images, so that the discriminator cannot accurately judge the generated image, and the adversarial loss is defined as:

[0042] L adv = E (log (1-D M (I F-Y )) + E (log (1-D P-Y (I F-Y )) + E (log (1-D D (I F-Y ))

[0043] I F-Y = G (I M , I P-Y , ID )

[0044] where I F-Y is the generated image by the generator (Y component of the generated PET image), I M is the original MRI image, I P-Y is the Y component of the source PET image after YCbCr transformation, I D is the difference image; G is the generator, D M is the discriminator that discriminates the generated image by the generator from the original MRI image, D P-Y is the discriminator that discriminates the generated image by the generator from the Y component of the source PET image, D D is the discriminator that discriminates the generated image by the generator from the difference image I D ; E(·) represents the average operation.

[0045] As a preferred technical solution of the present application, the structural similarity loss term constrains the three-branch generator from the aspects of brightness, contrast and structure, so that the generator balances the input MRI image, the Y component of the PET image and the difference image when generating the image, so that the generated fusion image has certain similar structure, which is specifically defined as:

[0046] L ssim = ω1(1-L ssim (I F-Y ,I M ))+ ω2(1-L ssim (I F-Y ,I P-Y ))+ ω3(1-L ssim (I F-Y ,I D )

[0047] where L ssim (·) represents the structural similarity calculation of the input two images, and the output value is a scalar between 0 and 1, and the larger the value is, the more similar structures the two images have;

[0048] The gradient loss is to measure the difference between the fusion image and the input image in terms of structure, texture and other information. Since the original MRI image usually contains a lot of dense structural information, an enhancement coefficient is applied to it. The difference image is applied to an attenuation coefficient as auxiliary information. Through this constraint, the fusion image contains as much detail and texture information of the source image as possible. The gradient loss is defined as follows:

[0049]

[0050] where ||·|| denotes the L2 norm. TVTV norm; s1 and s2 are enhancement and attenuation coefficients respectively.

[0051] As a preferred technical solution of the present application, the perception loss is used to measure the preservation of deep feature information of the input image in the fused image, so as to constrain the generator at the deep semantic level of the image.

[0052] The discriminator loss function is composed of three parts, which are the loss of discriminating MRI and fused image (Y component of PET fused), the loss of discriminating Y component of PET and fused image, and the loss of discriminating difference image and fused image.

[0053] The present application aims to better compensate for the shortcomings of the existing GAN-based medical image fusion method, that is, the insufficient information preservation of the fused image due to the limitations of the discriminator, so as to better assist the diagnosis of clinical diseases while preserving the original image information as much as possible.

[0054] The present method adopts an end-to-end network model, without the need to set a complex image fusion strategy; the overall model is composed of a generator and three discriminators; the input of the generator additionally introduces a difference image in addition to PET and MRI images; the feature extraction part adopts a three-branch structure, which directly connects the corresponding images (source MRI, Y component of PET, difference image) to make each branch have its own information bias, and adopts a skip connection in each branch to improve the efficiency of transmitting image feature information, thereby improving the image fusion effect; in addition, the performance of the generator is continuously improved through mutual confrontation with three PatchGAN discriminators of the same structure; in terms of loss function, a hybrid loss is adopted, and the information entropy, saliency, gradient, etc. are used to measure the amount of information contained in the three types of input images to set the weight of the loss function; in addition, the structural similarity, gradient, perception and adversarial losses used can simultaneously constrain the generator at the surface and deep levels.

[0055] The experimental results on the disclosed data set show that the method in this paper is more in line with the visual tendency of the human eye in terms of subjective visual effect, and has achieved better results in terms of objective quantitative evaluation, among which the structural similarity SSIM is about 14.28% better than the second best performer (DSAGAN algorithm), the gradient-based fusion performance index Q AB / F is about 26.67% better than the second best performer, the correlation coefficient CC is about 11.43% better than the second best performer, the spatial frequency SF is about 8.33% better than the second best performer, and the human visual sensitivity index Q cb is about 11.29% better than the second best performer. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 Flow chart of the method of the present application;

[0057] Figure 2 General framework chart for PET and MRI image fusion of the present application;

[0058] Figure 3 Model adversarial training framework chart for PET and MRI image fusion of the present application;

[0059] Figure 4 Generator network structure chart of the present application;

[0060] Figure 5 Discriminator network structure chart of the present application. DETAILED DESCRIPTION

[0061] The preferred embodiments of the present application are described in detail below with reference to the accompanying drawings, so that the advantages and features of the present application can be more easily understood by those skilled in the art, and the scope of protection of the present application can be more clearly defined.

[0062] Embodiment: Please refer to Figures 1-3 The present application provides a technical solution: a PET and MRI image fusion method based on a three-discriminator generative adversarial network, the specific steps are as follows:

[0063] S1: YCbCr color space conversion;

[0064] The PET image is converted from the RGB color space to the YCbCr color space, and the PET image after color space conversion contains three image components, Y, Cb and Cr image components, wherein the Y image component contains brightness information that can reflect the overall structure and brightness change of the PET image, and the Cb and Cr image components respectively represent the concentration deviation components of blue and red in the PET image;

[0065] S2: Y component image fusion;

[0066] The Y component of the PET image and the MRI image are subjected to a difference operation, and the obtained difference image is subjected to a joint operation with the Y component of the PET image and the MRI image, and then input into the proposed network model to obtain a fused Y component image;

[0067] S3: RGB image reconstruction;

[0068] The Y component image generated by the generator and the Cb and Cr components of the corresponding source PET image are subjected to inverse transformation from the YCbCr color space to the RGB color space, thereby obtaining the final fused image.

[0069] In the YCbCr color space conversion of S1, the PET image IP RGB color space to YCbCr color space to obtain the component images I P Y, Cb, and Cr of the three channels of I P-Y , I P-Cb , and I P-Cr respectively; the PET image is transformed from the RGB color space to the YCbCr color space as follows:

[0070]

[0071] The Y component image of S2 is fused, and the MRI image I M is subtracted from the PET image Y component I P-Y and the absolute value is taken to obtain the difference image I D , and then the MRI image I M , the PET image Y component I P-Y , and the difference image I D are input into the generator network after being "coupled";

[0072] The generator extracts the feature information of the input image and performs feature selection, and then fuses I F-Y , which requires training of the generator-discriminator network before fusion.

[0073] The training process of the generator-discriminator network is as follows:

[0074] First, the I M , I P-Y , and the calculated I D image are input into the generator network, and the generator uses three different branches to extract image features. The three branches can be called gradient branch, contrast branch, and difference branch based on the characteristics of the extracted image feature information.

[0075] The network levels and specific parameters used by the three branches are the same, and the difference between the branches is that the source images directly connected in the branch convolution layer are different. Each branch uses a skip connection to extract image features to improve information utilization. Specifically, first, the input image is feature-extracted through three different feature extraction branches; then the features extracted by different branches are coupled and the features of each branch are summarized; finally, the image features extracted by different branches are selected through two convolution layers, and a decision is made to generate an image. The step of the convolution layer in the generator network is 1, and the specific structure of the generator is shown in Figure 4 ;

[0076] Second, the I F-Y generated by the generator network is respectively subtracted from the I M , I P-Y , ID Input to the corresponding discriminator D M , D P-Y Discriminate with D D ;

[0077] In the discriminator network, a total of 4 convolutional layers are included, all using a 4*4 size convolution kernel, and all using a BN (Batch Normalization) layer for normalization operation; the first three layers of convolution in the discriminator extract features from the input image, and the last layer of convolution makes a decision on the receptive field based on the features extracted by the previous layer of convolution, and the step size of all convolutional layers in the discriminator is 2, the padding size of the first three feature extraction convolutional layers is 1, and the activation function is selected as the Relu function; the padding size of the last decision-making convolutional layer is 0, and a non-linear function Tanh is selected to generate the final decision map, and the last convolutional layer outputs a 4*4 size decision map, and the output decision map is then averaged to obtain a scalar value between 0 and 1, which is the final discrimination result of the discriminator on the input image, and the specific structure of the discriminator is as shown in Figure 5 ;

[0078] Finally, according to the output results of the generator and the discriminator, the network of the generator and the discriminator is trained and updated alternately through the loss function until the network balance is reached, at which time the generator network trained is the generator we need for PET and MRI image fusion.

[0079] The loss function includes the weight of the loss function, the generator loss function and the discriminator loss function;

[0080] Since the data input to the generator in the present method includes the Y component of the PET image, the MRI image and the difference image (the difference between the Y component of the PET image and the MRI image), the loss weight between different images has a very important influence on the fusion result; therefore, the present application balances the information between the three types of input images based on the weight, specifically, the Y component of the PET image, the MRI image and the difference image are subjected to information entropy measurement, saliency measurement and gradient measurement, respectively, and then the weight parameters are generated based on the measurement results, in addition, the enhancement and attenuation parameters are set according to the information performance characteristics of the PET and MRI images to balance the loss functions of different categories;

[0081] The information entropy measurement is to measure how much information the image contains by calculating the information entropy of the source image, which is used as a weight reference for the loss function, specifically:

[0082]

[0083] Where I EnH(I) is the information entropy of image I; l is the number of gray levels, set to 256; p is the probability of corresponding gray level; l H(I) is the information entropy of image I; l is the number of gray levels, set to 256; p is the probability of corresponding gray level;

[0084] The saliency measure measures the amount of highlight information contained in the source image as a reference for the weight, specifically:

[0085]

[0086] where I Sal is the saliency measure of image I; Sign(·) is the sign function; τ is the threshold for screening the salient region; H, W are the height and width of image I; I ij is the gray value of the pixel point at coordinate (i, j) in image I;

[0087] The gradient measure is measured by using the Laplacian operator, and the higher the measure value indicates that the image contains more texture details, thereby serving as a measure of the loss weight, specifically:

[0088]

[0089] where I Grad is the gradient measure of image I; is the Laplacian operator;

[0090] The weight of the total loss function is defined as:

[0091] η I =λ1I En +λ2I Sal +λ3I Grad

[0092]

[0093] where η I is the overall measure corresponding to image I, λ1, λ2, λ3 are proportional parameters for controlling different measures; similarly, are the overall measures of the Y component of the MRI image, the Y component of the PET image and the difference image respectively; ω1, ω2, ω3 are the loss parameter weights of the Y component of the MRI image, the Y component of the PET image and the difference image respectively.

[0094] The generator loss function adopts a hybrid loss, that is, the loss of the surface information of the image is combined with the loss of the deep semantic information excavated by the network to constrain the network; the loss function of the generator is defined as follows:

[0095] L G =L adv +αL ssim +βL grad +γLper

[0096] wherein L G is the total loss of the generator, L adv is the adversarial loss, L ssim is the structural similarity loss, L grad is the gradient loss, L per is the perceptual loss; and a, b, g are the proportion parameters for adjusting different losses, respectively.

[0097] The adversarial loss is a deep constraint imposed by the discriminator on the image generated by the generator, which is composed of three parts, respectively from three discriminators. The generator needs to ensure that the generated image can achieve a balance in the information retention of the three types of images in the training process. When the value of the adversarial loss is low, it indicates that the generated image can simultaneously retain the original information of the three types of input images, so that the discriminator cannot accurately judge the generated image. The adversarial loss is defined as:

[0098] L adv = E (log (1-D M (I F-Y )) + E (log (1-D P-Y (I F-Y )) + E (log (1-D D (I F-Y ))

[0099] I F-Y = G (I M , I P-Y , I D )

[0100] wherein I F-Y is the image generated by the generator (the Y component of the generated PET image), I M is the original MRI image, I P-Y is the Y component obtained by YCbCr transformation of the source PET image, I D is the difference image; G is the generator, D M is the discriminator for discriminating the image generated by the generator from the original MRI image, D P-Y is the discriminator for discriminating the image generated by the generator from the Y component of the source PET image, and D D is the discriminator for discriminating the image generated by the generator from the difference image I D ; E (·) represents the average value operation.

[0101] The structural similarity loss term constrains the three-branch generator in terms of brightness, contrast and structure, so that the generator balances between the input MRI image, the Y component of the PET image and the difference image when generating the image, so that the generated fusion image has certain similar structure, which is specifically defined as:

[0102] L ssim = ω1(1-L ssim (I F-Y ,I M ))+ ω2(1-L ssim (I F-Y ,I P-Y ))+ ω3(1-L ssim (I F-Y ,I D )

[0103] Where L ssim (·) represents the structural similarity calculation of the input two images, and the output value is a scalar between 0 and 1, and the larger the value, the more similar structures the two images have;

[0104] The gradient loss is to measure the difference between the information such as structure and texture of the fused image and the input image, and since the original MRI image usually contains a lot of dense structural information, an enhancement coefficient is applied to it; the difference image as auxiliary information applies a decay coefficient to it, through this constraint, the fusion image contains as much detail and texture information of the source image as possible, and the gradient loss is defined as follows:

[0105]

[0106] Where ||·|| TV is the TV norm; s1 and s2 are the enhancement and decay coefficients, respectively.

[0107] The perception loss is to measure the preservation of the deep feature information of the input image in the fusion image, so as to constrain the generator in the deep semantic level of the image; since the medical image data used for training in the present application is small in size, and the pooling layer in the network will reduce the extracted image information to a certain extent, when the VGG19 network is selected as the image information perception network, only four convolutional layers at different positions of the network are selected as the perception feature layers of the image, which are located at the front one layer of the first four pooling layers of the network. Specifically:

[0108] i) the fusion image (the Y component of the PET image fused) generated by the generator is input into the pre-trained VGG19 network together with the three types of input images (source MRI, Y component of PET and difference image);

[0109] ii) extracting deep image feature information of the input image through the VGG19 network to obtain deep semantic representation of the input image at different feature levels;

[0110] iii) calculating the difference between the fusion image and the input image in the deep semantic representation by using the mean square error to measure the preservation of the original image feature information in the fusion image; an enhancement coefficient is applied to the PET image which can provide rich functional information; similarly, an attenuation coefficient is applied to the difference image; the perceptual loss is specifically defined as:

[0111]

[0112]

[0113]

[0114] L per =ω1L per (I M )+s3×ω2L per (I P-Y )+s4×ω3L per (I D )

[0115] wherein φ t represents the calculation of the image feature map through the t-th perceptual feature layer of the pre-trained VGG19 network; H t , W t and T t respectively represent the height, width and channel number of the image feature map corresponding to the t-th perceptual feature layer; is the Frobenius norm; s3 and s4 are respectively the enhancement and attenuation coefficients.

[0116] The discriminator loss function is composed of three parts, which are respectively the loss of discriminating the MRI and the fusion image (the Y component of the fused PET), the loss of discriminating the Y component of the PET and the fusion image, and the loss of discriminating the difference image and the fusion image;

[0117] Three network structures of the PatchGAN-based discriminator are adopted, and the loss functions adopted by different discriminators are also similar, so as to distinguish the images generated by the generator and the three types of original input images, and then optimize the generator through the output results of the discriminator, so that the generator can generate the fusion image conforming to the texture and intensity of the input image; specifically, the loss function of the discriminator is:

[0118] L D =L D (I M )+LD (I P-Y )+L D (I D )

[0119] L D (I M )=E(-log(D M (I M )))+E(-log(1-D M (I F-Y )))

[0120] L D (I P-Y )=E(-log(D P-Y (I P-Y )))+E(-log(1-D P-Y (I F-Y )))

[0121] L D (I D )=E(-log(D D (I D )))+E(-log(1-D D (I F-Y )))。

[0122] The reconstruction of the RGB image, the PET component I F-Y generated by the generator is inversely transformed from the YCbCr color space to the RGB color space together with the Cb, Cr components I P-Cb , I P-Cr corresponding to the PET image, so as to obtain the final fusion image I F .

[0123] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it cannot be understood as the limitation of the patent scope of the present application. It should be pointed out that, for the ordinary skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A PET and MRI image fusion method based on a three-discriminator generative adversarial network, characterized in that: The specific steps are as follows: S1: YCbCr color space conversion, comprising: The PET image is transformed from the RGB color space to the YCbCr color space, and the PET image after the color space transformation contains three image components, namely Y, Cb and Cr image components, wherein the Y image component contains brightness information capable of reflecting the overall structure and brightness change of the PET image, and the Cb and Cr image components respectively represent the concentration offset components of blue and red in the PET image; S2: fusion of the Y image component, comprising: The MRI image I M The Y image component I P-Y of the PET image is subjected to difference and absolute value taking to obtain a difference image I D The Y image component I M of the PET image is subjected to difference and absolute value taking to obtain a difference image I P-Y The Y image component I D of the PET image is subjected to difference and absolute value taking to obtain a difference image I F-Y ; Before fusion, the generator network A1 is obtained by training the generator-discriminator network, and the training process comprises: I M , I P-Y and the calculated I D The image is input to the generator network A0, wherein the generator adopts 3 different branches to extract image features, and the 3 branches are respectively called gradient branch, contrast branch and difference branch based on the characteristics of the extracted image feature information. The I generated by the generator network A0 F-Y Respectively with I M , I P-Y , I D Input to the corresponding discriminator D M 、D P-Y With D D Among them, D M The discriminator D is used to distinguish the images generated by the generator from the original MRI images. P-Y The discriminator D is used to distinguish the Y image component of the image generated by the generator from the source PET image. D The image generated by the identification generator and the difference image I D The discriminator of According to the output results of the generator and the discriminator, the network of the generator and the discriminator is subjected to adversarial training and alternating update through a loss function until the network balance is reached, so that the generator network A1 is obtained, that is, the generator for PET and MRI image fusion; S3: reconstruction of the RGB image, comprising: I F-Y Cb, Cr image components of the corresponding source PET image, resulting in a final fused image I F .

2. The PET and MRI image fusion method based on three discriminator generative adversarial networks according to claim 1, characterized in that: In the YCbCr color space conversion of the S1, the PET image I P is converted into RGB color space, and the RGB image I P is converted into YCbCr color space, to obtain the image components I P-Y , I P-Cb and I P-Cr of the Y, Cb and Cr channels respectively; the conversion process of the PET image from RGB color space to YCbCr color space is as follows:

3. The PET and MRI image fusion method based on three discriminator generative adversarial networks according to claim 1, characterized in that: The loss function comprises a generator loss function and a discriminator loss function.

4. The PET and MRI image fusion method based on three discriminator generative adversarial networks according to claim 3, characterized in that: The weight of the loss function balances the information between the three types of input images by using a weight-based method. Specifically, the information entropy of the Y image component of the PET image, the MRI image and the difference image is measured, the saliency is measured, and the gradient is measured. Then, according to the measurement results, the weight parameters are generated, the enhancement and attenuation parameters are set according to the information expression characteristics of the PET and MRI images, and the loss functions of different categories are balanced. The information entropy measurement is to measure the amount of information contained in the source image by calculating the information entropy of the source image, which is used as a reference for the weight of the loss function. Specifically: where I En is the information entropy of the image I; / is the number of gray levels, set to 256; p l is the probability of the corresponding gray level; The saliency measurement measures the amount of highlight information contained in the source image, which is used as a reference for the weight. Specifically: where I Sal is the saliency measure of image I; Sign(·) is the sign function; τ is the threshold to filter salient regions; H, W are the height and width of image I; I ij is the intensity value of pixel at coordinate (i, j) in image I. The gradient measurement is measured by using the Laplacian operator. The higher the measurement value, the more texture details the image contains, which is used as a measurement of the loss weight. Specifically: where I Grad is a gradient measure of the image I; is a Laplacian operator; The weight of the total loss function is defined as: η I = λ1I En + λ2I Sal + λ3I Grad where η I is the overall metric for the corresponding image I, λ1, λ2, λ3 are the proportional parameters controlling the different metrics; are the overall metrics for the Y image components of the MRI image, the PET image and the difference image, respectively; ω1, ω2, ω3 are the loss parameter weights for the Y image components of the MRI image, the PET image and the difference image, respectively.

5. The PET and MRI image fusion method based on three discriminator generative adversarial networks according to claim 4, characterized in that: The generator loss function adopts a hybrid loss, that is, the loss of the surface information of the image is combined with the loss of the deep semantic information excavated by the network to constrain the network. The loss function of the generator is defined as follows: L G = L adv + αL ssim + βL grad + γL per where L G is the total loss of the generator, L adv is the adversarial loss, L ssim is the structural similarity loss, L grad is the gradient loss, L per is the perceptual loss; and α, β, γ are the proportion parameters for adjusting different losses, respectively.

6. The PET and MRI image fusion method based on three discriminator generative adversarial networks according to claim 5, characterized in that: The adversarial loss is a deep constraint imposed by the discriminator on the image generated by the generator, which is composed of three parts, respectively from the three discriminators. The generator needs to ensure that the generated image can balance the information preservation of the three types of input images in the training process. When the value of the adversarial loss is low, it means that the generated image can simultaneously preserve the original information of the three types of input images, so that the discriminator cannot accurately judge the generated image. The adversarial loss is defined as: L adv = E(log(l - D(I))) + E(log(l - D(I))) + E(log(l - D(I))) M (I F-Y )) + E(log(l - D(I))) + E(log(l - D(I))) P-Y (I F-Y )) + E(log(l - D(I))) + E(log(l - D(I))) D (I F-Y )) I F-Y = G(I M , I P-Y , I D ) Where G is the generator, and E(·) represents the average value operation.

7. The PET and MRI image fusion method based on three discriminator generative adversarial networks according to claim 5, characterized in that: The structural similarity loss constrains the three branch generators through brightness, contrast and structure, so that the generator balances the input MRI image, Y image component of the PET image and difference image when generating the image, so that the generated fusion image has a certain similar structure. The specific definition is as follows: L ssim = ω1(1 - L ssim (I F-Y , I M ))+ ω2(1 - L ssim (I F-Y , I P-Y ))+ ω3(1 - L ssim (I F-Y , I D )) where L ssim (·) denotes a structural similarity computation on the input pair of images, whose output value is a scalar between 0 and 1, the larger the value, the more similar structures the two images have; The gradient loss is to measure the difference between the structure and texture information of the fused image and the input image. Since the original MRI image contains dense structural information, an enhancement coefficient is applied to it. The difference image is auxiliary information, and an attenuation coefficient is applied to it, so that the fused image contains the details and texture information of the source image. The gradient loss is defined as follows: where ||·|| TV is the TV norm; s1 and s2 are the enhancement and attenuation coefficients respectively.

8. The PET and MRI image fusion method based on three discriminator generative adversarial networks according to claim 5, characterized in that: The perception loss is to measure the preservation of the deep feature information of the input image in the fused image, so as to constrain the generator at the deep semantic level of the image.

Images