PET-mri image fusion method based on adaptive generative adversarial network

Abstract

The application constructs an adaptive residual dense generative adversarial network and combines a YC b C r color space method to propose a PET-MRI image fusion method based on an adaptive generative adversarial network, which mainly adopts a regional residual learning module and an output cascade deep generative network to avoid feature loss, and dynamically guides the generator to generate a fusion image with the same distribution as the source image through an adaptive decision block, and performs an adversarial game between the fusion image gradient graph and the joint gradient graph of the input image to efficiently train the generator and the discriminator, so as to obtain a fusion image with rich details and clear texture; the method has the characteristics of end-to-end and unsupervised, does not need manual intervention, and does not need real data as a label, and can fuse images with different resolutions without introducing a traditional framework; in the test of the Harvard Medical School MRI / PET data set, the peak signal-to-noise ratio and the structural similarity reach 55.2124 and 0.4697 respectively, which are better than the most advanced algorithm at present, and are more helpful for clinical application diagnosis.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical imaging, and particularly relates to a PET-MRI image fusion method based on an adaptive generative adversarial network. BACKGROUND

[0002] Medical images are divided into structural systems and functional systems, and different imaging mechanisms of different systems can obtain different imaging information of the same part. For example, a magnetic resonance imaging (MRI) image can provide high-resolution brain soft tissue structure information, and a positron emission computed tomography (PET) image can reflect color information of tissue metabolism and function. However, images of different modalities have their own limitations. An MRI image lacks body metabolism and other motion information, and a PET image has low resolution and cannot accurately locate a lesion. Different images have their own specific characteristics, and limited information of a single modality image cannot meet the demand of clinical diagnosis and treatment for information quantity, and images from multiple imaging mechanisms need to be fused. In recent years, the success of PET-MRI fusion imaging in the clinical field has aroused great interest in non-invasive functional imaging and anatomical imaging.

[0003] In the fusion process, it is necessary to retain spatial information of the MRI image and spectral information of the PET image, or in other words, it is necessary to introduce spatial information existing in the MRI data into the PET, so as to overcome the limitation of a single modality medical image, improve imaging quality while retaining image characteristics, and improve the clinical applicability of the image in diagnosing and evaluating medical problems.

[0004] The most widely used technology in the field of traditional medical image fusion is pixel-level fusion, which can be divided into two categories: spatial domain and transform domain. The former directly acts on the pixels, and the rules are simple, but the fusion effect is poor. For example, in "He C T, Liu Q X, Li H L, et al. Multimodal medical image fusion based on IHS and PCA[J]. Procedia Engineering, 2010, 7: 280-285", the image is converted into intensity, hue, and saturation (IHS) channels. However, the IHS transformation will cause spectral and spatial distortion. The image fusion technology based on transform domain mostly uses multi-scale transform (MST) technology, which consists of three processes: decomposition, fusion, and reconstruction. The source image is first transformed into the frequency domain and fused according to certain rules, and then the fused coefficients are used to reconstruct the image with the transform basis. This method well preserves the detail information of the source image, but ignores the spatial consistency, resulting in distortion of the brightness and color of the fused image. The rules of traditional fusion methods need to be designed and selected manually, and different filter parameters in MST will result in significant differences in fusion effect. However, due to the diversity of feature extraction and the complexity of fusion rules, it is difficult to design a fusion method manually, which reduces the robustness of the model.

[0005] With the rise of deep learning in recent years, neural networks are used to solve the above problems, and most existing deep learning-based image fusion uses convolutional neural networks (CNN) to build models. In the past few years, research on deep learning-based image fusion has gradually become active, and scholars have proposed numerous fusion methods, gradually forming an important branch. In some methods, deep learning frameworks are used to extract image features in an end-to-end manner for reconstruction. For example, the document "Liu Y, Chen X, Ward R K, et al. Image fusion with convolutional sparse representation[J]. IEEE signal processing letters, 2016, 23(12): 1882-1886" applies convolutional sparse representation (CSR) to image fusion, extracts multi-layer features, and generates a fusion image using these features. IFCNN adds a convolutional neural network to the transform domain image fusion algorithm (Zhang Y, Liu Y, Sun P, et al. IFCNN: A general image fusion framework based on convolutional neural network[J]. Information Fusion, 2020, 54: 99-118). The document "Yousif A S, Omar Z, Sheikh U U. An improved approach for medical image fusion using sparse representation and Siamese convolutional neural network[J]. Biomedical Signal Processing and Control, 2022, 72: 103357" proposes a fusion method for medical images based on sparse representation and Siamese convolutional neural networks. The document "Hou R, Zhou D, Nie R, et al. Brain CT and MRI medical image fusion using convolutional neural networks and a dual-channel spiking cortical model[J].Medical & biological engineering & computing, 2019, 57(4): 887-900” selects to add deep learning technology in traditional image fusion scheme, CNN framework is used to fuse low frequency coefficient and high frequency coefficient; Densefuse includes convolution layer, fusion layer dense block, encoder is responsible for providing input for network, after network gets feature map, fusion image is reconstructed through decoder (Li H, Wu X J. DenseFuse: A fusion approach to infrared and visible images[J]. IEEE Transactions on Image Processing, 2018, 28(5): 2614-2623.); GCF is an unsupervised model of multi-focus image fusion based on gradient and connected region; Literature “Chen M, Zheng H, Lu C, et al. A spatio-temporal fully convolutional network for breast lesion segmentation in DCE-MRI[C] / / International Conference on Neural Information Processing. Springer, Cham, 2018: 358-368” extracts features by combining CNN and RNN, then fuses to segment, for the first time, generative adversarial network is introduced into infrared and visible light image fusion, the purpose of generator is to generate a fusion image mainly containing infrared information and a small amount of visible light information, the purpose of discriminator is to force fusion image to have more detailed information in visible light image; DDcGan constructs dual discriminator generative adversarial network (Ma J, Xu H, Jiang J, et al. DDcGAN: A dual-discriminator conditional generative adversarial network for multi-resolution image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4980-4995); Literature “Tang W, Liu Y, Zhang C, et al. Green fluorescent protein and phase-contrast image fusion via generative adversarial networks[J].Computational and Mathematical Methods in Medicine, 2019, 5450373:11” proposes to fuse biological images through generative adversarial networks; PMGI extracts information using image gradients and contrast, and performs feature reuse on the same path (Zhang H, Xu H, Xiao Y, et al. Rethinking the Image Fusion: A Fast Unified Image Fusion Network based on Proportional Maintenance of Gradient and Intensity [C] / / Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 12797-12804).

[0006] Deep learning-based research has become an active topic in the field of image fusion in the past few years, and many deep learning-based fusion methods have been proposed and gradually formed an important branch. Although these methods have achieved good results, most of the design of the fusion rules is still artificial, and the whole method cannot get rid of the limitations of traditional fusion methods; the biggest obstacle of using deep learning for image fusion is the lack of real label data, and it is difficult to directly obtain real label images for the MRI-PET fusion task.

[0007] Therefore, although these existing works have achieved good results, there are still some deficiencies: (1) the deep learning framework is only used to make up for some defects of traditional fusion methods, such as feature extraction, and the design of the whole fusion method is still based on traditional methods, and the complex fusion rule design based on the traditional fusion framework cannot realize the end-to-end result generation; (2) due to the lack of label data, the solution relying on the design of the loss function is not comprehensive, and due to the limitations of the physical imaging process, the fusion task cannot obtain a real fusion image as a label, and the existing deep learning methods rely heavily on artificial priori, and use manually made pseudo data labels, which greatly limits the performance of the algorithm; (3) the solution based on the traditional generative adversarial network can only make the result similar to one source image, that is, only the pixel-level L1 loss is used to train the generative network, and due to the existence of Nash equilibrium theory, part of the high-frequency detail information contained in the original image is lost. SUMMARY

[0008] In order to avoid the loss of spatial information in the image fusion process, protect the spatial texture structure of MRI and PET images, and further save the texture and detail information of high-resolution images and the structure information of low-resolution images, a PET-MRI image fusion method based on adaptive generative adversarial network is proposed, which comprises the following steps:

[0009] a) mapping the PET image from the RGB space to the YC b C r space and extracting the Y component;

[0010] b) inputting the PET image Y component and the MRI image into the generative network;

[0011] c) using the Laplacian operator to extract the joint gradient map of the input image and the gradient map of the output result of the generative network respectively;

[0012] d) inputting the two gradient maps into the discriminative network and making the real input label probability 0.7-1.2 (soft label), and the generated result input label probability 0-0.3 (soft label);

[0013] e) training the generative network and the discriminative network based on the adversarial generation strategy;

[0014] f) using the Adam optimizer for optimization;

[0015] g) obtaining the trained generative network model;

[0016] h) using the network model for prediction.

[0017] Among them, about step a), the de-correlated color model YC b C r is adopted, which divides the image information into three channels: Y channel, C b channel and C r channel, which respectively represent the luminance component of color and the chrominance offset component of blue and red; the Y channel stores the luminance information of the image, and the C b channel and C r channel store the red and blue color difference information of the image; therefore, only the Y component information of the MRI image and the PET image needs to be processed in the image fusion iteration, both of which are gray images; the specific transformation equation and inverse transformation equation are shown in equations (1) and (2) respectively:

[0018] (1)

[0019] (2)

[0020] The architecture of the adaptive generative adversarial network mainly includes a generator, a discriminator and a regional residual learning module. The network can fuse the low-resolution Y component (PET_y) of the PET image with the grayscale image MRI with higher spatial resolution to obtain a fused image including rich structural information and higher spatial resolution. In order to simultaneously save the texture and detail information of the high-resolution image and the structural information of the low-resolution image, an adjustment loss function mechanism is used to optimize the prediction result. The overall architecture of the adaptive generative adversarial network is as shown in Figure 1 The adaptiveness of the network is derived from the decision block shown in the preprocessing stage before the input of the discriminator network in Figure 1 The decision block can also be referred to as a maximum matching algorithm as shown in equation (3).

[0021] (3)

[0022] The maximum matching algorithm process is as follows: input the PET_y component Y and the MRI image M, extract the Laplacian gradient graph thereof through the Laplacian operator, then compare the pixel values of each pixel of the two graphs, and take the pixel value of the larger one as the fused gradient pixel, and finally calculate the joint gradient graph. The decision block can guide the fusion result to approximate the brightness and gradient distribution of the source image. The principle is to evaluate the sharpness of each pixel to generate a screening graph with an effective information position.

[0023] The structure of the generator is as shown in Figure 2 The generator is a double-branch fusion network that divides the PET image Y component and the MRI image into two paths for processing. The double-branch fusion network framework uses a set of 3x3 convolution layers for feature extraction, then deepens the network for feature processing, and finally uses a set of 1x1 convolution layers for reconstruction. The first convolution layer extracts shallow features through equation (4):

[0024] (4)

[0025] wherein, H conv represents a convolution operation with a 5x5 convolution kernel in the shallow feature extraction layer; the second layer output can be obtained through equation (5):

[0026] (5)

[0027] wherein, wherein H LRLP is a composite function of the LRLR layer operation; the third layer output is as shown in equation (6):

[0028] (6)

[0029] wherein,H RL represents residual connection; λ is the weight when residual fusion; the fourth layer is the same as the third layer, and the input is based on the output of the previous three layers in cascade; the fourth layer output is shown in formula (7):

[0030] (7)

[0031] The subsequently output layers of features are spliced and fused features using 3*3 convolution, and the output formula is shown in formula (8):

[0032] (8)

[0033] Wherein, H concat represents feature map splicing operation; the last layer of the extraction module is set to be spliced with 1*1 convolution after feature map, W is the weight matrix of the fusion of the first four layers of the extraction module, and the outputs of the two paths F ext,1 , F ext,2 enter the fusion module, and after the fusion operation, the finally fused image is shown in formula (9):

[0034] (9)

[0035] Wherein, H fuse represents the composite operation of the fusion module.

[0036] The structure of the discriminator is shown in Figure 3 The input of the discriminator has two sources, the gradient map calculated by the Laplacian operator of the input two images, and the joint gradient map obtained by the maximum function, and the gradient map calculated by the Laplacian operator of the fusion image as the two inputs of the discriminator; four convolution layers and a linear layer constitute the discriminator of the model, the convolution kernel size of the convolution layer is all set to 3*3, the step is set to 4, and ELU is used as the activation function; the last layer is a linear layer, which is used to calculate the probability, so as to judge the true and false of the generated data.

[0037] Regional residual learning module (LRLP):

[0038] In the forward transmission process of the convolutional neural network, with the increase of the network depth, the feature map obtained by convolution gradually reduces the information contained, in order to solve the above problems, the regional residual learning module is used, the information between different layers is directly mapped, and the features contained in each layer are saved as much as possible; wherein the LRLP module is as Figure 4As shown, the module first obtains image features of different depths through c different convolutions in series, then splices the features of different depths after convolution by weight, then compresses and reconstructs using a 1x1 convolution layer, and finally activates using ELU; if there are c convolution layers, then the final output thereof is as shown in formula (10):

[0039] (10)

[0040] wherein, F c is the output of the cth convolution layer; H concat represents a splicing function of the feature map; W is a joint function representing the weights of each convolution layer during splicing; H active represents ELU activation on the spliced data; in the LRLP block, the output of each previous layer is taken as the input of the next layer.

[0041] In step e), the discriminator defines the joint gradient map as real data, and continuously performs adversarial learning with the gradient map of the fusion image defined as pseudo data, and the objective function of the GAN is defined as shown in formula (11):

[0042] (11)

[0043] In the adversarial learning in the GAN, a set of distinguishable semi-supervised loss functions are designed, which are different from the fixed loss functions of traditional deep learning, and the loss functions are dynamically constructed using the screening map, and only those pixels that are judged as effective by the judgment block can participate in the loss function calculation in the optimization process, so as to protect the high-frequency information and make the outline of the fusion image clearer, specifically, the semi-supervised loss function includes the loss function of the generator and the loss function of the discriminator.

[0044] Generator loss function

[0045] The loss function of the generator is based on adversarial loss, pixel-level Euclidean loss and texture loss, and the loss function is as shown in formula (12):

[0046] (12)

[0047] wherein, is the adversarial loss from the generator-discriminator network; is the pixel-level Euclidean loss optimized by the screening map; represents the texture loss based on the gradient map; and are the weights of the pixel-level loss and the texture loss respectively, used to ensure that the three loss functions have the same importance;

[0048] Adversarial loss

[0049] In order to make the image generated by the generator closer to the ideal fusion image, it is necessary to establish a loss between the generator and the discriminator, and the traditional adversarial loss simplifies the max-min problem to But in the initial stage of training, It may be saturated, so the generator network is trained using maximization; in order to provide stronger gradients, a square operation is added on the basis of the maximization operation, Defined as shown in formula (13):

[0050] (13)

[0051] Wherein, M is the number of images in a batch during training; c is the rate label of the discriminator identifying true and false images; the present application uses Laplacian to calculate the gradient graph; M, Y represents the input MRI image and PET image Y channel;

[0052] Pixel-level Euclidean loss

[0053] The present application uses the Euclidean distance between the fusion image and the original image pixels to constrain the intensity distribution of the fusion image and the original image in the clear area, and the pixel-level Euclidean loss can be formulaized as shown in formula (14):

[0054] (14)

[0055] Wherein, h, w Represents the pixel value of the i-th row and the j-th column; H, W are the height and width of the image respectively; h w Map 1 , Map 2 Represents the screening image generated by the decision block based on the two input images;

[0056] Texture loss

[0057] The gradient of the image can partially represent the texture details, and the contrast of the MRI image is more so that the fusion image and the input image are required to have similar gradients, combined with the screening image, the texture loss can be formulaized as shown in formula (15):

[0058] (15).

[0059] Discriminator loss function

[0060] ​​Not only the generator needs a loss function to optimize the quality of the fusion image, but also the discriminator needs a loss function to accurately identify the true and false of the fusion image, and the application designs a loss function based on the gradient map for the discriminator, wherein the "false data" is the gradient map of the fusion image, which can be formulated as formula (16) shown:

[0061] (16)

[0062] The "true data" needed by the discriminator comes from the joint gradient map constructed by MRI and PET_y, which is formulated as formula (17) shown:

[0063] (17)

[0064] Wherein, abs represents the absolute value function; maximum represents the maximum function; based on the above two gradient maps, the loss function is represented as formula (18) shown:

[0065] (18)

[0066] Wherein, a is the label of "false data", which is set to 0; b is the label of "true data", which is set to 1; so that the discriminator regards the joint gradient map of the image as true data and the gradient map of the fusion image as false data; this constraint can guide the generator to adjust Grad union Grad fused , and enhance the texture of the fusion image in the confrontation.

[0067] The application has the following beneficial effects:

[0068] The application constructs an adaptive residual dense generative adversarial network, combines a YCbCr-based color space method, and proposes a new image fusion method, which enables the generation network to avoid gradient disappearance and gradient explosion, improves the network feature extraction performance, performs an adversarial game between the fusion image gradient map and the input image joint gradient map, and combines the designed adversarial loss, discriminator loss, pixel-level consistency loss and gradient consistency loss to obtain a fusion image with rich details and clear texture, which does not need real data as a label for training, can fuse images of different resolutions without introducing traditional frameworks, greatly optimizes the fusion rule design of traditional methods, realizes adaptive fusion without human intervention, the fusion image has more high-frequency details and greatly retains the MRI pseudo-color content information, in the test of the Harvard Medical School MRI / PET data set, the peak signal-to-noise ratio reaches PSNR=55.2124, the structural similarity reaches SSIM=0.4697, the RMSE reaches 0.1968, the Q abf =0.3635 and Q​cv =2009.348, which are better than the current most advanced algorithm, and are more helpful for assisting clinical application diagnosis. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 is a schematic diagram of an adversarial dense residual adversarial generative network;

[0070] Figure 2 is a schematic diagram of a generator network;

[0071] Figure 3 is a schematic diagram of a discriminator network;

[0072] Figure 4 is a schematic diagram of a regional residual learning module (LRLP);

[0073] Figure 5 is a schematic diagram of a qualitative comparison of the proposed method and other international leading methods. DETAILED DESCRIPTION

[0074] The application will be further described below through specific embodiments.

[0075] Embodiment 1

[0076] The PET and MRI images used in this embodiment come from a public data set on the website of Harvard Medical School, wherein the MRI image is a single-channel image with a size of 256x256; and the PET image is a pseudo-color image with a size of 256x256x3;

[0077] The generator and the discriminator are trained iteratively according to the adversarial process, the batch size is set to b, one iteration of training requires k steps, the ratio of the number of discriminator training to the number of generator training is p, and a total of M times of training are performed; through multiple experiments, it is obtained that b=32, p=2, M=300, the parameters in ADRGAN are updated by AdamOptimizer; in order to make the training of GAN more stable, the soft label is used for the loss term parameter: for the label that should be set to 1, it is set to a random number from 0.7 to 1.2;

[0078] The image is preprocessed from the RGB channel to the YC b C r color space, because the Y channel (luminance channel) can represent structural details and brightness changes, only the Y channel needs to be fused; the C b and C rThe fused components are then inversely transformed into RGB channels; the experimental environment of the embodiment is: Windows 10, CPU AMD R5 5600X, memory 16G GPU RTX-3060 (6G); the software environment is Python 3.7.6 and Pytorch 1.10.0, and the training set, the verification set and the test set of the data set are divided according to 7:2:1, and the specific training process is shown as algorithm 1:

[0079]

[0080] Quantitative evaluation index

[0081] Five evaluation indexes are used to objectively evaluate the method and the comparative method, and the five indexes are Q abf , Q cv , PSNR, SSIM and Rmse, Q abf The algorithm uses local measurement to estimate the performance of important information in the input of the fused image, and the higher the value, the better the quality of the fused image, as shown in formula (19):

[0082] (19)

[0083] Wherein, W is used to divide the local area; λ(w) represents the weight of the local area; A, B, F are two input images and a fused image, respectively;

[0084] Q cv The quality of the local area image is obtained by calculating the weighted difference image of the fused area image and the source area image, and finally the quality of the fused image is the weighted sum of the quality measurement of the local area image, which is formalized as formula (20):

[0085] (20)

[0086] Wherein, D is a local area similarity measurement function;

[0087] Peak signal-to-noise ratio (PSNR) is the ratio of peak power to noise power in the fused image, which reflects the distortion of the fused image, and the calculation method is shown as formula (21)-(24):

[0088] (21)

[0089] (22)

[0090] (23)

[0091] (24)

[0092] wherein, MSE is the mean square error, represents the image in i row j column of pixels; r represents the peak value of the fusion image, the greater the peak signal-to-noise ratio, the closer the fusion image to the source image;

[0093] Structural similarity (SSIM) is used to simulate the loss and distortion of the image, and the index is composed of three parts, which are: correlation loss, contrast loss, brightness loss; The product of the three components is the evaluation result of the fusion image, defined as follows:

[0094] (25)

[0095] wherein, x and f represent a block in the source image and the fusion image, respectively; is the covariance between the two blocks; , indicates the standard deviation (SD); u x , u y indicates the average value between the two blocks, and the addition of C1, C2, C3 makes the loss function more stable;

[0096] Root mean square error (RMSE) is based on MSE, which quantitatively describes the difference between the source image and the fusion image by calculating the mean square error of the source image and the result image, as shown in equation (26):

[0097] (26)

[0098] Quantitative and qualitative comparison results

[0099] In order to verify the effect of the model on PET-MRI image fusion and verify its robustness, five methods of DDcGan, Densefuse, GCF, IFCNN and PMGI are selected for comparison with the present application, and the above methods have achieved good results in the previous medical image fusion;

[0100] The visual experimental results of the six related methods are as follows: Figure 4As shown in the table, the results of DDcGan (the third column) have the problem of spectral distortion, and the edges are more blurred than the model; Densefuse (the fourth column) loses the color intensity in the PET image, loses part of the functional information, and increases the difficulty of finding lesions; GCF (the fifth column) has good color preservation, but large noise blocks appear in multiple images, and the structural information is directly lost, which can mislead clinical judgment and has poor robustness; IFCNN (the sixth column) will lose details near the boundary line and is not clear enough in the texture-intensive area; PMGI (the seventh column) has high color intensity and complete functional information preservation when fusing, but the background is blurred, high-frequency information is lost, and there is no texture detail; the fused image of the application has no above problems, and the structural and functional information is well preserved, the details are clear and the contrast is high, especially the contrast at the edge is obvious, the details in the texture-intensive area are clear, and the image information is sufficient to meet the needs of clinical diagnosis; Since most of these methods try to sharpen the edges by directly enhancing the target and gradient, the naturalness and authenticity of the fused image still differ greatly; In addition, almost all of these methods rely on large data sets, while the structural content loss function and the adversarial loss function proposed in the application protect high-frequency information and content information, respectively, and improve the effect of the fused image through separate nonlinear loss constraints.

[0101] Qualitative evaluation of the fusion effect by subjective feeling of the human eye has great limitations, in order to objectively verify the superiority of the application, the application selects an objective evaluation method to quantitatively evaluate the experimental results, and the results are shown in Table 1:

[0102] Experimental method PSNR↑ SSIM↑ RMSE↓ Q abf ↑]]> Q cv ↓]]> DDcGan 54.8162 0.3000 0.2146 0.1602 2534.607 DenseFuse 55.1830 0.3628 0.1986 0.1368 2242.367 GCF 54.4163 0.3347 0.2367 0.3401 2521.672 IFCNN 54.4163 0.4160 0.2083 0.3516 2226.219 PMGI 54.0151 0.1022 0.2581 0.0460 3469.525 OURS 55.2124 0.4697 0.1968 0.3635 2009.348

[0103] It can be seen that the application is better than the other five comparison methods in the five indicators; Q cv The index is based on the human visual system (HVS) and the regional mean square error, and the model can adaptively judge the pixel weight to improve the regional similarity due to the adaptive module; Compared with DDcGan, the application reduces by 20.7%, which proves that it is stronger than the other methods in the sense of human eye perception and has higher regional similarity; The adversarial game enables the model to have excellent denoising ability, thereby improving the PSNR, which proves that the application has less noise and less interference information compared with other methods; The SSIM index tends to verify the structural information, which is increased by 11.4% compared with the best IFCNN, and the higher index indicates that the texture structure of the application is complete and the blurred area is less, and the structural similarity of PMGI is only 21% of the method, and the structure is not well preserved; Compared with the qualitative comparison results, the application adopts a pixel scale control strategy, and the Euclidean distance between pixels is well controlled, and the pixel-level fusion index Q abfThe higher, the visual information is perfect, the gap between the fused image and the source image is small in pixel level; meanwhile, the smaller Rmse indicates that the fused image has less error and distortion.

[0104] The above detailed description is used to explain and illustrate the present application, rather than limiting the present application, any modification and change made to the present application within the spirit and protection scope of the claims, falls into the protection scope of the present application.

Claims

1. A PET-MRI image fusion method based on adaptive adversarial generative networks, characterized in that, Includes the following steps: a) Mapping PET images from RGB space to YC space b C r The space is then used to extract the Y component; the transformation equation is shown in equation (1), and the inverse transformation equation is shown in equation (2): （1） （2） b) Input the Y component of the PET image and the MRI image into the generator; the generator is a two-branch fusion network, whose inputs are the Y component of the PET image and the grayscale image of the MRI, respectively; the two-branch fusion network framework uses a set of 3×3 convolutional layers for feature extraction; then the network is deepened for feature processing; finally, a set of 1×1 convolutional layers is used for reconstruction; the region residual learning module LRLP is used in the feature processing of the network. This module first obtains different image features through c different convolutions in different network branches, then the convolutional features are weighted and concatenated, and finally activated using ELU. The process is shown in equation (3): （3） in, F c For the first c The output of each convolutional layer; H concat This represents the feature map concatenation function; W It is a set of joint functions representing the weights of each convolutional layer during the splicing process; H active This indicates that ELU activation is performed on the concatenated data; in the LRLP block, the output of each preceding layer serves as the input of the next layer; the first convolutional layer extracts shallow features using equation (4): （4） in, H conv This indicates a 5×5 convolution operation in the shallow feature extraction layer. The extracted shallow features are then fed into the next layer, and the output of the second layer is obtained through equation (5): （5） in, H LRLP It is a composite function of the LRLP layer operation; in the subsequent convolutional layers, the input is the concatenation of the outputs of all previous layers and the LRLP module; at the same time, parameter sharing is also set between the two paths; the output of the third layer is as shown in equation (6): （6） in, H RL λ represents the residual connection; λ is the weight during residual fusion; the fourth layer is similar to the third layer, the input of the fourth layer is based on the output of the first three layers concatenated, and the output of the fourth layer is as shown in equation (7): （7） The output features from each layer are then concatenated and fused using a 3×3 convolution, as shown in equation (8): （8） in, H concat The feature map concatenation operation is represented by the last layer of the extraction module, which is set to concatenate the feature maps and then add a 1×1 convolution. W is the weight matrix of the first four layers of the extraction module. The outputs of the subsequent two paths are... F ext,1 , F ext,2 After entering the fusion module and undergoing fusion calculation, the fused image is obtained, as shown in equation (9): （9） in, H fuse Composite operations representing the fusion module; c) Use the Laplacian operator to extract the fused image obtained in step b), the Y component of the real input PET image in step a), and the gradient map of MRI respectively. The extracted Y component of PET image and gradient map of MRI are processed by decision block. The result of decision block processing is shown in Equation (10). （10） Where abs represents the absolute value function; maximum represents the maximization function. The decision block processing includes: inputting the PET_y component Y and the MRI image M, extracting their Laplacian gradient map through the Laplacian operator, then comparing the pixel values ​​pixel by pixel and taking the pixel with the larger pixel value in each of the two images as the gradient pixel after fusion, and finally calculating the joint gradient map. d) Input the gradient map of the fused image extracted in step c) and the calculated joint gradient map into the discriminator and set the probability of the true input label to 0.7-1.2, and the probability of the generated input label to 0-0.3; the discriminator consists of four convolutional layers and one linear layer. The kernel size of the convolutional layers is set to 3×3, the stride is set to 4, and ELU is used as the activation function. The last layer is a linear layer used to calculate the probability, thereby determining the authenticity of the generated data; e) Based on the adversarial generation strategy, the generator network and the discriminator network are trained. The discriminator will define the joint gradient map as real data and perform continuous adversarial learning with the gradient map of the fused image, which is defined as pseudo data. The objective function of GAN is defined as shown in Equation (11): （11） The loss function of the generator is shown in equation (12): （12） in, It is an adversarial loss from the generator-discriminator network; It utilizes pixel-level Euclidean loss optimized using a screening map; This represents texture loss based on gradient maps; and These are the weights for pixel-level loss and texture loss, used to ensure that the three loss functions are equally important; （13） Where M is the number of images in a batch during training; c is the gradient label used by the discriminator to identify real and fake images; ▽ indicates that the Laplacian operator is used to calculate the gradient map; M, Y represent the Y channels of the input MRI and PET images. （14） in, h,w Indicates the first h Line 1 w The column's pixel values; H and W are the image's height and width, respectively; Map 1 , Map 2 This represents the filter map generated by the decision block based on the two input images; （15） The loss function of the discriminator is shown in equation (16): （16） Where 'a' is the label for "fake data", set to 0-0.3; "fake data" is the gradient map of the fused image, formulated as equation (17): （17） b is the label for "real data", set to 0.7-1.2; "real data" comes from the joint gradient map constructed by MRI and PET_y through decision blocks, which is formulated as equation (10). f) Optimize using the Adam optimizer; g) Obtain the trained adversarial generative network model; h) Predict using a network model.

Images