A network model and method for glioma SPECT-MRI image fusion

Abstract

The application belongs to the technical field of medical image fusion, and discloses a network model and method for glioma SPECT-MRI image fusion. The network model is referred to as DEFSD, which promotes image fusion by decomposing exclusive features and saliency masks. Specifically, first, the source SPECT image in the RGB space is converted into the YCbCr space, then the SPECT image and the MRI image in the Y channel are input into the DEFSD, the DEFSD combines the attribute vector and the object feature map, respectively reconstructs the two source images, and determines the exclusive features of the source images, meanwhile, the features of the highlighted regions are utilized by using the saliency mask; then the image output by the trained DEFSD is converted into the RGB space together with the SPECT images in the Cb and Cr channels, and finally, the output fusion image is obtained. The training is carried out on the medical image data set disclosed by Harvard Medical School, and the experimental results show that the DEFSD can produce a fusion image containing a large amount of information unique to each modality, and has obvious groove features in the lesion area, which is beneficial to early diagnosis of glioma.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical image fusion, and particularly relates to a network model and method for glioma SPECT-MRI image fusion. BACKGROUND

[0002] Many people in the world are suffering from brain tumors. As one of the most common malignant brain tumors, glioma is characterized by blurred edges, strong diffusion and unclear grooves of the surrounding tissues of the lesion. The mortality rate of advanced glioma is as high as 95%, however, early diagnosis and treatment can reduce the mortality rate by 25%.

[0003] Medical imaging is the most effective tool for early glioma diagnosis because it has the advantage of providing tumor localization, characteristics and post-treatment information. Different imaging techniques produce medical images with different information. Magnetic resonance imaging (MRI) is an imaging technique that provides soft tissue information specifically and is considered a reference standard for evaluating brain tumors. Because compared with computerized tomography (CT) which highlights bone information, it can provide clearer texture information about the tumor and the surrounding swollen area. However, early tumor cell proliferation does not cause obvious differences in brain tissue. Therefore, it is difficult to diagnose early glioma by referring to MRI images only. Single photon emission computed tomography (SPECT) as an imaging technique that provides functional features, not only can provide information about tumor activity and cell proliferation as positron emission tomography (PET) can, but also can provide key information about the malignancy of glioma. Therefore, for the diagnosis of early glioma, the use of fusion of MRI images and SPECT images can retain the texture features of MRI images while increasing the biological activity features possessed by SPECT images, providing a more comprehensive reference for judging the tumor grade and making relevant treatment plans.

[0004] However, there is a prerequisite for obtaining a glioma fusion image with diagnostic value, that is, to extract as many types of features as possible from the source images with ambiguity and diffusion. In order to facilitate the extraction of different types of features, some traditional methods decompose the features of each modality into different types by manually designed decomposition methods, then fuse the same type of features of different modalities, and finally fuse each type of feature into a fusion image. According to its principle, traditional image fusion methods can be roughly divided into six categories. 1) Pyramid-based methods; 2) Sparse representation-based methods; 3) Wavelet transform-based methods, including dual-tree dual-quaternion wavelet, two-dimensional dual-density wavelet, discrete wavelet, etc.; 4) salient feature-based methods; 5) subspace-based methods; 6) other methods, including contourlet, shearlet, Gabor, etc. However, designing a decomposition method is not only time-consuming, but also difficult to adapt to the different requirements of feature extraction of each modality of medical images. In some deep learning-based methods, different features are decomposed by an encoder, which can solve the problem of manually designed decomposition methods in traditional methods and extract more valuable features. Therefore, the current glioma image fusion usually uses deep learning network. As a representative of the supervised model stage, DenseFuse no longer uses the image decomposition method in the traditional method, but uses the convolutional layer to extract the feature; Hui Lal. proposed a method of adding a pre-trained VGG network to the traditional image fusion model, which improved the ability of the traditional model to extract detailed features. However, the generality of supervised or pre-trained models is poor, and the extraction of exclusive features for each model is also not valued. DDcGAN, MGMDcGAN as a representative of changing fusion network structure, through embedding multiple generators or discriminators, enhanced the ability to extract exclusive features; NestFuse, PMGI as a typical example of optimizing weights, proposed to adjust the weights through spatial and channel attention; FusionDN, U2Fusion improved the ability to extract exclusive features for each mode by designing two types of loss function.

[0005] CSF first proposes to extract the exclusive features by defining them explicitly, and the specific idea is to predict the feature maps obtained by decoding the source image A first, and then replace the feature maps of A one by one with the feature maps obtained by decoding the source image B. The feature map whose probability of being predicted as A is the lowest after replacement is the exclusive feature of A. But the main contribution of CSF is to define and determine the position of the exclusive feature, rather than to better fuse them. EMFusion defines the exclusive feature as the feature that cannot be obtained by reconstructing the features of other modalities, and according to this definition, the MRI image is reconstructed to find its exclusive feature. In addition, EMFusion uses exclusive feature loss to train the network in an unsupervised manner to improve the fusion effect. However, EMFusion can only identify the most unique exclusive feature, and there is more than one exclusive feature due to the imaging method. DRF uses an encoder to separate the attribute vector and shared space feature specific to each modality according to the imaging characteristics of each modality. Then, the fused image is obtained by fusing the attribute vector and spatial feature decoded by each modality, but how to better train DRF is a problem.

[0006] In summary, in order to extract effective features from each modality glioma image and use it for diagnosis, the present application proposes a network model and method for glioma SPECT-MRI image fusion. SUMMARY

[0007] In view of the problems that the typical features of glioma images are difficult to extract due to the entanglement with noise due to their diffuseness and ambiguity, and the tumor edge is difficult to determine due to the swelling around the lesion of glioma images, the present application proposes a network model and method for glioma SPECT-MRI image fusion.

[0008] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0009] The present application provides a network model for glioma SPECT-MRI image fusion, which comprises:

[0010] Two object editors And All include four residual blocks, three convolutional layers and a multi-head self-attention layer MSA, the first layer is a residual block Resblock1, followed by three convolutional layers, three residual blocks Resblock2, and the multi-head self-attention layer is placed at the last layer of the editor, the convolution kernel size of all convolutional layers is 3*3, the normalization method is batch normalization, and the activation function is ReLU;

[0011] Two attribute encoders And Both include six convolutional layers and a global pooling layer, the first five layers are convolutional layers, a 5*5 convolution kernel, down-sampling attribute information, and the activation function is ReLU, the sixth layer global pooling layer can map the attribute information into an attribute vector, and the last convolutional layer compresses the dimension of the attribute vector to 64;

[0012] Two fusion decoders D, each composed of 12 layers, in turn, 4 residual blocks Resblock2, 3 deconvolution blocks, 4 convolutional layers and a tanh function layer;

[0013] Four encoding-decoding encoders enc-dec, enc is composed of three convolutional layers, dec is also composed of three convolutional layers, the input enc image is converted into a feature map with a size of 256*256*96 through the three convolutional layers of enc, and then the feature map is restored to an image with the same shape as the input image through the three deconvolution layers of dec.

[0014] The application also provides a method for glioma SPECT-MRI image fusion based on the above network model, the method comprising the following steps:

[0015] Step 1, color space conversion of the image, specifically: in order to fuse the single-channel MRI image and the three-channel SPECT image, the chrominance information of the SPECT image needs to be separated first, and the conversion from the RGB space to the YCbCr space can be represented as:

[0016]

[0017] Step 2, according to the imaging characteristics of SPECT and MRI modalities, the attribute features and glioma features of the two modalities are respectively decomposed through the attribute encoder and the object encoder, to obtain attribute information A and object information O;

[0018] Step 3, in order to obtain more rich saliency features when fusing the image, a saliency mask k is used to constrain the training of the network model;

[0019] Step 4, in order to further avoid that some exclusive features of the non-salient regions of the two ways are ignored, the exclusive features are determined by reconstructing the image using the attribute vector and the object feature map, and the exclusive feature loss is used to optimize the model during the training process;

[0020] Step 5, constructing the loss function of the network model;

[0021] Step 6, obtaining the glioma SPECT-MRI fusion image through the trained network model.

[0022] Further, the step 2 is specifically:

[0023] source SPECT image s and source MRI image m, belonging to SPECT mode and MRI mode respectively using attribute encoder and attribute information is split from and attribute information obtained is defined as and Since the object information from the two modes has some similarities, the object information from the two modalities with similarities and differences is defined as and using object encoder and separated from and respectively;

[0024] Since the object information mainly contains the spatial and positional information of the glioma tissue, it is displayed in the form of feature map; since the attribute information does not contain spatial information, it is calculated using vector representation method; therefore, the object feature map o m and attribute vector a m of the source image m can be defined as:

[0025]

[0026] Similarly, the object feature map o s and attribute vector a s of the Y channel of the SPECT image s can be defined as:

[0027]

[0028] Further, the step 3 is specifically: first, fuse a m and a s to obtain the fused attribute vector fuse o m and o s to obtain the fused feature map a sm and o sm fuse the fused image I Y by the trained fusion decoder D, and the decoding process of D is as follows:

[0029]

[0030] EGNet is used to obtain the mask k of the MRI image, and and ​The image of k is superimposed on I Y and the image of k is superimposed on m, with and the difference as the surface saliency loss L saliency The surface saliency loss L saliency includes the pixel loss L pixel and the abundance loss L abun , L pixel tends the pixel intensity of the salient region of the fused image to the intensity of the source image, while L abun makes the fused image contain more information of the source image, the present application uses entropy to replace gradient to calculate the amount of information contained in the image, entropy determines the probability distribution of gray levels in the image by analyzing the overall pixel intensity distribution, avoiding the case of being misled by edges like gradient; therefore L saliency can be defined as:

[0031] L saliency = L pixel + ω abun L abun (5)

[0032] where ω abun represents a hyperparameter that controls the trade-off, L pixel and L abun can be defined as:

[0033]

[0034] where H, W represent the height and width of the image respectively, represents element multiplication,

[0035] L abun = EN(I Y ) (7)

[0036] The details of EN(*) are as follows:

[0037]

[0038] where L is the number of gray levels, set to 256, p l is the probability corresponding to the gray level, the larger the EN, the more information contained.

[0039] Further, the step 4 is specifically: first, assuming that the imaging of s and m is in the same patient's brain, a m and o s are sent into the fusion decoder D, and o s is converted into the reconstructed image I s using a m ; similarly, the reconstructed image I m is obtained by using as will be defined as: m transformed; I s and I m are defined as:

[0040]

[0041]

[0042] Next, taking I s and s as an example, the reconstruction loss L recon , I m and m are similar; first, I s and s are simultaneously input into two trained encoders-decoders enc-dec with a connected network structure, the input I m and m of enc-dec are the same as the input I s and s of enc-dec; the output of dec is named I s and s respectively, and the loss function of enc-dec is as follows:

[0043] L enc-dec = L sim (I s ,s)+L sim (I s ',s') (11)

[0044] wherein L sim is a structure loss;

[0045] The feature maps extracted from I s and s by enc are named f rec and f s respectively, then f rec and f s are compared layer by layer; after comparison, the feature with the largest feature difference on the feature map is regarded as the exclusive feature; f rec and f s both have channel numbers belonging to the set N = {N1, N2, …, N n}, in the present application, n = 30 is set, which is about 30% of the number of all channels, and the channel with the largest difference is found by (12):

[0046]

[0047] wherein f c is the unique feature of the cth channel, and ||·||F F represents Frobenius-norm, which is usually used to compare the similarity between real matrices and estimated matrices.

[0048] Further, the step 5 is specifically:

[0049] Deep exclusive feature loss L excl , including object encoding loss L object , attribute encoding loss L attr and reconstruction exclusive feature loss L rec Therefore, L excl is defined as follows:

[0050] L excl = L object + ω attr L attar + ω rec L rec (13)

[0051] Wherein, ω attr and ω rec are hyperparameters for controlling the trade-off;

[0052] L object measures the similarity between the object feature maps separated from the two modal images with the same imaging object, that is, the similarity between o s and o m ,

[0053] L object =||o s -o m || F (14)

[0054] L attar measures the mixing of object information and attribute information, and the application reduces L attar by making the attribute vector as close to the prior Gaussian distribution as possible to separate as much attribute information as possible. Specifically, the application fits the KL divergence of the Gaussian distribution with the KL divergence of the attribute vector distribution from s and m, that is, the KL divergence from the prior Gaussian distribution:

[0055]

[0056] L rec represents the similarity between the exclusive features of each mode obtained by (12) and the corresponding position features of the reconstructed image, including the exclusive feature loss L rec_s of the reconstructed SPECT image, the attribute feature loss L rec_m of the reconstructed MRI image, and L rec is defined as:

[0057] L rec = L rec_s + L rec_m (16)

[0058]

[0059]

[0060] and N * The meaning of (12) is the same, n refers to the total number of corresponding feature maps (0

[0061] In summary, the loss function of the network model is defined as:

[0062] L=L saliency +ω excl L excl (19)

[0063] Where ω excl is a hyperparameter that controls the trade-off.

[0064] Further, the step 6 is specifically: the fused image I Y output from the trained network model, then the image I Y and the images of the Cb and Cr channels of the SPECT are converted back to the RGB space, that is, the glioma SPECT-MRI fusion image YCbCr space to RGB space conversion formula is represented as:

[0065]

[0066] Further, the fusion rule FusionA is: in order to minimize the loss of attribute vectors, the weighted sum method is adopted to fuse the attribute vectors, and the fusion method is as follows:

[0067] a sm =λa s +(1-λ)a m (21)

[0068] Where λ represents a hyperparameter between 0 and 1, when λ=1, the fused image shows SPECT image features, and when λ=0, it shows MRI image features; according to different problems, λ can be adjusted to adjust the fusion effect of the fused image;

[0069] The fusion rule FusionO is: in order to fully fuse the object feature map, the average strategy is used to fuse the object feature map o s and o m , defined as:

[0070]

[0071] Compared with the prior art, the present application has the following advantages:

[0072] The application provides a network model for glioma SPECT-MRI image fusion, referred to as DEFSD, which promotes image fusion by decomposing exclusive features and saliency masks. Based on the above-mentioned model, the application first converts the source SPECT image in RGB space to YCbCr space, then inputs the Y channel SPECT image and the MRI image into DEFSD, which uses an attribute encoder and an object encoder to decompose the two source images into attribute vectors in the attribute domain and object feature maps in the object domain. On the one hand, the application combines the attribute vectors and the object feature maps to reconstruct the two source images respectively using a fusion decoder and determine the exclusive features of the source images. On the other hand, the application uses a saliency mask to highlight the features of the prominent regions, such as the glioma region and the edge. DEFSD is trained in an unsupervised manner using deep exclusive feature loss and shallow saliency loss, and finally the output image of DEFSD is converted to RGB space together with the Cb and Cr channel SPECT images to obtain the final output fusion image. Verification by the medical image dataset published by Harvard Medical School shows that the network model DEFSD and the method of the application can produce a fusion image containing a large amount of information unique to each modality, and have obvious sulcus features in the lesion area, which is beneficial to early glioma diagnosis. BRIEF DESCRIPTION OF DRAWINGS

[0073] Figure 1 The color conversion process diagram of the network model DEFSD of the application.

[0074] Figure 2 The fusion process diagram of the network model DEFSD of the application.

[0075] Figure 3 The training process diagram of the network model DEFSD of the application.

[0076] Figure 4 The structure diagram of the object encoder and . In the figure, Conv represents a convolution layer, the left side of ResBlock Conv is the number of input channels, and the right side is the number of output channels; D is the size of the image block, H is the number of attention heads of multi-attention, s is the stride, and p is the padding.

[0077] Figure 5 The structure diagram of the attribute encoder and . In the figure, Conv is a convolution layer, GAP is a global pooling layer, s is the stride, and p is the padding.

[0078] Figure 6The structure diagram of the fusion decoder D is shown in the figure, where Conv represents a convolutional layer, the Conv of LM-ReLU is a deconvolutional block, s is a span, and p is padding.

[0079] Figure 7 The structure of the encoder-decoder enc-dec is shown in the figure. Conv represents a convolutional layer, and n is the number of channels.

[0080] Figure 8 The qualitative comparison result figure of DEFSD and nine state-of-the-art methods on two pairs of MRI-T2 / SPECT-Tc is shown.

[0081] Figure 9 The qualitative comparison result figure of DEFSD and nine state-of-the-art methods on two pairs of MRI-T2 / SPECT-T1 is shown.

[0082] Figure 10 The quantitative comparison line chart is shown. DETAILED DESCRIPTION

[0083] The technical solutions of the present application will be specifically and concretely described below in combination with the embodiments of the present application and the accompanying drawings. It should be pointed out that, for those skilled in the art, without departing from the principles of the present application, a number of variations and improvements can be made, which should be regarded as falling within the protection scope of the present application.

[0084] A network model for glioma SPECT-MRI image fusion, referred to as DEFSD, comprises:

[0085] Two object editors and As shown in Figure 4 , each of them includes four residual blocks, three convolutional layers and one multi-head self-attention layer MSA, the first layer is a residual block Resblock1, followed by three convolutional layers, three residual blocks Resblock2, and the multi-head self-attention layer is placed at the last layer of the editor. The convolution kernel size of all convolutional layers is 3x3, the normalization method is batch normalization, and the activation function is ReLU; the residual block not only can more fully extract the local features of the image, but also is one of the three most effective network structures for current image fusion.

[0086] Two attribute encoders and As shown in Figure 5 , each of them includes six convolutional layers and one global pooling layer. The first five layers are convolutional layers, one of which has a 5x5 convolutional kernel, and the attribute information is down-sampled. The activation function is ReLU. The sixth layer global pooling layer can map the attribute information into an attribute vector. The last layer is the sixth convolutional layer, which compresses the dimension of the attribute vector to 64. In addition, in order to make the generator better distinguish and The present application to attribute vector A little offset is applied.

[0087] Two fusion decoders D, as Figure 6 shown, are composed of 12 layers, in order, 4 residual blocks Resblock2, 3 deconvolution blocks, 4 convolution layers and a tanh function layer; the object feature map is first compressed in channel number by ResBlock2, and then concatenated with the feature map into a 32x32x48 tile; the high-dimensional feature after concatenation is sent to the following three ResBlock2; followed by three deconvolution blocks, the size of the feature map output by the fourth ResBlock2 is restored to 256x256, in order to avoid losing some details due to deconvolution, the present application also stacks the feature map output by the ResBlock1 of the object encoder to supplement the lost details before the feature map is sent to the next Conv layer. Finally, the feature map is compressed through four convolution layers, and then reconstructed into a fusion image through a tanh function.

[0088] Four encoding decoders enc-dec, as Figure 7 shown, enc is composed of three convolution layers, dec is also composed of three convolution layers, the input enc image is converted into a 256x256x96 size feature map through the three convolution layers of enc, and then the feature map is restored to an image with the same shape as the input image through the three deconvolution layers of dec. Since the stride of enc-dec is 1, the encoding and decoding process only changes the number of feature map channels, and the length and width remain unchanged.

[0089] Taking the fusion of 256x256x3 SPECT image and 256x256x1 MRI-T2 image as an example, the MRI image is a gray-scale image, and the image types commonly used for image fusion are MRI-T1 and MRI-T2. The imaging characteristics of MRI-T1 image are: the longer the T1 signal of the lesion tissue, the weaker the imaging signal; on the contrary, the stronger the imaging signal. The imaging characteristics of MRI-T2 image are: the longer the T2 signal of the lesion tissue, the stronger the imaging signal; on the contrary, the weaker the imaging signal. Glioma tissue appears as long T1 signal and long T2 signal on MRI, so MRI-T2 image is suitable for glioma imaging.

[0090] SPECT is an RGB image, which is suitable for detecting the activity of brain tissue near early glioma. In order to fuse the single-channel MRI-T2 image and the three-channel SPECT image, the chrominance information of the SPECT image is first separated, and then it is fused with the MRI-T2. The common separation method is usually to convert the RGB image to the IHS space and separate it, but there is a problem of image distortion when the MRI-T2 is fused with the I channel. Therefore, the present application selects to convert the SPECT image to the YCrCb space separation, wherein the Y channel is a gray channel representing the brightness of the image, and contains more structure and texture information.

[0091] (1) Color space conversion: the process is as shown in Figure 1 The Cb and Cr channels are both color channels, mainly reflecting the color information of the image. In order to make the SPECT image and the gray image MRI-T2 more fully fused, the present application uses the Y channel of the SPECT image to fuse with the MRI-T2, and obtains I Y at the same time, the Cb and Cr channels are enhanced to retain the color information. The conversion from the RGB space to the YCbCr space can be expressed as:

[0092]

[0093] It is worth noting that if the Y channel SPECT image and the MRI-T2 image have different resolutions, they need to be unified by upsampling before being sent into the DEFSD (in this embodiment, the resolutions of the two are the same).

[0094] (2) The fusion process of DEFSD, as shown in Figure 2 The source SPECT image s and the source MRI-T2 image m belong to the SPECT mode and the MRI-T2 mode respectively. The two modes of images are split to obtain attribute information and object information O. is the attribute information obtained by splitting the attribute encoder and The attribute information obtained from and is defined as O refers to different information carried by the same tumor object due to different modalities, such as texture and structure. However, there are some similarities between these object information from the two modes, for example, the white matter of the glioma patient has unclear texture details. The present application defines the object information from the two modalities with similarities and differences as and and designs object encoders and separate them from and respectively. It is worth noting that the object and attribute encoders of the two modalities are twin networks, with the same structure and different weights and parameters.

[0095] Since the object information mainly contains the spatial and positional information of the glioma tissue, the object information is displayed in the form of feature maps. Since the attribute information does not contain spatial information, it is more convenient to calculate using vector representation. Therefore, the object feature map o m and the attribute vector a m of the source image m can be defined as:

[0096]

[0097] Similarly, the Y channel SPECT image s of o s and a s can be defined as.

[0098]

[0099] (3) In order to obtain more rich saliency features when fusing images, the present application uses a saliency mask k to constrain the training of DEFSD. First, the present application fuses a m and a s by fusion rule FusionA to obtain the fused attribute vector o m and o s are fused by fusion rule FusionO to obtain the fused feature map a sm and o sm are decoded by the trained decoder D to obtain the fused image I Y . The decoding process of D is as follows:

[0100]

[0101] FusionA: In order to minimize the loss of attribute vectors, the present application adopts a weighted summation method to fuse the attribute vectors. The fusion method is as follows:

[0102] a sm = λas +(1-λ)a m (5)

[0103] where λ represents a hyper-parameter between 0 and 1. When λ = 1, the fused image shows SPECT image features, and when λ = 0, it shows MRI image features. Depending on different problems, the fusion effect of the fused image can be adjusted by adjusting λ. In the work of the present application, λ is 0.3.

[0104] FusionO: In order to make the object feature map fully fused, the present application uses an average strategy to fuse the object feature map o s and o m , which is defined as:

[0105]

[0106] Next, the present application uses and to represent the image of k superimposed on I Y and the image of k superimposed on m, respectively, and uses the difference between and as the surface saliency loss L saliency . The saliency loss is introduced into the loss function to prevent the salient texture information in the MRI image from being weakened in the decomposition process, while providing dense structural information for the low signal-to-noise ratio SPECT. And most importantly, the features of the non-salient region covered by k will not be lost. The glioma image can be superimposed with the saliency mask to highlight the structural and texture information within the lesion area. Superimposing k on the glioma image can highlight the structural and texture information of the glioma area. The k mentioned in the present application is obtained by EGNet.

[0107] The surface saliency loss L saliency includes pixel loss L pixel and abundance loss L abun , L pixel tends to make the pixel intensity of the salient region of the fused image tend to the intensity of the source image, and L abun makes the fused image contain more information of the source image, and the present application uses entropy instead of gradient to calculate the amount of information contained in the image, and the entropy determines the probability distribution of gray levels in the image by analyzing the overall pixel intensity distribution, avoiding the case of being misled by edges like gradient; Therefore, L saliency can be defined as:

[0108] L saliency =L pixel +ω abun L abun (7)

[0109] where ω abunL, a hyper-parameter representing the trade-off of control pixel and L abun can be defined as:

[0110]

[0111] where H, W represent the height and width of the image respectively, represents element-wise multiplication,

[0112] L abun = EN(I Y ) (9)

[0113] The details of EN(*) are as follows:

[0114]

[0115] where L is the number of gray levels, set to 256, p l is the probability corresponding to the gray level, the larger the EN, the more information it contains.

[0116] (4) In order to further avoid the exclusive features of the non-significant regions of the two ways being ignored, the application determines the exclusive features by reconstructing the image using the attribute vector and the object feature map, and optimizes the model using the exclusive feature loss in the training process, as shown in Figure 3 First, assuming that the imaging of s and m is in the same patient's brain, the application inputs a m and o s into D, and uses a m to transform o s into the reconstructed image I s . Similarly, the reconstructed image I m is obtained by using a s to transform o m . I s and I m are defined as:

[0117]

[0118]

[0119] Next, the application takes I s and s as an example to describe the reconstruction loss L recon , I m and m are similar to them. First, I s and s are simultaneously input into two trained encoders-decoders enc-dec with a connected network structure. The input I m and m of enc-dec are similar to the input I sand s. The output of dec is named as I s and s. The loss function of enc-dec is as follows:

[0120] L enc-dec = L sim (I s ,s) + L sim (I s ',s') (13)

[0121] where L sim is the structure loss.

[0122] The feature maps extracted from I s and s by enc are named as f rec and f s respectively. Then, f rec and f s are compared layer by layer. After comparison, the feature with the largest feature difference on the feature map is regarded as the exclusive feature. f rec and f s both have channel numbers belonging to the set N = {N1, N2, …, N n}. In this experiment, the present application sets n = 30, which is about 30% of the number of all channels, and finds the channel with the largest difference through (14).

[0123]

[0124] where f c is the unique feature of the cth channel. ||·||F F represents the Frobenius-norm. It is usually used to compare the similarity between real matrices and estimated matrices.

[0125] (5) Construct the loss function of the network model:

[0126] The deep exclusive feature loss L excl includes the object encoding loss L object , the attribute encoding loss L attr , and the reconstructed exclusive feature loss L rec . Therefore, L excl is defined as follows:

[0127] L excl = L object + ω attr L attar + ω rec L rec (15)

[0128] where ω attr and ω recis a hyper-parameter for controlling the trade-off.

[0129] L object The similarity between the object feature maps separated from the two modal images with the same imaging object, i.e., o s and o m is measured.

[0130] Lo bject =||o s -o m || F (16)

[0131] L attar The object information is mixed with the attribute information, and the present application reduces L attar by making the attribute vectors as close to the prior Gaussian distribution as possible to split the attribute information as much as possible. Specifically, the present application fits the KL divergence of the Gaussian distribution with the KL divergence of the attribute vector distribution from s and m, i.e., the KL divergence from the prior Gaussian distribution:

[0132]

[0133] L rec represents the similarity between the exclusive features of each mode obtained by (14) and the corresponding position features of the reconstructed image, including the exclusive feature loss L rec_s of the reconstructed SPECT image, the attribute feature loss L rec_m of the reconstructed MRI image; L rec is defined as:

[0134] L rec = L rec_s + L rec_m (18)

[0135]

[0136]

[0137] and N * have the same meaning as (14). n refers to the total number of corresponding feature maps (0

[0138] In summary, the loss function of DEFSD is defined as:

[0139] L = L saliency + ω excl L excl (21)

[0140] where ω excl is a hyper-parameter for controlling the trade-off.

[0141] (6) The fused image I is output from the DEFSD Y , I Y is converted back to the RGB space together with the images of the Cb and Cr channels to obtain a glioma SPECT-MRI fusion image. The conversion formula from the YCbCr space to the RGB space is expressed as:

[0142]

[0143] The training data set and the test data set of the DEFSD are all from the Harvard Medical School. For the fusion of SPECT and MRI-T2 images, the present application selects 350 pairs of images and crops them into more than 20,000 pairs of images. The present application fixes the dimension of the attribute vector to 128 and uses the Adam optimizer to optimize the model, and the implementation platform is TensorFlow. The training parameters are set as follows: the batch size is 4, the learning rate is set to 0.0001, and the epoch is set to 4. abun = 7, ω attr = 0.001, ω rec = 0.8, ω comp = 10, λ = 0.3. The experiment is carried out on an RTX3060Ti 8GB GDR6 GPU and an i7-11700 memory 16G CPU.

[0144] In order to intuitively understand the image fusion effect of the present application, the present application selects nine advanced image fusion methods for comparison, including DDcGAN, MGMDcGAN and NestFuse with improved network structure; EMFusion, U2Fusion and VIF-Net with improved training method; and STDFusionNet, CSF and DRF with improved fusion rule. Since the comparison methods NestFuse, VIF-Net, STDFusionNet, CSF and DRF are single-channel image fusion methods based on infrared and visible light images, when the present application uses these methods, the input DEFSD image is the Y channel of the SPECT image and the MRI image in the YCbCr space. Finally, the DEFSD output gray-scale fusion image is converted back to the RGB space together with the Cb and Cr two channels to obtain a fusion image in the RGB space.

[0145] To demonstrate the effect of DEFSD in fusing MRI-T2 images with glioma SPECT images, we selected eight pairs of brain images of glioma patients, four of which were MRI-T2 / SPECT-Tc images and the other four were MRI-T2 / SPECT-T1 images. The reason for selecting the above two types of images for fusion with MRI images was to ensure the robustness of the experiment, in which SPECT-Tc images can complement the biological activity information of the fused images, and SPECT-T1 images can ensure the stability of the structural features in the fused images. The four pairs of MRI-T2 / SPECT-T1 images were the 12th and 52nd images of two patients, respectively. The four pairs of MRI-T2 / SPECT-Tc images were the 34th and 30th images of two patients, respectively.

[0146] Figure 8 Qualitative comparisons of the fusion of two pairs of MRI-T2 / SPECT-T1 images are shown. According to Figure 8 Four significant advantages of DEFSD can be found: first, the fused images of DEFSD can accurately depict the location and shape of gliomas. Second, the images fused with DEFSD have clear color gradations, which can clearly depict the activity of soft tissues in the brain. Third, the details of the bone can be observed in the fused images. Fourth, the fused images of DEFSD reduce the mosaic caused by the low resolution of SPECT.

[0147] The fusion results obtained by other methods have problems such as blurred bone edges, unclear textures due to too low brightness or contrast, and loss of unique features of each mode, which makes it difficult for doctors to determine the information of the lesion. In contrast, the fused images obtained by DEFSD show clear bone structures, as well as clear edge information and texture details of soft tissues. Overall, the visual effect of the DEFSD images is better than that of the images obtained by the other nine fusion methods.

[0148] Figure 9 Qualitative comparisons of the fusion of two pairs of MRI-T2 / SPECT-Tc images are shown. According to Figure 9 In addition to the four advantages of the fused images of DEFSD, which are the same as those of the fused images of MRI and SPECT-Tc, the fused images of DEFSD can also describe the biological activity of the lesion area and the biological sensitivity of the bone edge.

[0149] In addition, MRI images are used to describe the texture details in the fused images, and SPECT-Tc can highlight the biological activity of the lesion area. In Figure 9In the MRI-T2 images of the other competitors, texture features in the fused images were not clearly represented. For example, the fused images from U2Fusion showed mosaic residues in the glioma region. In contrast, the fused images from DEFSD not only displayed biological activity in different regions but also revealed clear sulci in the brain. As a result, the visual effect of the DEFSD fused images was closer to the source images than the images obtained by the other nine fusion methods.

[0150] In order to objectively evaluate the fusion effect of DEFSD, the present invention selects peak signal-to-noise ratio (PSNR), structural similarity (SSIM), spatial frequency (SF), edge information transfer coefficient (Qab / f), nonlinear correlation information entropy (NCIE) and normalized mutual information (NMI), among which PSNR represents the ratio of peak power to noise power in the fused image. SSIM is used to measure the brightness, contrast and structure of the fused image and the source image. SF reflects the overall activity of the image in the spatial domain. Qab / f is a gradient-based measurement that links important visual information to the "edge" information present in each pixel of the image. NCIE represents the nonlinear correlation between the source image and the fused image. NMI represents the similarity of the grayscale distribution of the source image and the grayscale distribution of the fused image. The higher the values ​​of the six evaluation indicators, the more similar the source image is to the fused image, and the better the fusion effect.

[0151] The present invention randomly selected 20 pairs of glioma fusion images and conducted quantitative comparisons with other comparison methods. The line graph of the quantitative comparison results is shown in Figure 10 . DEFSD achieved the best performance on PSNR, SSIM, SF, Qab / f and NCIE, and achieved suboptimal performance on NMI. The reason for the suboptimal performance is that since the source SPECT image is a low-resolution RGB image, it is difficult to ensure the grayscale similarity with the MRI image while ensuring the grayscale similarity between the fused image and the SPECT image. However, the competitor's source images have the same resolution, and their image grayscale similarity will be higher. But from the fusion of low-resolution RGB SPECT images and high-resolution MRI images, DEFSD can retain more color levels and structural information.

Claims

1. A network model for glioma SPECT-MRI image fusion, characterized in that: The network model comprises: Two object editors and Both include four residual blocks, three convolutional layers and a multi-head self-attention layer MSA, the first layer is a residual block Resblock1, followed by three convolutional layers, three residual blocks Resblock2, and the multi-head self-attention layer is placed at the last layer of the editor. The convolution kernel size of all convolutional layers is 3x3, the normalization method is batch normalization, and the activation function is ReLU. Two attribute encoders and Both include six convolutional layers and a global pooling layer, the first five layers are convolutional layers with a 5x5 convolutional kernel, down-sampling attribute information, and the activation function is ReLU. The sixth layer global pooling layer can map the attribute information into an attribute vector, and the last convolutional layer compresses the dimension of the attribute vector to 64. Two fusion decoders D, each consisting of 12 layers, in turn, 4 residual blocks Resblock2, 3 deconvolution blocks, 4 convolution layers and a tanh function layer; Four encoding decoders enc-dec, enc consisting of three convolution layers, dec also consisting of three convolution layers, the input enc image is converted into a feature map of 256x256x96 size through the three convolution layers of enc, and then the feature map is restored to an image with the same shape as the input image through the three deconvolution layers of dec.

2. A method for glioma SPECT-MRI image fusion based on the network model of claim 1, characterized in that, The method comprises the following steps: Step 1, color space conversion of the image, specifically: in order to fuse the single-channel MRI image and the three-channel SPECT image, the chrominance information of the SPECT image needs to be separated first, and the conversion from the RGB space to the YCbCr space can be expressed as: （1）； Step 2, according to the imaging characteristics of SPECT and MRI modalities, the attribute features and object features of the two modalities are respectively decomposed by attribute encoders and object encoders, respectively, to obtain attribute information and object information ; Step 3, in order to obtain more rich saliency features when fusing images, a saliency mask k is used to constrain the training of the network model; Step 4, in order to further avoid that some exclusive features of the non-salient regions of the two ways are ignored, exclusive features are determined by using attribute vectors and object feature maps to reconstruct images, and exclusive feature loss is used to optimize the model during the training process; Step 5, constructing the loss function of the network model; Step 6, obtaining the glioma SPECT-MRI fusion image through the trained network model.

3. The method for glioma SPECT-MRI image fusion according to claim 2, characterized in that, The step 2 is specifically: source SPECT image s and source MRI image m, belonging to SPECT modality and MRI modality respectively ; using attribute encoder , attribute information is split from and , the attribute information obtained from and is defined as and ; since the object information from the two modalities has some similarities, the object information from the two modalities with similarities and differences is defined as and , using object encoder and , they are separated from and respectively; Since the object information mainly contains the spatial and location information of the glioma tissue, it is displayed in the form of feature maps; since the attribute information does not contain spatial information, it is calculated using a vector representation method; therefore, the object feature map of the source image m and the attribute vector can be defined as: （2） Likewise, the object feature map of the Y channel of the SPECT image s and the attribute vector may be defined as: （3）。 4. The method for glioma SPECT-MRI image fusion according to claim 2, characterized in that, The step 3 is specifically as follows: First, the fusion rule FusionA is used to fuse and , get the fused attribute vector , through the fusion rule FusionO fusion and , get the fused feature map ; and The fused image is decoded by the trained fusion decoder D , the decoding process of D is as follows: （4）； The trained EGNet is used to obtain the mask k of the MRI image, and the image of k is superimposed on m to obtain the image of k and respectively, and the image of k is superimposed on m to obtain the image of k and k and The difference between and is used as the surface saliency loss The surface saliency loss includes pixel loss and abundance loss , The pixel intensity of the salient region of the fusion image tends to the intensity of the source image, while The fusion image contains more information of the source image, and the amount of information contained in the image is calculated by replacing the gradient with entropy, which determines the probability distribution of gray levels in the image by analyzing the overall pixel intensity distribution, avoiding the situation of being misled by edges like gradient; therefore It can be defined as: （5） wherein, a hyperparameter that represents a control trade-off, and may be defined as: （6） where H, W represent the image length and width, respectively, representative element multiplication, （7） The details of EN(*) are as follows: （8） where L is the number of gray levels, set to 256, is the probability of the corresponding gray level, the greater EN, the more information contained.

5. The method for glioma SPECT-MRI image fusion according to claim 2, characterized in that, The step 4 is specified as follows: first, assuming that the imaging of s and m are in the same patient's brain, we have and are sent into the fusion decoder D and using we have transformed into the reconstructed image ; similarly, the reconstructed image is obtained by transforming and are defined as follows:​​ （9） （10） Next, we describe the reconstruction loss for the case of , and m are similar; first, and s are simultaneously input into two well-trained encoder-decoders enc-dec with a Siamese network structure. The input of enc-dec is and m are similar; first, and s are simultaneously input into two well-trained encoder-decoders enc-dec with a Siamese network structure. The input of enc-dec is and The loss function of enc-dec is as follows: （11） wherein is the structural loss; The enc is extracted from The feature maps extracted from the and s are respectively named as and Then, the and and are compared layer by layer; after comparison, the feature with the largest feature difference on the feature map is regarded as the exclusive feature; and Both have channel serial numbers belonging to the set , and n=30 is set as 30% of the number of all channels, and the channel with the largest difference is found through (12): （12） wherein is a unique feature of the cth channel, denotes the Frobenius-norm, often used to compare the similarity between real and estimated matrices.

6. The method for glioma SPECT-MRI image fusion according to claim 5, characterized in that, The step 5 is specifically: Deep dedicated feature loss , including object encoding loss , attribute encoding loss and reconstruction dedicated feature loss Thus, is defined as follows: （13） wherein, and are hyperparameters for controlling the trade-off. measuring the similarity between object feature maps separated from two modal images having the same imaged object, i.e. and between （14） The object information is mixed with the attribute information, reducing To split the attribute information as much as possible, specifically, fitting the KL divergence of the Gaussian distribution with the KL divergence of the attribute vector distribution from s and m, i.e., the KL divergence from the prior Gaussian distribution: （15） representing the similarity between the exclusive features of each modality obtained from (12) and the corresponding location features of the reconstructed image, including the exclusive feature loss of the reconstructed SPECT image , the attribute feature loss of the reconstructed MRI image , is defined as: （16） （17） （18） , , , and have the same meaning as (12), n refers to the total number of corresponding feature maps, ; In summary, the loss function of the network model is defined as: （19） wherein, is a hyperparameter that controls the trade-off.

7. The method for glioma SPECT-MRI image fusion according to claim 2, characterized in that, The step 6 is specifically: outputting the fused image I from the trained network model Y Then the image I Y is converted back to the RGB space together with the images of the Cb and Cr channels of the SPECT, that is, the glioma SPECT-MRI fusion image Y is obtained, and the conversion formula from the YCbCr space to the RGB space is represented as: （20）。 8. The method for glioma SPECT-MRI image fusion according to claim 4, characterized in that: The fusion rule FusionA is: in order to minimize the loss of attribute vectors, a weighted summation method is adopted to fuse the attribute vectors, and the fusion method is as follows: （21） wherein, represents a hyper parameter between 0 and 1, when =1, the fusion image shows SPECT image features, when =0, it shows MRI image features; according to different problems, the fusion effect of the fusion image can be adjusted by adjusting ; The fusion rule FusionO is: in order to make the object feature map fully fused, the object feature map is fused by using an averaging strategy and , defined as: （22）。

Images