Multi-modal medical image fusion method based on image denoising and illumination enhancement

Abstract

The invention relates to a multi-modal medical image fusion method based on image denoising and illumination enhancement. The method comprises the following steps: inputting a multi-modal medical image into an illumination and contrast enhancement network for illumination intensity correction to generate an enhanced image; adding random noise to the multi-modal medical image to obtain a noise image; inputting the noise image into a denoising and reconstruction network to carry out denoising and reconstruction tasks; using the enhanced image and the multi-modal medical image as labels to jointly supervise and train a denoising and reconstruction network to obtain a feature encoder and a feature decoder with denoising and illumination enhancement functions, using the trained feature encoder to respectively extract features of different modal images to obtain modal feature maps, and using the modal feature maps to obtain the multi-modal medical image; and carrying out feature fusion on the modal feature maps of the different modal images, and reconstructing the fused feature maps by adopting a trained feature decoder to obtain a multi-modal medical image fusion result. The problems that image fusion is interfered by noise, illumination is degraded and fusion efficiency is low are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal image fusion, and in particular to a multimodal medical image fusion method based on image denoising and illumination enhancement. Background Art

[0002] Multimodal medical image fusion methods are usually divided into traditional image fusion methods and image fusion methods based on deep learning. Common traditional fusion methods are based on spatial domain, transform domain, sparse representation and traditional models. Among them, spatial domain fusion involves combining segmented image blocks, which will cause artifacts in the fused image. The transform domain method first transforms the original image, then fuses the obtained information layers, and finally reconstructs the fused image through inverse transformation. Based on sparse representation, it is necessary to create a dictionary to represent the weight information, and then continuously update the dictionary to generate a fused image. Fusion methods based on traditional models often need to select appropriate optimization schemes to continuously optimize the model. Multimodal fusion methods based on deep learning use convolutional neural network models to extract and represent features, effectively improving feature robustness and model generalization. In addition, deep learning models such as generative adversarial networks and visual transformers are also often used for image fusion.

[0003] In multimodal medical image fusion, the fusion methods often used for noisy images are mainly based on traditional variational models and sparse representation methods. However, these two fusion and denoising methods have certain limitations: most of the existing fusion and denoising methods are based on traditional variational models and sparse representation methods, which are essentially traditional fusion methods. They require manual adjustment of a large number of parameters according to specific scenes, and the efficiency of fusion and denoising is low. The illumination attenuation during the fusion process will cause texture blur and color distortion in the fusion results. Although some existing fusion methods will enhance the illumination during the fusion process, they cannot effectively remove noise while enhancing the illumination, resulting in poor robustness and generalization, and cannot effectively remove random noise in real scenes, and the denoising effect of the fused image is poor. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention proposes a multimodal medical image fusion method based on image denoising and illumination enhancement to solve the problems that multimodal images are susceptible to noise interference, illumination degradation, and low fusion efficiency. The method includes:

[0005] S1: Input the multimodal medical image into the illumination and contrast enhancement network to perform illumination intensity correction to generate an enhanced image;

[0006] S2: Add random noise to the multimodal medical image to obtain a noisy image; construct a denoising and reconstruction network and input the noisy image into the denoising and reconstruction network to perform denoising and reconstruction tasks; the denoising and reconstruction network extracts noise features through a feature encoder and generates a reconstructed image through a feature decoder;

[0007] S3: Use enhanced images and multimodal medical images as labels to jointly supervise the training of denoising and reconstruction networks to obtain feature encoders and feature decoders with denoising and illumination enhancement functions;

[0008] S4: Use the trained feature encoder to extract the features of different modal images respectively to obtain the modal feature map, use the LGLE or GLME fusion rule to fuse the modal feature maps of different modal images, and use the trained feature decoder to reconstruct the fused feature map to obtain the multimodal medical image fusion result.

[0009] The present invention has at least the following beneficial effects

[0010] The present invention jointly trains an illumination and contrast enhancement network and a denoising and reconstruction network, and uses enhanced images and multimodal medical images as labels to jointly supervise the training of denoising and reconstruction networks, so that the model learns a feature encoder and a feature decoder with denoising and illumination enhancement, which can effectively remove random noise in the image and improve the robustness and generalization performance of the model. The feature encoder and the feature decoder respectively realize effective removal of random noise and enhancement of illumination in the process of feature extraction and feature reconstruction, solve the problems of noise interference and illumination degradation, and improve the quality of image fusion. At the same time, the present invention designs two LGLE fusion rules and GLME fusion rules according to the characteristics of different fusion tasks, that is, the features of fusion between different modal images, increases the retention of semantic information such as texture details, physiological metabolism and significant targets, solves the problem of key semantic information loss, avoids the change of pixel distribution between different feature maps, and causes the problem of information loss in the feature reconstruction process. Compared with the fusion method based on traditional variational models and sparse representation, the present invention has higher fusion efficiency and effectively alleviates the problem of illumination decay in the fusion process. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a schematic diagram of the model framework of the present invention;

[0012] Figure 2 Schematic diagram of the architecture of the illumination estimation module of the present invention;

[0013] Figure 3 Schematic diagram of the architecture of the feature encoder of the present invention;

[0014] Figure 4 Schematic diagram of the architecture of the feature decoder of the present invention. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the specific invention embodiments to clearly and completely describe the technical solutions in the embodiments of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0016] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0017] See also Figure 1 to Figure 4 , a multimodal medical image fusion method based on image denoising and illumination enhancement of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0018] S1: Input the multimodal medical image into the illumination and contrast enhancement network to perform illumination intensity correction to generate an enhanced image;

[0019] Preferably, the illumination and contrast enhancement network comprises: an illumination estimation module, a Fourier transform module;

[0020] The illumination estimation module is used to perform illumination estimation on the input multimodal medical image to obtain an estimated illumination; and based on the Retinex theory, the multimodal medical image is divided by the estimated illumination to obtain a corrected image of the multimodal medical image;

[0021] The Fourier transform module includes: a fast Fourier transform module, a high-pass filter module and a fast Fourier inverse transform module;

[0022] The fast Fourier transform module is used to perform Fourier transform on the multimodal medical image and the corrected image to generate corresponding frequency spectrograms;

[0023] The high-pass filtering module is used to perform high-pass filtering on the spectrum of the multimodal medical image to obtain a high-frequency image with high-frequency detail information;

[0024] The fast inverse Fourier transform module is used to superimpose the frequency spectrum of the corrected image and the high-frequency image, and perform inverse Fourier transform on the superimposed frequency spectrum to obtain an enhanced image.

[0025] Preferably, the process of acquiring multimodal medical images includes: magnetic resonance imaging (MRI) and computed tomography (CT) images of acute stroke cases and glioma cases; magnetic resonance imaging (MRI) and single photon emission computed tomography (SPECT) images of Alzheimer's disease and subacute stroke; normal disease-free magnetic resonance imaging (MRI) and positron emission tomography (PET), etc.

[0026] Specifically, the illumination and contrast networks use a pre-trained illumination estimation module ( Figure 1 The illumination estimation module is used to estimate the image illumination. The illumination estimation module is composed of three conventional convolutional layers cascaded through residual connections; the conventional convolutional layers include a 3×3 convolution kernel, a ReLu activation layer or a Sigmoid activation layer (such as Figure 2 As shown); then, based on the Retinex theory, the multimodal medical image is divided by the estimated illumination to obtain a corrected image of the multimodal medical image;

[0027] Specifically, the Fourier transform module performs fast Fourier transform on the original image and the corrected image respectively and generates a frequency domain map, uses high-pass filtering to operate on the frequency domain map of the multimodal medical image and generates a high-frequency spectrum map, which represents the high-frequency texture information in the original image; the high-frequency spectrum map and the spectrum map of the corrected image are superimposed and the superimposed spectrum map is subjected to inverse fast Fourier transform to obtain an enhanced image with enhanced illumination and contrast, hereinafter referred to as the enhanced image.

[0028] S2: Add random noise to the multimodal medical image to obtain a noisy image; construct a denoising and reconstruction network and input the noisy image into the denoising and reconstruction network to perform denoising and reconstruction tasks; the denoising and reconstruction network extracts noise features through a feature encoder and generates a reconstructed image through a feature decoder;

[0029] Preferably, the random noise includes: Gaussian noise, salt and pepper noise, Poisson noise or speckle noise with a standard deviation of random size.

[0030] Preferably, the feature encoder is composed of N conventional convolutional layers in cascade; the conventional convolutional layer includes a cascaded 3×3 convolution kernel, a ReLu activation layer and a BN layer; and the feature decoder adopts a reversible neural network model.

[0031] like Figure 1As shown, a multimodal medical image is converted into a grayscale image and random noise is added; the adding of random noise to the original multimodal image includes: adding Gaussian noise, salt and pepper noise, Poisson noise, and speckle noise with a standard deviation of random size to the original multimodal image. A denoising and reconstruction network is constructed and the noisy image is input into the denoising and reconstruction to perform image denoising and reconstruction tasks; the denoising and reconstruction network extracts more representative noise features through a feature encoder, and generates a reconstructed image through a feature decoder;

[0032] Specifically, the feature encoder consists of N conventional convolutional layers, which include: a 3×3 convolution kernel, a ReLu activation layer, and a BN layer. Figure 3 The input noise image tensor dimensions are B, C, H, W; where B is the batch number, C is the number of channels, H is the number of feature map rows, and W is the number of feature map columns. In this case, C is 1. The first convolution layer maps the input feature map of C = 1 to C = 64, and B, H, and W remain unchanged. The second to ninth convolution layers keep C = 64.

[0033] Preferably, the feature decoder uses a reversible neural network model to reconstruct the fused feature map.

[0034] Specifically, the feature decoder uses a reversible neural network model for mapping and reconstruction of feature maps, see Figure 4 The process through the feature decoder can be expressed as:

[0035]

[0036] Among them, I r Represents the reconstructed image, Convi(i=1,2,3,4) represents the convolution layer with a convolution kernel of 3×3, Represents the feature map on the 1st to (C / 2)th dimension channel in the feature map extracted by the encoder, It represents the feature map on the (C / 2+1)th to Cth dimension channels in the feature map extracted by the encoder, and exp represents the exponential function calculation.

[0037] S3: Use enhanced images and multimodal medical images as labels to jointly supervise the training of denoising and reconstruction networks to obtain feature encoders and feature decoders with denoising and illumination enhancement functions;

[0038] Preferably, the method of using the enhanced image and the multimodal medical image as labels to jointly supervise the training of the denoising and reconstruction network includes: using the enhanced image as a training pseudo-label, constructing a first loss function based on the L2 norm between the enhanced image and the reconstructed image; constructing a second loss function based on the gradient information difference between the reconstructed image and the input multimodal medical image; constructing a third loss function based on the L2 norm between the reconstructed image and the enhanced image; constructing a total loss function of the model based on the first loss function, the second loss function and the third loss function, and tuning the parameters of the denoising and reconstruction network based on the constructed total loss function.

[0039] Specifically, a loss function for denoising and reconstruction network training is constructed, which can be expressed as:

[0040]

[0041] in, represents the reconstruction loss completed during the denoising and reconstruction network training process, represents the content loss and is used to calculate the L2 norm of the reconstructed image and the input multimodal medical image; Represents the gradient loss and is used to calculate the gradient information difference between the reconstructed image and the input multimodal medical image; represents the driving loss and is used to calculate the L2 norm between the enhanced image and the reconstructed image; γ, η, ξ represent the balancing factors.

[0042] Specifically, the illumination and contrast enhancement network forms a joint training method with the denoising and reconstruction network to train the feature encoder and feature decoder in the denoising and reconstruction network, so that the feature encoder and feature decoder have the functions of removing noise and enhancing illumination when performing image feature extraction and reconstruction.

[0043] S4: Use the trained feature encoder to extract the features of different modal images respectively to obtain the modal feature map, fuse the modal feature maps of different modal images, and use the trained feature decoder to reconstruct the fused feature map to obtain the multimodal medical image fusion result.

[0044] Specifically, the trained feature encoder with denoising and illumination enhancement functions extracts the features of noise images of different modes respectively to obtain image feature maps. According to different fusion tasks, fusion rules are selected to fuse the feature maps to generate fused feature maps. The trained feature decoder is used to reconstruct the fused feature maps to obtain the fusion results. Please refer to Figure 1 Multimodal image fusion process.

[0045] Preferably, the step of fusing the modal feature maps of the different modal images using the LGLE or GLME fusion rule comprises: when the input multimodal medical image is an MRI image and a CT image, fusing the modal feature maps of the different modal images using the LGLE fusion rule;

[0046] When the input multimodal medical images are MRI images and PET images, or MRI images and SPECT images, the GLME fusion rule is used to fuse the modal feature maps of different modal images.

[0047] Specifically, the fusion rule of MRI images and CT images adopts the LGLE fusion rule; the specific steps of LGLE are as follows:

[0048] Step 1: Use L1-Norm channel superposition to integrate the channel information of the two modal feature maps and generate the corresponding single-pass feature map. The process is expressed as:

[0049]

[0050] Among them, N k represents a single channel feature map, represents the modal feature map, k=1 or k=2 represents different modes, and N represents the number of feature map channels;

[0051] Step 2: Use the Sobel gradient operator to calculate the single-channel feature maps N of the two modes respectively k The gradient information in the horizontal and vertical directions generates the corresponding gradient map G k , and calculate the local gradient activity level map GL for the gradient map k , the process is expressed as:

[0052]

[0053] Among them, W l Indicates a convolution with a convolution kernel of 3×3, W l =[0,1,0;1,1,1;0,1,0];

[0054] Step 3: Use the local energy maximum method to calculate the feature information of the high signal area in the single-channel feature map of the two modalities and generate the local energy map E k :

[0055]

[0056] Among them, W e Indicates a convolution with a convolution kernel of 3×3, W e =[1,1,1; 1,1,1; 1,1,1];

[0057] Step 4: Combine the local gradient activity level map and the local energy map to calculate the corresponding weight map M k , and combine the two modal feature maps according to the weight maps corresponding to the two modalities to generate a fusion feature map

[0058]

[0059] In the LGLE fusion algorithm, L1-Norm integrates the information in the channel, and the gradient information and local energy of the feature map are calculated to help extract the MRI detail texture information and the skull area with dense structure in the CT image, respectively, and enhance the retention of texture details and brightness information in the fused image;

[0060] Preferably, the fusion rules of the MRI image and the PET image, or the MRI image and the SPECT image are consistent, and the GLME fusion rule is adopted. The specific steps of GLME are as follows:

[0061] Step 1: Use the Sobel gradient operator to calculate the gradient information of the two modal feature maps in the horizontal and vertical directions respectively to generate the gradient map G k , calculate the local gradient activity level map GL for the gradient map k Please refer to step 2 of LGLE fusion rule;

[0062] Step 2: Use the local gradient average energy operator to calculate two sets of different modal feature maps respectively Local average energy information graph ME k :

[0063]

[0064] Among them, W m Indicates a convolution with a convolution kernel of 3×3, W m =1 / 16×[1,2,1;2,4,2;1,2,1]

[0065] Step 3: Standardize the local gradient activity level map and local energy map of the same mode and add them together to obtain the weight map C k :

[0066] C k =Norm(GL k )+β×Norm(GL k )

[0067] Among them, Norm means standardization, and β means the balance factor;

[0068] Step 4: Use the maximum decision method to calculate the binary decision score map M for the weight maps of different modes; combine the feature maps of different modes according to the results of the binary decision score map to obtain the fusion feature map

[0069]

[0070] The GLME fusion rule helps to capture the texture details in MRI images and the physiological metabolism in PET or SPECT images, and effectively retains these significant medical semantic information in the fused images.

[0071] In summary, the present invention jointly trains an illumination and contrast enhancement network and a denoising and reconstruction network, uses enhanced images and multimodal medical images as labels to jointly supervise the training of denoising and reconstruction networks, and trains feature encoders and feature decoders with denoising and illumination enhancement, which can effectively remove random noise in images and improve the robustness and generalization performance of the model; uses pre-trained feature encoders and decoders to extract features of medical images of different modalities, respectively, to obtain modality feature maps; performs feature fusion of feature maps of different modalities through LGLE fusion rule or GLME fusion rule to obtain a fused feature map; performs feature map reconstruction of the fused feature map through the trained feature decoder to obtain a multimodal medical image fusion result; in this process The feature encoder and feature decoder remove noise and enhance illumination in the process of feature extraction and reconstruction respectively, solve the problems of noise interference and illumination degradation, and improve the fusion quality of the image; at the same time, the present invention designs two LGLE fusion rules and GLME fusion rules according to the characteristics of different fusion tasks, that is, the features of fusion between different modal images, increases the retention of semantic information such as texture details, physiological metabolism and significant targets, solves the problem of key semantic information loss, avoids the change of pixel distribution between different feature maps, and causes the problem of information loss in the feature reconstruction process. Compared with the fusion method based on traditional variational model and sparse representation, the present invention has higher fusion efficiency and effectively alleviates the problem of illumination decay in the fusion process.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A multimodal medical image fusion method based on image denoising and illumination enhancement, characterized in that: The following steps are involved: S1: Input the multimodal medical image into the illumination and contrast enhancement network to perform illumination intensity correction to generate an enhanced image; S2: adding random noise to the multimodal medical image to obtain a noisy image; constructing a denoising and reconstruction network and inputting the noisy image into the denoising and reconstruction network for denoising and reconstruction tasks; The denoising and reconstruction network extracts noise features through a feature encoder and generates a reconstructed image through a feature decoder; S3: Use enhanced images and multimodal medical images as labels to jointly supervise the training of denoising and reconstruction networks to obtain feature encoders and feature decoders with denoising and illumination enhancement functions; S4: Use the trained feature encoder to extract the features of different modal images respectively to obtain the modal feature map, use the LGLE or GLME fusion rule to fuse the modal feature maps of different modal images, and use the trained feature decoder to reconstruct the fused feature map to obtain the multimodal medical image fusion result.

2. The multimodal medical image fusion method based on image denoising and illumination enhancement according to claim 1, characterized in that: The illumination and contrast enhancement network includes: an illumination estimation module and a Fourier transform module; The illumination estimation module is used to perform illumination estimation on the input multimodal medical image to obtain an estimated illumination; and based on the Retinex theory, the multimodal medical image is divided by the estimated illumination to obtain a corrected image of the multimodal medical image; The Fourier transform module includes: a fast Fourier transform module, a high-pass filter module and a fast Fourier inverse transform module; The fast Fourier transform module is used to perform Fourier transform on the multimodal medical image and the corrected image to generate corresponding frequency spectrograms; The high-pass filtering module is used to perform high-pass filtering on the spectrum of the multimodal medical image to obtain a high-frequency image with high-frequency detail information; The fast inverse Fourier transform module is used to superimpose the frequency spectrum of the corrected image and the high-frequency image, and perform inverse Fourier transform on the superimposed frequency spectrum to obtain an enhanced image.

3. The multimodal medical image fusion method based on image denoising and illumination enhancement according to claim 1, characterized in that: The random noise includes: Gaussian noise, salt and pepper noise, Poisson noise or speckle noise with a standard deviation of random size.

4. The multimodal medical image fusion method based on image denoising and illumination enhancement according to claim 1, characterized in that: The feature encoder is composed of N conventional convolutional layers in cascade; the conventional convolutional layer includes a cascaded 3×3 convolution kernel, a ReLu activation layer and a BN layer; the feature decoder adopts a reversible neural network model.

5. The multimodal medical image fusion method based on image denoising and illumination enhancement according to claim 1, characterized in that: The method of using enhanced images and multimodal medical images as labels to jointly supervise the training of denoising and reconstruction networks includes: using the enhanced image as a training pseudo-label, constructing a first loss function based on the L2 norm between the enhanced image and the reconstructed image; constructing a second loss function based on the gradient information difference between the reconstructed image and the input multimodal medical image; constructing a third loss function based on the L2 norm between the reconstructed image and the enhanced image; constructing a total loss function of the model based on the first loss function, the second loss function and the third loss function, and tuning the parameters of the denoising and reconstruction network based on the constructed total loss function.

6. The multimodal medical image fusion method based on image denoising and illumination enhancement according to claim 1, characterized in that: The step of fusing the modal feature maps of images of different modalities using the LGLE or GLME fusion rule includes: When the input multimodal medical images are MRI images and CT images, the LGLE fusion rule is used to fuse the modal feature maps of different modal images; When the input multimodal medical images are MRI images and PET images, or MRI images and SPECT images, the GLME fusion rule is used to fuse the modal feature maps of different modal images.

7. The multimodal medical image fusion method based on image denoising and illumination enhancement according to claim 6, characterized in that: The LGLE fusion rules include: The L1-Norm channel superposition method is used to integrate the channel information of the two modal feature maps and generate the corresponding single-pass feature map; The Sobel gradient operator is used to calculate the gradient information of the single-pass feature maps of the two modes in the horizontal and vertical directions, generate the corresponding gradient map, and calculate the local gradient activity level map of the gradient map; The local energy maximum method is used to calculate the feature information of the high signal area in the single-channel feature map of the two modalities to generate a local energy map; The corresponding weight map is calculated by combining the local gradient activity level map and the local energy map, and the two modal feature maps are combined according to the weight maps corresponding to the two modalities to generate a fusion feature map.

8. The multimodal medical image fusion method based on image denoising and illumination enhancement according to claim 6, characterized in that: The GLME fusion rules include: The Sobel gradient operator is used to calculate the gradient information of the two modal feature maps in the horizontal and vertical directions respectively, and a gradient map is generated. The local gradient activity level map is calculated for the gradient map. The local gradient average energy operator is used to calculate the local energy information graphs of two groups of different modal feature graphs respectively; The local gradient activity level map and the local energy map of the same mode are normalized and added together to obtain a weight map; The maximum decision method is used to calculate the binary decision score map for the weight maps of different modes; the feature maps of different modes are combined according to the results of the binary decision score map to obtain the fusion feature map.

9. The multimodal medical image fusion method based on image denoising and illumination enhancement according to claim 1, characterized in that: The feature decoder uses a reversible neural network model to reconstruct the fused feature map.

Images