A multi-modal medical image fusion method based on image denoising and illumination enhancement

Abstract

The present application relates to a kind of multimodal medical image fusion methods based on image denoising and illumination enhancement, comprising: the multimodal medical image input illumination and contrast enhancement network is corrected to generate enhanced image with light intensity;For multimodal medical image adds random noise and obtains noise image;Noise image is input into denoising and reconstruction network and carries out denoising and reconstruction task;Enhanced image and multimodal medical image are as label jointly supervised training denoising and reconstruction network, obtain the feature encoder and feature decoder with denoising and illumination enhancement function, using the feature encoder of trained feature respectively extract the feature of different modal images, obtain modal feature map, the modal feature map of different modal images is fused, and the feature decoder of trained feature is used to reconstruct the fused feature map to obtain multimodal medical image fusion result.The present application solves the problem that image fusion is interfered by noise noise, illumination degradation and low fusion efficiency.

Description

Technical Field

[0001] This invention relates to the field of multimodal image fusion technology, and in particular to a multimodal medical image fusion method based on image denoising and illumination enhancement. Background Technology

[0002] Multimodal medical image fusion methods are generally divided into traditional image fusion methods and deep learning-based image fusion methods. Common traditional fusion methods are based on spatial domain, transform domain, sparse representation, and traditional models. Spatial domain fusion involves combining segmented image patches, which can lead to artifacts in the fused image. Transform domain methods first transform the original image, then fuse the resulting information layers, and finally reconstruct the fused image through inverse transform. Sparse representation-based methods require creating a dictionary to represent weight information and continuously updating the dictionary to generate the fused image. Fusion methods based on traditional models often require continuous optimization of the model using appropriate optimization schemes. Deep learning-based multimodal fusion methods use convolutional neural network models for feature extraction and representation, effectively improving feature robustness and model generalization. Furthermore, deep learning models such as generative adversarial networks and visual Transformers are also frequently used in image fusion.

[0003] In multimodal medical image fusion, common fusion methods for noisy images include traditional variational models and sparse representation methods. However, these two fusion and denoising methods have certain limitations: most existing fusion and denoising methods are based on traditional variational models and sparse representation methods, essentially traditional fusion methods. They require manual adjustment of a large number of parameters according to specific scenes, resulting in low efficiency in fusion and denoising. Illumination attenuation during the fusion process leads to texture blurring and color distortion in the fused result. Although some existing fusion methods enhance illumination during the fusion process, they cannot effectively remove noise while enhancing illumination, resulting in poor robustness and generalization. They cannot effectively remove random noise in real-world scenes, leading to poor denoising effects in the fused image. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a multimodal medical image fusion method based on image denoising and illumination enhancement, aiming to solve the problems of multimodal images being susceptible to noise interference, illumination degradation, and low fusion efficiency. The method includes:

[0005] S1: Input multimodal medical images into an illumination and contrast enhancement network to perform illumination intensity correction and generate enhanced images;

[0006] S2: Add random noise to multimodal medical images to obtain noisy images; construct a denoising and reconstruction network and input the noisy image into the denoising and reconstruction network to perform denoising and reconstruction tasks; the denoising and reconstruction network extracts noise features through a feature encoder and generates reconstructed images through a feature decoder;

[0007] S3: The denoising and reconstruction network is trained under the joint supervision of the enhanced image and the multimodal medical image as labels to obtain the feature encoder and feature decoder with denoising and illumination enhancement functions;

[0008] S4: Use the trained feature encoder to extract features from different modal images to obtain modal feature maps. Use LGLE or GLME fusion rules to fuse the modal feature maps of different modal images. Use the trained feature decoder to reconstruct the fused feature maps to obtain the multimodal medical image fusion result.

[0009] The present invention has at least the following beneficial effects

[0010] This invention employs joint training of an illumination and contrast enhancement network and a denoising and reconstruction network. By using enhanced images and multimodal medical images as labels for joint supervised training of the denoising and reconstruction network, the model learns a feature encoder and feature decoder with denoising and illumination enhancement capabilities. This effectively removes random noise from images, improving the model's robustness and generalization performance. The feature encoder and feature decoder effectively remove random noise and enhance illumination during feature extraction and feature reconstruction, respectively, solving the problems of noise interference and illumination degradation, thus improving the quality of image fusion. Furthermore, this invention designs two LGLE fusion rules and GLME fusion rules specifically for different fusion tasks, namely, the features of fusing images from different modalities. These rules enhance the preservation of semantic information such as texture details, physiological metabolism, and salient targets, solving the problem of lost key semantic information and preventing information loss during feature reconstruction due to changes in pixel distribution between different feature maps. Compared to fusion methods based on traditional variational models and sparse representations, this invention offers higher fusion efficiency and effectively alleviates the illumination degradation problem during fusion. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the model framework of the present invention;

[0012] Figure 2 This is a schematic diagram of the architecture of the illuminance estimation module of the present invention;

[0013] Figure 3 This is a schematic diagram of the architecture of the feature encoder of the present invention;

[0014] Figure 4 This is a schematic diagram of the feature decoder architecture of the present invention. Detailed Implementation

[0015] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of specific embodiments. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention, and the following embodiments and features can be combined with each other unless otherwise specified.

[0016] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0017] Please see Figures 1-4 The present invention provides a multimodal medical image fusion method based on image denoising and illumination enhancement, such as... Figure 1 As shown, the method includes the following steps:

[0018] S1: Input multimodal medical images into an illumination and contrast enhancement network to perform illumination intensity correction and generate enhanced images;

[0019] Preferably, the illuminance and contrast enhancement network includes: an illuminance estimation module and a Fourier transform module;

[0020] The illuminance estimation module is used to estimate the illuminance of the input multimodal medical image to obtain the estimated illuminance; and based on Retinex theory, the multimodal medical image is divided by the estimated illuminance to obtain the corrected image of the multimodal medical image;

[0021] The Fourier transform module includes: a fast Fourier forward transform module, a high-pass filter module, and a fast Fourier inverse transform module;

[0022] The Fast Fourier Transform module is used to perform Fourier transform on multimodal medical images and corrected images to generate corresponding spectrograms.

[0023] The high-pass filtering module is used to perform high-pass filtering on the spectrogram of multimodal medical images to obtain high-frequency images with high-frequency detail information;

[0024] The inverse fast Fourier transform module is used to superimpose the spectrogram of the corrected image and the high-frequency image, and then perform an inverse Fourier transform on the superimposed spectrogram to obtain the enhanced image.

[0025] Preferably, the process of acquiring multimodal medical images includes: magnetic resonance imaging (MRI) and computed tomography (CT) images of acute stroke cases and glioma cases; magnetic resonance imaging (MRI) and single-photon emission computed tomography (SPECT) images of Alzheimer's disease and subacute stroke; and magnetic resonance imaging (MRI) and positron emission tomography (PET) images of normal, disease-free individuals.

[0026] Specifically, the illuminance and contrast network uses a pre-trained illuminance estimation module ( Figure 1 An illumination estimation module is used to estimate image illumination. This module consists of three cascaded conventional convolutional layers connected by residual connections. The conventional convolutional layers include 3×3 convolutional kernels, ReLU activation layers, or Sigmoid activation layers (such as...). Figure 2 (as shown); then, based on Retinex theory, the multimodal medical image is divided by the estimated illuminance to obtain the corrected image of the multimodal medical image;

[0027] Specifically, the Fourier transform module performs Fast Fourier Transform on the original image and the corrected image respectively to generate frequency domain maps. High-pass filtering is used to operate on the frequency domain map of the multimodal medical image to generate a high-frequency spectrum map, which represents the high-frequency texture information in the original image. The high-frequency spectrum map and the spectrum map of the corrected image are superimposed, and the superimposed spectrum map is subjected to inverse Fast Fourier Transform to obtain an enhanced image with enhanced illumination and contrast, which is referred to as the enhanced image below.

[0028] S2: Add random noise to multimodal medical images to obtain noisy images; construct a denoising and reconstruction network and input the noisy image into the denoising and reconstruction network to perform denoising and reconstruction tasks; the denoising and reconstruction network extracts noise features through a feature encoder and generates reconstructed images through a feature decoder;

[0029] Preferably, the random noise includes: Gaussian noise, salt and pepper noise, Poisson noise, or speckle noise with a random standard deviation.

[0030] Preferably, the feature encoder consists of N cascaded conventional convolutional layers; the conventional convolutional layers include cascaded 3×3 convolutional kernels, ReLU activation layers, and BN layers; the feature decoder adopts a reversible neural network model.

[0031] like Figure 1As shown, multimodal medical images are converted into grayscale images and random noise is added. Adding random noise to the original multimodal image includes adding Gaussian noise, salt-and-pepper noise, Poisson noise, and speckle noise with random standard deviations. A denoising and reconstruction network is constructed, and the noisy image is input into the denoising and reconstruction network to perform image denoising and reconstruction tasks. The denoising and reconstruction network extracts more representative noise features through a feature encoder and generates a reconstructed image through a feature decoder.

[0032] Specifically, the feature encoder comprises N conventional convolutional layers, each including a 3×3 convolutional kernel, a ReLU activation layer, and a Batch Normalization (BN) layer. See also... Figure 3 The input noisy image tensor has dimensions B, C, H, W; where B is the batch number, C is the number of channels, H is the number of feature map rows, and W is the number of feature map columns, with C being 1 at this point; the first convolutional layer maps the input feature map with C=1 to C=64, while B, H, and W remain unchanged; the second to ninth convolutional layers maintain C=64.

[0033] Preferably, the feature decoder employs a reversible neural network model to reconstruct the fused feature map.

[0034] Specifically, the feature decoder employs a reversible neural network model for feature map mapping and reconstruction. (See also...) Figure 4 The process after feature decoding can be represented as:

[0035]

[0036] Among them, I r Represents the reconstructed image, and Convi(i=1,2,3,4) represents a convolutional layer with a 3×3 kernel. This represents the feature map in the first to (C / 2)th dimension channels of the feature map extracted by the encoder. This represents the feature map from the (C / 2+1)th to the Cth dimension in the feature map extracted by the encoder, and exp represents the calculation of the exponential function.

[0037] S3: The denoising and reconstruction network is trained under the joint supervision of the enhanced image and the multimodal medical image as labels to obtain the feature encoder and feature decoder with denoising and illumination enhancement functions;

[0038] Preferably, the step of using the enhanced image and the multimodal medical image as labels to jointly supervise the training of the denoising and reconstruction network includes: using the enhanced image as a training pseudo-label; constructing a first loss function based on the L2 norm between the enhanced image and the reconstructed image; constructing a second loss function based on the gradient information difference between the reconstructed image and the input multimodal medical image; constructing a third loss function based on the L2 norm between the reconstructed image and the enhanced image; constructing a total loss function for the model based on the first, second, and third loss functions; and optimizing the parameters of the denoising and reconstruction network based on the constructed total loss function.

[0039] Specifically, a loss function for training the denoising and reconstruction network is constructed, which can be expressed as:

[0040]

[0041] in, This represents the reconstruction loss achieved during the denoising and reconstruction network training process. This represents the content loss and is used to calculate the L2 norm of the reconstructed image and the input multimodal medical image; It represents the gradient loss and is used to calculate the difference in gradient information between the reconstructed image and the input multimodal medical image; denoted by and used to calculate the L2 norm between the augmented and reconstructed images; γ, η, and ξ represent balancing factors.

[0042] Specifically, the illumination and contrast enhancement network and the denoising and reconstruction network form a joint training method to train the feature encoder and feature decoder in the denoising and reconstruction network, so that the feature encoder and feature decoder have the functions of noise removal and illumination enhancement when performing image feature extraction and reconstruction.

[0043] S4: Use the trained feature encoder to extract features from different modal images to obtain modal feature maps. Then, fuse the modal feature maps of different modal images and use the trained feature decoder to reconstruct the fused feature maps to obtain the multimodal medical image fusion result.

[0044] Specifically, the trained feature encoder, equipped with denoising and illumination enhancement functions, extracts features from noisy images of different modalities to obtain image feature maps. Based on different fusion tasks, fusion rules are selected to fuse these feature maps, generating a fused feature map. The trained feature decoder is then used to reconstruct the fused feature map to obtain the fusion result. Please refer to [reference needed]. Figure 1 Multimodal image fusion process.

[0045] Preferably, the feature fusion of modal feature maps of different modal images using LGLE or GLME fusion rules includes: when the input multimodal medical images are MRI images and CT images, using LGLE fusion rules to fuse the modal feature maps of different modal images;

[0046] When the input multimodal medical images are MRI and PET images, or MRI and SPECT images, the GLME fusion rule is used to fuse the modal feature maps of different modal images.

[0047] Specifically, the fusion rule for MRI and CT images adopted the LGLE fusion rule; the specific steps of LGLE are as follows:

[0048] Step 1: Integrate the channel information of the two modal feature maps using L1-Norm channel overlay and generate the corresponding single-pass feature map. This process is represented as follows:

[0049]

[0050] Where, N k Represents a single-channel feature map. The modal feature map is represented by k=1 or k=2, which indicates different modes, and N represents the number of channels in the feature map.

[0051] Step 2: Calculate the single-channel feature map N for both modes using the Sobel gradient operator. k The gradient information in the horizontal and vertical directions is used to generate the corresponding gradient map G. k And calculate the local gradient activity level map GL on the gradient map. k The process is represented as:

[0052]

[0053] Among them, W l W represents a convolution with a 3×3 kernel. l = [0,1,0; 1,1,1; 0,1,0];

[0054] Step 3: Use the local energy maximization method to calculate the feature information of the high-signal region in the single-channel feature map of the two modes, and generate the local energy map E. k :

[0055]

[0056] Among them, W e W represents a convolution with a 3×3 kernel. e =[1,1,1;1,1,1;1,1,1];

[0057] Step 4: Calculate the corresponding weight map M by combining the local gradient activity level map and the local energy map. k The feature maps of the two modalities are combined based on the weight maps corresponding to the two modalities to generate a fused feature map.

[0058]

[0059] In the LGLE fusion algorithm, L1-Norm integrates information from the channels, and the calculation of gradient information and local energy of the feature map helps to extract detailed texture information from MRI and dense skull regions from CT images, respectively, thus enhancing the preservation of texture details and brightness information in the fused image.

[0060] Preferably, the fusion rules for MRI images and PET images, or MRI images and SPECT images, are consistent, and the GLME fusion rule is adopted. The specific steps of GLME are as follows:

[0061] Step 1: Calculate the gradient information of the two modal feature maps in the horizontal and vertical directions using the Sobel gradient operator to generate the gradient map G. k The local gradient activity level map (GL) is calculated from the gradient map. k Please refer to step 2 of the LGLE fusion rules;

[0062] Step 2: Calculate the feature maps of two different modes using the local gradient average energy operator. Local average energy information map ME k :

[0063]

[0064] Among them, W m W represents a convolution with a 3×3 kernel. m = 1 / 16 × [1,2,1; 2,4,2; 1,2,1]

[0065] Step 3: Standardize the local gradient activity level map and local energy map of the same mode, then sum them to obtain the weight map C. k :

[0066] C k =Norm(GL k )+β×Norm(GL k )

[0067] Where Norm represents standardization and β represents the balance factor;

[0068] Step 4: Use the maximum decision method to calculate the binary decision score map M for the weight maps of different modalities; combine the feature maps of different modalities based on the results of the binary decision score map to obtain the fused feature map.

[0069]

[0070] GLME fusion rules help capture texture details in MRI images and physiological metabolism in PET or SPECT images, effectively preserving this significant medical semantic information in the fused images.

[0071] In summary, this invention, through the joint training of an illumination and contrast enhancement network and a denoising and reconstruction network, trains the denoising and reconstruction network by using enhanced images and multimodal medical images as labels for joint supervision. This training of a feature encoder and decoder with denoising and illumination enhancement capabilities effectively removes random noise from images, improving the model's robustness and generalization performance. The pre-trained feature encoder and decoder extract features from different modalities of the medical image, obtaining modal feature maps. These feature maps are then fused using LGLE or GLME fusion rules to obtain a fused feature map. Finally, the fused feature map is reconstructed using the trained feature decoder to obtain the multimodal medical image fusion result. In this process... The feature encoder and feature decoder remove noise and enhance illumination during feature extraction and reconstruction, respectively, solving the problems of noise interference and illumination degradation and improving the image fusion quality. Furthermore, this invention designs two LGLE fusion rules and GLME fusion rules to address the characteristics of different fusion tasks, namely the features of fusion between images of different modalities. These rules increase the preservation of semantic information such as texture details, physiological metabolism, and salient targets, solving the problem of lost key semantic information and avoiding information loss during feature reconstruction caused by changes in pixel distribution between different feature maps. Compared with fusion methods based on traditional variational models and sparse representations, this invention has higher fusion efficiency and effectively alleviates the illumination degradation problem during fusion.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multimodal medical image fusion method based on image denoising and illumination enhancement, characterized in that, Includes the following steps: S1: Input multimodal medical images into an illumination and contrast enhancement network to perform illumination intensity correction and generate enhanced images; S2: Add random noise to multimodal medical images to obtain noisy images; construct a denoising and reconstruction network and input the noisy images into the denoising and reconstruction network to perform denoising and reconstruction tasks; The denoising and reconstruction network extracts noise features through a feature encoder and generates a reconstructed image through a feature decoder; S3: The denoising and reconstruction network is trained under the joint supervision of the enhanced image and the multimodal medical image as labels to obtain the feature encoder and feature decoder with denoising and illumination enhancement functions; S4: Use the trained feature encoder to extract features from different modal images to obtain modal feature maps. Then, use the LGLE or GLME fusion rules to fuse the modal feature maps of different modal images. Finally, use the trained feature decoder to reconstruct the fused feature maps to obtain the multimodal medical image fusion result. The LGLE fusion rules include: Step 1: Integrate the channel information of the two modal feature maps using L1-Norm channel overlay and generate the corresponding single-pass feature map. This process is represented as follows: in, Represents a single-channel feature map. The modal feature map is represented by k=1 or k=2, where k=1 or k=2 represents different modes, and N represents the number of channels in the feature map. Step 2: Calculate the single-channel feature maps of the two modes using the Sobel gradient operator. The gradient information in the horizontal and vertical directions is used to generate corresponding gradient maps. And calculate the local gradient activity level map of the gradient map. The process is represented as: in, This indicates a convolution with a 3×3 kernel. ; Step 3: Use the local energy maximization method to calculate the feature information of the high-signal region in the single-channel feature map of the two modes, and generate the local energy map. : in, This indicates a convolution with a 3×3 kernel. ; Step 4: Calculate the corresponding weight map by combining the local gradient activity level map and the local energy map. The feature maps of the two modalities are combined based on the weight maps corresponding to the two modalities to generate a fused feature map. : In the LGLE fusion algorithm, L1-Norm integrates information from the channels, and the calculation of gradient information and local energy of the feature map helps to extract detailed texture information from MRI and dense skull regions from CT images, respectively, thus enhancing the preservation of texture details and brightness information in the fused image. The GLME fusion rules include: Step 1: Calculate the gradient information of the two modal feature maps in the horizontal and vertical directions using the Sobel gradient operator to generate gradient maps. Calculate the local gradient activity level map from the gradient map. Please refer to step 2 of the LGLE fusion rules; Step 2: Calculate the feature maps of two different modes using the local gradient average energy operator. Local average energy information map : in, This indicates a convolution with a 3×3 kernel. ; Step 3: Standardize the local gradient activity level map and local energy map of the same mode, then sum them to obtain the weight map. : in, Indicates standardization, Indicates the balance factor; Step 4: Use the maximum decision method to calculate the binary decision score map M for the weight maps of different modalities; combine the feature maps of different modalities based on the results of the binary decision score map to obtain the fused feature map. : GLME fusion rules help capture texture details in MRI images and physiological metabolism in PET or SPECT images, effectively preserving this significant medical semantic information in the fused images.

2. The multimodal medical image fusion method based on image denoising and illumination enhancement according to claim 1, characterized in that, The illuminance and contrast enhancement network includes: an illuminance estimation module and a Fourier transform module; The illuminance estimation module is used to estimate the illuminance of the input multimodal medical image to obtain the estimated illuminance; and based on Retinex theory, the multimodal medical image is divided by the estimated illuminance to obtain the corrected image of the multimodal medical image; The Fourier transform module includes: a fast Fourier forward transform module, a high-pass filter module, and a fast Fourier inverse transform module; The Fast Fourier Transform module is used to perform Fourier transform on multimodal medical images and corrected images to generate corresponding spectrograms. The high-pass filtering module is used to perform high-pass filtering on the spectrogram of multimodal medical images to obtain high-frequency images with high-frequency detail information; The inverse fast Fourier transform module is used to superimpose the spectrogram of the corrected image and the high-frequency image, and then perform an inverse Fourier transform on the superimposed spectrogram to obtain the enhanced image.

3. The multimodal medical image fusion method based on image denoising and illumination enhancement according to claim 1, characterized in that, The random noise includes: Gaussian noise, salt and pepper noise, Poisson noise, or speckle noise with a random standard deviation.

4. The multimodal medical image fusion method based on image denoising and illumination enhancement according to claim 1, characterized in that, The feature encoder consists of N cascaded conventional convolutional layers; the conventional convolutional layers include cascaded 3×3 convolutional kernels, ReLU activation layers, and BN layers; the feature decoder adopts a reversible neural network model.

5. The multimodal medical image fusion method based on image denoising and illumination enhancement according to claim 1, characterized in that, The method of using enhanced images and multimodal medical images as labels to jointly supervise the training of the denoising and reconstruction network includes: using the enhanced images as training pseudo-labels; constructing a first loss function based on the L2 norm between the enhanced and reconstructed images; constructing a second loss function based on the gradient information difference between the reconstructed image and the input multimodal medical image; constructing a third loss function based on the L2 norm between the reconstructed and enhanced images; constructing a total loss function for the model based on the first, second, and third loss functions; and optimizing the parameters of the denoising and reconstruction network based on the constructed total loss function.

6. The multimodal medical image fusion method based on image denoising and illumination enhancement according to claim 1, characterized in that, The feature fusion of modal feature maps of different modal images using LGLE or GLME fusion rules includes: When the input multimodal medical images are MRI images and CT images, the LGLE fusion rule is used to fuse the modal feature maps of different modal images; When the input multimodal medical images are MRI and PET images, or MRI and SPECT images, the GLME fusion rule is used to fuse the modal feature maps of different modal images.

7. The multimodal medical image fusion method based on image denoising and illumination enhancement according to claim 1, characterized in that, The feature decoder uses a reversible neural network model to reconstruct the fused feature map.

Images