A Multimodal Medical Image Fusion Method Based on a Global Optimization Model

Abstract

This invention discloses a multimodal medical image fusion method based on a global optimization model, comprising: acquiring registered image pairs: anatomical medical images as grayscale images and functional images as three-channel color images; inputting the extracted illumination component of the functional image and the single-channel grayscale image into the image module of global optimization decomposition respectively to obtain their corresponding base layer and detail layer; fusing the obtained base layer and detail layer respectively; linearly adding the obtained fused base layer and fused detail layer to obtain the fused illumination component; and merging the new illumination component with the color component of the input functional image to obtain the final fused image. This invention effectively solves the uncertainty of parameter selection in medical image fusion algorithms and directly obtains the ideal fused image through an optimization function method.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing, and in particular relates to a multimodal medical image fusion method based on a global optimization model. Background Technology

[0002] Medical image fusion is an interdisciplinary field combining medical imaging and computer image processing. It aims to overcome the limitations of images acquired by single-type imaging devices, generating single images with complementary and rich information to provide auxiliary information for clinical disease diagnosis and surgical planning. Based on the imaging modality, images are categorized into single-channel anatomical images (CT images, MRI images, PC images, etc.) and three-channel pseudo-color functional images (PET images, SPECT images, GFP images, etc.). These images have different imaging mechanisms and principles, and provide varying information about lesions. For example, CT images can clearly depict the space-occupying information of bones or calcified lesions; MRI images are better at reflecting soft tissue structures; and PC images can provide subcellular structural information. Unlike these, functional images describe more cellular metabolic information from different tissue sites. Image fusion technology can effectively combine significant information from different modalities of medical images at the same level, achieving a synergistic effect greater than the sum of its parts.

[0003] Existing medical image fusion methods can be mainly categorized into spatial domain-based, transformation domain-based, and deep learning-based methods. Spatial domain-based image fusion methods directly process the source image pairs, extracting features pixel-by-pixel or image block-by-image block, offering high computational efficiency. However, the generated images are prone to spatial pixel intensity discontinuities, resulting in suboptimal perceptual quality of the fused image. Transform domain-based multi-scale image decomposition frameworks are widely used. These methods transform images from the spatial domain to different frequency domains, separating energy information and texture details. Appropriate image fusion rules are designed for each frequency domain, and the fused result is then transformed back to the spatial domain to obtain the final fused image. Commonly used transform methods include wavelet transform, contourlet transform, shear wave transform, and curvelet transform. These methods have different sensitivities to edge information in different directions and shapes, but lack translation invariance. Building upon this, non-subsampled shear wave transform and non-subsampled contourlet transform achieve better separation of multi-scale image information, further improving the quality of the fused image, but also increasing the algorithm's implementation complexity. With the continuous development of convolutional neural networks, deep learning-based image fusion methods have rapidly emerged. These methods do not require complex feature extraction algorithms; instead, they train the constructed network model on labeled data, allowing it to learn parameters on its own, thus making the fusion framework adaptive and flexible. However, its interpretability and the need for a large number of labeled medical images remain unresolved issues.

[0004] Currently, improving the quality of fused images and the execution efficiency of fusion algorithms, as well as achieving adaptive decomposition and fusion of images in the spatial domain, remain unsolved technical challenges. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal medical image fusion method based on a global optimization model to solve the problems existing in the prior art.

[0006] To achieve the above objectives, this invention provides a multimodal medical image fusion method based on a global optimization model, comprising:

[0007] Acquire multimodal medical images, including grayscale images and color images; perform color space transformation on the color images to obtain color component images and illumination component images;

[0008] Construct a global optimization decomposition model; based on the global optimization decomposition model, decompose the illumination component image and the grayscale image respectively to obtain a first base layer image, a first detail layer image, a second base layer image and a second detail layer image;

[0009] A global optimization fusion model is constructed; based on the global optimization fusion model, the first detail layer image and the second detail layer image are fused to obtain a multimodal detail layer image; the first base layer image and the second base layer image are fused to obtain a multimodal base layer image;

[0010] The multimodal base layer image, the multimodal detail layer image, and the color component image are fused to obtain a multimodal fused image.

[0011] Preferably, the process of performing color space transformation on the color image includes:

[0012] Based on color encoding, the color image is transformed in color space to obtain a color component image and an illumination component image.

[0013] Preferably, the process of constructing a global optimization decomposition model includes:

[0014] Obtain an ideal base layer image, and construct a global optimization decomposition model based on the pixel intensity relationship between the ideal base layer image, the grayscale image, and the illumination component image.

[0015] Preferably, the process of decomposing the grayscale image and the illumination component image includes:

[0016] The illumination component image is input into the global optimization decomposition model to obtain the first base layer image and the first detail layer image;

[0017] The grayscale image is input into the global optimization decomposition model to obtain the second base layer image and the second detail layer image.

[0018] Preferably, the process of constructing a global optimization fusion model includes:

[0019] An ideal detail layer image is obtained, and a global optimization fusion model is constructed based on the pixel intensity variation relationship between the ideal detail layer image, the grayscale image, and the illumination component image.

[0020] Preferably, the process of obtaining the multimodal detail layer image includes:

[0021] Based on the global optimization fusion model, the first detail layer image and the second detail layer image are fused, and a multimodal detail layer image is obtained by using the semi-quadratic splitting method and the gradient descent method.

[0022] Preferably, the process of obtaining the multimodal base layer image includes:

[0023] Based on the pixel intensity values ​​of the first base layer image and the second base layer image, a multimodal base layer image is obtained by selecting the maximum value.

[0024] Preferably, the process of obtaining a multimodal fused image includes:

[0025] The multimodal base layer image and the multimodal detail layer image are linearly added pixel by pixel to obtain a new illumination component image. The new illumination component image and the color component image are then merged by channel to obtain a multimodal fused image.

[0026] The technical effects of this invention are as follows:

[0027] Compared with existing image fusion techniques, this invention provides a multimodal medical image fusion method based on a global optimization model. The image decomposition process does not rely on various tools. By constructing a global optimization decomposition model, the basic layer and detail layer of the corresponding medical image are obtained.

[0028] For the detail layer, which contains the most organizational information, a global optimization fusion model is constructed to fuse the detail layer images of the grayscale image and the illumination component image, so that the obtained fused image avoids the phenomenon of spatial discontinuity of pixel intensity information.

[0029] This invention directly decomposes and fuses input image pairs in the spatial domain, achieving multi-scale representation of image features without the need for transformation tools, thereby improving the execution efficiency of the algorithm and enhancing the visual quality of the fused image. Attached Figure Description

[0030] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0031] Figure 1 This is a flowchart of the multimodal medical image fusion method based on a global optimization model in an embodiment of the present invention;

[0032] Figure 2 The fusion result of the MRI-CT test images in the embodiments of the present invention;

[0033] Figure 3 This refers to the fusion result of MRI-PET test images in this embodiment of the invention;

[0034] Figure 4 This is the fusion result of CT-SPECT test images in an embodiment of the present invention. Detailed Implementation

[0035] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0036] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0037] Example 1

[0038] like Figure 1 As shown, this embodiment provides a multimodal medical image fusion method based on a global optimization model, including the following steps:

[0039] S1. Obtain the registered image pairs: anatomical medical images as single-channel grayscale images and functional images as three-channel color images;

[0040] S2. Convert the functional image with R, G, and B channels into the YUV color space, extract its illumination component Y as a single-channel grayscale image I1, and input it and the anatomical medical image I2 into the globally optimized image decomposition module to obtain their respective base layers B1 and B2 and detail layers T1 and T2.

[0041] S3. The base layer B1 of the obtained anatomical image and the base layer B2 of the Y component of the functional image are fused using the maximum value method to obtain the fused base layer B. F The obtained detail layers T1 and T2 are input into the globally optimized fusion module to obtain the fused detail layer T. F ;

[0042] S4. Obtain the fusion base layer B F and the detail layer T of the fusion F Linear addition yields a new fused illumination component Y';

[0043] S5. Merge the new illumination component Y' with the color components U and V of the input functional image to obtain the final fused image F.

[0044] In some embodiments, the input anatomical image in step S1 is a single-channel grayscale image. If it is a three-channel image, it needs to be preprocessed into a single channel.

[0045] In some embodiments, the three components of the commonly used RGB color space in step S2 are not independent of each other. Direct manipulation of functional medical images may lead to color distortion or aberration. However, in the YUV color space, the illumination component Y is independent of the color components U and V, allowing for direct feature extraction without affecting color information. Based on this, according to the ideal input image I... i (i = 1, 2) Basic layer B iThe imaging characteristics of the image, and its relationship with the source image I. i The pixel intensity relationship between them is used to construct a global optimization function G1(I i B i ), defined as follows:

[0046]

[0047] Where m is used to calculate the intensity difference between the center pixel and its neighboring pixels, and is defined as follows:

[0048]

[0049] The ideal base layer B of the source image is obtained using the Fast Fourier Transform method. i And by linear subtraction T i =I i -B i Obtain the detail layer T of the corresponding image. i .

[0050] In some embodiments, the source image I input in step S3 i Base layer B i Since the pixel intensity values ​​are all greater than or equal to 0, we iterate through all pixels in B1 and B2, compare the pixel intensity values ​​at corresponding positions, and then perform a fusion in the base layer B. F Retain larger values, calculation method

[0051]

[0052] Where (x, y) represents the coordinates of a pixel. Based on the ideal blending detail layer T... F Based on the characteristics, construct the global optimization function G2(T1,T2,T) F ), defined as follows:

[0053]

[0054] in, Let α denote the gradient operator. α is the balance coefficient, and λ is the penalty coefficient. The fused detail layer T is obtained using the semi-quadratic splitting method and gradient descent method. F .

[0055] In some embodiments, the fusion base layer B obtained in step S4 F With the fusion detail layer T F The new illumination component Y' is obtained by linearly adding each pixel, and the calculation method is as follows:

[0056] Y'(x,y)=B F (x,y)+T F (x,y)

[0057] In some embodiments, in step S5, the obtained component Y' is merged with the components U and V in step S2, and the final fused image F is obtained by linear transformation of the YUV-RGB color space, which is used to assist in clinical treatment and diagnostic analysis.

[0058] In this embodiment, the fusion result of the MRI-CT test images is as follows: Figure 2 As shown; the fusion results of MRI-PET test images, as follows Figure 3 As shown; the fusion results of CT-SPECT test images, as follows. Figure 4 As shown.

[0059] To objectively evaluate the quality of the fused image obtained by this invention, simulation experiments were conducted on two public datasets and compared with seven common traditional image fusion algorithms. The results are shown in Table 1. Higher values ​​for the five objective indicators in the table indicate higher quality fused images and more information retained from the source images. The table shows that the fused image obtained by this invention scores highest on all five objective indicators.

[0060] Table 1

[0061] method NCIE PSNR <![CDATA[Q0]]> RMI SSIM GFF 0.807 32.8351 0.3687 0.6801 0.6285 SEWT 0.8061 32.0598 0.2921 0.5909 0.3411 FIGM 0.8063 33.0578 0.3548 0.6230 0.6476 LRD 0.8072 37.4817 0.3736 0.6872 0.6784 NSSTMSMG 0.8061 31.4217 0.3258 0.5956 0.6968 THREELAYER 0.8068 38.1295 0.2892 0.6419 0.5798 LLF 0.8061 34.2392 0.3554 0.6074 0.7700 This invention 0.8078 42.4447 0.3758 0.7417 0.8273

[0062] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-modal medical image fusion method based on a global optimization model, characterized in that, The method comprises the following steps: acquiring a multi-modal medical image, the multi-modal medical image comprising a grayscale image and a color image; performing color space transformation on the color image to obtain a color component image and an illumination component image; constructing a global optimization decomposition model; based on the global optimization decomposition model, decomposing the illumination component image and the grayscale image respectively to obtain a first base layer image, a first detail layer image, a second base layer image, and a second detail layer image; constructing a global optimization fusion model; based on the global optimization fusion model, fusing the first detail layer image and the second detail layer image to obtain a multi-modal detail layer image; fusing the first base layer image and the second base layer image to obtain a multi-modal base layer image; the process of obtaining the multi-modal detail layer image comprises: based on the global optimization fusion model, fusing the first detail layer image and the second detail layer image to obtain a multi-modal detail layer image by using a semi-quadratic splitting method and a gradient descent method; fusing the multi-modal base layer image, the multi-modal detail layer image, and the color component image to obtain a multi-modal fusion image.

2. The multi-modality medical image fusion method based on a global optimization model according to claim 1, characterized in that, the process of performing color space transformation on the color image comprises: based on color coding, performing color space transformation on the color image to obtain a color component image and an illumination component image. 3.The multi-modal medical image fusion method based on a global optimization model according to claim 1, characterized in that, the process of constructing the global optimization decomposition model comprises: acquiring an ideal base layer image, and constructing a global optimization decomposition model based on a pixel intensity relationship between the ideal base layer image, the grayscale image, and the illumination component image.

4. The multi-modal medical image fusion method based on a global optimization model according to claim 3, characterized in that, the process of decomposing the grayscale image and the illumination component image comprises: inputting the illumination component image into the global optimization decomposition model to obtain a first base layer image and a first detail layer image; inputting the grayscale image into the global optimization decomposition model to obtain a second base layer image and a second detail layer image.

5. The multi-modal medical image fusion method based on a global optimization model according to claim 4, characterized in that, the process of constructing the global optimization fusion model comprises: acquiring an ideal detail layer image, and constructing a global optimization fusion model based on a pixel intensity variation relationship between the ideal detail layer image, the grayscale image, and the illumination component image. 6.The multi-modal medical image fusion method based on a global optimization model according to claim 1, characterized in that, the process of obtaining the multi-modal base layer image comprises: based on pixel intensity values of the first base layer image and the second base layer image, obtaining a multi-modal base layer image by selecting a maximum value.

7. The multi-modal medical image fusion method based on a global optimization model according to claim 1, characterized in that, the process of obtaining the multi-modal fusion image comprises: performing pixel linear addition on the multi-modal base layer image and the multi-modal detail layer image to obtain a new illumination component image; performing channel merging on the new illumination component image and the color component image to obtain a multi-modal fusion image.

Images