A medical image fusion method based on modality difference information

Abstract

The present invention discloses a medical image fusion method based on modal difference information, which belongs to the field of image fusion technology. The technical points are: comprising the following steps: step 1: image decomposition, image decomposition includes cross-modal decomposition and self-modal decomposition; cross-modal decomposition treats two source images as a whole; self-modal decomposition is based on the characteristics of the detail layer of the image; step 2: modal information difference estimation, modal information difference estimation includes structural similarity calculation and modal difference estimation; step 3: image fusion, image fusion includes fusion rule design and image reconstruction; fusion rule design includes designing fusion rules according to the imaging characteristics of different layers; image reconstruction includes linearly merging the fused image layers and outputting the fusion result, which has the advantage of solving how to use imaging modal features for fusion in the image fusion process to reduce information loss.

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and in particular to a medical image fusion method based on modality difference information. Background Art

[0002] In the field of medical image fusion, feature extraction and fusion strategy optimization are key research topics. Traditional methods typically rely on a fixed decomposition framework to process images of different modalities, aiming to fuse the image's structure and texture information separately. However, when processing complex image features, these methods often overlook subtle differences between modalities, which can lead to the loss or distortion of key information, thus limiting the application effect of the fused image. To address this issue, this study proposes a medical image fusion method guided by modal information differences, aiming to improve the quality and practical application value of the fused image and reduce the loss of effective modal information.

[0003] This method expands the concepts of local and global features in medical image fusion, treating the image pair to be fused as a global entity and a single image as a local entity. Cross-modal and self-modal features are extracted separately to simulate the dual-branch processing structure and attention mechanism in deep learning models. First, the shared intensity of the source image pair is extracted through a joint filtering method and used as the base layer of the source image. Then, the modal information is obtained through linear subtraction and used as the detail layer of the source image. Next, an adaptive guided filter is constructed to smooth the detail layer of the image to obtain the structural layer and energy layer of the corresponding image. Simultaneously, based on the structural similarity between the detail layer and the source image, the modal difference between the source image pair is estimated to guide the design of the fusion rules. Finally, corresponding fusion rules are designed for different image layers, and the fused image is obtained through image reconstruction.

[0004] Although existing medical image fusion methods can generate fused images with high visual quality, they still have significant limitations in clinical auxiliary diagnosis. Specifically, in the multi-scale image decomposition framework, traditional fusion methods usually treat a single image as an independent research object, ignoring the modality matching between the different modal images to be fused. Although this method can effectively extract different features of the image in the image decomposition stage, it does not consider the complementarity and consistency of information in multi-modal images, which may lead to modality information mismatch during the fusion process. Even if a single-modality image has been successfully decomposed into multi-scale features, the intensity difference between different modalities may still affect the final fusion effect. In the subsequent fusion process, the intensity difference between sub-layers from different modalities may cause high-intensity irrelevant information to dominate the fused image, thereby masking the low-intensity but diagnostically valuable modality information.

[0005] In response to the above problems, the present invention proposes a medical image fusion method based on modal difference information. Through a joint optimization strategy, the input image pair is taken as a whole, the common structure is extracted, and the modal information is obtained. Based on this, the modal information difference is estimated to guide the subsequent fusion rule design. By constructing an adaptive guided filter, the detail information of the image is further decomposed into energy layer and structure layer, reducing the difference between image layers, which is conducive to improving the subsequent fusion effect. Fusion rules are constructed based on the characteristics of the image layer and the modal difference information to retain the effective information of medical images of different modalities and reduce information loss, providing more accurate and reliable image support for clinical diagnosis. Therefore, the purpose of the present invention is to improve the adaptability, robustness and accuracy of multimodal medical image fusion technology, thereby providing more powerful support for diagnostic assistance of medical images. Summary of the Invention

[0006] In view of the deficiencies in the prior art, an embodiment of the present invention aims to provide a medical image fusion method based on modality difference information to solve the problems in the above-mentioned background technology.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A medical image fusion method based on modality difference information includes the following steps:

[0009] Step 1: Image decomposition, which includes cross-modal decomposition and self-modal decomposition;

[0010] Cross-modal decomposition treats the two source images as a whole, designs an objective function, performs joint optimization steps, extracts the common information of the multimodal medical image pair, and uses the results as the respective base layers. Through linear operations on the images, the detail layers of the corresponding images are obtained;

[0011] Based on the characteristics of the detail layer of the image, the self-modal decomposition calculates the pixel intensity skewness, constructs an adaptive guided filter, smoothes the detail layer, and obtains the structure layer and energy layer of the image;

[0012] Step 2: Modal information difference estimation, which includes structural similarity calculation and modal difference estimation;

[0013] Structural similarity calculation includes calculating the structural similarity of the source image and the corresponding detail layer;

[0014] Modality difference estimation includes estimating the input modality difference based on the two obtained structural similarity values;

[0015] Step 3: Image fusion, which includes fusion rule design and image reconstruction;

[0016] The fusion rule design includes designing fusion rules based on the imaging characteristics of different layers, while introducing modal difference estimation results to reduce image information loss;

[0017] Image reconstruction involves linearly merging the fused image layers and outputting the fused result.

[0018] As a further solution of the present invention, the cross-modal decomposition in step 1 converts the source image into and Input the common structure extractor and perform the following joint optimization steps:

[0019]

[0020] in, and Represents the source images and The base layer represents the intensity consistency area extracted from the original image; and constrains the base layer to preserve the contents of the source image, and Suppress the noise of the base layer so that it has sparse edges, and are weight hyperparameters, controlling the strength of edge preservation and structure alignment respectively;

[0021] Based on the optimization solution and , calculate the modal information and use it as the detail layer of the corresponding image:

[0022]

[0023] in, Represents the detail layer, For image retrieval.

[0024] As a further solution of the present invention, the self-modal decomposition in step 1 is analyzed by Based on the pixel intensity distribution characteristics, the guided filter can automatically calculate the size of the filter window and the filter weight according to the image content. The specific definitions are as follows:

[0025]

[0026] in, Represents guided filtering, where both the input image and the guided image are set to , window size coefficient Control the spatial scale of the filter area, The larger the value, the smoother the filtered image is, and the regularization coefficient Affects the degree of edge preservation, in order to perform adaptive filtering operations based on the content of the image, and The value of is automatically calculated based on the pixel intensity skewness of the image:

[0027]

[0028]

[0029] in, The pixel intensity skewness is calculated as follows:

[0030]

[0031] in, Describing an image The median value of Describing an image The mean of Describing an image The standard deviation of Measures the asymmetry and discreteness of image pixel distribution;

[0032] Then, you can get The structural layer :

[0033] .

[0034] As a further solution of the present invention, the modal information difference estimation in step 2 is approximately estimated by the structural similarity between the decomposed modal features and the source image;

[0035]

[0036] in, Calculate the structural similarity of two images if High, indicating that the proportion of non-shared information in the source image is higher. , illustrating the source image More modal information than , and the shared information is less than ,Will Introducing the fusion rules of the base layer to reduce the information loss caused by modal differences, The same applies to time.

[0037] As a further solution of the present invention, the fusion rule design in step 3 includes a base layer fusion rule:

[0038] For the base layer and , select according to pixel differences and obtain the fusion result , specifically:

[0039]

[0040] in, is the estimated modal difference information.

[0041] As a further solution of the present invention, the fusion rule design in step 3 further includes a structure layer fusion rule:

[0042] First, the structural layer of the image Perform Wiener filtering to suppress interference information:

[0043]

[0044] Then, the intensity gradient changes in the horizontal and vertical directions are calculated to obtain the structural change intensity:

[0045]

[0046] Constructing the weight matrix , defined as follows:

[0047]

[0048] Perform structure layer fusion according to the weight matrix to obtain the fused structure layer :

[0049]

[0050] in, Represents an element-wise weighted multiplication operation.

[0051] As a further solution of the present invention, the fusion rule design in step 3 also includes energy layer fusion rules:

[0052] According to energy level and The intensity value of the corresponding pixel position is used to obtain the fusion energy map by performing the maximum selection method :

[0053] .

[0054] As a further solution of the present invention, the image reconstruction in step 3: the final fused image It is a weighted combination of the above three fusion results:

[0055] .

[0056] In summary, the embodiments of the present invention have the following beneficial effects compared with the prior art:

[0057] By designing a staged image decomposition framework, we separate the features of multimodal medical images at different scales, and then estimate the modal difference information of the input image pair, guiding the design of subsequent image fusion rules. Cross-modal decomposition treats the source image pair as a whole and leverages the spatial correlation of the registered medical image pair to improve the matching of the decomposed image feature intensities, enhancing the subsequent fusion effect. Automodal decomposition is adaptively performed based on the features of the image detail layer, further extracting residual energy information and improving the sparsity and integrity of the image structure layer. During the fusion stage, modal difference information is introduced into the fusion rules to reduce the loss of weak modal features and improve the perceptual quality and information richness of the fusion results.

[0058] In order to more clearly illustrate the structural features and effects of the present invention, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is an MRI-CT fusion image of an embodiment of the invention.

[0060] Figure 2 This is an MRI-SPECT fusion image in an embodiment of the invention.

[0061] Figure 3 This is an MRI-PET fusion image in an embodiment of the invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0063] The specific implementation of the present invention is described in detail below with reference to specific embodiments.

[0064] In one embodiment, a medical image fusion method based on modality difference information is provided. Figures 1 to 3 , including the following steps:

[0065] Step 1: Image decomposition, which includes cross-modal decomposition and self-modal decomposition;

[0066] Cross-modal decomposition treats the two source images as a whole, designs an objective function, performs joint optimization steps, extracts the common information of the multimodal medical image pair, and uses the results as the respective base layers. Through linear operations on the images, the detail layers of the corresponding images are obtained;

[0067] Based on the characteristics of the detail layer of the image, the self-modal decomposition calculates the pixel intensity skewness, constructs an adaptive guided filter, smoothes the detail layer, and obtains the structure layer and energy layer of the image;

[0068] Step 2: Modal information difference estimation, which includes structural similarity calculation and modal difference estimation;

[0069] Structural similarity calculation includes calculating the structural similarity of the source image and the corresponding detail layer;

[0070] Modality difference estimation includes estimating the input modality difference based on the two obtained structural similarity values;

[0071] Step 3: Image fusion, which includes fusion rule design and image reconstruction;

[0072] The fusion rule design includes designing fusion rules based on the imaging characteristics of different layers, while introducing modal difference estimation results to reduce image information loss;

[0073] Image reconstruction involves linearly merging the fused image layers and outputting the fused result.

[0074] For further information, see Figures 1 to 3 In the step 1, the cross-modal decomposition converts the source image into and Input the common structure extractor and perform the following joint optimization steps:

[0075]

[0076] in, and Represents the source images and The base layer represents the intensity consistency area extracted from the original image; and constrains the base layer to preserve the contents of the source image, and Suppress the noise of the base layer so that it has sparse edges, and are weight hyperparameters, controlling the strength of edge preservation and structure alignment respectively;

[0077] Based on the optimization solution and , calculate the modal information and use it as the detail layer of the corresponding image:

[0078]

[0079] in, Represents the detail layer, For image retrieval.

[0080] For further information, see Figures 1 to 3 In step 1, the self-modal decomposition is analyzed by Based on the pixel intensity distribution characteristics, the guided filter can automatically calculate the size of the filter window and the filter weight according to the image content. The specific definitions are as follows:

[0081]

[0082] in, Represents guided filtering, where both the input image and the guided image are set to , window size coefficient Control the spatial scale of the filter area, The larger the value, the smoother the filtered image is, and the regularization coefficient Affects the degree of edge preservation, in order to perform adaptive filtering operations based on the content of the image, and The value of is automatically calculated based on the pixel intensity skewness of the image:

[0083]

[0084]

[0085] in, The pixel intensity skewness is calculated as follows:

[0086]

[0087] in, Describing an image The median value of Describing an image The mean of Describing an image The standard deviation of Measures the asymmetry and discreteness of image pixel distribution;

[0088] Then, you can get The structural layer :

[0089] .

[0090] For further information, see Figures 1 to 3 , the modal information difference estimation in the step 2 is approximately estimated by the structural similarity between the decomposed modal features and the source image;

[0091]

[0092] in, Calculate the structural similarity of two images if High, indicating that the proportion of non-shared information in the source image is higher. , illustrating the source image More modal information than , and the shared information is less than ,Will Introducing the fusion rules of the base layer to reduce the information loss caused by modal differences, The same applies to time.

[0093] For further information, see Figures 1 to 3 The fusion rule design in step 3 includes the basic layer fusion rule:

[0094] For the base layer and , select according to pixel differences and obtain the fusion result , specifically:

[0095]

[0096] in, is the estimated modal difference information.

[0097] For further information, see Figures 1 to 3 The fusion rule design in step 3 also includes structural layer fusion rules:

[0098] First, the structural layer of the image Perform Wiener filtering to suppress interference information:

[0099]

[0100] Then, the intensity gradient changes in the horizontal and vertical directions are calculated to obtain the structural change intensity:

[0101]

[0102] Constructing the weight matrix , defined as follows:

[0103]

[0104] Perform structure layer fusion according to the weight matrix to obtain the fused structure layer :

[0105]

[0106] in, Represents an element-wise weighted multiplication operation.

[0107] For further information, see Figures 1 to 3The fusion rule design in step 3 also includes energy layer fusion rules:

[0108] According to energy level and The intensity value of the corresponding pixel position is used to obtain the fusion energy map by performing the maximum selection method :

[0109]

[0110] For further information, see Figures 1 to 3 , the image reconstruction in step 3: the final fused image It is a weighted combination of the above three fusion results:

[0111]

[0112] In this example, the image is decomposed:

[0113] Cross-modal decomposition: Treating the two source images as a whole, designing an objective function, and performing joint optimization steps to extract the shared information of the multimodal medical image pair, using the result as the base layer for each. Through linear operations on the images, the detail layer of the corresponding image is obtained.

[0114] Self-modal decomposition: Based on the characteristics of the detail layer of the image, the pixel intensity skewness is calculated, an adaptive guided filter is constructed, the detail layer is smoothed, and the structure layer and energy layer of the image are obtained.

[0115] Modal information difference estimation:

[0116] Structural similarity calculation: Calculate the structural similarity between the source image and the corresponding detail layer.

[0117] Modal Difference Estimation: Estimate the input modal difference based on the two obtained structural similarity values.

[0118] Image Fusion:

[0119] Fusion rule design: Fusion rules are designed based on the imaging characteristics of different layers, and modal difference estimation results are introduced to reduce image information loss.

[0120] Image reconstruction: Linearly merge the fused image layers and output the fused result.

[0121] 1. Image Decomposition

[0122] In this phase, the model extracts multi-scale features of multimodal image pairs through a global-local phase, including the following key tasks:

[0123] 1) Cross-modal decomposition

[0124] The source image and Input the common structure extractor and perform the following joint optimization steps:

[0125]

[0126] in, and Represents the source images and The base layer represents the intensity consistency area extracted from the original image; and Constrain the base layer to preserve the main content of the source image. and Suppress the noise of the base layer so that it has sparse edges. and are weight hyperparameters that control the strength of edge preservation and structure alignment respectively.

[0127] Based on the optimization solution and , calculate the modal information and use it as the detail layer of the corresponding image:

[0128]

[0129] in, Represents the detail layer, For image retrieval.

[0130] 2) Automodal Decomposition

[0131] In order to further extract the significant energy information remaining in the image details The present invention analyzes Based on the pixel intensity distribution characteristics, an adaptive guided filtering operation is designed and performed. Different from the traditional guided filtering method with fixed parameters, the adaptive guided filter can automatically calculate the size of the filtering window and the filtering weight according to the image content. The specific definition is as follows:

[0132]

[0133] in, Represents guided filtering, where both the input image and the guided image are set to , window size coefficient Control the spatial scale of the filter area, The larger it is, the smoother the filtered image will be. Affects the degree of edge preservation. In order to perform adaptive filtering operations based on the content of the image, and The value of is automatically calculated based on the pixel intensity skewness of the image:

[0134]

[0135]

[0136] in, The pixel intensity skewness is calculated as follows:

[0137]

[0138] in, Describing an image The median value of Describing an image The mean of Describing an image The standard deviation of . Measures the asymmetry and dispersion of image pixel distribution.

[0139] Then, you can get The structural layer :

[0140]

[0141] 2. Modal Contribution Estimation

[0142] The modal difference information is approximately estimated by the structural similarity between the decomposed modal features and the source image:

[0143]

[0144] in, Calculate the structural similarity of two images. If If the value is high, it means that the proportion of non-shared information in the source image is higher, that is, the modal information is richer. , illustrating the source image More modal information than , and the shared information is less than , so Introducing the fusion rules of the base layer can reduce the information loss caused by modal differences. The same is true when.

[0145] 3. Image fusion stage

[0146] 1) Basic layer fusion rules:

[0147] For the base layer and , select according to pixel differences and obtain the fusion result Specifically:

[0148]

[0149] in, is the estimated modal difference information.

[0150] 2) Structural layer fusion rules:

[0151] First, the structural layer of the image Perform Wiener filtering to suppress interference information:

[0152]

[0153] Then, the intensity gradient changes in the horizontal and vertical directions are calculated to obtain the structural change intensity:

[0154]

[0155] Constructing the weight matrix , defined as follows:

[0156]

[0157] Perform structure layer fusion according to the weight matrix to obtain the fused structure layer :

[0158]

[0159] in, Represents an element-wise weighted multiplication operation.

[0160] 3) Energy layer fusion rules:

[0161] According to energy level and The intensity value of the corresponding pixel position is used to obtain the fusion energy map by performing the maximum selection method :

[0162]

[0163] 4) Image reconstruction:

[0164] Final fused image It is a weighted combination of the above three fusion results:

[0165]

[0166] The medical image fusion method guided by modal difference information can better alleviate the modal information phenomenon. Three groups of image fusion examples are shown, namely MRI-CT fusion, MRI-SPECT fusion and MRI-PET fusion. Among them, the first and second images are source images respectively, and the third image is the fusion result. In each image, two red rectangular frames are used to mark the key observation areas and zoom in to the bottom of each image. It can be seen that the method proposed in the present invention can retain richer modal information. Specifically, in the MRI-CT fusion results, the skull base tumor part can better observe the correlation between the tumor and the skull. In the MRI-SPECT fusion results, the present invention retains more complete tissue information of the MRI image and alleviates the information loss caused by modal differences. In the MRI-PET fusion results, the present invention better handles the part where tissue information and metabolic information meet, and there is no distortion of pseudo-color information and loss of tissue information.

[0167] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A medical image fusion method based on modality difference information, characterized in that: The following steps are involved: Step 1: Image decomposition, which includes cross-modal decomposition and self-modal decomposition; Cross-modal decomposition treats the two source images as a whole, designs an objective function, performs joint optimization steps, extracts the common information of the multimodal medical image pair, and uses the results as the respective base layers. Through linear operations on the images, the detail layers of the corresponding images are obtained; Based on the characteristics of the detail layer of the image, the self-modal decomposition calculates the pixel intensity skewness, constructs an adaptive guided filter, smoothes the detail layer, and obtains the structure layer and energy layer of the image; Step 2: Modal information difference estimation, which includes structural similarity calculation and modal difference estimation; Structural similarity calculation includes calculating the structural similarity of the source image and the corresponding detail layer; Modality difference estimation includes estimating the input modality difference based on the two obtained structural similarity values; Step 3: Image fusion, which includes fusion rule design and image reconstruction; The fusion rule design includes designing fusion rules based on the imaging characteristics of different layers, while introducing modal difference estimation results to reduce image information loss; Image reconstruction involves linearly merging the fused image layers and outputting the fused result; The modal information difference estimation in the step 2 is approximately estimated by the structural similarity between the decomposed modal features and the source image; ； in, Calculate the structural similarity of two images if High, indicating that the proportion of non-shared information in the source image is higher. , illustrating the source image More modal information than , and the shared information is less than ,Will Introducing the fusion rules of the base layer to reduce the information loss caused by modal differences, The same applies to time; The fusion rule design in step 3 includes the following basic layer fusion rules: For the base layer and , select according to pixel differences and obtain the fusion result , specifically: ； in, is the estimated modal difference information.

2. The medical image fusion method based on modality difference information according to claim 1, characterized in that: In step 1, the cross-modal decomposition transforms the source image into and Input the common structure extractor and perform the following joint optimization steps: ； in, and Represents the source images and The base layer represents the intensity consistency area extracted from the original image; and constrains the base layer to preserve the contents of the source image, and Suppress the noise of the base layer so that it has sparse edges, and are weight hyperparameters, controlling the strength of edge preservation and structure alignment respectively; Based on the optimization solution and , calculate the modal information and use it as the detail layer of the corresponding image: ； in, Represents the detail layer, For image retrieval.

3. The medical image fusion method based on modality difference information according to claim 2, characterized in that: In step 1, the self-modal decomposition is carried out by analyzing The pixel intensity distribution characteristics of the image are used to guide the filter to calculate the size of the filter window and the filter weight according to the image content. The energy layer of the image is specifically defined as follows: ； in, Represents guided filtering, where both the input image and the guided image are set to , window size coefficient Control the spatial scale of the filter area, The larger the value, the smoother the filtered image is, and the regularization coefficient Affects the degree of edge preservation, and The value of is automatically calculated based on the pixel intensity skewness of the image: ； ； in, The pixel intensity skewness is calculated as follows: ； in, Describing an image The median value of Describing an image The mean of Describing an image The standard deviation of Measures the asymmetry and discreteness of image pixel distribution; Then, get The structural layer : 。 4. The medical image fusion method based on modality difference information according to claim 3, characterized in that: The fusion rule design in step 3 also includes structural layer fusion rules: First, the structural layer of the image Perform Wiener filtering to suppress interference information: ； Then, the intensity gradient changes in the horizontal and vertical directions are calculated to obtain the structural change intensity: ； Constructing the weight matrix , defined as follows: ； Perform structure layer fusion according to the weight matrix to obtain the fused structure layer : ； in, Represents an element-wise weighted multiplication operation.

5. The medical image fusion method based on modality difference information according to claim 4, characterized in that: The fusion rule design in step 3 also includes energy layer fusion rules: According to energy level and The intensity value of the corresponding pixel position is used to obtain the fusion energy map by performing the maximum selection method : 。 6. The medical image fusion method based on modality difference information according to claim 5, characterized in that: Image reconstruction in step 3: final fused image It is a weighted combination of the above three fusion results: 。

Images