Image fusion method based on detail enhancement and double-branch feature fusion

Abstract

The invention discloses an image fusion method based on detail enhancement and double-branch feature fusion, and relates to the technical field of image processing, and the method comprises the steps: S1, image detail enhancement: employing guided filtering to carry out the detail enhancement of a source image; s2, performing double-branch feature fusion based on guide filtering and rolling filtering; selecting a guide image under a double-branch channel based on guide filtering and rolling filtering, extracting a bright feature map, a dark feature map and a basic image, and fusing the image; s21, selection of a guide graph; the characteristics of the two filtering methods are comprehensively considered. According to the image fusion method based on detail enhancement and double-branch feature fusion, detail information of a fusion image is enhanced, and energy and structure significant edge information in a medical image are effectively reserved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image fusion method based on detail enhancement and dual-branch feature fusion. Background Art

[0002] In recent years, medical imaging technology has continued to develop and has played an increasingly important role in modern medical diagnosis. According to the different representation information of medical images, they can be roughly divided into two categories: structural images and functional images. Structural images, such as Computer Tomography (CT) and Magnetic Resonance Imaging (MRI), often have high resolution and can clearly reflect the structural information of relevant organs of the human body. Functional images, such as Positron Emission Computed Tomography (PET) and Single-Photon Emission Computed Tomography (SPECT), mainly provide metabolic information and functional status of relevant organs of the human body.

[0003] However, in clinical applications, although single-modality medical images can provide unique biomedical information, they all have their limitations, and medical information is often not comprehensive enough. Doctors often need to comprehensively analyze multiple modal medical images from the same location for medical diagnosis. However, manually integrating and analyzing multiple modal medical images is time-consuming and laborious, and the diagnostic accuracy is often not high. In order to solve the above problems, multimodal medical image fusion technology came into being. By effectively integrating information from multiple different modalities to provide more comprehensive and accurate diagnosis and treatment guidance, it has become one of the important research directions in the field of medical imaging.

[0004] At present, multimodal medical image fusion methods are increasingly used in the medical field. Generally speaking, most multimodal medical image fusion methods are studied at the pixel level. According to different technical fields, medical image fusion methods can be divided into four categories: sparse representation (SR-based), deep learning (DL-based), multi-scale transform (MST-based), and edge preserving filtering (EPF-based).

[0005] Sparse representation methods mainly realize the fusion of multimodal medical images based on dictionary learning and sparse coding. Medical image fusion based on sparse representation can not only improve the contrast and clarity of the image, but also enhance the recognizability and quantitative analysis capabilities of the targets in the image. Medical image fusion methods based on sparse representation rely on the quality and diversity of the data in dictionary selection and sparse coding, and need to deal with the challenges of computational complexity and parameter selection. In addition, the quality of the fused image is highly dependent on the reasonable fusion strategy of the sparse coefficients.

[0006] DL-based medical image fusion methods usually take advantage of the powerful feature extraction capabilities of deep learning models and use end-to-end training strategies to automatically learn and fuse information from different modalities. However, deep learning models are usually complex and require a lot of computing resources and efficient training algorithms, which limits their widespread application in clinical practice. In addition, the generalization ability and stability of the model remain challenges, especially when faced with data collected by different devices or different case types, the fusion effect may not be satisfactory. Finally, deep learning models have poor interpretability, and it is difficult to explain how the model fuses and processes information from different modalities, which may limit its application in medical imaging diagnosis and treatment decision-making.

[0007] MST-based medical image fusion methods usually transform multimodal images into a unified domain by applying transformation techniques of different scales, and then integrate the image information using certain fusion rules. However, MST-based medical image fusion methods rely on pre-selected transformation types and scales, which may lead to information loss or increase noise in the fusion process. Secondly, the combination and parameter settings of different scale transformations are sensitive to the fusion effect and require fine-tuning and optimization. In addition, when processing images of different modalities, how to effectively integrate multi-scale information and avoid artifacts using the MST method still needs further exploration.

[0008] EPF-based medical image fusion methods usually apply filters that retain image edge information to process images of different modalities, and then fuse the filtered images to enhance the contrast and clarity of the target structure. This method can effectively reduce noise and artifacts and maintain important structural details in the image, and is suitable for improving the diagnosis and analysis capabilities of medical images. For EPF-based medical image fusion methods, the selection of filter parameters has a significant impact on the fusion results, and careful adjustment is required to balance the edge preservation effect and fusion quality. Secondly, edge-preserving filters may cause loss or blurring of image details, especially in complex structures or low-contrast areas. In addition, the computational complexity of the filter is high, especially the processing of large-size medical images requires a lot of computing resources and time.

[0009] In summary, most of the current medical image fusion methods do not fully consider the impact of image details that reflect the condition of the lesion. Moreover, EPF-based medical image fusion methods often only use a single filtering method, the feature extraction capability of the image filter is insufficient, and the edge preservation effect and fusion quality need to be improved. Summary of the invention

[0010] The technical problem to be solved by the present invention is to provide an image fusion method based on detail enhancement and dual-branch feature fusion to enhance the detail information of the fused image and effectively retain the energy and structural significant edge information in the medical image.

[0011] The present invention adopts the following technical solutions to achieve the invention objectives: An image fusion method based on detail enhancement and dual-branch feature fusion, characterized by comprising: S1: image detail enhancement, using guided filtering to enhance the details of the source image; S2: Dual-branch feature fusion based on guided filtering and rolling filtering; The selection of the guide image under the dual-branch channel based on the guide filter and the rolling filter, the extraction of the bright feature map, the dark feature map and the basic image and the image fusion strategy are adopted; S21: Selection of guide map; considering the characteristics of the two filtering methods, a dual-branch feature extraction model of joint guide filtering and rolling guide filtering is established to extract more comprehensive and representative image features; S22: feature map fusion; S221: For an image, including bright features and dark features, the image is decomposed into three parts: a bright feature map, a dark feature map and a base image; S222: Based on the image decomposition, the bright features, dark features and the base image of the two images are fused respectively, and then the fused three parts are further fused to obtain a final fused image; S3: Experimental results and analysis.

[0012] As a further limitation of the present technical solution, the specific process of S1 is as follows: S11: Given X g represents the guide image, X represents the source image, that is, the input image. Y represents the output image after guided filtering; Corresponding source image X Any location k The two-dimensional filter window is w k , the window size is (2 r +1)×(2 r -1) indicates thatr Represents the side length of the window. k The center pixel i , its linear relationship can be expressed as follows: (1) in: a k and b k Represents the linear constant coefficient in the window; S12: It can be seen from formula (1) that the linear transformation is composed of linear coefficients a k and b k Determine, and find the gradient of both ends of the above equation respectively, and we can get: (2) From formula (2), we can see that the gradient of the output image is linearly related to the gradient of the guide image. a k Determines the gradient retention capability and can be regarded as an adaptive adjustment factor of the output image gradient. a k and b k The value of is evaluated by the following formula; (3) in: represents the regularization parameter; S13: Obtain the optimal linear coefficient according to the least squares method a k and b k ,as follows: (4) (5) in: and express mean and standard deviation within the window; Indicates the total number of pixel blocks in the window; Indicates the input image in the window The mean of By averaging the linear coefficients under different windows, the corresponding pixels of the final output image are as follows: (6) in: ; ; express w conjugation of; S14: Detail enhancement of the source image is achieved through equation (7); (7) in: represents the image after detail enhancement; represents the enhancement coefficient, which is taken in this method .

[0013] As a further limitation of the present technical solution, the specific process of S21 is as follows: S211: for guided filtering; The input image is filtered using the local minimum map of the input image as a guide image; (8) in: represents guided filtering; j represents the current filter model scale, , n Represents the total number of scales; represents the original input image after detail enhancement according to formula (7); represent( j -1) Local minimum map under scale; Indicates j The filter window size at each scale; The local minimum map of images at different scales can be obtained by the following formula: (9) in: represents the image erosion operator; Representative j Structural element objects at different scales; represent j The original input image at scale; Based on the local maximum map as the guide image, the formula (8) is further filtered as follows: (10) in: and denote the input image and the guided image of the guided filtering process respectively; represent The local maximum map of is obtained by the following formula: (11) in: represents the image dilation operator; S212: for bilateral filtering; Given an input detail-enhanced image and local extrema guidance image , guiding the output of bilateral filtering It can be calculated by the following formula: (12) in: Represents the pixel in the bilateral filtering output image p The value of represents the normalization factor; Represents all pixels p a collection of related pixel locations; and Represent the image after detail enhancement and local extrema guidance image Medium Pixels q The value of Represents the weight parameter of the bilateral filter, which is determined by the spatial distance Distance from pixel value Control can be obtained by the following formula: (13) in: and Represent the spatial domain and pixel domain weights respectively; Considering equations (12) and (13) comprehensively, the iterative process of guided bilateral filtering can be summarized as follows: (14) in: stands for guided bilateral filtering; t Indicates the number of iterations; The guided bilateral filter in (14) is extended to multiple scales and the image after detail enhancement is The local minimum map of the image is used as a guide image to enhance the details of the image Filter as follows: (15) in: Represents the current guided bilateral filter model scale j The output image below; Based on the local maximum map as the guide image, the Further filtering is performed as follows: (16) in: and Represent the input and guidance images of the bilateral filtering process respectively; represent The local maximum map of .

[0014] As a further limitation of the present technical solution, the specific process of S221 is as follows: For the dual-branch fusion model, the bright feature map and dark feature map at different scales can be obtained from the difference image of two adjacent filtered output images in turn through equations (17) and (18), as follows: (17) (18) in: and Respectively represent j Bright feature maps and dark feature maps obtained based on guided filtering at different scales; and They represent the j Bright feature maps and dark feature maps obtained based on guided bilateral filtering at different scales; The last filter scale n The output image obtained is used as the low-frequency basis image of the input image. The implementation process is as follows: (19) in: and Represent the basis images obtained based on guided filtering and guided bilateral filtering, respectively.

[0015] As a further limitation of the technical solution, the specific process of S222 is as follows: The high-intensity high-frequency features of an image often correspond to the prominent sharp features in the image. Therefore, the bright feature maps in the two images are fused based on the element maximum criterion of the bright feature maps in the two images at each scale. The process is as follows: (20) Where: superscripts 1 and 2 represent the indexes of the two input images; Represents the image after the bright features of the two images are fused based on guided filtering; It is an image obtained by fusion of bright features of two images based on guided bilateral filtering; Indicates for and The fused bright feature map fused image; The dark feature maps in the two images are fused based on the minimum element criterion of the dark feature maps in the two images at each scale, as follows: (twenty one) in: and They represent the images obtained by fusion of dark features of two images obtained based on guided filtering and guided bilateral filtering respectively; Indicates for and The fused bright feature map fused image; Guided filtering and guided bilateral filtering are performed based on local extreme value maps. The obtained bright feature maps and dark feature maps are often not representative enough. Therefore, the weights of each image at different scales are determined based on their entropy values. Then, the comprehensiveness and representativeness of the fused image features are enhanced by multiplying the bright feature and dark feature maps at each scale by the relevant weights. The integration strategy of the bright feature maps and dark feature maps at each scale is as follows: (twenty two) in: and Respectively represent j The weights of the bright feature map and the dark feature map at each scale; and They represent the features after the fusion of bright feature maps and dark feature maps of all scales; and The calculation method is as follows: (twenty three) in: and Respectively and( )’s entropy value; It can be seen from formula (23) that the feature map with the smallest entropy value is assigned a weight of 1, while the weights of feature maps at other scales are all greater than 1, which makes the bright feature maps and dark feature maps at most scales enhanced to varying degrees; In order to retain most of the basic information of the medical image in the final fused image, the element maximum criterion is used to fuse the basic images of the two input images, as shown in the following formula: (twenty four) in: and They represent the feature maps after the fusion of the basic images obtained based on guided filtering and guided bilateral filtering respectively; Indicates for and The fused base image is a fused image; They represent the base images of the two input images obtained based on guided filtering; Respectively represent the base images of the two input images obtained based on guided bilateral filtering; The fused bright feature maps obtained by equations (22) and (24) are , dark feature map and the base image Fusion is performed to obtain the final fusion result of the two input images , as shown in formula (25): (25) Through the proposed dual-branch feature fusion strategy, the sharp features of the image can be significantly enhanced, and the basic information of the source image can be retained, ultimately obtaining a more comprehensive and representative fused image.

[0016] Compared with the prior art, the advantages and positive effects of the present invention are: The present invention uses guided filtering to preprocess the original medical image, highlight the important details in the image, achieve image detail enhancement, and improve the fusion effect and visualization effect. A dual-branch feature fusion model is built, using the local extreme value map as a guide, combining guided filtering and bilateral filtering, enhancing the feature extraction ability of the image filter, and further improving the edge preservation effect and fusion quality of the fused image. Through multimodal brain medical image fusion experiments and comparison with other advanced methods, the superiority of the DEDF method is verified from the perspectives of qualitative and quantitative analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is the overall framework diagram of the present invention.

[0018] Figure 2 CT and MRI medical images of the brain.

[0019] Figure 2 (a) is the original CT image. Figure 2 (b) is the original MRI image.

[0020] Figure 3 PET and MRI medical images of the brain.

[0021] Figure 3 (a) is the original PET image. Figure 3 (b) is the original MRI image.

[0022] Figure 4 These are the SPECT and MRI medical images of the brain of the present invention.

[0023] Figure 4 (a) is the original SPECT image. Figure 4 (b) is the original MRI image.

[0024] Figure 5 This is the fusion image result of CR and MRI of the present invention.

[0025] Figure 6 This is the fusion image result of PET and MRI of the present invention.

[0026] Figure 7 This is the fusion image result of SPECT and MRI of the present invention. DETAILED DESCRIPTION

[0027] A specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings, but it should be understood that the protection scope of the present invention is not limited by the specific implementation.

[0028] The present invention comprises the following steps: S1: Image detail enhancement, using guided filtering to enhance the details of the source image; The detail enhancement of the original image has an important impact on the effect of medical image fusion. In medical image fusion, the original image usually contains rich detail information, including the subtle features of the lesion area and the structure of the surrounding normal tissue. If the details of the original image are unclear or missing, key information may be lost during the fusion process, resulting in the final fused image not being clear or accurate enough. Therefore, in order to make the fused image clearer and more accurate, this paper first enhances the details of the source image before the image is fused; Guided filtering (GF) is an edge-aware image filtering technique. It has been widely used in many fields because of its good visual quality, high model operation efficiency, easy implementation, and no influence from gradient reversal artifacts. Therefore, guided filtering is used in this section to enhance the details of the source image. The specific implementation process is as follows: The specific process of S1 is as follows: S11: Given X g represents the guide image, X represents the source image, that is, the input image.Y represents the output image after guided filtering; Corresponding source image X Any location k The two-dimensional filter window is w k , the window size is (2 r +1)×(2 r -1) indicates that r Represents the side length of the window. k The center pixel i , its linear relationship can be expressed as follows: (1) in: a k and b k Represents the linear constant coefficient in the window; S12: It can be seen from formula (1) that the linear transformation is composed of linear coefficients a k and b k Determine, and find the gradient of both ends of the above equation respectively, and we can get: (2) From formula (2), we can see that the gradient of the output image is linearly related to the gradient of the guide image. a k Determines the gradient retention capability and can be regarded as an adaptive adjustment factor of the output image gradient. a k and b k The value of is evaluated by the following formula; (3) in: represents the regularization parameter, preventing a k Oversize and control smoothness; S13: Obtain the optimal linear coefficient according to the least squares method a k and b k ,as follows: (4) (5) in: and express mean and standard deviation within the window; Indicates the total number of pixel blocks in the window; Indicates the input image in the window The mean of By averaging the linear coefficients under different windows, the final output image (a pixel in the output image) can be obtained. Y i The value of varies with the linear coefficients in the filter windows of different sizes. At this time, the pixels of the output image Y i There are multiple, and then the average is calculated to get the corresponding pixels of 1 output image pixel) as follows: (6) in: ; ; express w conjugation of; S14: Detail enhancement of the source image is achieved through equation (7); (7) in: represents the image after detail enhancement; represents the enhancement coefficient, which is taken in this method .

[0029] S2: Dual-branch feature fusion based on guided filtering and rolling filtering; The selection of the guide image under the dual-branch channel based on the guide filter and the rolling filter, the extraction of the bright feature map, the dark feature map and the basic image and the image fusion strategy are adopted; S21: Selection of guide map; Guided filtering can retain the edge information of the image, maintain the overall structure and clarity of the image, but it may cause excessive edge enhancement when processing images with complex textures and structures. Rolling filtering is suitable for fast smoothing of images and has certain noise reduction capabilities, but it is easy to cause image edge blur. Considering the characteristics of the two filtering methods, a dual-branch feature extraction model of joint guided filtering and rolling guided filtering is established to extract more comprehensive and representative image features; Both filtering methods in the dual-branch model require a guide image, and the selection of the guide image is crucial to ensure the filter feature extraction capability. Using the local extreme value map as the guide image can effectively suppress the significant features of the source image, thereby extracting more significant image features.

[0030] The specific process of S21 is as follows: The local extreme value graph is used as the guide graph, and its filtering process is extended to multiple scales. The construction method is as follows: S211: for guided filtering; The input image is filtered using the local minimum map of the input image as a guide image; (8) in: represents guided filtering; j represents the current filter model scale, , n Represents the total number of scales; represents the original input image after detail enhancement according to formula (7); represent( j -1) Local minimum map under scale; Indicates j The filter window size at each scale; The local minimum map of images at different scales can be obtained by the following formula: (9) in: represents the image erosion operator; Representative j Structural element objects at different scales; represent j The original input image at scale; Based on the local maximum map as the guide image, the formula (8) is further filtered as follows: (10) in: and denote the input image and the guided image of the guided filtering process respectively; represent The local maximum map of is obtained by the following formula: (11) in: represents the image dilation operator; S212: for bilateral filtering; Bilateral filtering comprehensively considers the influence of spatial domain and value domain on filtering, so it can achieve the effect of maintaining image edges, reducing noise and smoothing. In order to enhance the ability to maintain image details and structures during the filtering process and facilitate fusion with the results of guided filtering, bilateral filtering is extended to guided bilateral filtering. The local extreme value map is used as the guided map, and its filtering process is extended to multiple scales. Its construction method is introduced as follows: Given an input detail-enhanced image and local extrema guidance image , guiding the output of bilateral filtering It can be calculated by the following formula: (12) in: Represents the pixel in the bilateral filtering output image p The value of represents the normalization factor; Represents all pixels p a collection of related pixel locations; and Represent the image after detail enhancement and local extrema guidance image Medium Pixels q The value of Represents the weight parameter of the bilateral filter, which is determined by the spatial distance Distance from pixel value Control can be obtained by the following formula: (13) in: and Represent the spatial domain and pixel domain weights respectively; Considering equations (12) and (13) comprehensively, the iterative process of guided bilateral filtering can be summarized as follows: (14) in: stands for guided bilateral filtering; t Indicates the number of iterations; The guided bilateral filter in (14) is extended to multiple scales and the image after detail enhancement is The local minimum map of the image is used as a guide image to enhance the details of the image Filter as follows: (15) in: Represents the current guided bilateral filter model scale j The output image below; Based on the local maximum map as the guide image, the Further filtering is performed as follows: (16) in: and Represent the input and guidance images of the bilateral filtering process respectively; represent The local maximum map of .

[0031] and The calculation of can refer to equations (9) and (11) respectively, which will not be repeated here.

[0032] S22: Feature map fusion.

[0033] S221: For an image, including bright features and dark features, the image is decomposed into three parts: a bright feature map, a dark feature map and a base image.

[0034] The specific process of S221 is as follows: For the dual-branch fusion model, the bright feature map and dark feature map at different scales can be obtained from the difference image of two adjacent filtered output images in turn through equations (17) and (18), as follows: (17) (18) in: and Respectively represent j Bright feature maps and dark feature maps obtained based on guided filtering at different scales; and They represent the j Bright feature maps and dark feature maps obtained based on guided bilateral filtering at different scales; The last filter scale n The output image obtained is used as the low-frequency basis image of the input image. The implementation process is as follows: (19) in: and Represent the basis images obtained based on guided filtering and guided bilateral filtering, respectively.

[0035] S222: Based on the image decomposition, the bright features, dark features and the base image of the two images are fused respectively, and then the fused three parts are further fused to obtain a final fused image.

[0036] The specific process of S222 is as follows: In general, high-intensity high-frequency features of an image often correspond to prominent sharp features in the image. Therefore, the bright feature maps of the two images are fused based on the element maximum criterion of the bright feature maps in the two images at each scale. The process is as follows: (20) Where: superscripts 1 and 2 represent the indexes of the two input images; Represents the image after the bright features of the two images are fused based on guided filtering; It is an image obtained by fusion of bright features of two images based on guided bilateral filtering; Indicates for and The fused bright feature map fused image; The dark feature maps in the two images are fused based on the minimum element criterion of the dark feature maps in the two images at each scale, as follows: (twenty one) in: and They represent the images obtained by fusion of dark features of two images obtained based on guided filtering and guided bilateral filtering respectively; Indicates for and The fused bright feature map fused image; Guided filtering and guided bilateral filtering are performed based on local extreme value maps. The obtained bright feature maps and dark feature maps are often not representative enough. Therefore, the weights of each image at different scales are determined based on their entropy values. Then, the comprehensiveness and representativeness of the fused image features are enhanced by multiplying the bright feature and dark feature maps at each scale by the relevant weights. The integration strategy of the bright feature maps and dark feature maps at each scale is as follows: (twenty two) in: and Respectively represent j The weights of the bright feature map and the dark feature map at each scale; and They represent the features after the fusion of bright feature maps and dark feature maps of all scales; and The calculation method is as follows: (twenty three) in: and Respectively and( )’s entropy value; It can be seen from formula (23) that the feature map with the smallest entropy value is assigned a weight of 1, while the weights of feature maps at other scales are all greater than 1, which makes the bright feature maps and dark feature maps at most scales enhanced to varying degrees; In order to retain most of the basic information of the medical image in the final fused image, the element maximum criterion is used to fuse the basic images of the two input images, as shown in the following formula: (twenty four) in: and They represent the feature maps after the fusion of the basic images obtained based on guided filtering and guided bilateral filtering respectively; Indicates for and The fused base image is a fused image; They represent the base images of the two input images obtained based on guided filtering; Respectively represent the base images of the two input images obtained based on guided bilateral filtering; The fused bright feature maps obtained by equations (22) and (24) are , dark feature map and the base image Fusion is performed to obtain the final fusion result of the two input images , as shown in formula (25): (25) Through the proposed dual-branch feature fusion strategy, the sharp features of the image can be significantly enhanced, and the basic information of the source image can be retained, ultimately obtaining a more comprehensive and representative fused image.

[0037] S3: Experimental results and analysis.

[0038] S31: Dataset Description In order to verify the fusion effect of the proposed method, three groups of commonly used brain medical images are selected as the experimental data of this section. These medical images include: 10 pairs of CT and MRI medical images of the brain, 10 pairs of PET and MRI medical images of the brain, and 10 pairs of SPECT and MRI medical images of the brain. The above three groups of images are respectively as follows: Figure 2-Figure 4 In addition, the resolution of all images is 256×256.

[0039] S32: Comparison method and parameter setting In order to verify the effectiveness of the proposed method, six the-state-of-the-art image fusion methods were selected as comparison methods in this experiment, which are described as follows.

[0040] LEGFF, this method combines local extrema graph and guided filtering to achieve image fusion. CFL, this method is a multimodal image fusion method based on coupled dictionary learning. FGF, this method achieves effective image fusion from the perspective of protecting image edge information. MST based method, this method processes the high-pass and low-pass subbands of the image respectively to obtain the final fused image. NSFB, this method has made in-depth development and application of the nonsubsampled contourlet transform (NSCT). GF, this method mainly achieves image fusion based on rolling image filtering.

[0041] The parameter settings of the six comparison methods all adopt the optimal parameters given in the relevant literature. For the proposed method, the parameter settings are as follows. In the multi-scale fusion process, the number of scales is set to 4, and the spatial distance of the bilateral filter is Distance from pixel value Set to 5 and 20 respectively.

[0042] S33: Fusion effect evaluation index Evaluation of fusion results by visual perception is mostly subjective. Due to different visual levels and psychological factors of different observers, the evaluation results will vary greatly. Therefore, in order to prove the effectiveness of DEDF, qualitative and quantitative comparisons are made with the contrast method, and 10 commonly used evaluation indicators in the field of medical image fusion are selected to evaluate the fusion effect. The evaluation indicators used include: standard deviation (STD), Peak Signal-to-Noise Ratio (PSNR), Mutual Information (MI), average gradient (AG), coefficient of correlation (CC), the sum of the correlations of differences (SCD), the entropy (EN), spatial frequency (SF), structural similarity index measure (SSIM) and the Feature Mutual Information (FMI).

[0043] STD and AG are mainly used to evaluate the detail retention of the fused image, EN can measure the intensity distribution of the fused image, MI is used to calculate the amount of information transferred from the input image to the fused image, CC is an indicator used to measure the linear correlation between the fused image and the source image, PSNR is used to measure the ratio between the effective information and noise of the image, and SSIM is used to measure the degree of retention of the structural information of the input image during the fusion process. VIF evaluates the image fusion quality from the perspective of information communication and sharing, SCD uses the sum of difference correlations as the quality measure of the fused image, and FMI is used to evaluate the dependence of the fused image on the input image. It should be noted that among the 10 evaluation indicators selected in this paper, higher values ​​indicate better image fusion performance.

[0044] S34: Experimental results and analysis In order to verify the superiority of DEDF, the fused images obtained by different fusion methods are first analyzed intuitively. Then, the quality of fused images of different fusion methods is discussed based on multiple evaluation indicators, and the information reflected by the fused images is further explained.

[0045] S341: Qualitative results on CT-MRI dataset In order to save space and facilitate discussion, in this part of the experiment, the fourth pair of CR and MRI fusion images are used as an example for qualitative analysis. The fourth pair of CR and MRI fusion images obtained by different methods are shown in the figure below. Figure 5 shown.

[0046] The goal of CT / MRI fusion is to combine images from two different modalities, CT and MRI, to comprehensively utilize their respective advantages and obtain images that contain more comprehensive information. Figure 5 (g) It can be seen that the fusion image based on NSFB loses the energy information of the CT and MRI images, resulting in reduced image contrast and poor visual effects. Figure 5 (h) It can be seen that although the contrast of the rolling filter fusion image is good, the information of the outer edge of the MRI is lost. The fusion image of LEGFF is darker overall, resulting in a blurred image texture. Although the fusion images of CFL, FGF and MST have high visual contrast, they all lose the texture and other detail information of the MRI image to a certain extent. The method in this paper focuses on improving the image contrast and retaining the edge and detail information of the input image. Therefore, compared with the comparative method, it can be concluded that DEDF retains the important features of single-modality medical images, and can effectively improve the image contrast, strengthen the details and grayscale information of the image, and achieve better fusion effect.

[0047] S342: Qualitative results on the PET-MRI dataset In this part of the experiment, the fused images of the fourth pair of PET and MRI are taken as an example for demonstration and qualitative analysis. The fused images obtained by different methods are shown in Figure 2. Figure 6 As shown in Figure 2, PET is a functional image that can reflect the metabolic state of tissues. MRI is a structural image that mainly characterizes tissue structure and texture information.

[0048] Figure 6 (a) and Figure 6 (b) are the original images of PET and MRI respectively. It can be seen that except for the NSFB method, other methods have well preserved the color information in the PET functional image. The main difference lies in the preservation of texture and detail information in MRI. It can be found through observation that the MRI part of the fusion image obtained by the LEGFF method has a darkened background, resulting in poor contrast effect, but the functional information and detail information are well preserved. For the GF method, the bright hot spots in PET are weakened too much, which weakens the important physiological information in the PET image, and the soft tissue texture information in the MRI modality is seriously lost. Figure 6 (f) It can be seen that the MST fusion result retains the color information of the PET image to the greatest extent, but the bright hot spots obviously cover up some of the detailed information in the MRI. Figure 6 (d), 6(e) and 6(i), CFL, FGF and DEDF obtain the best visual effects among all the methods, and the fused images have high contrast and intensity while retaining clear structural details and functional information.

[0049] S343: Qualitative results on the SPECT-MRI dataset Taking the 7th pair of SPECT-MRI fusion images as an example, Figure 6 The fusion image results obtained by different methods are shown. Figure 7 (a) and 7(b) show the original images of SPECT and MRI, respectively. Figure 7 The image fusion results in (g) show that the NSFB method weakens the bright hot spots in the SPECT modality too much, and loses a lot of structural details, contrast and intensity. Figure 7 As shown in (h), GF has a poor fusion effect on the functional information in SPECT, and a lot of energy information is lost, resulting in a black block effect, making it impossible to observe the lesion area normally. The fusion result of CFL shows uneven brightness, and edge information loss can be seen in the bone area of ​​the MRI part. The fusion effect of the three methods of LEGFF, FGF and MST is good, but there is also edge information loss in the bone area of ​​the MRI part, and the detailed information of the soft tissue part is not comprehensive enough. For the DEDF method, the contrast and brightness of the fused image are the highest among all methods, and the structural details and edge information in MRI are well preserved. When fusing the SPECT part, artifacts appeared at the edge, but it had little effect on the observation of the image.

[0050] S344: Quantitative results analysis The qualitative analysis of the fusion results of the above three groups of multimodal medical images shows that the proposed method has better image fusion effect and more sufficient structural details extraction. In order to further verify the superiority of the proposed method, in this section, the fusion results of each method are quantitatively analyzed from an objective perspective. As can be seen from Section 3.1, each medical image dataset contains 10 pairs of images. According to the fusion results of each pair of images, the measurement value of the quantitative evaluation index is calculated, and the average value of the evaluation index of the 10 pairs of images is taken as the quantitative evaluation result of the reorganized dataset. The quantitative evaluation values ​​of the fusion results of the three groups of multimodal medical images are shown in Table 1, Table 2 and Table 3 respectively. In each table, the bold font represents the best fusion result, and the underlined font represents the second best result.

[0051] Table 1 Comparison of quantitative index evaluation results of CT-MRI dataset LEGFF CFL FGF MST NSFB GF Proposed STD 84.6919 87.5501 87.6461 84.7438 61.4701 <![CDATA[ 88.8702 ]]> 90.2069 PSNR <![CDATA[ 61.9333 ]]> 61.4766 61.4552 61.6463 63.7905 61.0476 61.0607 MI 2.6107 2.7360 2.7708 2.5080 <![CDATA[ 3.0973 ]]> 3.7744 2.3366 AG 7.3490 <![CDATA[ 7.5222 ]]> 7.1211 7.1618 4.5904 6.4248 9.3516 CC 0.8109 0.8080 0.8059 0.7858 <![CDATA[ 0.8206 ]]> 0.8012 0.8280 SCD 1.4582 <![CDATA[ 1.4972 ]]> 1.4969 1.2435 0.9608 1.4824 1.5927 EN 4.2334 4.2872 4.3789 4.6927 <![CDATA[ 4.4098 ]]> 3.1749 4.0754 SF 32.7452 <![CDATA[ 33.2718 ]]> 32.8199 30.1645 19.1697 31.3285 37.9424 SSIM 0.9511 <![CDATA[ 0.9406 ]]> 0.9402 0.9138 0.8684 0.8926 0.8910 FMI 0.8748 0.8772 0.8796 0.8729 0.8726 <![CDATA[ 0.8788 ]]> 0.8532 Table 2 Comparison of quantitative index evaluation results of PET-MRI dataset LEGFF CFL FGF MST NSFB GF Proposed STD 89.4779 107.2109 103.4998 95.5617 103.1732 104.1639 <![CDATA[ 106.8004 ]]> PSNR 60.9945 58.6834 59.0432 <![CDATA[ 59.1001 ]]> 57.914 58.9183 58.7849 MI 3.1852 3.5493 3.5899 3.0489 <![CDATA[ 4.0950 ]]> 4.2780 3.1820 AG 12.2038 <![CDATA[ 12.3625 ]]> 11.9477 12.3117 9.6047 11.2689 12.9130 CC 0.8079 0.7971 0.8020 0.7587 <![CDATA[ 0.8044 ]]> 0.8002 0.7970 SCD 1.2107 0.6594 0.5604 0.2152 0.6171 0.5451 <![CDATA[ 0.6798 ]]> EN <![CDATA[ 5.2996 ]]> 4.8342 5.0909 5.5776 4.9710 4.7672 4.9306 SF 40.4445 <![CDATA[ 45.0113 ]]> 41.9103 41.7700 38.3704 40.9439 48.5636 SSIM 0.8657 0.8437 <![CDATA[ 0.8677 ]]> 0.8632 0.7210 0.8078 0.8824 FMI 0.7937 0.8392 0.8531 0.8492 0.8387 <![CDATA[ 0.8508 ]]> 0.8245 Table 3 Comparison of quantitative index evaluation results of SPECT-MRI dataset LEGFF CFL FGF MST NSFB GF Proposed STD 54.9613 58.3583 56.0242 52.9884 <![CDATA[ 77.2246 ]]> 55.5805 77.6194 PSNR <![CDATA[ 66.5259 ]]> 65.9119 66.2823 66.6726 59.7267 65.0564 62.1653 MI 3.1954 2.9878 3.6186 3.4023 <![CDATA[ 3.8866 ]]> 5.0913 2.7398 AG 6.5455 6.6011 6.0959 5.9737 5.7464 <![CDATA[ 8.0067 ]]> 8.3322 CC 0.8435 0.8398 0.8526 0.8295 0.8489 0.7723 0.8430 SCD 1.3521 1.4250 1.5329 1.0568 1.9680 0.7528 <![CDATA[ 1.6700 ]]> EN 5.4110 5.1441 5.4443 5.5244 <![CDATA[ 5.5428 ]]> 4.9844 5.5538 SF 19.4449 19.7665 17.696 17.5986 16.7509 33.0779 <![CDATA[ 27.4068 ]]> SSIM 0.9520 0.9317 0.9461 0.9421 <![CDATA[ 0.9466 ]]> 0.7118 0.8645 FMI 0.8585 0.8590 <![CDATA[ 0.8672 ]]> 0.8711 0.8501 0.8004 0.8347

[0052] As shown in Table 1, on the CT-MRI dataset, the DEDF algorithm proposed in this paper achieved the best performance in five indicators: STD, AG, CC, SCD, and SF. The values ​​of STD, CC, and SF are the highest, indicating that DEDF better preserves the structural features of the original CT and MRI images, including edges, textures, and structural details, compared with the fused images of other comparison methods. DEDF has the highest AG value, indicating that the proposed method has higher visual quality. The value of SCD is the highest, indicating that the fused image obtained by DEDF is smoother and more coherent. In addition, NSFB achieved the best performance in PSNR and the second best performance in the other three indicators (i.e., MI, CC, and EN). CFL achieved the second best performance in four evaluation indicators (i.e., AG, SCD, SF, and SSIM). In summary, consistent with the qualitative comparison results, the fusion effect obtained by the DEDF method on the CT-MRI dataset is significantly better than that of other comparison methods.

[0053] According to the evaluation index results of each method in Table 2, on the PET-MRI dataset, the DEDF method performs best in the three evaluation indicators of AG, SF and SSIM, and obtains the second best performance in the two indicators of STD and SCD. It shows that the fusion image of DEDF can obtain better visual effects, while better retaining structural features and functional information. DEDF performs poorly on the EN evaluation index, which is due to the fact that we fuse images based on the element-by-element maximum strategy. In addition, the LEGFF method performs best in three indicators and second best in one indicator, and is the best performing method among the comparison methods. In summary. The quantitative evaluation results of the PET-MRI dataset show that the indicators of the DEDF fusion image are relatively balanced, and the overall performance is better than other comparison methods.

[0054] From the comparison results in Table 3, it can be seen that on the SPECT-MRI dataset, the DEDF method performs best in the three evaluation indicators of STD, AG and EN, reaching 77.6149, 8.3322 and 5.5538 respectively. The second best performance is obtained in the two indicators of SCD and SF, reaching 1.6700 and 27.4068 respectively. Due to the addition of the pre-processing process of detail enhancement in DEDF, the ability to retain the content in the original functional image may be weakened, and the MI value is low. In addition, the evaluation indicators of the fusion effect of LEGFF and MST also perform well, which is consistent with the qualitative evaluation results. Overall, the objective evaluation results show that DEDF can provide higher quality fusion results on the SPECT-MRI dataset.

[0055] The above disclosure is only a specific embodiment of the present invention, but the present invention is not limited thereto, and any changes that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. An image fusion method based on detail enhancement and dual-branch feature fusion, characterized in that: include: S1: Image detail enhancement, using guided filtering to enhance the details of the source image; S2: Dual-branch feature fusion based on guided filtering and rolling filtering; The selection of the guide image under the dual-branch channel based on the guide filter and the rolling filter, the extraction of the bright feature map, the dark feature map and the basic image and the image fusion strategy are adopted; S21: Selection of guide map; considering the characteristics of the two filtering methods, a dual-branch feature extraction model of joint guide filtering and rolling guide filtering is established to extract more comprehensive and representative image features; S22: feature map fusion; S221: For an image, including bright features and dark features, the image is decomposed into three parts: a bright feature map, a dark feature map and a base image; S222: Based on the image decomposition, the bright features, dark features and the base image of the two images are fused respectively, and then the fused three parts are further fused to obtain a final fused image; S3: Experimental results and analysis.

2. The image fusion method based on detail enhancement and dual-branch feature fusion according to claim 1, characterized in that: The specific process of S1 is as follows: S11: Given X g represents the guide image, X represents the source image, that is, the input image. Y represents the output image after guided filtering; Corresponding source image X Any location k The two-dimensional filter window is w k , the window size is (2 r +1)×(2 r -1) indicates that r Represents the side length of the window. k The center pixel i , its linear relationship can be expressed as follows: （1） in: a k and b k Represents the linear constant coefficient in the window; S12: It can be seen from formula (1) that the linear transformation is composed of linear coefficients a k and b k Determine, and find the gradient of both ends of the above equation respectively, and we can get: （2） From formula (2), we can see that the gradient of the output image is linearly related to the gradient of the guide image. a k Determines the gradient retention capability and can be regarded as an adaptive adjustment factor of the output image gradient. a k and b k The value of is evaluated by the following formula; （3） in: represents the regularization parameter; S13: Obtain the optimal linear coefficient according to the least squares method a k and b k ,as follows: （4） （5） in: and express mean and standard deviation within the window; Indicates the total number of pixel blocks in the window; Indicates the input image in the window The mean of By averaging the linear coefficients under different windows, the corresponding pixels of the final output image are as follows: （6） in: ; ； express w conjugation of; S14: Detail enhancement of the source image is achieved through equation (7); （7） in: represents the image after detail enhancement; represents the enhancement coefficient, which is taken in this method .

3. The image fusion method based on detail enhancement and dual-branch feature fusion according to claim 2, characterized in that: The specific process of S21 is as follows: S211: for guided filtering; The input image is filtered using the local minimum map of the input image as a guide image; （8） in: represents guided filtering; j represents the current filter model scale, , n Represents the total number of scales; represents the original input image after detail enhancement according to formula (7); represent( j -1) Local minimum map under scale; Indicates j The filter window size at each scale; The local minimum map of images at different scales can be obtained by the following formula: （9） in: represents the image erosion operator; Representative j Structural element objects at different scales; represent j The original input image at scale; Based on the local maximum map as the guide image, the formula (8) is further filtered as follows: （10） in: and denote the input image and the guided image of the guided filtering process respectively; represent The local maximum map of is obtained by the following formula: （11） in: represents the image dilation operator; S212: for bilateral filtering; Given an input detail-enhanced image and local extremum guidance image , guiding the output of bilateral filtering It can be calculated by the following formula: （12） in: Represents the pixel in the bilateral filtering output image p The value of represents the normalization factor; Represents all pixels p a collection of related pixel locations; and Represent the images after detail enhancement and local extremum guidance image Medium Pixels q The value of Represents the weight parameter of the bilateral filter, which is determined by the spatial distance Distance from pixel value Control can be obtained by the following formula: （13） in: and Represent the spatial domain and pixel domain weights respectively; Considering equations (12) and (13) comprehensively, the iterative process of guided bilateral filtering can be summarized as follows: （14） in: stands for guided bilateral filtering; t Indicates the number of iterations; The guided bilateral filter in (14) is extended to multiple scales and the image after detail enhancement is The local minimum map of the image is used as a guide image to enhance the details of the image Filter as follows: （15） in: Represents the current guided bilateral filter model scale j The output image below; Based on the local maximum map as the guide image, the Further filtering is performed as follows: （16） in: and Represent the input and guidance images of the bilateral filtering process respectively; represent The local maximum map of .

4. The image fusion method based on detail enhancement and dual-branch feature fusion according to claim 3, characterized in that: The specific process of S221 is as follows: For the dual-branch fusion model, the bright feature map and dark feature map at different scales can be obtained from the difference image of two adjacent filtered output images in turn through equations (17) and (18), as follows: （17） （18） in: and Respectively represent j Bright feature maps and dark feature maps obtained based on guided filtering at different scales; and They represent the j Bright feature maps and dark feature maps obtained based on guided bilateral filtering at different scales; The last filter scale n The output image obtained is used as the low-frequency basis image of the input image. The implementation process is as follows: （19） in: and Represent the basis images obtained based on guided filtering and guided bilateral filtering, respectively.

5. The image fusion method based on detail enhancement and dual-branch feature fusion according to claim 4, characterized in that: The specific process of S222 is as follows: The high-intensity high-frequency features of an image often correspond to the prominent sharp features in the image. Therefore, the bright feature maps in the two images are fused based on the element maximum criterion of the bright feature maps in the two images at each scale. The process is as follows: （20） Where: superscripts 1 and 2 represent the indexes of the two input images; Represents the image after the bright features of the two images are fused based on guided filtering; It is an image obtained by fusion of bright features of two images based on guided bilateral filtering; Indicates for and The fused bright feature map fused image; The dark feature maps in the two images are fused based on the minimum element criterion of the dark feature maps in the two images at each scale, as follows: （21） in: and They represent the images obtained by fusion of dark features of two images obtained based on guided filtering and guided bilateral filtering respectively; Indicates for and The fused bright feature map fused image; Guided filtering and guided bilateral filtering are performed based on local extreme value maps. The obtained bright feature maps and dark feature maps are often not representative enough. Therefore, the weights of each image at different scales are determined based on their entropy values. Then, the comprehensiveness and representativeness of the fused image features are enhanced by multiplying the bright feature and dark feature maps at each scale by the relevant weights. The integration strategy of the bright feature maps and dark feature maps at each scale is as follows: （22） in: and Respectively represent j The weights of the bright feature map and the dark feature map at each scale; and They represent the features after the fusion of bright feature maps and dark feature maps of all scales; and The calculation method is as follows: （23） in: and Respectively and( )’s entropy value; It can be seen from formula (23) that the feature map with the smallest entropy value is assigned a weight of 1, while the weights of feature maps at other scales are all greater than 1, which makes the bright feature maps and dark feature maps at most scales enhanced to varying degrees; In order to retain most of the basic information of the medical image in the final fused image, the element maximum criterion is used to fuse the basic images of the two input images, as shown in the following formula: （24） in: and They represent the feature maps after the fusion of the basic images obtained based on guided filtering and guided bilateral filtering respectively; Indicates for and The fused base image is a fused image; They represent the base images of the two input images obtained based on guided filtering; Respectively represent the base images of the two input images obtained based on guided bilateral filtering; The fused bright feature maps obtained by equations (22) and (24) are , dark feature map and the base image Fusion is performed to obtain the final fusion result of the two input images , as shown in formula (25): （25） Through the proposed dual-branch feature fusion strategy, the sharp features of the image can be significantly enhanced, and the basic information of the source image can be retained, ultimately obtaining a more comprehensive and representative fused image.

Images