A multi-focus image fusion method
By processing images using the Sobel operator and morphological filtering functions, combined with a Gaussian filter, the problems of difficult focus region evaluation and visual artifact boundary seams in multi-focus image fusion are solved, achieving a clear image fusion effect.
Patent Information
- Application Number
- CN202210949597.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-09
- Publication Date
- 2025-12-19
- Estimated Expiration
- 2042-08-09
AI Technical Summary
Existing multi-focus image fusion methods have difficulty evaluating the focal region of the input image and are prone to visual artifacts and boundary seams when fusing unregistered image pairs.
The Sobel operator is used to transform the images to be fused into gradient feature images. The focus region is determined by using variance and gradient comparison maps. The initial focus feature map is processed by a morphological filtering function and the boundary is smoothed by a Gaussian filter. Finally, the fusion of multi-focus images is achieved through an image fusion algorithm.
It effectively removes background noise, solves the problem of difficult focus area evaluation, and removes visual artifacts and boundary seams, resulting in a clear fused image.
Smart Images
Figure CN115294003B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of image fusion, and particularly relates to a multi-focus image fusion method. BACKGROUND
[0002] Multi-focus image fusion is a branch of image fusion, which combines the complementary features of different images taken in the same scene or similar scenes to generate an image. Multi-focus image fusion has broad application space, such as digital photography, surveillance, non-diffracting imaging systems, remote sensing, and mobile microscope processing software.
[0003] The multi-focus image fusion methods in the prior art mostly use traditional methods and deep learning-based methods. For example, Petrovic et al. proposed a multi-focus image fusion algorithm based on gradient pyramid decomposition. The algorithm uses a new "fusion first and then decomposition" strategy, obtains fusion subbands at different scales, and inversely transforms the fusion subbands to reconstruct the final fusion image. Chai et al. adopted a multi-focus image fusion method based on lifting stationary wavelet transform (LSWT). Li et al. proposed a NSCT multi-focus image fusion method based on multi-scale curvature. Guo et al. proposed a multi-focus image fusion method based on a full convolutional network.
[0004] However, there are still many difficulties in evaluating the focus regions of input images, removing visual artifacts, and boundary seams in the prior art. At the same time, both traditional methods and deep learning-based methods currently have the problem of visual artifacts and boundary seams in the edge region caused by misalignment of image edges when fusing unregistered image pairs. SUMMARY
[0005] In order to overcome the deficiencies of the prior art, the application provides a multi-focus image fusion method.
[0006] In order to achieve the above purpose, the application provides the following technical scheme:
[0007] A multi-focus image fusion method, comprising:
[0008] using a Sobel operator to convert any two images to be fused I1 and I2 into two gradient feature images G1 and G2;
[0009] According to the two gradient feature images G1 and G2, obtain their respective variance comparison images M1 and M2, and their respective average gradient comparison images S1 and S2, obtain their respective focus region comparison images C1 and C2 from M1, M2, S1, and S2, and obtain a first initial focus feature image D according to the focus region comparison images C1 and C2;
[0010] The first initial focus feature map D is processed by using a morphological image processing function to obtain a second initial focus feature map D1;
[0011] The second initial focus feature map D1, the two images to be fused I1 and I2 are input into an image fusion algorithm to perform image fusion to obtain a final fused image If;
[0012] The image fusion algorithm is used to fuse If with the remaining images to be fused two by two to realize the fusion of multiple multi-focus images.
[0013] The image fusion algorithm is:
[0014] If = I1⊙D1 + I2⊙(1-D1)
[0015] In the above formula, I1 is the first image to be fused, I2 is the second image to be fused, and ⊙ represents pixel-by-pixel multiplication.
[0016] The image fusion algorithm If = I1⊙D1 + I2⊙(1-D1) is used to fuse If with the remaining images to be fused two by two to realize the fusion of multiple multi-focus images.
[0017] Further, the calculation method of the Sobel operator is:
[0018]
[0019]
[0020] wherein, is a convolution operator, I is an image to be fused, G0 is an initial gradient feature image, HI is an image gray value for horizontal edge detection, and HV is an image gray value for vertical edge detection.
[0021] Further, it further comprises: further processing the initial gradient feature image G0 to obtain a gradient feature image G, and the processing algorithm is:
[0022]
[0023] Further, the morphological image processing function uses the bwareaopen(·) function.
[0024] Further, it further comprises: using a Gaussian filter to smooth the boundary of the second initial focus feature map D1.
[0025] Further, the algorithm for comparing the variances of the maps M1 and M2 is:
[0026]
[0027]
[0028] Where V1 and V2 are the variances of G1 and G2; ε is used to prevent M1 and M2 from tending to infinity and can be set to 0.001.
[0029] The algorithm for comparing the average gradients S1 and S2 is as follows:
[0030]
[0031]
[0032] in, and Here are the local average gradients of G1 and G2, where, The algorithm is as follows:
[0033]
[0034] in, r is the average gradient value within the local sliding window; r is the size of the local window.
[0035] The algorithm for comparing the focused regions C1 and C2 is as follows:
[0036] C1=M1+λ×S1
[0037] C2=M2+λ×S2
[0038] Where λ is a coefficient used to balance M and S;
[0039] The algorithm for the first initial focused feature map D is as follows:
[0040]
[0041] Where T is a threshold set manually.
[0042] Furthermore, the Gaussian filter window is set to [9,9], and the filter variance is set to 2.
[0043] The multi-focus image fusion method provided by this invention has the following beneficial effects:
[0044] This application solves the problem of difficulty in evaluating the focus region of the input image in the prior art by converting the images I1 and I2 to be fused into gradient feature images to remove background noise interference; it uses variance and gradient comparison maps to fuse the respective focus regions and uses morphological filtering functions to process the first initial focus feature map D, thus solving the problems of difficulty in removing visual artifacts and boundary seams in the prior art, and the visual artifacts and boundary seams in the edge region caused by edge misalignment when fusing unregistered image pairs. Attached Figure Description
[0045] In order to make the technical scheme of the present application clearer, the accompanying drawings needed in the embodiments of the present application will be briefly introduced. The accompanying drawings in the following description are only some embodiments of the present application, and other accompanying drawings can be obtained by those skilled in the art without any creative effort on the basis of the accompanying drawings.
[0046] Figure 1 Structure diagram of a multi-focus image fusion method of the embodiments of the present application;
[0047] Figure 2 Two gradient feature images G of the embodiments of the present application;
[0048] Figure 3 The first initial focus feature map D of the embodiments of the present application;
[0049] Figure 4 The second initial focus feature map D1 of the embodiments of the present application;
[0050] Figure 5 The focus feature map D2 of the embodiments of the present application;
[0051] Figure 6 Experimental results of the embodiments of the present application for fusing image pairs;
[0052] Figure 7 Experimental results of the embodiments of the present application for fusing image pairs. DETAILED DESCRIPTION
[0053] In order to make those skilled in the art better understand the technical scheme of the present application and implement it, the present application will be described in detail in the following with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical scheme of the present application, and cannot be used to limit the protection scope of the present application.
[0054] In the description of the present application, it should be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "axial", "radial", "circumferential" and the like indicate the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the technical scheme of the present application and simplify the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.
[0055] In addition, the terms "first", "second" and the like are used only for descriptive purposes and do not imply or imply relative importance. In the description of the present application, it should be noted that unless otherwise explicitly specified or limited, the terms "connected", "connected" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances. In the description of the present application, unless otherwise stated, the meaning of "a plurality of" is two or more, which will not be described here.
[0056] Embodiment:
[0057] The present application provides a multi-focus image fusion method, specifically as shown in the figure, comprising: Figures 1-5
[0058] Any two images to be fused I1 and I2 are converted into two gradient feature images G1 and G2 using Sobel operator; According to two gradient features G1 and G2, the variance comparison map M1 and M2 of each other and the average gradient comparison map S1 and S2 of each other are obtained, and the focus area comparison map C1 and C2 of each other is obtained from M1, M2, S1, S2, and a first initial focus feature map D is obtained according to the focus area comparison map C1 and C2; The first initial focus feature map D is processed by using morphological image processing function, and the influence of image noise or artifact on image is eliminated, and a second initial focus feature map D1 is further obtained; The boundary of the second initial focus feature map D1 is smoothed by using a Gaussian filter, and a focus feature map D2 is further obtained; The focus feature map D2 is fused by using an image fusion algorithm, and a final fusion image If is obtained; If and the rest of the to-be-fused images are fused two by two by using the image fusion algorithm, and the fusion of multiple multi-focus images is realized.
[0059] In this embodiment, the image fusion algorithm is:
[0060] If = I1 o D2 + I2 o (1-D2)
[0061] Wherein, I1 is the first image to be fused, I2 is the second image to be fused, o represents pixel-by-pixel multiplication, and D2 is the second initial focus feature map D1 after smoothing by Gaussian filter.
[0062] Specifically, the calculation method of Sobel operator is:
[0063]
[0064]
[0065] wherein, is a convolution operator, I is the image to be fused, G0 is the initial gradient feature image, HI is the image gray value for horizontal edge detection, and HV is the image gray value for vertical edge detection.
[0066] Specifically, the initial gradient feature image G0 needs to be further processed to obtain the gradient feature image G, and the processing algorithm is:
[0067]
[0068] Specifically, the first initial focus feature map D is obtained using the gradient feature image G, and the first initial focus feature map D specifically includes:
[0069] Selecting any two gradient feature images G1 and G2, the respective variance comparison maps M1 and M2 and the respective average gradient comparison maps S1 and S2 are obtained, and the respective focus region comparison maps C1 and C2 are obtained from M1, M2, S1 and S2, and the respective first initial focus feature maps D are obtained according to the focus region comparison maps C1 and C2. The algorithm of M1 and M2 is:
[0070]
[0071]
[0072] wherein, V1 and V2 are the variances of G1 and G2; ε is used to prevent M1 and M2 from tending to infinity, and can be set to 0.001.
[0073] The algorithm of S1 and S2 is:
[0074]
[0075]
[0076] wherein, and are the local average gradients of G1 and G2, can be obtained according to the following formula:
[0077]
[0078] wherein, is the average gradient value in the local sliding window; and r is the size of the local window. can be obtained in the same way.
[0079] Combining the above formulas, the focus region comparison maps C1 and C2 can be obtained.
[0080] C1=M1+λ×S1
[0081] C2 = M2 + λ x S2
[0082] wherein λ is a coefficient for balancing M and S.
[0083] The algorithm of D is:
[0084]
[0085] Specifically, the Gaussian filter window is set to [9, 9] and the filter variance is set to 2 to obtain the boundary-smoothed focus feature map D2.
[0086] As shown in Figure 4 :
[0087] The bwareaopen(·) function is used to remove small connected regions in the focus map. If the number of pixels in a connected region is less than a threshold value, the region is considered to be a small connected region and will be removed. The processed focus feature map c is as follows:
[0088] D A = bwareaopen(D, area)
[0089] wherein the connected region area threshold value area is R*H*W; R is a ratio factor which determines the minimum area of the connected region to be filtered out; H and W represent the height and width of the source image respectively, and 8-connected regions are used here.
[0090] As shown in Figure 6 , 7 :
[0091] Figure 6 , Figure 7 Two sets of experimental results using the present application are given, wherein Figure 6 is the result of fusing a registered image pair using the present application, Figure 7 is the result of fusing an unregistered image pair using the present application. It can be seen that whether for a registered image pair or an unregistered image pair, the fused image obtained by the present application has clear texture, natural transition region, retains the most features and detail information in the source image, and thus has the best fusion effect.
[0092] The above-described embodiments are merely preferred specific embodiments of the present application, and the protection scope of the present application is not limited thereto. Any simple changes or equivalent replacements of the technical solutions within the technical range disclosed by the present application, which can be obviously obtained by those skilled in the art, shall fall within the protection scope of the present application.
Claims
1. A multi-focus image fusion method, characterized by, The method comprises the following steps: transforming any two images to be fused I1 and I2 into two gradient feature images G1 and G2 by using a Sobel operator; obtaining a variance comparison image M1 and M2 and an average gradient comparison image S1 and S2 of each of the two gradient feature images G1 and G2, obtaining a focus area comparison image C1 and C2 of each of the two gradient feature images G1 and G2 from M1, M2, S1 and S2, and obtaining a first initial focus feature image D from the focus area comparison image C1 and C2; processing the first initial focus feature image D by using a morphological image processing function to obtain a second initial focus feature image D1; inputting the second initial focus feature image D1 and the two images to be fused I1 and I2 into an image fusion algorithm to perform image fusion and obtain a final fused image If; fusing If with the remaining images to be fused two by two by using the image fusion algorithm to realize the fusion of multiple multi-focus images; wherein the image fusion algorithm is: If = I1⊙D1 + I2⊙(1-D1) In the above formula, I1 is the first image to be fused, I2 is the second image to be fused, and ⊙ represents pixel-by-pixel multiplication; the algorithm for the variance comparison images M1 and M2 is: wherein V1 and V2 are the variances of G1 and G2; ε is used to prevent M1 and M2 from tending to infinity and is set to 0.001; the algorithm for the average gradient comparison images S1 and S2 is: wherein and are the local average gradients of G1 and G2, respectively, wherein The algorithm for is: wherein, is the average gradient value within the local sliding window; r is the size of the local window; the algorithm for the focus area comparison images C1 and C2 is: C1 = M1 + λ × S1 C2 = M2 + λ × S2 wherein λ is a coefficient for balancing M and S; the algorithm for the first initial focus feature image D is: wherein T is a threshold set artificially.
2. The multi-focus image fusion method of claim 1, wherein, the calculation method of the Sobel operator is: wherein, is a convolution operator, I is the image to be fused, G0is the initial gradient feature image, HIis the image gray value for horizontal edge detection, and HVis the image gray value for vertical edge detection.
3. The multi-focus image fusion method of claim 2, wherein, The method further comprises the following steps: further processing the initial gradient feature image G0 to obtain a gradient feature image G, and the processing algorithm is:
4. The multi-focus image fusion method of claim 1, wherein, the morphological image processing function uses a bwareaopen(·) function.
5. The multi-focus image fusion method of claim 1, wherein, The method further comprises the following steps: smoothing the boundary of the second initial focus feature image D1 by using a Gaussian filter to further obtain a focus feature image D2.
6. The multi-focus image fusion method of claim 5, wherein, The window of the Gaussian filter is set to [9, 9] and the filter variance is set to 2.