Multi-focus Image Fusion Method Based on Significant Edge Enhancement and Related Devices

Abstract

This application belongs to the field of image processing technology, and discloses a multi-focus image fusion method and related devices based on significant edge enhancement. The method includes: obtaining an initial fusion image of two source images based on the transform domain method; the two source images have the same size and are mutually registered, and the focused areas of the two source images are complementary; generating a saliency map of the initial fusion image and saliency maps of the two source images; generating a difference map between the saliency map of the initial fusion image and the saliency maps of each source image; obtaining an initial decision map corresponding to each source image according to the saliency map of the initial fusion image, the saliency maps of the two source images, and the difference map; performing optimization processing on each initial decision map to obtain an optimized decision map corresponding to each source image; fusing the two source images based on the optimized decision map to obtain a final fusion image; thereby enabling the edge information of the fusion image to be more abundant and effectively solving the problems of edge blurring and artifacts, and obtaining a clearer high-quality fusion image.

Description

Technical Field

[0001] The present application relates to the field of image processing technologies, and in particular, to a multi-focus image fusion method based on significant edge enhancement and related devices. Background Art

[0002] Due to the limited depth of field of a camera, during the shooting process, some scenes in the image will be in focus while other scenes will be blurred, which is not conducive to extracting useful information from the image. The depth of field is the range within which an optical system can form a clear image and reflects the imaging ability of the imaging system. The larger the depth of field, the wider the clear imaging range of the optical system. Only the objects within the depth of field have a clear appearance in the photo, while other objects will become blurred. Multi-focus image fusion aims to break the limitation of the depth of field, integrate different focus information in multiple source images, and obtain a fully focused fusion image for a comprehensive and objective interpretation of the scene. Moreover, the multi-focus image fusion technology has played a huge role in many fields such as biomedical imaging, photography, and robot vision.

[0003] Existing multi-focus image fusion methods mainly include: deep learning-based methods, spatial domain-based methods, and transform domain-based methods.

[0004] Deep learning-based methods use deep learning models such as convolutional neural network models, generative adversarial network models, and self-attention mechanism models for multi-focus image fusion.

[0005] Spatial domain-based methods directly consider the intensity information of pixels on the source image, extract the features of the source image in the spatial domain for activity measurement, and then use a certain fusion rule for fusion according to the activity.

[0006] Transform domain-based methods mainly consist of three stages: image transformation, coefficient fusion, and inverse transformation. First, the source image is converted to the transform domain to capture the features in the image, then the coefficients of different source images are fused by formulating different fusion strategies, and finally the fused coefficients are inversely transformed to reconstruct the fusion image.

[0007] The fusion images obtained by existing multi-focus image fusion methods still have problems such as blurred boundaries, boundary artifacts, and loss of boundary details at the focus and defocus boundaries. Therefore, the existing technologies need to be improved. Summary of the Invention

[0008] The purpose of the present application is to provide a multi-focus image fusion method based on significant edge enhancement and related devices, which can make the edge information of the fusion image richer and effectively solve the problems of edge blurring and artifacts, and obtain a clearer high-quality fusion image.

[0009] In a first aspect, the present application provides a multi-focus image fusion method based on significant edge enhancement, including the steps of:

[0010] A1. Based on the transform domain method, obtain the initial fusion image of two source images; the two source images are of the same size and are registered with each other, and the focus areas of the two source images are complementary;

[0011] A2. Generate the saliency map of the initial fusion image and the saliency maps of the two source images;

[0012] A3. Generate the difference map between the saliency map of the initial fusion image and the saliency maps of the respective source images;

[0013] A4. According to the saliency map of the initial fusion image, the saliency maps of the two source images, and the difference map, obtain the initial decision maps corresponding to the respective source images;

[0014] A5. Perform optimization processing on each of the initial decision maps to obtain the optimized decision maps corresponding to the respective source images;

[0015] A6. Based on the optimized decision maps, fuse the two source images to obtain the final fusion image.

[0016] Based on the initial fusion image obtained by the transform domain method, the difference map between the saliency map of the initial fusion image and the saliency maps of the respective source images is used to obtain and optimize the decision maps, so as to better highlight the boundaries between the focus areas and the defocus areas, and better retain the edge features of the source images in the final fusion image. Therefore, the edge information of the fusion image can be made more abundant, and the problems of edge blurring and artifacts can be effectively solved, and a clearer high-quality fusion image can be obtained.

[0017] Preferably, step A1 includes:

[0018] A101. Calculate the fusion luminance gradient according to the luminance of the two source images;

[0019] A102. Perform gradient reconstruction according to the fusion luminance gradient to obtain the first fusion image;

[0020] A103. Perform non-linear mapping on the first fusion image to obtain the initial fusion image.

[0021] The initial fusion image obtained in this way can better retain the details of the edges and the transition regions between the focus areas and the defocus areas in the source images, which is beneficial to reducing the generation of edge blurring and artifacts; the edge information of the final fusion image obtained based on this initial fusion image is more abundant, and the problems of edge blurring and artifacts are smaller.

[0022] Preferably, step A2 includes:

[0023] Based on the Sobel operator, the Tenengrad detection method is used to perform significant information detection on the initial fusion image and the two source images respectively, and the saliency maps of the initial fusion image and the two source images are obtained.

[0024] The Sobel operator is sensitive to pixel changes, so it can better judge the focusing attribute in edge detection, which is more conducive to accurately determining the boundary between the focused area and the defocused area.

[0025] Preferably, step A4 includes:

[0026] A401. Using the Tenengrad detection method to perform significant information detection on each of the difference images to obtain the corresponding difference image saliency maps;

[0027] A402. Calculate the enhanced saliency map according to the saliency maps of the two source images and the difference image saliency maps;

[0028] A403. Calculate the initial decision maps corresponding to the source images according to the saliency map of the initial fusion image and the enhanced saliency map.

[0029] Preferably, step A402 includes:

[0030] Calculate the enhanced saliency map according to the following formula:

[0031] ;

[0032] ;

[0033] where A and B are the representation symbols of the two source images respectively, is the enhanced saliency map corresponding to source image A, is the enhanced saliency map corresponding to source image B, is the saliency map of source image A, is the saliency map of source image B, is the difference image saliency map corresponding to the difference image of source image A, is the difference image saliency map corresponding to the difference image of source image B, is the preset proportionality coefficient.

[0034] Preferably, step A403 includes:

[0035] Calculate the initial decision maps corresponding to the source images according to the following formula:

[0036] ;

[0037] ;

[0038] wherein, x and y are respectively the horizontal pixel coordinate and the vertical pixel coordinate, is the pixel value of the (x, y) pixel point of the initial decision map corresponding to the source image A, is the pixel value of the (x, y) pixel point of the initial decision map corresponding to the source image B, is the pixel value of the (x, y) pixel point of is the pixel value of the (x, y) pixel point of is the pixel value of the (x, y) pixel point of the saliency map of the initial fusion image.

[0039] Preferably, step A5 includes:

[0040] A501. Filter each of the initial decision maps by using a bwareaopen filter to obtain corresponding initially filtered decision maps;

[0041] A502. Respectively use each of the source images as a guidance image to perform guided filtering on the corresponding initially filtered decision map to obtain corresponding guided decision maps;

[0042] A503. Respectively perform consistency verification processing on each of the guided decision maps to obtain corresponding verified decision maps;

[0043] A504. Assign values to the pixel values of the pixel points with unclear focus attributes in each of the verified decision maps to obtain corresponding optimized decision maps.

[0044] Preferably, step A6 includes:

[0045] Fuse the two source images according to the following formula:

[0046] ;

[0047] wherein, x and y are respectively the horizontal pixel coordinate and the vertical pixel coordinate, is the brightness of the (x, y) pixel point of the final fusion image, A and B are respectively the representation symbols of the two source images, is the pixel value of the (x, y) pixel point of the optimized decision map corresponding to the source image A, is the pixel value of the (x, y) pixel point of the optimized decision map corresponding to the source image B, is the brightness of the (x, y) pixel point of the source image A, is the brightness of the (x, y) pixel point of the source image B.

[0048] In a second aspect, the present application provides an electronic device, including a processor and a memory. The memory stores a computer program executable by the processor. When the processor executes the computer program, it runs the steps in the multi-focus image fusion method based on significant edge enhancement as described above.

[0049] In a third aspect, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it runs the steps of the multi-focus image fusion method based on significant edge enhancement as described above.

[0050] Beneficial effects: The multi-focus image fusion method and related devices based on significant edge enhancement provided by the present application obtain an initial fusion image of two source images based on a transform domain method; the two source images are of the same size and registered with each other, and the focus areas of the two source images are complementary; generate a saliency map of the initial fusion image and saliency maps of the two source images; generate a difference map between the saliency map of the initial fusion image and the saliency maps of the respective source images; obtain an initial decision map corresponding to each source image according to the saliency map of the initial fusion image, the saliency maps of the two source images, and the difference map; perform optimization processing on each initial decision map to obtain an optimized decision map corresponding to each source image; fuse the two source images based on the optimized decision map to obtain a final fusion image; thereby enabling the edge information of the fusion image to be more abundant and effectively solving the problems of edge blurring and artifacts, and obtaining a clearer high-quality fusion image. Description of the Drawings

[0051] Figure 1 It is a flowchart of the multi-focus image fusion method based on significant edge enhancement provided by an embodiment of the present application.

[0052] Figure 2 It is a schematic structural diagram of the electronic device provided by an embodiment of the present application.

[0053] Figure 3 It is a set of schematic source images.

[0054] Figure 4 For Figure 3 The source images are comparison charts of fusion results of fusion using different image fusion methods. Detailed Embodiments

[0055] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Usually, the components of the embodiments of the present application described and illustrated herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application claimed, but merely represents the selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts belong to the scope of protection of the present application.

[0056] It should be noted that similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present application, the terms "first", "second", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0057] Please refer to Figure 1 , a multi-focus image fusion method based on significant edge enhancement in some embodiments of the present application includes the steps:

[0058] A1. Based on the transform domain method, obtain the initial fusion image of two source images; the two source images are of the same size and registered with each other, and the focused areas of the two source images are complementary;

[0059] A2. Generate the saliency map of the initial fusion image and the saliency maps of the two source images;

[0060] A3. Generate the difference map between the saliency map of the initial fusion image and the saliency maps of each source image;

[0061] A4. According to the saliency map of the initial fusion image, the saliency maps of the two source images, and the difference map, obtain the initial decision maps corresponding to each source image;

[0062] A5. Perform optimization processing on each initial decision map to obtain the optimized decision maps corresponding to each source image;

[0063] A6. Based on the optimized decision maps, fuse the two source images to obtain the final fusion image.

[0064] Based on the initial fusion image obtained by the transform domain method, the difference map between the saliency map of the initial fusion image and the saliency maps of each source image is used to obtain and optimize the decision map, so as to better highlight the boundary between the focused area and the defocused area, and better retain the edge features of the source image in the final fusion image. Therefore, the edge information of the fusion image can be made richer, and the problems of edge blurring and artifacts can be effectively solved, and a clearer high-quality fusion image can be obtained.

[0065] Among them, each source image includes a focused area and a defocused area. The focused area is the area with clear imaging, and the defocused area is the area outside the focused area, where the imaging in the defocused area is blurred. The complementary focused areas of two source images mean that the focused area of one source image is the same as the defocused area of the other source image. For example Figure 3 in, the focused area of A is the area where the flower is imaged, and the focused area of B is the area outside the area where the flower is imaged, and the focused areas of the two are complementary.

[0066] Specifically, step A1 includes:

[0067] A101. Calculate the fusion luminance gradient according to the luminance of the two source images;

[0068] A102. Perform gradient reconstruction according to the fusion luminance gradient to obtain the first fused image;

[0069] A103. Perform non-linear mapping on the first fused image to obtain the initial fused image.

[0070] The initial fused image obtained in this way can better retain the details of the edges and the transition regions between the focused area and the defocused area in the source image, which is beneficial to reducing the generation of edge blurring and artifacts; the edge information of the final fused image obtained based on this initial fused image is richer, and the problems of edge blurring and artifacts are smaller.

[0071] Among them, step A101 includes:

[0072] Calculate the gradient magnitudes of the two source images according to the luminance of the two source images;

[0073] Compare the gradient magnitudes of the two source images to obtain the maximum gradient magnitude map;

[0074] Obtain the gradient of the maximum gradient magnitude map as the fusion luminance gradient.

[0075] Since the luminance channel contains the main structural details of the source image, when performing gradient-domain fusion, only using the luminance channel for fusion can reduce the algorithm burden and improve the processing efficiency while retaining the main structural details of the source image.

[0076] Specifically, the gradient magnitudes of the two source images can be calculated according to the following formula:

[0077] ;

[0078] ;

[0079] ;

[0080] ;

[0081] ;

[0082] ;

[0083] Wherein, A and B are respectively the representation symbols of two source images, x and y are respectively the horizontal pixel coordinate and the vertical pixel coordinate, 、 are respectively the gradient magnitudes of the source image A and the source image B at the pixel point (x, y), 、 are respectively the horizontal gradients of the source image A and the source image B at the pixel point (x, y), 、 are respectively the horizontal gradients of the source image A and the source image B at the pixel point (x - 1, y), 、 are respectively the vertical gradients of the source image A and the source image B at the pixel point (x, y), 、 are respectively the vertical gradients of the source image A and the source image B at the pixel point (x, y - 1), 、 are respectively the brightnesses of the source image A and the source image B at the pixel point (x + 1, y), 、 are respectively the brightnesses of the source image A and the source image B at the pixel point (x, y), 、 are respectively the brightnesses of the source image A and the source image B at the pixel point (x, y + 1), M is the vertical pixel size of the source image, and N is the horizontal pixel size of the source image.

[0084] Furthermore, by comparing the gradient magnitudes of the two source images, a maximum gradient magnitude map is obtained. Specifically, the maximum gradient magnitude map is determined according to the following formula:

[0085] ;

[0086] Wherein, is the gradient magnitude of the pixel point (x, y) of the maximum gradient magnitude map.

[0087] Furthermore, the gradient of the maximum gradient magnitude map can be calculated by the following formula:

[0088] ;

[0089] ;

[0090] Wherein, 、 are respectively the horizontal gradient and the vertical gradient of the pixel point (x, y) of the maximum gradient magnitude map, is the horizontal gradient of the pixel point (x - 1, y) in the maximum gradient magnitude map, is the vertical gradient of the pixel point (x, y - 1) in the maximum gradient magnitude map, is the gradient magnitude of the pixel point (x + 1, y) in the maximum gradient magnitude map, is the gradient magnitude of the pixel point (x, y) in the maximum gradient magnitude map, is the gradient magnitude of the pixel point (x, y + 1) in the maximum gradient magnitude map.

[0091] Therefore, we have:

[0092] ;

[0093] ;

[0094] Among them, and are respectively the horizontal fusion luminance gradient and the vertical fusion luminance gradient corresponding to the pixel point (x, y) of the source image (that is, the fusion luminance gradient includes the horizontal fusion luminance gradient and the vertical fusion luminance gradient).

[0095] When performing gradient reconstruction, when the fusion luminance gradient violates the zero curl condition, it may lead to no solution for the fusion luminance, resulting in reconstruction failure. To avoid this problem, when performing gradient reconstruction, an iterative Poisson filter can be used to iteratively solve the following relational expressions (the iterative solution process is a prior art and will not be elaborated here) to obtain the first fusion image:

[0096] ;

[0097] ;

[0098] ;

[0099] Among them, is the gradient operator, is the differential symbol, is the fusion luminance gradient matrix, is the horizontal fusion luminance gradient matrix (that is, the matrix composed of all horizontal fusion luminance gradients, and the element in the x-th row and y-th column of this matrix is ), is the vertical fusion luminance gradient matrix (that is, the matrix composed of all vertical fusion luminance gradients, and the element in the x-th row and y-th column of this matrix is ), is the first fusion image.

[0100] Among them, the recurrence formula of the iterative Poisson filter is:

[0101] ;

[0102] where kk is the iteration index, is the 2D convolution symbol, is the first fused image after the (kk + 1)-th iteration, is the first fused image after the kk-th iteration, is the horizontal fused luminance gradient matrix after the kk-th iteration, is the vertical fused luminance gradient matrix after the kk-th iteration. Using this iterative Poisson filter can achieve fast convergence and improve the solution efficiency.

[0103] Since the fused gradient is obtained by fusing multiple image gradients, there may be large differences in the gradients in adjacent regions, resulting in high-dynamic-range pixel intensities in the reconstructed first fused image, that is, the brightness of some pixel points in the first fused image exceeds the standard range of the brightness channel. To solve this problem while retaining the contrast information in the image, a non-linear mapping is performed on the first fused image to obtain the initial fused image.

[0104] Specifically, step A103 includes:

[0105] Perform a non-linear mapping on the first fused image according to the following formula to obtain the initial fused image:

[0106] ;

[0107] ;

[0108] ;

[0109] where, is the brightness of the pixel point (x, y) of the initial fused image, is the brightness of the pixel point (x, y) of the first fused image, is the minimum value of the brightness of all pixel points of the first fused image, is the maximum value of the brightness of all pixel points of the first fused image, is the conversion parameter, is the width of the brightness channel of the source image, is the upper limit value of the brightness channel of the source image, is the lower limit value of the brightness channel of the source image, is the preset reference channel width. If the source image is a color image, then is 235, is 19, so that is 216; if the source image is a grayscale image, then is 255, is 0, thus is 255.

[0110] Furthermore, in step A103, local histogram equalization processing can also be performed on the brightness of the initial fusion image to ensure that the brightness is correctly distributed within the display range.

[0111] In some embodiments, step A2 includes:

[0112] Based on the Sobel operator, the Tenengrad detection method is used to detect the significant information of the initial fusion image and the two source images respectively, and the saliency maps of the initial fusion image and the two source images are obtained.

[0113] The Sobel operator is sensitive to pixel changes, so it can better judge the focusing attribute in edge detection, which is more conducive to accurately determining the boundary between the focused area and the defocused area.

[0114] Specifically, based on the Sobel operator, using the Tenengrad detection method to detect the significant information of the initial fusion image and the two source images respectively can be expressed as:

[0115] ;

[0116] ;

[0117] ;

[0118] Among them, , , are the pixel values of the (x, y) pixel points of the saliency maps of source image A, source image B, and the initial fusion image respectively, , , are the Sobel gradient amplitudes of the (x, y) pixel points of the saliency maps of source image A, source image B, and the initial fusion image respectively.

[0119] Among them, , , ; , are respectively 's horizontal and vertical components, , are respectively 's horizontal and vertical components, , are respectively 's horizontal and vertical components.

[0120] Among them:

[0121] ;

[0122] ;

[0123] 、 、 、 、 、 、 、 are respectively the brightness of the pixel points (x - 1, y - 1), (x - 1, y), (x - 1, y + 1), (x + 1, y - 1), (x + 1, y), (x + 1, y + 1), (x, y - 1), (x, y + 1) of the source image A.

[0124] Among them:

[0125] ;

[0126] ;

[0127] 、 、 、 、 、 、 、 are respectively the brightness of the pixel points (x - 1, y - 1), (x - 1, y), (x - 1, y + 1), (x + 1, y - 1), (x + 1, y), (x + 1, y + 1), (x, y - 1), (x, y + 1) of the source image B.

[0128] Among them:

[0129] ;

[0130] ;

[0131] 、 、 、 、 、 、 、 are respectively the brightness of the pixel points (x - 1, y - 1), (x - 1, y), (x - 1, y + 1), (x + 1, y - 1), (x + 1, y), (x + 1, y + 1), (x, y - 1), (x, y + 1) of the initial fused image.

[0132] Specifically, in step A3, the difference image is calculated according to the following formula:

[0133] ;

[0134] ;

[0135] Among them, is the difference map between the saliency map of the initial fused image and the saliency map of source image A (i.e., is the difference map corresponding to source image A), is the difference map between the saliency map of the initial fused image and the saliency map of source image B (i.e., is the difference map corresponding to source image B), is the saliency map of the initial fused image, is the saliency map of source image A, is the saliency map of source image B.

[0136] In this embodiment, step A4 includes:

[0137] A401. Use the Tenengrad detection method to detect the significant information of each difference map to obtain the corresponding difference map saliency map;

[0138] A402. Calculate the enhanced saliency map according to the saliency maps of the two source images and the difference map saliency map;

[0139] A403. Calculate the initial decision map corresponding to each source image according to the saliency map of the initial fused image and the enhanced saliency map.

[0140] Among them, the method of using the Tenengrad detection method to detect the significant information of each difference map can refer to the method of using the Sobel operator in the previous text. The method of using the Tenengrad detection method to detect the significant information of the initial fused image and the two source images respectively can be expressed as:

[0141] ;

[0142] ;

[0143] Among them, , are respectively the pixel values of the (x, y) pixel points of the difference map saliency map corresponding to and , , are respectively the Sobel gradient amplitudes of the (x, y) pixel points of and (the specific calculation process refers to the calculation processes of the previous text , , , and will not be elaborated here).

[0144] The difference map saliency map obtained in the above manner may have small regions where defocused pixels and focused pixels alternate at the edges, resulting in unclear edges. Therefore, in step A402, the complementary information between the images is used to perform a comprehensive operation on the saliency maps of the two source images and the difference map saliency maps of the two difference maps to amplify the difference between focus and defocus in the saliency map and highlight the boundary between the focused area and the defocused area.

[0145] Specifically, step A402 includes:

[0146] Calculate the enhanced saliency map according to the following formula:

[0147] ;

[0148] ;

[0149] where is the enhanced saliency map corresponding to source image A, is the enhanced saliency map corresponding to source image B, is the difference map saliency map of the difference map corresponding to source image A (the pixel value of its (x, y) pixel is ), is the difference map saliency map of the difference map corresponding to source image B (the pixel value of its (x, y) pixel is ), is a preset proportionality coefficient (which can be set according to actual needs, for example, set to 0.5, but not limited to this).

[0150] Furthermore, step A403 includes:

[0151] Calculate the initial decision map corresponding to each source image according to the following formula:

[0152] ;

[0153] ;

[0154] where is the pixel value of the (x, y) pixel of the initial decision map corresponding to source image A, is the pixel value of the (x, y) pixel of the initial decision map corresponding to source image B, is the pixel value of the (x, y) pixel of , is the pixel value of the (x, y) pixel of . The edge information of the initial decision map obtained in the above manner is richer and smoother.

[0155] Preferably, step A5 includes:

[0156] A501. Filter each initial decision map using the bwareaopen filter to obtain the corresponding initially filtered decision map;

[0157] A502. Use each source image as a guiding image to perform guided filtering on the corresponding initially filtered decision map to obtain the corresponding guided decision map;

[0158] A503. Perform consistency verification processing on each guided decision map to obtain the corresponding verified decision map;

[0159] A504. Assign pixel values to the pixel points with unclear focusing attributes in each verified decision map to obtain the corresponding optimized decision map.

[0160] In the process of obtaining the initial decision map, the focusing attributes of some pixel points (the focusing attributes are used to distinguish whether a pixel point belongs to the pixel points in the focused area or the defocused area) may be misclassified, resulting in problems such as small holes, protrusions, and breaks in the initial decision map. To avoid these problems from affecting the quality of the final fused image, this application uses the bwareaopen filter (the bwareaopen filter is a filter provided by the Python platform) to perform filtering processing on the initial decision map.

[0161] Filtering each initial decision map using the bwareaopen filter to obtain the corresponding initially filtered decision map can be expressed as:

[0162] ;

[0163] ;

[0164] where is the initial decision map corresponding to the source image A (the pixel value of its (x, y) pixel point is ), is the initial decision map corresponding to the source image B (the pixel value of its (x, y) pixel point is ), is the initially filtered decision map corresponding to , is the initially filtered decision map corresponding to , t is the threshold, and t = th * S, S is the size of the source image (equal to M * N), th is a preset parameter (which can be set according to actual needs, for example, 0.02, but not limited to this), is the filtering function of the bwareaopen filter.

[0165] After removing small regions such as small holes, protrusions, and breaks in the initial decision map through the bwareaopen filter, there may be boundary problems such as boundary errors and artifacts in the boundary region of the decision map. Therefore, in step A502, guided filtering is performed on the initially filtered decision map, so as to better retain the edge features of the source image in the fused image.

[0166] Taking each source image as the guidance image respectively, guided filtering is performed on the corresponding initially filtered decision map to obtain the corresponding guided decision map, which can be expressed as:

[0167] ;

[0168] ;

[0169] Among them, is the guided decision map corresponding to the source image A, is the guided decision map corresponding to the source image B, is the GF guided filtering function, is the source image A, is the source image B, r is the window size (which can be set according to actual needs, for example, 5, but not limited to this), is the regularization parameter (which can be set according to actual needs, for example, 0.3, but not limited to this).

[0170] After guided filtering, although the edge region of the decision map can become clearer, some error information may be introduced, resulting in a pseudo-edge effect at the edge of the final fused image. To solve this problem, in step A503, consistency verification processing is performed on the guided decision map.

[0171] Among them, in step A503, the guided decision map can be processed for consistency verification according to the following formula:

[0172] ;

[0173] ;

[0174] Among them, is the pixel value of the (x, y) pixel point of the verification decision map corresponding to , is the pixel value of the (x, y) pixel point of the verification decision map corresponding to , , are the horizontal offset and vertical offset respectively, is the neighborhood size (equal to the product of q and S, q is the reference ratio, which can be set according to actual needs, for example, 5×10 -5 , but not limited to this), It means that the following conditions are satisfied: The pixel point (x + a, y + b) is within the neighborhood centered at the pixel point (x, y) and with a size of , and are respectively and the pixel values of the pixel point (x + a, y + b).

[0175] After the consistency verification of the decision diagram, it may lead to a situation where some pixels in the focused and defocused edge regions of the decision diagram cannot confirm their focusing attributes. Therefore, in step A504, the pixel values of the pixel points with unclear focusing attributes in each verified decision diagram are assigned values. Specifically, the assignment process can be carried out according to the following formula:

[0176] ;

[0177] ;

[0178] wherein, is the pixel value of the pixel point (x, y) of the optimized decision diagram corresponding to the source image A, is the pixel value of the pixel point (x, y) of the optimized decision diagram corresponding to the source image B.

[0179] Specifically, step A6 includes:

[0180] Fusing the two source images according to the following formula:

[0181] ;

[0182] wherein, is the brightness of the pixel point (x, y) of the final fused image.

[0183] In one embodiment, five existing image fusion methods and the multi - focus image fusion method based on significant edge enhancement of the present application are respectively used to fuse Figure 3 a set of source images shown (in the figure, A and B are respectively the source image A and the source image B). The five existing image fusion methods include the fusion algorithm based on joint bilateral filtering and local gradient energy operator (INS), the unified unsupervised image fusion network (U2Fusion), the general end - to - end image fusion network based on memory cells (MUFusion), the compressive decomposition network (SDNet), and the small - area perception method (SAMF). The fusion results are shown in Figure 4, in the figure, a is the fusion result of the INS method, b is the fusion result of the U2Fusion method, c is the fusion result of MUFusion, d is the fusion result of the SDNet method, e is the fusion result of the SAMF method, and f is the fusion result of the multi-focus image fusion method based on significant edge enhancement of the present application.

[0184] From Figure 4 it can be seen that, except for e and f, other fusion results still retain a small amount of background information in the defocused area, resulting in the clarity of the fusion result being affected; e is not as accurate as f in the extraction of focused edge details. Therefore, the multi-focus image fusion method based on significant edge enhancement of the present application is superior to other several image fusion methods.

[0185] As can be seen from the above, the multi-focus image fusion method based on significant edge enhancement, based on the transform domain method, obtains the initial fusion image of two source images; the two source images are of the same size and are mutually registered, and the focused areas of the two source images are complementary; generates the saliency map of the initial fusion image and the saliency maps of the two source images; generates the difference map between the saliency map of the initial fusion image and the saliency maps of the source images; obtains the initial decision maps corresponding to the source images according to the saliency map of the initial fusion image, the saliency maps of the two source images and the difference map; performs optimization processing on each initial decision map to obtain the optimized decision maps corresponding to the source images; fuses the two source images based on the optimized decision maps to obtain the final fusion image; thus, it can make the edge information of the fusion image richer and effectively solve the problems of edge blurring and artifacts, and obtain a clearer high-quality fusion image.

[0186] Please refer to Figure 2, which is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The present application provides an electronic device, including: a processor 301 and a memory 302. The processor 301 and the memory 302 are interconnected and communicate with each other through a communication bus 303 and / or other forms of connection mechanisms (not marked). The memory 302 stores a computer program executable by the processor 301. When the electronic device runs, the processor 301 executes the computer program to perform the multi-focus image fusion method based on significant edge enhancement in any optional implementation manner of the above embodiment to achieve the following functions: obtaining an initial fusion image of two source images; the two source images have the same size and are registered with each other, and the focus areas of the two source images are complementary; generating a saliency map of the initial fusion image and saliency maps of the two source images; generating a difference map between the saliency map of the initial fusion image and the saliency maps of the respective source images; obtaining an initial decision map corresponding to each source image according to the saliency map of the initial fusion image, the saliency maps of the two source images, and the difference map; performing an optimization process on each initial decision map to obtain an optimized decision map corresponding to each source image; fusing the two source images based on the optimized decision map to obtain a final fusion image.

[0187] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it executes the multi-focus image fusion method based on significant edge enhancement in any optional implementation manner of the above embodiment to achieve the following functions: obtaining an initial fusion image of two source images; the two source images are of the same size and are mutually registered, and the focus areas of the two source images are complementary; generating a saliency map of the initial fusion image and saliency maps of the two source images; generating a difference map between the saliency map of the initial fusion image and the saliency maps of the respective source images; obtaining an initial decision map corresponding to each source image according to the saliency map of the initial fusion image, the saliency maps of the two source images, and the difference map; performing an optimization process on each initial decision map to obtain an optimized decision map corresponding to each source image; and fusing the two source images based on the optimized decision map to obtain a final fusion image. Among them, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, abbreviated as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, abbreviated as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, abbreviated as EPROM), programmable read-only memory (Programmable Red-Only Memory, abbreviated as PROM), read-only memory (Read-OnlyMemory, abbreviated as ROM), magnetic memory, flash memory, magnetic disk or optical disc.

[0188] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces. The indirect coupling or communication connection of the devices or units can be in an electrical, mechanical or other form.

[0189] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] Furthermore, in each embodiment of this application, the various functional modules may be integrated together to form an independent part, or each module may exist alone, or two or more modules may be integrated to form an independent part.

[0191] In this text, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0192] The above description is only for the embodiments of this application and is not used to limit the protection scope of this application. For those skilled in the art, this application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included in the protection scope of this application.

Claims

1. A multi-focus image fusion method based on significant edge enhancement, characterized in that, Including the steps: A1. Based on the transform domain method, obtain the initial fused image of two source images; the two source images are of the same size and registered with each other, and the focus areas of the two source images are complementary; A2. Generate the saliency map of the initial fused image and the saliency maps of the two source images; A3. Generate the difference map between the saliency map of the initial fused image and the saliency maps of the respective source images; A4. According to the saliency map of the initial fused image, the saliency maps of the two source images, and the difference map, obtain the initial decision maps corresponding to the respective source images; A5. Perform optimization processing on each of the initial decision maps to obtain the optimized decision maps corresponding to the respective source images; A6. Based on the optimized decision maps, fuse the two source images to obtain the final fused image; Step A4 includes: A401. Adopt the Tenengrad detection method to perform saliency information detection on each of the difference maps to obtain the corresponding difference map saliency maps; A402. Calculate the enhanced saliency map according to the saliency maps of the two source images and the difference map saliency maps; A403. Calculate the initial decision maps corresponding to the respective source images according to the saliency map of the initial fused image and the enhanced saliency map; Step A403 includes: Calculate the initial decision maps corresponding to the respective source images according to the following formula: ； ； where x and y are the horizontal and vertical pixel coordinates respectively, and A and B are the representation symbols of the two source images, is the pixel value of the (x, y) pixel point of the initial decision map corresponding to the source image A, is the pixel value of the (x, y) pixel point of the initial decision map corresponding to the source image B, is the enhanced saliency map corresponding to the source image A, is the enhanced saliency map corresponding to the source image B, is the pixel value of the (x, y) pixel point of is the pixel value of the (x, y) pixel point of is the pixel value of the (x, y) pixel point of the saliency map of the initial fused image.

2. The multi-focus image fusion method based on significant edge enhancement according to claim 1, wherein Step A1 includes: A101. Calculate the fusion luminance gradient according to the luminances of the two source images; A102. Perform gradient reconstruction according to the fusion luminance gradient to obtain the first fused image; A103. Perform non-linear mapping on the first fused image to obtain the initial fused image.

3. The multi-focus image fusion method based on significant edge enhancement according to claim 1, wherein, Step A2 includes: Based on the Sobel operator, adopt the Tenengrad detection method to perform saliency information detection on the initial fused image and the two source images respectively to obtain the saliency map of the initial fused image and the saliency maps of the two source images.

4. The multi-focus image fusion method based on significant edge enhancement according to claim 1, wherein, Step A402 includes: Calculate the enhanced saliency map according to the following formula: ； ； wherein, is the saliency map of the source image A, is the saliency map of the source image B, is the saliency map of the difference map corresponding to the difference map of the source image A, is the saliency map of the difference map corresponding to the difference map of the source image B, is a preset proportionality coefficient.

5. The multi-focus image fusion method based on significant edge enhancement according to claim 1, characterized in that Step A5 includes: A501. Use the bwareaopen filter to perform filtering processing on each of the initial decision maps to obtain the corresponding initially filtered decision maps; A502. Respectively use each of the source images as the guiding image to perform guided filtering on the corresponding initially filtered decision maps to obtain the corresponding guided decision maps; A503. Respectively perform consistency verification processing on each of the guided decision maps to obtain the corresponding verified decision maps; A504. Perform value assignment processing on the pixel values of the pixel points with unclear focus attributes in each of the verified decision maps to obtain the corresponding optimized decision maps.

6. The multi-focus image fusion method based on significant edge enhancement according to claim 1, characterized in that Step A6 includes: Fuse the two source images according to the following formula: ； where x and y are the horizontal pixel coordinate and the vertical pixel coordinate respectively, is the brightness of the pixel at (x, y) of the final fused image, and A and B are the representation symbols of the two source images respectively, is the pixel value of the pixel at (x, y) of the optimized decision diagram corresponding to the source image A, is the pixel value of the pixel at (x, y) of the optimized decision diagram corresponding to the source image B, is the brightness of the pixel at (x, y) of the source image A, is the brightness of the pixel at (x, y) of the source image B.

7. An electronic device, characterized in that, Including a processor and a memory, the memory stores a computer program executable by the processor, and when the processor executes the computer program, it runs the steps in the multi-focus image fusion method based on saliency edge enhancement according to any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it runs the steps of the multi-focus image fusion method based on saliency edge enhancement according to any one of claims 1-6.

Images