Multi-focus image fusion method based on salient edge enhancement and related equipment

Abstract

The invention belongs to the technical field of image processing, and discloses a multi-focus image fusion method and related equipment based on salient edge enhancement, and the method comprises the steps: obtaining an initial fusion image of two source images based on a transform domain method; the two source images have the same size and are mutually registered, and the focal areas of the two source images are complementary; generating a saliency map of the initial fusion image and saliency maps of the two source images; generating a difference map between the saliency map of the initial fusion image and the saliency map of each source image; obtaining an initial decision diagram corresponding to each source image according to the saliency map of the initial fusion image and the saliency maps and the difference maps of the two source images; performing optimization processing on each initial decision diagram to obtain an optimized decision diagram corresponding to each source image; fusing the two source images based on the optimized decision diagram to obtain a final fused image; therefore, the edge information of the fused image is richer, the problems of edge blurring and artifacts are effectively solved, and a clearer high-quality fused image is obtained.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multi-focus image fusion method based on significant edge enhancement and related equipment. Background Art

[0002] Due to the limited depth of field of the camera, part of the scene in the image will be in focus while other scenes will be blurred during the shooting process, which is not conducive to extracting useful information from the image. The depth of field is the range in which the optical system can form a clear image, reflecting the imaging capability of the imaging system. The larger the depth of field, the wider the clear imaging range of the optical system. Only objects within the depth of field have a clear appearance in the photo, while other objects will become blurred. Multi-focus image fusion aims to break the limitation of depth of field, integrate different focus information from multiple source images, and obtain a fully focused fused image to provide a comprehensive and objective interpretation of the scene. In addition, multi-focus image fusion technology has played a huge role in many fields such as biomedical imaging, photography, and robotic vision.

[0003] Existing multi-focus image fusion methods mainly include: deep learning-based methods, spatial domain-based methods, and change domain-based methods.

[0004] The deep learning based method uses deep learning models such as convolutional neural network models, generative adversarial network models, and self-attention mechanism models to perform multi-focus image fusion.

[0005] The spatial domain-based method directly considers the intensity information of pixels on the source image, extracts the features of the source image in the spatial domain to measure the activity, and then fuses them using a certain fusion rule based on the activity.

[0006] The method based on the change domain mainly consists of three stages: image transformation, coefficient fusion and inverse transformation. First, the source image is converted to the transform domain to capture the features in the image, then the coefficients of different source images are fused by formulating different fusion strategies, and finally the fusion coefficients are inversely transformed to reconstruct the fused image.

[0007] The fused images obtained by the existing multi-focus image fusion method still have problems such as boundary blur, boundary artifacts and boundary detail loss at the focus and defocus boundaries, so the existing technology needs to be improved and enhanced. Summary of the invention

[0008] The purpose of this application is to provide a multi-focus image fusion method and related equipment based on significant edge enhancement, which can enrich the edge information of the fused image and effectively solve the problems of edge blur and artifacts, thereby obtaining a clearer high-quality fused image.

[0009] In a first aspect, the present application provides a multi-focus image fusion method based on significant edge enhancement, comprising the steps of: A1. Based on the transform domain method, an initial fused image of two source images is obtained; the two source images are of the same size and are registered with each other, and the focus areas of the two source images are complementary; A2. Generate a saliency map of the initial fused image and saliency maps of the two source images; A3. generating a difference map between the saliency map of the initial fused image and the saliency map of each of the source images; A4. obtaining an initial decision map corresponding to each of the source images according to the saliency map of the initial fused image, the saliency maps of the two source images and the difference map; A5. Optimizing each of the initial decision graphs to obtain an optimized decision graph corresponding to each of the source images; A6. Fusing the two source images based on the optimization decision graph to obtain a final fused image.

[0010] On the basis of the initial fused image obtained based on the transform domain method, the difference map between the saliency map of the initial fused image and the saliency map of each source image is used to obtain and optimize the decision map, so as to better highlight the boundary between the focused area and the defocused area, and better retain the edge features of the source image in the final fused image. Therefore, the edge information of the fused image can be enriched and the problems of edge blur and artifacts can be effectively solved, so as to obtain a clearer and high-quality fused image.

[0011] Preferably, step A1 comprises: A101. Calculate the fusion brightness gradient according to the brightness of the two source images; A102. Performing gradient reconstruction according to the fused brightness gradient to obtain a first fused image; A103. Perform nonlinear mapping on the first fused image to obtain the initial fused image.

[0012] The initial fused image obtained in this way can better preserve the details of the edges and the transition areas between the focused and defocused areas in the source image, which is beneficial to reducing the generation of edge blur and artifacts; the final fused image obtained based on the initial fused image has richer edge information and less problems with edge blur and artifacts.

[0013] Preferably, step A2 comprises: Based on the Sobel operator, a Tenengrad detection method is used to perform salient information detection on the initial fused image and the two source images respectively, so as to obtain a salient map of the initial fused image and a salient map of the two source images.

[0014] The Sobel operator is sensitive to pixel changes, so it can better judge the focus properties in edge detection, which is more conducive to accurately determining the boundaries of the focused area and the defocused area.

[0015] Preferably, step A4 comprises: A401. Performing significant information detection on each of the difference maps using the Tenengrad detection method to obtain a corresponding difference map significant map; A402. Calculate an enhanced saliency map according to the saliency maps of the two source images and the saliency map of the difference image; A403. According to the saliency map of the initial fused image and the enhanced saliency map, an initial decision map corresponding to each of the source images is calculated.

[0016] Preferably, step A402 includes: The enhanced saliency map is calculated according to the following formula: ; ; Wherein, A and B are the representation symbols of the two source images respectively. is the enhanced saliency map corresponding to the source image A, is the enhanced saliency map corresponding to the source image B, is the saliency map of the source image A, is the saliency map of the source image B, is the difference saliency map corresponding to the difference map of the source image A, is the difference saliency map corresponding to the difference map of the source image B, is the preset scale factor.

[0017] Preferably, step A403 includes: The initial decision graph corresponding to each source image is calculated according to the following formula: ; ; Among them, x and y are the horizontal pixel coordinates and the vertical pixel coordinates respectively. is the pixel value of the (x, y) pixel of the initial decision graph corresponding to the source image A, is the pixel value of the (x, y) pixel of the initial decision graph corresponding to the source image B, for The pixel value of the (x, y) pixel point, for The pixel value of the (x, y) pixel point, is the pixel value of the pixel point (x, y) of the saliency map of the initial fused image.

[0018] Preferably, step A5 comprises: A501. Filter each of the initial decision diagrams using a bwareaopen filter to obtain a corresponding initial filter decision diagram; A502. Using each of the source images as a guide image, respectively, the corresponding preliminary filter decision graph is subjected to guided filtering to obtain a corresponding guided decision graph; A503. Perform consistency verification on each of the guided decision diagrams to obtain a corresponding verification decision diagram; A504. Assign values ​​to the pixel points with unclear focus attributes in each of the verification decision graphs to obtain a corresponding optimization decision graph.

[0019] Preferably, step A6 comprises: The two source images are fused according to the following formula: ; Among them, x and y are the horizontal pixel coordinates and the vertical pixel coordinates respectively. is the brightness of the (x, y) pixel of the final fused image, A and B are the representation symbols of the two source images respectively, is the pixel value of the (x, y) pixel corresponding to the optimized decision graph of the source image A, is the pixel value of the (x, y) pixel corresponding to the optimized decision graph of the source image B, is the brightness of the pixel (x, y) of the source image A, is the brightness of the pixel (x, y) of the source image B.

[0020] In a second aspect, the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, it runs the steps in the multi-focus image fusion method based on significant edge enhancement as described above.

[0021] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the multi-focus image fusion method based on significant edge enhancement as described above are executed.

[0022] Beneficial effects: The multi-focus image fusion method based on significant edge enhancement and related equipment provided by the present application obtain an initial fused image of two source images based on a transform domain method; the two source images are of the same size and aligned with each other, and the focus areas of the two source images are complementary; a saliency map of the initial fused image and a saliency map of the two source images are generated; a difference map between the saliency map of the initial fused image and the saliency maps of each source image is generated; an initial decision map corresponding to each source image is obtained according to the saliency map of the initial fused image, the saliency maps of the two source images and the difference map; each initial decision map is optimized to obtain an optimized decision map corresponding to each source image; the two source images are fused based on the optimized decision map to obtain a final fused image; thereby, the edge information of the fused image can be enriched and the problems of edge blur and artifacts can be effectively solved, thereby obtaining a clearer high-quality fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flowchart of a multi-focus image fusion method based on significant edge enhancement provided in an embodiment of the present application.

[0024] Figure 2 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0025] Figure 3 A set of schematic source images.

[0026] Figure 4 To use different image fusion methods Figure 3 Comparison chart of fusion results of fusion of source images. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0028] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0029] Please refer to Figure 1 In some embodiments of the present application, a multi-focus image fusion method based on significant edge enhancement comprises the steps of: A1. Based on the transform domain method, an initial fused image of two source images is obtained; the two source images are of the same size and are mutually registered, and the focus areas of the two source images are complementary; A2. Generate a saliency map of the initial fused image and saliency maps of the two source images; A3. Generate a difference map between the saliency map of the initial fused image and the saliency map of each source image; A4. Obtaining an initial decision map corresponding to each source image based on the saliency map of the initial fused image, the saliency maps of the two source images, and the difference map; A5. Optimize each initial decision graph to obtain an optimized decision graph corresponding to each source image; A6. Fuse the two source images based on the optimized decision graph to obtain a final fused image.

[0030] On the basis of the initial fused image obtained based on the transform domain method, the difference map between the saliency map of the initial fused image and the saliency map of each source image is used to obtain and optimize the decision map, so as to better highlight the boundary between the focused area and the defocused area, and better retain the edge features of the source image in the final fused image. Therefore, the edge information of the fused image can be enriched and the problems of edge blur and artifacts can be effectively solved, so as to obtain a clearer and high-quality fused image.

[0031] Each source image includes a focus area and a defocus area. The focus area is the area with clear image, and the defocus area is the area outside the focus area. The image in the defocus area is blurred. The complementary focus areas of two source images refer to that the focus area of ​​one source image is the same as the defocus area of ​​the other source image. For example Figure 3 In the figure, the focusing area of ​​A is the flower imaging area, and the focusing area of ​​B is the area outside the flower imaging area, and the focusing areas of the two are complementary.

[0032] Specifically, step A1 includes: A101. Calculate the fusion brightness gradient according to the brightness of the two source images; A102. Perform gradient reconstruction according to the fused brightness gradient to obtain a first fused image; A103. Perform nonlinear mapping on the first fused image to obtain an initial fused image.

[0033] The initial fused image obtained in this way can better preserve the details of the edges and the transition areas between the focused and defocused areas in the source image, which is beneficial to reducing the generation of edge blur and artifacts; the final fused image obtained based on the initial fused image has richer edge information and less problems with edge blur and artifacts.

[0034] Wherein, step A101 includes: Calculate the gradient magnitude of the two source images according to the brightness of the two source images; Compare the gradient amplitudes of the two source images and obtain the maximum gradient amplitude map; Get the gradient of the maximum gradient magnitude map as the fused brightness gradient.

[0035] Since the brightness channel contains the main structural details of the source image, only using the brightness channel fusion during gradient domain fusion can reduce the algorithm burden and improve processing efficiency while retaining the main structural details of the source image.

[0036] Specifically, the gradient magnitude of the two source images can be calculated according to the following formula: ; ; ; ; ; ; Among them, A and B are the symbols of the two source images, x and y are the horizontal pixel coordinates and the vertical pixel coordinates, respectively. , are the gradient amplitudes of source image A and source image B at the pixel point (x, y), , are the lateral gradients of source image A and source image B at the pixel point (x, y), , are the lateral gradients of source image A and source image B at the pixel point (x-1, y), , are the longitudinal gradients of source image A and source image B at the pixel point (x, y), , are the longitudinal gradients of source image A and source image B at the pixel point (x, y-1), , are the brightness of the source image A and the source image B at the pixel (x+1, y), , are the brightness of the source image A and the source image B at the pixel (x, y), , are the brightness of source image A and source image B at the pixel (x, y+1), M is the vertical pixel size of the source image, and N is the horizontal pixel size of the source image.

[0037] Furthermore, the gradient amplitudes of the two source images are compared to obtain a maximum gradient amplitude map, which is determined specifically according to the following formula: ; in, is the gradient magnitude of the pixel (x, y) in the maximum gradient magnitude map.

[0038] Furthermore, the gradient of the maximum gradient magnitude map can be calculated by the following formula: ; ; in, , are the horizontal and vertical gradients of the (x, y) pixel of the maximum gradient amplitude map, respectively. is the lateral gradient of the pixel (x-1, y) of the maximum gradient amplitude map, is the longitudinal gradient of the pixel (x, y-1) of the maximum gradient amplitude map, is the gradient amplitude of the pixel (x+1, y) of the maximum gradient amplitude map, is the gradient amplitude of the (x, y) pixel of the maximum gradient amplitude map, is the gradient magnitude of the pixel (x, y+1) in the maximum gradient magnitude map.

[0039] Thus: ; ; in, , They are respectively the horizontal fusion brightness gradient and the vertical fusion brightness gradient corresponding to the pixel point (x, y) of the source image (ie, the fusion brightness gradient includes the horizontal fusion brightness gradient and the vertical fusion brightness gradient).

[0040] When performing gradient reconstruction, when the fused brightness gradient violates the zero curl condition, it may cause the fused brightness to have no solution, resulting in reconstruction failure. To avoid this problem, when performing gradient reconstruction, an iterative Poisson filter can be used to iteratively solve the following relationship (the iterative solution process is a prior art and will not be described in detail here) to obtain the first fused image: ; ; ; in, is the gradient operator, is the differential symbol, is the fused brightness gradient matrix, is the horizontal fusion brightness gradient matrix (i.e., a matrix composed of all horizontal fusion brightness gradients, the elements of the matrix in the xth row and yth column are ), is the vertical fusion brightness gradient matrix (i.e., the matrix composed of all vertical fusion brightness gradients, the elements of the matrix in the xth row and yth column are ), is the first fused image.

[0041] Among them, the recursive formula of iterative Poisson filter is: ; Among them, kk is the iteration index, is the symbol for 2D convolution, is the first fused image after the kk+1th iteration, is the first fused image after the kkth iteration, is the horizontal fusion brightness gradient matrix after the kkth iteration, is the vertical fusion brightness gradient matrix after the kkth iteration. The iterative Poisson filter can achieve fast convergence and improve the solution efficiency.

[0042] Since the fused gradient is obtained by fusing the gradients of multiple images, there may be large differences in the gradients of adjacent areas, resulting in the presence of pixel intensities with a high dynamic range in the reconstructed first fused image, that is, the brightness of some pixels in the first fused image exceeds the standard range of the brightness channel. In order to solve this problem while retaining the contrast information in the image, the first fused image is nonlinearly mapped to obtain the initial fused image.

[0043] Specifically, step A103 includes: The first fused image is nonlinearly mapped according to the following formula to obtain the initial fused image: ; ; ; in, is the brightness of the (x, y) pixel of the initial fused image, is the brightness of the pixel (x, y) of the first fused image, is the minimum brightness of all pixels in the first fused image, is the maximum value of the brightness of all pixels in the first fused image, is the conversion parameter, is the width of the brightness channel of the source image, is the upper limit of the brightness channel of the source image, is the lower limit of the brightness channel of the source image, is the preset reference channel width. If the source image is a color image, then is 235, is 19, so is 216; if the source image is a grayscale image, then is 255, is 0, so is 255.

[0044] Furthermore, in step A103, local histogram equalization processing may be performed on the brightness of the initial fused image to ensure that the brightness is correctly distributed within the display range.

[0045] In some embodiments, step A2 comprises: Based on the Sobel operator, the Tenengrad detection method is used to detect the salient information of the initial fused image and the two source images respectively, and the salient map of the initial fused image and the salient maps of the two source images are obtained.

[0046] The Sobel operator is sensitive to pixel changes, so it can better judge the focus properties in edge detection, which is more conducive to accurately determining the boundaries of the focused area and the defocused area.

[0047] Specifically, based on the Sobel operator, the Tenengrad detection method is used to detect significant information of the initial fused image and the two source images respectively, which can be expressed as: ; ; ; in, , , are the pixel values ​​of the (x, y) pixel points of the saliency maps of source image A, source image B, and the initial fused image, respectively. , , are the Sobel gradient magnitudes of the (x, y) pixel points of the saliency maps of the source image A, the source image B, and the initial fused image, respectively.

[0048] in, , , ; , They are The transverse and longitudinal components of , They are The transverse and longitudinal components of , They are The transverse and longitudinal components.

[0049] in: ; ; , , , , , , , are the brightness of pixels (x-1, y-1), (x-1, y), (x-1, y+1), (x+1, y-1), (x+1, y), (x+1, y+1), (x, y-1), and (x, y+1) of the source image A respectively.

[0050] in: ; ; , , , , , , , are the brightness of pixels (x-1, y-1), (x-1, y), (x-1, y+1), (x+1, y-1), (x+1, y), (x+1, y+1), (x, y-1), and (x, y+1) of the source image B respectively.

[0051] in: ; ; , , , , , , , are the brightness of the pixels (x-1, y-1), (x-1, y), (x-1, y+1), (x+1, y-1), (x+1, y), (x+1, y+1), (x, y-1), and (x, y+1) of the initial fused image respectively.

[0052] Specifically, in step A3, the difference map is calculated according to the following formula: ; ; in, is the difference map between the saliency map of the initial fused image and the saliency map of the source image A (i.e. is the difference image corresponding to the source image A), is the difference map between the saliency map of the initial fused image and the saliency map of the source image B (i.e. is the difference image corresponding to the source image B), is the saliency map of the initial fused image, is the saliency map of the source image A, is the saliency map of the source image B.

[0053] In this embodiment, step A4 includes: A401. Use the Tenengrad detection method to detect significant information of each difference map and obtain the corresponding difference map saliency map; A402. Calculate an enhanced saliency map according to the saliency maps of the two source images and the difference saliency map; A403. According to the saliency map of the initial fused image and the enhanced saliency map, an initial decision map corresponding to each source image is calculated.

[0054] Among them, the method of using the Tenengrad detection method to detect significant information of each difference image can refer to the method based on the Sobel operator in the previous article, which uses the Tenengrad detection method to detect significant information of the initial fusion image and the two source images respectively, which can be expressed as: ; ; in, , Corresponding to and The pixel value of the (x, y) pixel of the difference map saliency map, , They are and The Sobel gradient amplitude of the (x, y) pixel point (the specific calculation process is referred to the previous article , , The calculation process will not be described here).

[0055] The difference image saliency map obtained by the above method may have small areas where defocused pixels and focused pixels appear alternately at the edge, resulting in unclear edges. Therefore, in step A402, the complementary information between the images is used to perform a comprehensive operation on the saliency maps of the two source images and the two difference image saliency maps to amplify the difference between focus and defocus in the saliency map and highlight the boundary between the focused area and the defocused area.

[0056] Specifically, step A402 includes: The enhanced saliency map is calculated according to the following formula: ; ; in, is the enhanced saliency map corresponding to the source image A, is the enhanced saliency map corresponding to the source image B, is the difference image saliency map corresponding to the difference image of the source image A (the pixel value of the (x, y) pixel point is ), is the difference image saliency map corresponding to the difference image of the source image B (the pixel value of the (x, y) pixel point is ), is a preset proportionality factor (which can be set according to actual needs, for example, to 0.5, but not limited thereto).

[0057] Further, step A403 includes: The initial decision graph corresponding to each source image is calculated according to the following formula: ; ; in, is the pixel value of the (x, y) pixel of the initial decision graph corresponding to the source image A, is the pixel value of the (x, y) pixel of the initial decision graph corresponding to the source image B, for The pixel value of the (x, y) pixel point, for The edge information of the initial decision graph obtained in the above manner is richer and smoother.

[0058] Preferably, step A5 comprises: A501. Filter each initial decision graph using a bwareaopen filter to obtain a corresponding initial filter decision graph; A502. Using each source image as a guide image, perform guided filtering on the corresponding preliminary filter decision graph to obtain a corresponding guided decision graph; A503. Perform consistency verification on each guided decision diagram to obtain a corresponding verification decision diagram; A504. Assign pixel values ​​of pixel points with unclear focus attributes in each verification decision graph to obtain a corresponding optimization decision graph.

[0059] In the process of obtaining the initial decision graph, the focus attributes of some pixels (focus attributes are used to distinguish whether the pixels belong to the focused area or the defocused area) may be misclassified, resulting in problems such as small holes, protrusions and breaks in the initial decision graph. In order to prevent these problems from affecting the quality of the final fused image, this application uses the bwareaopen filter (bwareaopen filter is a filter provided by the Python platform) to filter the initial decision graph.

[0060] The bwareaopen filter is used to filter each initial decision graph to obtain the corresponding initial filter decision graph, which can be expressed as: ; ; in, is the initial decision graph corresponding to the source image A (the pixel value of its (x, y) pixel is ), is the initial decision graph corresponding to the source image B (the pixel value of its (x, y) pixel is ), For the corresponding The initial filtration decision diagram, For the corresponding The initial filter decision diagram is , t is the threshold, and t=th*S, S is the size of the source image (equal to M*N), th is a preset parameter (can be set according to actual needs, such as 0.02, but not limited to this), It is the filter function of bwareaopen filter.

[0061] After removing small areas such as small holes, protrusions and breaks in the initial decision graph through the bwareaopen filter, there may be boundary errors and artifacts in the boundary area of ​​the decision graph. For this reason, guided filtering is performed on the initial filter decision graph in step A502 to better preserve the edge features of the source image in the fused image.

[0062] Using each source image as a guide image, the corresponding preliminary filter decision graph is guided and filtered to obtain the corresponding guided decision graph, which can be expressed as: ; ; in, is the guided decision graph corresponding to the source image A, is the guided decision graph corresponding to the source image B, is the GF guided filter function, is the source image A, is the source image B, r is the window size (can be set according to actual needs, for example, 5, but not limited to this), is a regularization parameter (which can be set according to actual needs, for example, 0.3, but not limited to this).

[0063] After guided filtering, although the edge area of ​​the decision graph can become clearer, some erroneous information may be introduced, causing a pseudo edge effect to appear at the edge of the final fused image. To solve this problem, in step A503, consistency verification processing is performed on the guided decision graph.

[0064] In step A503, the consistency verification process of the guided decision graph can be performed according to the following formula: ; ; in, For the corresponding The pixel value of the (x, y) pixel of the verification decision graph, For the corresponding The pixel value of the (x, y) pixel of the verification decision graph, , are the lateral offset and the longitudinal offset respectively, is the neighborhood size (equal to the product of q and S, q is the reference ratio, which can be set according to actual needs, for example, 5×10 -5 , but not limited to this), Indicates that the following conditions are met: the pixel (x+a, y+b) is centered at the pixel (x, y) and is In the neighborhood of , They are and The pixel value of the pixel point (x+a, y+b).

[0065] After the consistency verification of the decision graph, some pixels in the focused and defocused edge areas of the decision graph may not be able to confirm their focus attributes. Therefore, in step A504, the pixel values ​​of the pixels with unclear focus attributes in each verification decision graph are assigned values. Specifically, the assignment process can be performed according to the following formula: ; ; in, is the pixel value of the (x, y) pixel of the optimization decision graph corresponding to the source image A, is the pixel value of the (x, y) pixel corresponding to the optimized decision graph of the source image B.

[0066] Specifically, step A6 includes: The two source images are fused according to the following formula: ; in, is the brightness of the (x, y) pixel of the final fused image.

[0067] In one embodiment, five existing image fusion methods and the multi-focus image fusion method based on significant edge enhancement of the present application are used to respectively Figure 3 A set of source images (A and B in the figure are source images A and source image B, respectively) are fused. Five existing image fusion methods include a fusion algorithm based on joint bilateral filtering and local gradient energy operator (INS), a unified unsupervised image fusion network (U2Fusion), a universal end-to-end image fusion network based on storage unit (MUFusion), a compression decomposition network (SDNet) and a small area perception method (SAMF). The fusion results are shown in Figure 4 In the figure, a is the fusion result of the INS method, b is the fusion result of the U2Fusion method, c is the fusion result of the MUFusion method, d is the fusion result of the SDNet method, e is the fusion result of the SAMF method, and f is the fusion result of the multi-focus image fusion method based on significant edge enhancement of the present application.

[0068] from Figure 4 It can be seen that, except for e and f, other fusion results still retain a small amount of background information in the defocused area, which affects the clarity of the fusion results; e is not as accurate as f in extracting the focus edge details. Therefore, the multi-focus image fusion method based on significant edge enhancement in this application is superior to other image fusion methods.

[0069] As can be seen from the above, the multi-focus image fusion method based on significant edge enhancement obtains an initial fused image of two source images based on a transform domain method; the two source images are of the same size and are aligned with each other, and the focus areas of the two source images are complementary; a saliency map of the initial fused image and a saliency map of the two source images are generated; a difference map between the saliency map of the initial fused image and the saliency maps of each source image is generated; an initial decision map corresponding to each source image is obtained according to the saliency map of the initial fused image, the saliency maps of the two source images and the difference map; each initial decision map is optimized to obtain an optimized decision map corresponding to each source image; the two source images are fused based on the optimized decision map to obtain a final fused image; thereby, the edge information of the fused image can be enriched and the problems of edge blur and artifacts can be effectively solved, so as to obtain a clearer high-quality fused image.

[0070] Please refer to Figure 2 , is a structural diagram of an electronic device provided in an embodiment of the present application. The present application provides an electronic device, including: a processor 301 and a memory 302. The processor 301 and the memory 302 are interconnected and communicate with each other through a communication bus 303 and / or other forms of connection mechanisms (not shown). The memory 302 stores a computer program executable by the processor 301. When the electronic device is running, the processor 301 executes the computer program to execute the multi-focus image fusion method based on significant edge enhancement in any optional implementation of the above embodiment to achieve the following functions: obtaining an initial fusion of two source images; The method comprises the following steps: generating a saliency map of the initial fused image and a saliency map of the two source images; generating a difference map between the saliency map of the initial fused image and the saliency map of each source image; obtaining an initial decision map corresponding to each source image according to the saliency map of the initial fused image, the saliency maps of the two source images and the difference map; optimizing each initial decision map to obtain an optimized decision map corresponding to each source image; and fusing the two source images based on the optimized decision map to obtain a final fused image.

[0071] The embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the multi-focus image fusion method based on significant edge enhancement in any optional implementation of the above embodiment is executed to achieve the following functions: obtaining an initial fused image of two source images; the two source images are of the same size and are aligned with each other, and the focus areas of the two source images are complementary; generating a saliency map of the initial fused image and a saliency map of the two source images; generating a difference map between the saliency map of the initial fused image and the saliency maps of each of the source images; obtaining an initial decision map corresponding to each of the source images according to the saliency map of the initial fused image, the saliency maps of the two source images and the difference map; optimizing each of the initial decision maps to obtain an optimized decision map corresponding to each of the source images; fusing the two source images based on the optimized decision map to obtain a final fused image. Among them, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0072] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0073] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0074] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0075] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0076] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-focus image fusion method based on significant edge enhancement, characterized in that: Includes steps: A1. Based on the transform domain method, an initial fused image of two source images is obtained; the two source images are of the same size and are registered with each other, and the focus areas of the two source images are complementary; A2. Generate a saliency map of the initial fused image and saliency maps of the two source images; A3. generating a difference map between the saliency map of the initial fused image and the saliency map of each of the source images; A4. obtaining an initial decision map corresponding to each of the source images according to the saliency map of the initial fused image, the saliency maps of the two source images and the difference map; A5. Optimizing each of the initial decision graphs to obtain an optimized decision graph corresponding to each of the source images; A6. Fusing the two source images based on the optimization decision graph to obtain a final fused image.

2. The multi-focus image fusion method based on significant edge enhancement according to claim 1, characterized in that: Step A1 includes: A101. Calculate the fusion brightness gradient according to the brightness of the two source images; A102. Performing gradient reconstruction according to the fused brightness gradient to obtain a first fused image; A103. Perform nonlinear mapping on the first fused image to obtain the initial fused image.

3. The multi-focus image fusion method based on significant edge enhancement according to claim 1, characterized in that: Step A2 includes: Based on the Sobel operator, a Tenengrad detection method is used to perform salient information detection on the initial fused image and the two source images respectively, so as to obtain a salient map of the initial fused image and a salient map of the two source images.

4. The multi-focus image fusion method based on significant edge enhancement according to claim 1, characterized in that: Step A4 includes: A401. Performing significant information detection on each of the difference maps using the Tenengrad detection method to obtain a corresponding difference map significant map; A402. Calculate an enhanced saliency map according to the saliency maps of the two source images and the saliency map of the difference image; A403. According to the saliency map of the initial fused image and the enhanced saliency map, an initial decision map corresponding to each of the source images is calculated.

5. The multi-focus image fusion method based on significant edge enhancement according to claim 4, characterized in that: Step A402 includes: The enhanced saliency map is calculated according to the following formula: ； ； Wherein, A and B are the representation symbols of the two source images respectively. is the enhanced saliency map corresponding to the source image A, is the enhanced saliency map corresponding to the source image B, is the saliency map of the source image A, is the saliency map of the source image B, is the difference saliency map corresponding to the difference map of the source image A, is the difference saliency map corresponding to the difference map of the source image B, is the preset scale factor.

6. The multi-focus image fusion method based on significant edge enhancement according to claim 5, characterized in that: Step A403 includes: The initial decision graph corresponding to each source image is calculated according to the following formula: ； ； Among them, x and y are the horizontal pixel coordinates and the vertical pixel coordinates respectively. is the pixel value of the (x, y) pixel of the initial decision graph corresponding to the source image A, is the pixel value of the (x, y) pixel of the initial decision graph corresponding to the source image B, for The pixel value of the (x, y) pixel point, for The pixel value of the (x, y) pixel point, is the pixel value of the pixel point (x, y) of the saliency map of the initial fused image.

7. The multi-focus image fusion method based on significant edge enhancement according to claim 1, characterized in that: Step A5 includes: A501. Filter each of the initial decision diagrams using a bwareaopen filter to obtain a corresponding initial filter decision diagram; A502. Using each of the source images as a guide image, respectively, the corresponding preliminary filter decision graph is subjected to guided filtering to obtain a corresponding guided decision graph; A503. Perform consistency verification on each of the guided decision diagrams to obtain a corresponding verification decision diagram; A504. Assign values ​​to the pixel points with unclear focus attributes in each of the verification decision graphs to obtain a corresponding optimization decision graph.

8. The multi-focus image fusion method based on significant edge enhancement according to claim 1, characterized in that: Step A6 includes: The two source images are fused according to the following formula: ； Among them, x and y are the horizontal pixel coordinates and the vertical pixel coordinates respectively. is the brightness of the (x, y) pixel of the final fused image, A and B are the representation symbols of the two source images respectively, is the pixel value of the (x, y) pixel corresponding to the optimized decision graph of the source image A, is the pixel value of the (x, y) pixel corresponding to the optimized decision graph of the source image B, is the brightness of the pixel (x, y) of the source image A, is the brightness of the pixel (x, y) of the source image B.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, the steps in the multi-focus image fusion method based on significant edge enhancement as described in any one of claims 1 to 8 are executed.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multi-focus image fusion method based on significant edge enhancement as described in any one of claims 1 to 8 are executed.

Images