Multi-focus image fusion method and system based on salient feature difference

Abstract

The application discloses a kind of multi-focus image fusion method and system based on salient feature difference, which comprises: obtaining multi-focus image;Based on Hessian matrix, feature extraction is carried out on multi-focus image;Based on improved modified Laplacian energy, salient feature extraction is carried out on feature map, and improved modified Laplacian graph is obtained;Significant pixel difference analysis is carried out on improved modified Laplacian graph, and initial decision graph is obtained;Region screening optimization is carried out on initial decision graph, and final decision graph is obtained;According to final decision graph and multi-focus image reconstruction fusion result, fusion image is obtained.The system comprises: image acquisition module, first feature extraction module, salient feature extraction module, difference analysis module, optimization module and fusion module.By using the application, focused area on source image can be accurately distinguished, and high-quality fusion result is provided.The application can be widely applied in the field of image fusion.

Description

Technical Field

[0001] This invention relates to the field of image fusion, and in particular to a multi-focus image fusion method and system based on significant feature differences. Background Technology

[0002] Due to the limited depth of field of a camera, it can only focus on a specific area of ​​a scene and cannot focus on all objects simultaneously. Multi-focus image fusion aims to integrate multiple images of the same scene captured at different focal length settings into a single fused all-focus image. This fused image can be considered as a collection of the best-focused pixels extracted from a set of source images. Compared to a single source image, the fused image provides a more comprehensive, objective, and reasonable interpretation of the scene. Existing multi-focus fused images suffer from low clarity and an inability to effectively distinguish between in-focus and out-of-focus areas. Summary of the Invention

[0003] To address the aforementioned technical problems, the present invention aims to provide a multi-focus image fusion method and system based on significant feature differences, which can accurately distinguish the focus regions on the source image and provide high-quality fusion results.

[0004] The first technical solution adopted in this invention is: a multi-focus image fusion method based on significant feature differences, comprising the following steps:

[0005] Obtain multi-focus images from source images;

[0006] Feature maps are obtained by extracting features from multi-focus images based on the Hessian matrix.

[0007] An improved modified Laplacian graph is obtained based on the improved modified Laplacian energy and the extraction of salient features from the feature map.

[0008] Significant pixel difference analysis was performed on the improved modified Laplacian plot to obtain the initial decision map;

[0009] The initial decision map is optimized by region selection to obtain the final decision map;

[0010] The fused image is obtained based on the final decision map and the reconstruction and fusion results of the multi-focus image.

[0011] Furthermore, the step of extracting features from the multi-focus image based on the Hessian matrix to obtain a feature map specifically includes:

[0012] The Hessian matrix is ​​improved into a multi-scale Hessian matrix weighted at different scales, resulting in the improved matrix.

[0013] A threshold is set and feature extraction is performed on the multi-focus image based on the improved matrix to obtain a feature map.

[0014] Furthermore, the improved matrix formula is expressed as follows:

[0015]

[0016] In the above formula, j is the j-th scale, and n is the number of scales. Let H(x,y,σ) be the weight for scale j. j H(x,y) represents the Hessian matrix before improvement, and MSH(x,y) represents H(x,y,σ) at different scales. j ) Using weights The new matrix is ​​obtained by summing.

[0017] Furthermore, the step of obtaining an improved modified Laplacian graph based on the improved modified Laplacian energy and salient feature extraction from the feature map specifically includes:

[0018] Considering the pixels on the diagonal, we improve the modified Laplacian to obtain the improved modified Laplacian energy.

[0019] An improved modified Laplacian graph is obtained by using an improved modified Laplacian energy and extracting salient features from the feature map.

[0020] Furthermore, the expression for the improved modified Laplace energy sum is as follows:

[0021]

[0022] In the above equation, PSML(f(x,y),α) represents the improved modified Laplacian energy of the image f(x,y) calculated using a window of radius α. Let f(x,y) represent the modified Laplacian of the image f(x,y), where α represents the radius of the window, L represents the preset threshold, and (i,j) represents the pixels within the window of radius α, excluding the center pixel (x,y).

[0023] Furthermore, the step of performing significant pixel difference analysis on the improved modified Laplacian graph to obtain the initial decision map specifically includes:

[0024] The difference between the improved modified Laplace plot and the original plot is used to obtain the first type of difference information.

[0025] The maximum and minimum modified Laplacian plots of the source image are calculated based on the modified Laplacian energy.

[0026] The difference between the maximum modified Laplace plot and the minimum modified Laplace plot is used to obtain the second type of difference information;

[0027] Based on preset rules, an initial decision map is generated by combining the first type of difference information and the second type of difference information.

[0028] Furthermore, the preset rule formula is expressed as follows:

[0029]

[0030] In the above formula, IDM represents the initial decision graph, MMD represents the first type of difference information, and MSD represents the second type of difference information.

[0031] Furthermore, the step of optimizing the initial decision map by region selection to obtain the final decision map specifically includes:

[0032] Small structural regions are removed from the initial decision graph to obtain the intermediate decision graph;

[0033] Considering the consistency of objectives, the intermediate decision graph is optimized based on the adaptive multi-scale consistency method to obtain the final decision graph.

[0034] Furthermore, the formula for reconstructing the fusion result is expressed as follows:

[0035]

[0036] In the above formula, FM(x,y) represents the final decision graph, f A (x,y) represents the multifocal image A, f B (x,y) represents the multifocus image B.

[0037] The second technical solution adopted in this invention is: a multi-focus image fusion system based on significant feature differences, comprising:

[0038] The image acquisition module is used to acquire multi-focus images based on the source image;

[0039] The first feature extraction module extracts features from the multi-focus image based on the Hessian matrix to obtain a feature map;

[0040] The salient feature extraction module extracts salient features from the feature map based on the improved modified Laplacian energy, resulting in an improved modified Laplacian map.

[0041] The difference analysis module is used to perform significant pixel difference analysis on the improved modified Laplacian plot to obtain the initial decision map;

[0042] The optimization module is used to perform regional filtering and optimization on the initial decision map to obtain the final decision map;

[0043] The fusion module is used to reconstruct the fusion result based on the final decision map and the multi-focus image to obtain the fused image.

[0044] The beneficial effects of the method and system of this invention are as follows: This invention utilizes Hessian matrix decomposition of multi-focus images to obtain the feature regions of the images, uses improved modified Laplacian energy and extracts the salient features of the images, and then performs saliency analysis, decision graph optimization and fusion result reconstruction. It can accurately distinguish the focus regions on the source images and can adaptively process images of different sizes, providing high-quality fusion results. Attached Figure Description

[0045] Figure 1 This is a flowchart of the steps of a multi-focus image fusion method based on significant feature differences according to the present invention;

[0046] Figure 2 This is a schematic diagram of the data flow in a specific embodiment of the present invention;

[0047] Figure 3 This is a structural block diagram of a multi-focus image fusion system based on significant feature differences according to the present invention;

[0048] Figure 4 These are multi-focus source images from specific embodiments of the present invention;

[0049] Figure 5 This is a schematic diagram comparing the fused image obtained using this method with other methods. Detailed Implementation

[0050] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0051] Reference Figure 1 and Figure 2 This invention provides a multi-focus image fusion method based on significant feature differences, the method comprising the following steps:

[0052] S1. Obtain the corresponding multifocal image f based on the source image. A and f B ;

[0053] S2. Feature extraction is performed on the multi-focus image based on the Hessian matrix to obtain the feature map;

[0054] Input source image f A and f B The Hessian matrix is ​​defined as follows:

[0055]

[0056] Where L xx (x,y,σ) is the Gaussian second-order partial derivative δ 2 / δx 2 g(σ) is convolved with the image f at point (x,y), L xy (x,y,σ) and L yy The same applies to (x,y,σ).

[0057] S2.1. Improve the Hessian matrix into a multi-scale Hessian matrix weighted at different scales to obtain the improved matrix;

[0058] Since the Hessian matrix can extract rotation-invariant features from the source image at multiple scales, it can be improved into a multi-scale Hessian matrix weighted at different scales, as shown below.

[0059]

[0060] Where j is the j-th scale, and n is the number of scales. Let be the weight of scale j.

[0061] S2.2 Set a threshold and extract features from the multi-focus image based on the improved matrix to obtain a feature map.

[0062] Based on formula (2), the source image f can be extracted by setting a threshold λ. A and f B Feature Image FIM A (x,y) and FIM B (x,y)

[0063]

[0064]

[0065] Where λ is set to 0.0002.

[0066] S3. Based on the improved modified Laplacian energy and the extraction of salient features from the feature map, an improved modified Laplacian graph is obtained.

[0067] S3.1 Considering the pixels on the diagonal, improve the modified Laplacian to obtain the improved modified Laplacian energy;

[0068] The Modified Laplacian Energy (SML) is an effective tool for representing salient information in images. The mathematical expression of the Modified Laplacian (ML) is as follows:

[0069]

[0070] Where step represents the step size. Using a variable spacing (step size) between pixels to calculate ML allows it to adapt to changes in texture size, and is typically set to 1. SML is defined as follows:

[0071]

[0072] Where α is the radius of the window, L is the threshold set to 0 in this invention, and α is set to 3.

[0073] Traditional machine learning (ML) only considers pixels surrounding the center pixel in the horizontal and vertical directions. (x, y-step) represents the pixel below (x, y), (x, y+step) represents the pixel above, and so on for the remaining two. However, the improved ML considers the pixel in the lower left corner (x-step, y-step) to take into account pixels in the diagonal region.

[0074] This invention improves upon it by also considering the other four pixels on the diagonal that contain key information. The improved ML expression is as follows:

[0075]

[0076] The improved version of SML is named PSML, and its mathematical expression is as follows.

[0077]

[0078] S3.2. Based on the improved modified Laplacian energy and the extraction of salient features from the feature map, an improved modified Laplacian graph is obtained.

[0079] The feature map FIM obtained by decomposition using formulas (3) and (4) A and FIM B The PSML value for each pixel is expressed as follows:

[0080] M A (x,y)=PSML(FIM A (x,y),α) (9)

[0081] M B (x,y)=PSML(FIM B (x,y),α) (10)

[0082] In the above formula, we use a window of radius α to calculate the image FIM. A (x,y) and FIM B The improved modified Laplace energy of (x,y) is used to obtain the image M. A (x,y) and M B (x,y), where (x,y) represents the pixels in the image.

[0083] S4. Perform significant pixel difference analysis on the improved modified Laplacian plot to obtain the initial decision map;

[0084] S4.1 Calculate the maximum and minimum modified Laplacian plots of the source image based on the modified Laplacian energy;

[0085] S4.2. Subtract the maximum modified Laplace plot and the minimum modified Laplace plot to obtain the second type of difference information;

[0086] First, the feature map FIM is computed within a small window near the center pixel (x,y). A and FIM B Modified Laplace (ML) plot,

[0087]

[0088]

[0089] To find the most salient and least salient pixels in the source image, we need to find the maximum and minimum ML maps of the source image, respectively. The mathematical expression is as follows:

[0090] S max (x,y)=max(S A (x,y),S B (x,y)) (13)

[0091] S min (x,y)=min(S A (x,y),S B (x,y)) (14)

[0092] Where S max (x,y) and S min (x, y) represent the maximum and minimum ML maps of all source images, respectively. Since ML can reflect the focus information of an image, and S... A (cx,y) and S B The salient information for each position (x, y) is contained in S. max Therefore, S max It can be approximated as a fully focused saliency map, while S min This is a salience map that is completely out of focus. Salience map S max and S min The difference between them can be calculated as

[0093] MSD=S max -S min (15)

[0094] S4.3. Subtract the improved modified Laplace plot to obtain the first type of difference information;

[0095] The saliency map PSML difference of the source images can be calculated as follows:

[0096] MMD = M A -M B (16)

[0097] S max With S min These represent the salient pixel information of fully focused and fully defocused images, respectively, while MSD reflects the maximum difference in pixel salient information within the source image. Therefore, by comparing MMD and MSD, we can determine the strength of pixel salientity in the source image, thus reflecting the focusing characteristics of the pixels.

[0098] S4.4. Based on preset rules, and combining the first type of difference information and the second type of difference information, generate an initial decision map.

[0099] However, in MMD, M A With M B These are PSML diagrams of the source images; the differences between them are not S. max With S min So obvious. In fact, M A With M B The difference between each pixel in S is always less than or equal to the difference in S. max With S min The difference between mid-pixel density. This invention utilizes the difference between MSD and MMD to propose the following rules for obtaining the initial decision map.

[0100]

[0101] Where IDM is the source image f A The initial decision graph is given by μ, which is a custom threshold set to 0.55 in this invention.

[0102] S5. Optimize the initial decision map by filtering regions to obtain the final decision map;

[0103] S5.1 Remove small structural regions from the initial decision graph to obtain the intermediate decision graph;

[0104] Because the initial decision map inevitably contains some pixel regions that are misselected, meaning that the focus area in the decision map also contains some small isolated areas composed of misselected pixels, this invention uses a "bwareaopen" filter to remove isolated areas or holes containing erroneous pixels within the focus area.

[0105] MDM=bwareaopen(IMD,S / 25) (18)

[0106] Where S represents the area of ​​the source image, formula (18) uses the "bwareaopen" filter to eliminate isolated regions in the IMD with an area smaller than S / 45, resulting in the intermediate decision map MDM. Compared to the IMD, the MDM can further correct misselected pixels in the decision map and improve the focusing detection accuracy.

[0107] S5.2 Considering the consistency of objectives, the intermediate decision graph is optimized based on the adaptive multi-scale consistency method to obtain the final decision graph.

[0108] Meanwhile, considering the consistency of the target, this invention employs a consistency verification technique to optimize MDM. Traditional consistency verification techniques only use a single window to determine whether a pixel is within the focal region.

[0109]

[0110] Here, δ represents a square neighborhood window centered at (x,y). However, such methods have a fixed window size, which cannot effectively consider pixel values ​​under different window sizes, and can easily lead to pixel judgment errors in the boundary region between the focused and defocused areas. Furthermore, due to the diversity of images, a fixed window size may have different effects on different decision maps, and may even cause severe damage to the focused area in the decision map, introducing a large area of ​​erroneous pixels. Therefore, the selection of the window size for consistency verification is crucial. To effectively solve this problem, this invention proposes a new adaptive multi-scale consistency verification scheme. This invention sets up two windows to determine whether a pixel is focused, and the mathematical formula is expressed as follows:

[0111]

[0112] Where δ A With δ B For two square neighborhood windows of different sizes centered at (m,n), δ A =T,δ B =T-14, assuming the source image size is M×N, then FM represents the final decision graph. In this invention, T is set to 17.

[0113] S6. Based on the final decision map and the multi-focus image, the fusion result is reconstructed to obtain the fused image.

[0114] After obtaining the final decision graph FM(x,y), the fusion result can be obtained by the following formula.

[0115]

[0116] In the above formula, FM(x,y) represents the final decision graph, f A (x,y) represents the multifocal image A, f B (x,y) represents the multifocus image B. When the value of FM(x,y) is 1, the pixel F(x,y) in the fusion result takes the pixel f from the multifocus image A. A (x,y), when the value in FM(x,y) is 0, the pixel F(x,y) in the fusion result is taken as the pixel f in the multifocus image B. B (x,y).

[0117] To further demonstrate the advantages and effectiveness of this invention, we conducted a comparative experiment with five state-of-the-art image fusion algorithms, analyzing the merits and demerits of each algorithm from a subjective visual evaluation perspective. Figure 4 (a) and (b) are two multifocal source images (both are 256×256 in size). Figure 5 (a) through (f) show the fusion results and difference maps of the following algorithms: Non-subsampled contourlet transform and residual removal-based multi-focus image fusion algorithm (NSCT-RR); Multi-scale weighted gradient-based multi-focus image fusion algorithm (MWGF); Hessian matrix factorization-based multi-focus image fusion algorithm (HDM); Convolutional sparse representation-based image fusion algorithm (CSR); Ensembled convolutional neural network algorithm for multi-focus image fusion (ECNN); and the fusion algorithm presented in this paper. Figure 5 The difference graph shows that Figure 5 (e) Residual background information remains in the out-of-focus areas, which reduces the sharpness of the fused result. Figure 5 (b) Artifacts appeared at the boundaries, affecting the image fusion quality. Besides Figure 5 Apart from (f), the fusion results of the remaining methods failed to distinguish well between the focused and out-of-focus areas, and could not accurately determine the activity level of pixels at the boundary of the focused area. This phenomenon is particularly evident in… Figure 5 This is especially evident in the magnified red area of ​​(d). Observe Figure 5 (f) It can be observed that the method proposed in this invention outperforms other comparative methods in maintaining the sharpness of the fused image and the accuracy of detecting focused pixels. In summary, the fusion performance of this invention is leading among the five comparative methods.

[0118] like Figure 3 As shown, a multi-focus image fusion system based on significant feature differences includes:

[0119] The image acquisition module is used to acquire multi-focus images based on the source image;

[0120] The first feature extraction module extracts features from the multi-focus image based on the Hessian matrix to obtain a feature map;

[0121] The salient feature extraction module extracts salient features from the feature map based on the improved modified Laplacian energy, resulting in an improved modified Laplacian map.

[0122] The difference analysis module is used to perform significant pixel difference analysis on the improved modified Laplacian plot to obtain the initial decision map;

[0123] The optimization module is used to perform regional filtering and optimization on the initial decision map to obtain the final decision map;

[0124] The fusion module is used to reconstruct the fusion result based on the final decision map and the multi-focus image to obtain the fused image.

[0125] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A multi-focus image fusion method based on significant feature differences, characterized in that, Includes the following steps: Obtain multi-focus images from source images; Feature maps are obtained by extracting features from multi-focus images based on the Hessian matrix. An improved modified Laplacian graph is obtained based on the improved modified Laplacian energy and the extraction of salient features from the feature map. Significant pixel difference analysis was performed on the improved modified Laplacian plot to obtain the initial decision map; The initial decision map is optimized by region selection to obtain the final decision map; The fused image is obtained based on the final decision map and the reconstruction fusion result of the multi-focus image; The step of performing significant pixel difference analysis on the improved modified Laplacian graph to obtain the initial decision map specifically includes: The difference between the improved modified Laplace plot and the original plot is used to obtain the first type of difference information. The maximum and minimum modified Laplacian plots of the source image are calculated based on the modified Laplacian energy. The difference between the maximum modified Laplace plot and the minimum modified Laplace plot is used to obtain the second type of difference information; Based on preset rules, an initial decision map is generated by combining the first type of difference information and the second type of difference information. The preset rule formula is expressed as follows: In the above formula, Represents the initial decision graph. This represents the first type of difference information. This indicates the second type of difference information; The step of optimizing the initial decision map by region selection to obtain the final decision map specifically includes: The bwareaopen filter is used to remove small structural regions from the initial decision graph to obtain an intermediate decision graph. Considering the consistency of objectives, the intermediate decision graph is optimized based on the adaptive multi-scale consistency method to obtain the final decision graph.

2. The multi-focus image fusion method based on significant feature differences according to claim 1, characterized in that, The step of extracting features from multi-focus images based on the Hessian matrix to obtain feature maps specifically includes: The Hessian matrix is ​​improved into a multi-scale Hessian matrix weighted at different scales; A threshold is set and feature extraction is performed on the multi-focus image based on the multi-scale Hessian matrix to obtain a feature map.

3. The multi-focus image fusion method based on significant feature differences according to claim 2, characterized in that, The multi-scale Hessian matrix formula is expressed as follows: In the above formula, For the first One scale, For the number of scales, For scale The weight, This represents the Hessian matrix before improvement. Representing different scales Using weights The new matrix is ​​obtained by summing.

4. The multi-focus image fusion method based on significant feature differences according to claim 3, characterized in that, The step of obtaining an improved modified Laplacian graph based on the improved modified Laplacian energy and salient feature extraction from the feature map specifically includes: Considering the pixels on the diagonal, we improve the modified Laplacian to obtain the improved modified Laplacian energy. An improved modified Laplacian graph is obtained by using an improved modified Laplacian energy and extracting salient features from the feature map.

5. The multi-focus image fusion method based on significant feature differences according to claim 4, characterized in that, The expression for the improved modified Laplace energy sum is as follows: In the above formula, This means we use a radius of Window size for image calculation The improved modified Laplace energy and, Representing an image The modified Laplace, Indicates the radius of the window. Indicates the preset threshold. Representing radius excluding the center pixel within the window Pixels other than those in the image.

6. The multi-focus image fusion method based on significant feature differences according to claim 1, characterized in that, The formula for reconstructing the fusion result is expressed as follows: In the above formula, This represents the final decision diagram. Represents a multifocal image A. This represents the multifocus image B.

7. A multi-focus image fusion system based on significant feature differences, characterized in that, For performing the multi-focus image fusion method based on significant feature differences as described in claim 1, comprising: The image acquisition module is used to acquire multi-focus images based on the source image; The first feature extraction module extracts features from the multi-focus image based on the Hessian matrix to obtain a feature map; The salient feature extraction module extracts salient features from the feature map based on the improved modified Laplacian energy, resulting in an improved modified Laplacian map. The difference analysis module is used to perform significant pixel difference analysis on the improved modified Laplacian plot to obtain the initial decision map; The optimization module is used to perform regional filtering and optimization on the initial decision map to obtain the final decision map; The fusion module is used to reconstruct the fusion result based on the final decision map and the multi-focus image to obtain the fused image.

Images