Multi-focus image fusion method based on focus area detection and boundary processing

Abstract

The invention discloses a multi-focus image fusion method based on focus area detection and boundary processing. The method comprises the following steps: performing focus measurement processing and small area removal processing on a source image pair through improved Laplace energy to obtain a focus area decision diagram; performing initial fusion on the focus area decision diagram and the source image pair to obtain an initial fusion diagram; obtaining a boundary region decision diagram from the focus region decision diagram through a boundary extraction algorithm; inputting the source image pair into a multi-scale feature interactive twin network to generate a network fusion graph; and according to the boundary region decision diagram, performing weighted summation on the initial fusion diagram and the network fusion diagram to obtain a final fusion diagram. According to the invention, high-quality image fusion is realized.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a multi-focus image fusion method based on focus area detection and boundary processing. Background Art

[0002] Due to the limitation of the depth of field of optical lenses, only the focused area in the image can present a clear effect. Therefore, it is difficult to obtain a fully clear image of the entire scene. Multi-focus image fusion (MFIF) technology is an effective means to solve the above problem. By extracting the complementary information of two or more local focused images of the same scene, a fully clear fused image is generated. At present, image fusion has been widely used in medicine, military, agriculture, security and other fields.

[0003] MFIF is divided into three categories: transform domain methods, spatial domain methods, and deep learning methods.

[0004] The transform domain method is an image fusion algorithm based on mathematical transformation, which mainly includes three steps: image decomposition, coefficient fusion, and image reconstruction. Common methods include: pyramid transform-based method, wavelet transform-based method, and multi-scale geometric analysis algorithm.

[0005] The spatial domain method directly fuses the source images, which can retain the original information to the greatest extent and obtain a focus area without information residue. It is divided into three methods: block-based, region-based, and pixel-based. The first two have problems such as block artifacts and blocking effects. The pixel-based method first calculates the degree of focus and defocus of corresponding pixels in multiple source images through focus measurement, and generates a weight map; then the weight map is used to achieve image fusion. Common focus measurement methods include variance, energy gradient (EOG), spatial frequency (SF), Laplace energy sum (EOL), and improved Laplace energy sum (SML).

[0006] Deep learning methods benefit from powerful feature learning capabilities, including classification models and regression models. The former detects the focus area of ​​the image by classifying or segmenting pixels; the latter achieves end-to-end mapping from source image to fused image through network training. Tang et al. divided the image into blocks and further divided them into three categories: focus area, defocus area and unknown area. Through network training, they solved the problems of difficulty in designing artificial activity level measurement in traditional methods. However, this method is highly dependent on image block division and it is difficult to determine the pixel ownership of the boundary area. In 2020, Xu et al. proposed an unsupervised end-to-end image fusion network. The algorithm can achieve clear and artifact-free focus and defocus boundary areas, but there is information residual in the focus area. In order to solve the problem of boundary fusion quality, Ma et al. proposed a cascaded, boundary-aware neural network to refine the boundary fusion results through sub-networks, but the network did not use the attention mechanism to improve the boundary guidance map, and the network structure needs to be improved. Lai et al. proposed a multi-scale visual attention deep convolutional neural network (MADCNN), which uses a multi-scale feature extraction unit to extract features at different spatial scales and introduces an attention unit to focus on the details of the focal area, but there is a problem of computational complexity.

[0007] The disadvantage of the transform domain method is that the spatial consistency is relatively poor, that is, it is easy to lose spatial frequency information, which in turn causes different degrees of distortion in the fused image. Spatial domain method: Since the activity level measurement and fusion rules of the image need to be manually designed, it is difficult to adapt to all complex situations. Although the deep learning method is universal, it has problems such as insufficient feature utilization and single extraction scale, resulting in information residue in the focus area. At the same time, since the characteristics of the boundary area and the non-boundary area are obviously different, it is difficult to obtain a clear fusion effect by simply using the deep learning network. Therefore, we consider: combining the advantages of the spatial domain method and the deep learning method, respectively extracting information from the focus area and the boundary area to effectively solve the above problems.

[0008] In the prior art, the transform domain method has the disadvantage of poor spatial consistency, that is, it is easy to lose spatial frequency information, which in turn causes different degrees of distortion in the fused image. Spatial domain method: Since the activity level measurement and fusion rules of the image need to be manually designed, it is difficult to adapt to all complex situations. Although the deep learning method is universal, it has problems such as insufficient feature utilization and single extraction scale, which leads to consequences such as residual information in the focused area. At the same time, since the features of the boundary area and the non-boundary area are obviously different, it is difficult to obtain a completely clear fusion effect by simply using deep learning network processing. How to obtain a clear fused image without residual information in the focused area and boundary artifacts is a technical problem that technicians in this field need to solve.

[0009] The Chinese invention application with application number 202310767148.7 discloses "A multi-focus image fusion method based on multi-scale context perception", and its technical solution is: data preparation and preprocessing, construction of a multi-scale context perception network, including: encoder, coarse positioning decoder, receptive field enhancement module and multi-scale feature interaction module, wherein the encoder is a visual converter for multi-scale feature extraction, the decoder includes: convolution and activation functions for multi-scale feature coarse decoding, the receptive field enhancement module includes convolution and activation functions for feature receptive field enhancement, and the multi-scale feature interaction module includes: convolution and activation functions for multi-level feature fusion; the input multi-focus images are fused, including: network training and multi-focus image fusion. Its technical solution does not solve the technical problem of residual focus area information and boundary artifacts in the fused image. Summary of the invention

[0010] The object of the present invention is to provide a multi-focus image fusion method based on focus area detection and boundary processing, so as to realize image fusion without focus area information residue and boundary artifact.

[0011] In order to solve the above technical problems, the present invention provides a multi-focus image fusion method based on focus area detection and boundary processing, the method comprising:

[0012] The focus region decision map is obtained by improving the Laplace energy and performing focus measurement processing and small region removal processing on the source image pair.

[0013] Initially fusing the focus region decision map with the source image pair to obtain an initial fusion map;

[0014] The focus area decision diagram is subjected to a boundary extraction algorithm to obtain a boundary area decision diagram;

[0015] Inputting the source image pair into a multi-scale feature interactive Siamese network to generate a network fusion graph;

[0016] According to the boundary area decision graph, the initial fusion graph and the network fusion graph are weighted summed to obtain a final fusion graph.

[0017] As a preferred solution, the method of obtaining a focus area decision map by improving Laplace energy and performing focus measurement processing and small area removal processing on a source image pair includes:

[0018] The source image f is respectively processed by the improved Laplace operator A and f B Perform focus measurement and obtain the focus map S A and S B ; The calculation formula of the focus metric is as follows:

[0019] ML (i,j)=|2f(i,j)-f(i-step,j)-f(i+step,j)|+|2f(i,j)-f(i,j-step)-f(i,j+step)| (1)

[0020] In formula (1), M L is the improved Laplacian operator, f(i,j) is the pixel value of the source image, step is the variable spacing between pixels, where step = 1;

[0021]

[0022] In formula (2), SML is the focus metric at point (i, j), and (2M+1)×(2N+1) is the metric window size;

[0023] Using formulas (3) and (4) to calculate the source image f A and the source image f B Perform focus measurement and obtain the focus map S A and S B , the formula is as follows:

[0024] S A (i,j)=SML(f A (i,j)) (3)

[0025] S B (i,j)=SML(f B (i,j)) (4)

[0026] According to the focus map S A and S B f A , f B Fill the pixels and get the initial decision graph DM init , the formula is as follows:

[0027] DM init (i,j)=S A (i,j)-S B (i,j) (5);

[0028] The initial decision graph DM is filtered by a double threshold filtering method. init Then, the binary image is used to obtain the number of connected domains and calculate the area of ​​the connected domain. When the area of ​​the connected domain is less than the preset threshold, the focus attribute value of the connected domain is corrected to obtain the focus area decision map DM. nb .

[0029] As a preferred solution, the method of initially fusing the focus region decision map and the source image pair to obtain an initial fusion map includes:

[0030] According to the focus area decision map DM nb Determine the focus area of ​​the source image and obtain the initial fusion image f init , the formula is as follows:

[0031]

[0032] Where (i, j) is the pixel position.

[0033] As a preferred solution, the method of obtaining the boundary area decision by using the boundary extraction algorithm on the focus area decision graph includes:

[0034] Focus Area Decision Diagram DM nb Perform Canny edge detection to obtain the boundary; then, use guided filtering to process the boundary to obtain the boundary area decision map DM b .

[0035] As a preferred solution, the source image pair is input into a multi-scale feature interactive twin network, and the method for generating a network fusion graph includes:

[0036] The source image f A and f B The images are input into two feature extraction branches of the multi-scale feature interaction twin network respectively, and feature extraction is performed through 3×3 convolution, 1×1 convolution, and 3×3 convolution respectively;

[0037] The features of the two feature extraction branches are superimposed on the channel through dense connections to obtain fused features;

[0038] The fusion features are subjected to image reconstruction processing to obtain a network fusion graph.

[0039] As a preferred solution, the method of performing image reconstruction processing on the fusion features to obtain a network fusion graph includes:

[0040] The fusion feature is reduced in dimension by 3×3 convolution, and then the convolution result is input into the BRB structure to perform three-stage feature extraction operations in sequence;

[0041] The features obtained by the first stage feature extraction operation and the features obtained by the second stage feature extraction operation are subjected to multi-scale feature fusion through the first MSFF module to obtain the first fused feature;

[0042] The features obtained from the feature extraction operations in the three stages are subjected to multi-scale feature fusion through the second MSFF module to obtain the second fused features;

[0043] Inputting the first fused feature and the second fused feature into a coordinate attention upsampling CAU module for feature enhancement;

[0044] The features output by the CAU module are processed by 1×1 convolution and 3×3 convolution in sequence, and then the convolution result and the features output by the first 3×3 convolution in the two feature extraction branches are fused to obtain the third fused feature;

[0045] The third fused feature is processed by 1×1 convolution and 3×3 convolution in sequence, and then the convolution result and the feature output by the second 3×3 convolution in the two feature extraction branches are fused to obtain a fourth fused feature;

[0046] The fourth fusion feature is processed by 1×1 convolution, 3×3 convolution, 1×1 convolution, and 3×3 convolution in sequence to obtain the network fusion graph f net .

[0047] As a preferred solution, the outputs of the feature extraction operations in the three stages are I1, I2, and I3, respectively, where I1∈R 2H×2W×(C / 2) , 2H is the height of the source image, 2W is the width of the source image, and C is the total number of channels I2∈R in the second stage feature extraction operation H×W×C , I3∈R (H / 2)×(W / 2)×2C ;

[0048] The implementation method of the second MSFF module includes:

[0049] Use 1×1 convolution on I1 and I3 so that their number of channels is the same as I2, which is C.

[0050] Through 3×3 convolution and downsampling with a step size of 2, I1 is transformed into I'1, and the size of I'1 is (2H / 2)×(2W / 2), that is, I'1∈R H×W×C ; Through a 3×3 convolution with a step size of 1, I2 is converted to I'2, and the size of I'2 remains unchanged; I3 is upsampled using bilinear interpolation to obtain I'3, I'3∈R H×W×C ; I'1, I'2 and I'3 are the same size, add them together to get I:

[0051] I=I'1+I'2+I'3 (7)

[0052] Perform global average pooling on I to obtain the global feature F, F∈R 1×1×C , the c(c∈{1,2,…,C})th channel F c for:

[0053]

[0054] Among them Ic is the cth channel of I.

[0055] Use 1×1 convolution to reduce the dimension of F and get V, V∈R 1×1×D ; Then, batch normalize V; Finally, use the ReLU activation function to get Z, Z∈R 1×1×D , where D = 4;

[0056]

[0057] Where B is the batch normalization (BN) operation, and δ(·) is the ReLU activation function;

[0058] Three groups of 1×1 convolutions are used to increase the dimension of Z, and three C-dimensional attention vectors n1, n2, and n3 are obtained by splicing and reshaping, where n1, n2, and n3∈R 1×1×C ; Then, n1, n2, and n3 are normalized using the softmax activation function to obtain F1, F2, and F3, respectively, where F1, F2, and F3∈R 1×1×C ; F i The cth channel F i,c for:

[0059]

[0060] Among them, n i,c n i The cth channel of , i∈{1,2,3};

[0061] The multi-scale features I'1, I'2, I'3 are weighted summed with F1, F2, F3 to obtain the fusion feature IF2, IF2∈R H ×W×C , the cth channel IF of IF2 2,c for:

[0062] IF 2,c =F 1,c I' 1,c +F 2,c I' 2,c +F 3,c I' 3,c (11)

[0063] Among them, I' 1,c , I' 2,c , I' 3,c They are the cth channels of I'1, I'2, and I'3 respectively;

[0064] The input branches of the first MSFF module are I1 and I2. The inputs I1, I2 and I3 of the second MSFF module are replaced by 0, I1 and I2, which are the inputs of the first MSFF module. The output of the first MSFF module is IF1, IF1∈R 2H ×2W×(C / 2) .

[0065] As a preferred solution, the two inputs of the coordinate attention upsampling CAU module are IF1 and IF2, the CAU module includes a coordinate attention block CA, and the implementation method of the CAU module includes:

[0066] IF2 is processed by 1×1 convolution, batch normalization, and ReLU activation function to obtain IF'2, which has the same number of channels as IF1.

[0067] Upsample IF'2 to obtain IF"2, which has the same size as IF1;

[0068] Concatenate IF'2 and IF2 to obtain X(X∈R 2H×2W×C );

[0069] Take X as the input of CA and generate the coordinate attention map T through CA h 、T w ;

[0070] The output of CA is T h 、T w Multiplying the three by IF1 gives CA out , C.A. out ∈R 2H×2W×(C / 2) ;

[0071] CA out Sum it with IF"2 to get the final output CAU of the module out , CAU out ∈R 2H×2W×(C / 2) .

[0072] As a preferred solution, X is used as the input of CA and the coordinate attention map T is generated by CA h 、T w The methods include:

[0073] Perform average pooling on X in the vertical and horizontal directions respectively to obtain K h , K h ∈R 2H×1×C , h∈{1,2,…,2H}, K w , K w ∈R 1×2W×C , w∈{1,2,…,2W},K h , K wThe Cth channel of The calculation formula is as follows:

[0074]

[0075]

[0076] X c is the Cth channel of X, X c Ability to capture orientation and position across channels;

[0077] For S w The 2H and 2W dimensions are exchanged and combined with S h Concatenate to get G, G∈R (2H+2W)×1×C ; Then, use 1×1 convolution to reduce the dimension of G and get U, U∈R (2H+2W)×1×D , where D = 8; batch normalization and ReLu6 activation function processing are performed on U to obtain J, J∈R (2H+2W)×1×D ;

[0078] Split J into two separate tensors J h , J w , where J h ∈R 2H×1×D , J w ∈R 2W×1×D , and J w 2H and 2W are exchanged back to their original dimensions; then, h and J w Perform 1×1 convolution respectively; finally, perform sigmoid activation function processing to obtain the coordinate attention map T h 、T w , where T h ∈R 2H×1×(C / 2) , T w ∈R 1×2W×(C / 2) , the specific formula is as follows:

[0079] T h =σ(E h J h )(14)

[0080] T w =σ(E w J w )(15)

[0081] Among them, σ is the sigmoid activation function, E h 、E w It plays the role of 1×1 convolution.

[0082] As a preferred solution, the multi-scale feature interaction twin network uses the mean square error loss L MSEand the structural similarity loss L SSIM Combined to construct the loss function L, the specific formula is as follows:

[0083] L SSIM =1-SSIM

[0084]

[0085] Among them, O and P represent the network output image and label image respectively, μ O and μ P represents the mean of the image, σ O and σ P represents the image variance, σ O,P is the covariance of the two images, C1, C2, and C3 are constants used to avoid the denominator being 0, where W and H are the width and height of the image, and (i, j) is the pixel position.

[0086] Compared with the prior art, the present invention has the following beneficial effects:

[0087] The present invention retains image texture information through improved Laplace energy sum, achieves no information residue in the focus area, and enhances the clarity of the focus area.

[0088] The present invention realizes a network structure of multi-scale feature extraction, interaction and reconstruction through a multi-scale feature interactive twin network, thereby obtaining a clear boundary area without artifacts. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 A flow chart of a multi-focus image fusion method based on focus area detection and boundary processing provided in this embodiment;

[0090] Figure 2 A framework diagram of a multi-focus image fusion method based on focus area detection and boundary processing provided in this embodiment;

[0091] Figure 3 This is a graph of the SML focus detection results provided in this embodiment;

[0092] Figure 4 A multi-scale feature interaction twin network diagram provided for this embodiment;

[0093] Figure 5 A second MSFF module framework diagram provided for this embodiment;

[0094] Figure 6 A CAU module framework diagram provided for this embodiment;

[0095] Figure 7 A multi-scale feature interactive twin network training flow chart provided for this embodiment;

[0096] Figure 8 This is a graph showing the fusion results of the “golf” image using different methods provided in this embodiment;

[0097] Fig. 9 This is a differential image between the “golf” image fusion result provided in this embodiment and the source image. DETAILED DESCRIPTION

[0098] The accompanying drawings are only used for illustrative purposes and are not to be construed as limiting the present invention;

[0099] It should be clear that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.

[0100] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0101] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims. In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and do not have to be used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0102] In addition, in the description of this application, unless otherwise specified, "multiple" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0103] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0104] Example 1

[0105] Please refer to Figure 1 as well as Figure 2 This embodiment provides a multi-focus image fusion method based on focus area detection and boundary processing, the method comprising:

[0106] S1: The focus region decision graph is obtained by improving the Laplace energy and performing focus measurement processing and small region removal processing on the source image pair;

[0107] In a specific embodiment, please refer to Figure 3 , a method for obtaining a focus region decision map by performing focus measurement processing and small region removal processing on a source image pair through improved Laplace energy includes:

[0108] The source image f is respectively processed by the improved Laplace operator A and f B Perform focus measurement and obtain the focus map S A and S B ; The calculation formula of the focus metric is as follows:

[0109] M L (i,j)=|2f(i,j)-f(i-step,j)-f(i+step,j)|+|2f(i,j)-f(i,j-step)-f(i,j+step)| (1)

[0110] In formula (1), M L is the improved Laplacian operator, f(i,j) is the pixel value of the source image, step is the variable spacing between pixels, where step = 1;

[0111]

[0112] In formula (2), SML is the focus metric at point (i, j), and (2M+1)×(2N+1) is the metric window size;

[0113] Using formulas (3) and (4) to calculate the source image f A and the source image f B Perform focus measurement and obtain the focus map S A and S B , the formula is as follows:

[0114] S A (i,j)=SML(f A (i,j)) (3)

[0115] S B (i,j)=SML(f B (i,j)) (4)

[0116] According to the focus map S A and S B f A , f B Fill the pixels and get the initial decision graph DM init , the formula is as follows:

[0117] DM init (i,j)=S A (i,j)-S B (i,j) (5);

[0118] The initial decision graph DM is filtered by a double threshold filtering method. init Then, the binary image is used to obtain the number of connected domains and calculate the area of ​​the connected domain. When the area of ​​the connected domain is less than the preset threshold, the focus attribute value of the connected domain is corrected to obtain the focus area decision map DM. nb .

[0119] S2: The focus area decision map DM nb Perform initial fusion with the source image pair to obtain the initial fusion image f init ;

[0120] In a specific embodiment, the method of initially fusing the focus region decision map with the source image pair to obtain an initial fusion map includes:

[0121] According to the focus area decision map DM nb Determine the focus area of ​​the source image and obtain the initial fusion image f init , the formula is as follows:

[0122]

[0123] Where (i, j) is the pixel position.

[0124] S3: applying a boundary extraction algorithm to the focus area decision graph to obtain a boundary area decision graph;

[0125] In a specific embodiment, the method of obtaining a boundary area decision by using a boundary extraction algorithm on the focus area decision graph includes:

[0126] Focus Area Decision Diagram DM nb Perform Canny edge detection to obtain the boundary; then, use guided filtering to process the boundary to obtain the boundary area decision map DM b .

[0127] S4: inputting the source image pair into a multi-scale feature interaction twin network (MFIS) network to generate a network fusion graph;

[0128] It should be noted that the MFIS network adopts a Siamese Network structure, which consists of two sub-networks that share parameters and weights. In order to reduce the difficulty of network training and improve network efficiency, we add a 1×1 convolution after the 3×3 convolution, which can reduce parameters, increase the nonlinear characteristics of the network, and improve the network's understanding ability. The network does not have a fully connected layer, so images of any size can be input. In addition, through dense connections, features of different scales can be fully utilized.

[0129] The entire network mainly consists of three parts: feature extraction, feature fusion and image reconstruction. The feature extraction part contains two identical branches, using 3×3 and 1×1 convolutions to extract the features of the source image. The feature fusion part uses dense connections to superimpose the features extracted by the two branches on the channel to obtain fused features. The image reconstruction part uses the Basic Residual Block (BRB) in ResNet to build the backbone network.

[0130] In a specific embodiment, please refer to Figure 4 , the source image pair is input into the MFIS network, and the method for generating a network fusion graph includes:

[0131] The source image f A and f B The images are input into two feature extraction branches of the MFIS network and feature extracted through 3×3 convolution, 1×1 convolution, and 3×3 convolution respectively;

[0132] The features of the two feature extraction branches are superimposed on the channel through dense connections to obtain fused features;

[0133] The fusion features are subjected to image reconstruction processing to obtain a network fusion graph.

[0134] In a specific embodiment, the method of performing image reconstruction processing on the fusion features to obtain a network fusion graph includes:

[0135] The fusion feature is reduced in dimension through 3×3 convolution, and then the convolution result is input into the BRB structure to perform three-stage feature extraction operations in sequence; the three-stage feature extraction operations are recorded as Stages1, Stages2, and Stages3 in sequence.

[0136] The features obtained by the first stage feature extraction operation and the features obtained by the second stage feature extraction operation are subjected to multi-scale feature fusion through the first MSFF module to obtain the first fused feature;

[0137] The features obtained from the feature extraction operations in the three stages are subjected to multi-scale feature fusion through the second MSFF module to obtain the second fused features;

[0138] The MSFF module takes the features of the current and adjacent stages as input to achieve the interaction of adjacent features and the fusion of multi-scale features, which can avoid information redundancy and noise interference caused by the fusion of multi-scale features;

[0139] Inputting the first fused feature and the second fused feature into the coordinate attention upsampling CAU module for feature enhancement, so as to realize the transmission of global semantic information from the deep layer to the shallow layer;

[0140] The features output by the CAU module are processed by 1×1 convolution and 3×3 convolution in sequence, and then the convolution result and the features output by the first 3×3 convolution in the two feature extraction branches are fused to obtain the third fused feature;

[0141] The third fused feature is processed by 1×1 convolution and 3×3 convolution in sequence, and then the convolution result and the feature output by the second 3×3 convolution in the two feature extraction branches are fused to obtain a fourth fused feature;

[0142] The fourth fusion feature is processed by 1×1 convolution, 3×3 convolution, 1×1 convolution, and 3×3 convolution in sequence to obtain the network fusion graph f net ;

[0143] Specifically, the feature extraction operation parameter settings of each stage are shown in Table 1:

[0144] Table 1 Parameter settings for each stage

[0145]

[0146] Among them, Ks is the convolution kernel size, I C is the number of input channels, O C is the number of output channels.

[0147] It should be noted that the MSFF module merges information from adjacent stages. Features from shallower stages can enhance detail information, while features from deeper stages can enrich semantic information.

[0148] In a specific embodiment, the outputs of the feature extraction operations in the three stages are I1, I2, and I3, respectively, where I1∈R 2H×2W×(C / 2) , 2H is the height of the source image, 2W is the width of the source image, and C is the total number of channels I2∈R in the second stage feature extraction operation H×W×C , I3∈R (H / 2)×(W / 2)×2C ;

[0149] Please refer to Figure 5 , the implementation method of the second MSFF module includes:

[0150] Use 1×1 convolution on I1 and I3 so that their number of channels is the same as I2, which is C.

[0151] Through 3×3 convolution and downsampling with a step size of 2, I1 is transformed into I'1, and the size of I'1 is (2H / 2)×(2W / 2), that is, I'1∈R H×W×C ; Through a 3×3 convolution with a step size of 1, I2 is converted to I'2, and the size of I'2 remains unchanged; I3 is upsampled using bilinear interpolation to obtain I'3, I'3∈R H×W×C ; I'1, I'2 and I'3 are the same size, add them together to get I:

[0152] I=I'1+I'2+I'3 (7)

[0153] Perform global average pooling on I to obtain the global feature F, F∈R 1×1×C , the c(c∈{1,2,…,C})th channel F c for:

[0154]

[0155] Among them I c is the cth channel of I.

[0156] Use 1×1 convolution to reduce the dimension of F and get V, V∈R 1×1×D ; Then, batch normalize V; Finally, use the ReLU activation function to get Z, Z∈R 1×1×D , where D = 4;

[0157]

[0158] Where B is the batch normalization (BN) operation, and δ(·) is the ReLU activation function;

[0159] Three groups of 1×1 convolutions are used to increase the dimension of Z, and three C-dimensional attention vectors n1, n2, and n3 are obtained by splicing and reshaping, where n1, n2, and n3∈R 1×1×C ; Then, n1, n2, and n3 are normalized using the softmax activation function to obtain F1, F2, and F3, respectively, where F1, F2, and F3∈R 1×1×C ; F i The cth channel F i,c for:

[0160]

[0161] Among them, ni,c n i The cth channel of , i∈{1,2,3};

[0162] The multi-scale features I'1, I'2, I'3 are weighted summed with F1, F2, F3 to obtain the fusion feature IF2, IF2∈R H ×W×C , the cth channel IF of IF2 2,c for:

[0163] IF 2,c =F 1,c I' 1,c +F 2,c I' 2,c +F 3,c I' 3,c (11)

[0164] Among them, I' 1,c , I' 2,c , I' 3,c They are the cth channels of I'1, I'2, and I'3 respectively;

[0165] The input branches of the first MSFF module are I1 and I2. The inputs I1, I2 and I3 of the second MSFF module are replaced by 0, I1 and I2, which are the inputs of the first MSFF module. The output of the first MSFF module is IF1, IF1∈R 2H ×2W×(C / 2) .

[0166] It should be noted that the traditional squeeze-excite (SE) block ignores the position information, and the existing spatial attention methods such as the convolutional block attention module (CBAM) usually cannot obtain long-range dependencies. In order to enhance the multi-scale feature interaction, this paper introduces the coordinate attention upsampling CAU module, please refer to Figure 6 ,This module includes a coordinate attention (CA) block. CA embeds position information in the channel attention, which can effectively obtain spatial multi-scale features.

[0167] In a specific embodiment, please refer to Figure 6 , the two inputs of the coordinate attention upsampling CAU module are IF1 and IF2, the CAU module includes a coordinate attention block CA, and the implementation method of the CAU module includes:

[0168] IF2 is processed by 1×1 convolution, batch normalization, and ReLU activation function to obtain IF'2, which has the same number of channels as IF1.

[0169] Upsample IF'2 to obtain IF"2, which has the same size as IF1;

[0170] Concatenate IF'2 and IF2 to obtain X(X∈R 2H×2W×C );

[0171] Take X as the input of CA and generate the coordinate attention map T through CA h 、T w ;

[0172] The output of CA is T h 、T w Multiplying the three by IF1 gives CA out , C.A. out ∈R 2H×2W×(C / 2) ;

[0173] CA out Sum it with IF"2 to get the final output CAU of the module out , CAU out ∈R 2H×2W×(C / 2) .

[0174] In a specific embodiment, X is used as the input of CA, and the coordinate attention map T is generated by CA. h 、T w The methods include:

[0175] Perform average pooling on X in the vertical and horizontal directions respectively to obtain K h , K h ∈R 2H×1×C , h∈{1,2,…,2H}, K w , K w ∈R 1×2W×C , w∈{1,2,…,2W},K h , K w The Cth channel of The calculation formula is as follows:

[0176]

[0177]

[0178] X c is the Cth channel of X, X c Ability to capture orientation and position across channels;

[0179] For S w The 2H and 2W dimensions are exchanged and combined with S h Concatenate to get G, G∈R (2H+2W)×1×C ; Then, use 1×1 convolution to reduce the dimension of G and get U, U∈R (2H+2W)×1×D , where D = 8; batch normalization and ReLu6 activation function processing are performed on U to obtain J, J∈R(2H+2W)×1×D ;

[0180] Split J into two separate tensors J h , J w , where J h ∈R 2H×1×D , J w ∈R 2W×1×D , and J w 2H and 2W are exchanged back to their original dimensions; then, h and J w Perform 1×1 convolution respectively; finally, perform sigmoid activation function processing to obtain the coordinate attention map T h 、T w , where T h ∈R 2H×1×(C / 2) , T w ∈R 1×2W×(C / 2) , the specific formula is as follows:

[0181] T h =σ(E h J h ) (14)

[0182] T w =σ(E w J w ) (15)

[0183] Among them, σ is the sigmoid activation function, E h 、E w It plays the role of 1×1 convolution.

[0184] In a specific embodiment, the MFIS network uses a mean square error loss L MSE and the structural similarity loss L SSIM Combined to construct the loss function L, the specific formula is as follows:

[0185] L SSIM =1-SSIM

[0186]

[0187]

[0188] Among them, O and P represent the network output image and label image respectively, μ O and μ P represents the mean of the image, σ O and σ P represents the image variance, σ O,P is the covariance of the two images, C1, C2, and C3 are constants used to avoid the denominator being 0, where W and H are the width and height of the image, and (i, j) is the pixel position.

[0189] S5: According to the boundary area decision graph, weighted sum is performed on the initial fusion graph and the network fusion graph to obtain a final fusion graph.

[0190] Example 2

[0191] This embodiment can be regarded as an improved or extended embodiment based on Embodiment 1, specifically: a multi-focus image fusion method based on focus area detection and boundary processing, the method comprising:

[0192] The focus region decision map is obtained by improving the Laplace energy and performing focus measurement processing and small region removal processing on the source image pair.

[0193] Initially fusing the focus region decision map with the source image pair to obtain an initial fusion map;

[0194] The focus area decision diagram is subjected to a boundary extraction algorithm to obtain a boundary area decision diagram;

[0195] Inputting the source image pair into the MFIS network to generate a network fusion graph;

[0196] According to the boundary area decision graph, the initial fusion graph and the network fusion graph are weighted summed to obtain a final fusion graph.

[0197] Specifically, please refer to Figure 7 , the training method of the MFIS network includes:

[0198] The data set is screened, and then the source images A, source images B and label images T in the data set are preprocessed. Then the source images A and source images B in the data set are input into the MFIS network for feature extraction, feature fusion, and feature reconstruction in sequence, and the fused image is output. The loss function is calculated by the output fused image and the preprocessed label image T, and the model parameters are updated and optimized according to the calculation results. After several rounds of training, the trained MFIS network is output.

[0199] Specifically, due to the lack of large labeled datasets, this embodiment selected 2000 images from the VOC2007 public dataset and cropped them into labeled images of size 256×256 to form a training dataset. Complementary cross-Gaussian blur processing was performed on different areas of each image, with a blur radius of 3, and each labeled image was divided into 8×8 blocks (each block size was 32×32).

[0200] Test dataset: contains 33 pairs of images, 13 of which are widely used multi-focus images and the other 20 pairs are from the lytro dataset.

[0201] Specifically, during the network training process, this embodiment uses Adam as the optimization function, the learning rate is 0.0005, the batch size is 1, and the epoch is 100.

[0202] In a specific embodiment, in order to verify the effectiveness of the present invention, 8 representative methods were selected for comparison, namely, a method based on finding boundaries (BF), a method based on convolutional neural networks (CNN), a method based on classification models (ECNN), a method based on end-to-end networks (GACN), a method based on guided filtering (GFDF), a method based on regression models (IFCNN), a method based on unsupervised networks (SESF), and a method based on the combination of dense connections and Siamese structures (SF_MFIF).

[0203] Please refer to Figure 8 , 9 , the fusion results of different methods for the "golf" image are as follows Figure 8 As shown, Figure 8 The golf balls in (d) and (i) are completely blurred, and information is seriously lost; Figure 8 (c), (f), and (g) The boundary areas are fuzzy, texture details are lost, and the golf ball is not clear; Figure 8 (e), (h), (j), and (k) have clear boundaries and no artifacts. Fig. 9 It can be seen that: Fig. 9 (f) and (h) have information differences between the near-focus area and the source image, and the small flag has pixel diffusion phenomenon; Fig. 9 (d) The features of the focused area and the source image are obviously different, and the hammer of the golf club disappears; Fig. 9 (a)~(c), (e), (g), and (i) have no information residue and can better preserve the texture detail information of the focused area.

[0204] In summary, the algorithm of the present invention performs better in the image fusion task. It can not only retain the characteristics of the source image, but also be clear and artifact-free at the boundary.

[0205] In order to avoid the one-sidedness of using a single indicator for evaluation, the present invention uses the average value of multiple indicators to evaluate the algorithm performance. Here, we select the metrics based on image features (SD, SF and EI), the metrics based on information theory (MI, AG), the metrics based on structural similarity (Q Y ) and multi-category indicators such as human perception-based metrics (VIF).

[0206] Tables 2 and 3 show the quantitative evaluation of widely used images and lytro images, respectively. The bold values ​​in the tables represent the best indicators, and the underlined values ​​represent the suboptimal indicators.

[0207] It can be seen from Table 2 that compared with the suboptimal algorithm, the algorithm of the present invention improves the SD and SF indicators by 5.503% and 3.710% respectively, retaining more texture details; the VIF indicator is improved by 8.832%, and a better visual effect can be obtained. Y In terms of indicators, the algorithm of the present invention is 2.122% lower than the optimal algorithm (BF). Except for the medium performance in the structural similarity indicator, the algorithm of the present invention is superior to other algorithms in other indicators.

[0208] It can be seen from Table 3 that the algorithm of the present invention is optimal in multiple indicators such as image features, information theory and human perception, and performs moderately in structural similarity indicators.

[0209] Table 2 Average values ​​of multiple indicators (widely used images)

[0210]

[0211] Table 3 Average values ​​of multiple indicators (Lytro dataset)

[0212]

[0213] The above process is an overall evaluation of the fused image. Next, we will evaluate the focus area and the boundary area respectively. These two areas are evaluated by the boundary decision map DM. b In the focus area, the mutual information (MI), feature mutual information (FMI), edge information preservation (Q AB / F ), human visual perception (Q CB ) four indicators, the first two are used to evaluate the degree of time-frequency information preservation, and the last two are used to evaluate visual similarity.

[0214] In the boundary area, because the source images are not clear, the non-reference evaluation indicators are selected here: entropy (EN), average gradient (AG), edge intensity (EI) and point acuity algorithm (EAV).

[0215] Table 4 and Table 5 respectively compare the objective performance of lytro images in the focus area and boundary area. It can be seen from Table 4 that compared with the suboptimal algorithm, the proposed algorithm has good performance in MI, FMI, Q AB / F , Q CB The indicators are improved by 1.146%, 13.543%, 0.946% and 1.677% respectively. It can be seen from Table 5 that compared with the suboptimal algorithm, the algorithm of the present invention improves EN, AG, EI and EAV indicators by 37.676%, 5.112%, 4.062% and 7.749% respectively.

[0216] Table 4 Average values ​​of multiple indicators in the focus area (Lytro dataset)

[0217]

[0218] Table 5 Average values ​​of multiple indicators in the boundary area (Lytro dataset)

[0219]

[0220] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A multi-focus image fusion method based on focus area detection and boundary processing, characterized in that: The method comprises: The focus region decision map is obtained by improving the Laplace energy and performing focus measurement processing and small region removal processing on the source image pair. Initially fusing the focus region decision map with the source image pair to obtain an initial fusion map; The focus area decision diagram is subjected to a boundary extraction algorithm to obtain a boundary area decision diagram; Inputting the source image pair into a multi-scale feature interactive Siamese network to generate a network fusion graph; According to the boundary area decision graph, the initial fusion graph and the network fusion graph are weighted summed to obtain a final fusion graph.

2. The multi-focus image fusion method based on focus area detection and boundary processing according to claim 1, characterized in that: The method of obtaining a focus region decision map by improving Laplace energy and performing focus measurement processing and small region removal processing on a source image pair includes: The source image f is respectively processed by the improved Laplace operator A and f B Perform focus measurement and obtain the focus map S A and S B ; The calculation formula of the focus metric is as follows: M L (i,j)=|2f(i,j)-f(i-step,j)-f(i+step,j)+2f(i,j)-f(i,j-step)-f(i,j+step)|(1) In formula (1), M L is the improved Laplacian operator, f(i,j) is the pixel value of the source image, step is the variable spacing between pixels, where step = 1; In formula (2), SML is the focus metric at point (i, j), and (2M+1)×(2N+1) is the metric window size; Using formulas (3) and (4) to calculate the source image f A and the source image f B Perform focus measurement and obtain the focus map S A and S B , the formula is as follows: S A (i,j)=SML(f A (i,j)) (3) S B (i,j)=SML(f B (i,j)) (4) According to the focus map S A and S B f A , f B Fill the pixels and get the initial decision graph DM init , the formula is as follows: DM init (i,j)=S A (i,j)-S B (i,j) (5); The initial decision graph DM is filtered by a double threshold filtering method. init Then, the binary image is used to obtain the number of connected domains and calculate the area of ​​the connected domain. When the area of ​​the connected domain is less than the preset threshold, the focus attribute value of the connected domain is corrected to obtain the focus area decision map DM. nb .

3. The multi-focus image fusion method based on focus area detection and boundary processing according to claim 2, characterized in that: The method of initially fusing the focus region decision map and the source image pair to obtain an initial fusion map includes: According to the focus area decision map DM nb Determine the focus area of ​​the source image and obtain the initial fusion image f init , the formula is as follows: Where (i, j) is the pixel position.

4. The multi-focus image fusion method based on focus area detection and boundary processing according to claim 1, characterized in that: The method of obtaining a boundary area decision by using the focus area decision graph through a boundary extraction algorithm comprises: Focus Area Decision Diagram DM nb Perform Canny edge detection to obtain the boundary; then, use guided filtering to process the boundary to obtain the boundary area decision map DM b .

5. The multi-focus image fusion method based on focus area detection and boundary processing according to claim 1, characterized in that: Inputting the source image pair into a multi-scale feature interactive twin network to generate a network fusion graph includes: The source image f A and f B The images are input into two feature extraction branches of the multi-scale feature interaction twin network respectively, and feature extraction is performed through 3×3 convolution, 1×1 convolution, and 3×3 convolution respectively; The features of the two feature extraction branches are superimposed on the channel through dense connections to obtain fused features; The fusion features are subjected to image reconstruction processing to obtain a network fusion graph.

6. The multi-focus image fusion method based on focus area detection and boundary processing according to claim 5, characterized in that: The method of performing image reconstruction processing on the fusion features to obtain a network fusion graph includes: The fusion feature is reduced in dimension by 3×3 convolution, and then the convolution result is input into the BRB structure to perform three-stage feature extraction operations in sequence; The features obtained by the first stage feature extraction operation and the features obtained by the second stage feature extraction operation are subjected to multi-scale feature fusion through the first MSFF module to obtain the first fused feature; The features obtained from the feature extraction operations in the three stages are subjected to multi-scale feature fusion through the second MSFF module to obtain the second fused features; Inputting the first fused feature and the second fused feature into a coordinate attention upsampling CAU module for feature enhancement; The features output by the CAU module are processed by 1×1 convolution and 3×3 convolution in sequence, and then the convolution result and the features output by the first 3×3 convolution in the two feature extraction branches are fused to obtain the third fused feature; The third fused feature is processed by 1×1 convolution and 3×3 convolution in sequence, and then the convolution result and the feature output by the second 3×3 convolution in the two feature extraction branches are fused to obtain a fourth fused feature; The fourth fusion feature is processed by 1×1 convolution, 3×3 convolution, 1×1 convolution, and 3×3 convolution in sequence to obtain the network fusion graph f net .

7. The multi-focus image fusion method based on focus area detection and boundary processing according to claim 6, characterized in that: The outputs of the feature extraction operations in the three stages are I1, I2, and I3, respectively, where I1∈R 2H×2W×(C / 2) , 2H is the height of the source image, 2W is the width of the source image, and C is the total number of channels I2∈R in the second stage feature extraction operation H×W×C , I3∈R (H / 2)×(W / 2)×2C ; The implementation method of the second MSFF module includes: Use 1×1 convolution on I1 and I3 so that their number of channels is the same as I2, which is C. Through 3×3 convolution and downsampling with a step size of 2, I1 is transformed into I1', the size of I1' is (2H / 2)×(2W / 2), that is, I1'∈R H×W×C ; Through a 3×3 convolution with a step size of 1, I2 is converted to I'2, and the size of I'2 remains unchanged; I3 is upsampled using bilinear interpolation to obtain I3', I3'∈R H×W×C ; I1', I'2 and I3' are the same size, add them together to get I: I=I1'+I'2+I3' (7) Perform global average pooling on I to obtain the global feature F, F∈R 1×1×C , the c(c∈{1,2,…,C})th channel F c for: Among them I c is the cth channel of I; Use 1×1 convolution to reduce the dimension of F and get V, V∈R 1×1×D ; Then, batch normalize V; Finally, use the ReLU activation function to get Z, Z∈R 1×1×D , where D = 4; Where B is the batch normalization (BN) operation, and δ(·) is the ReLU activation function; Three groups of 1×1 convolutions are used to increase the dimension of Z, and three C-dimensional attention vectors n1, n2, and n3 are obtained by splicing and reshaping, where n1, n2, and n3∈R 1×1×C ; Then, n1, n2, and n3 are normalized using the softmax activation function to obtain F1, F2, and F3, respectively, where F1, F2, and F3∈R 1×1×C ; F i The cth channel F i,c for: Among them, n i,c n i The cth channel of , i∈{1,2,3}; The multi-scale features I1', I'2, I3' are weighted summed with F1, F2, F3 to obtain the fusion feature IF2, IF2∈R H×W×C , the cth channel IF of IF2 2,c for: IF 2,c =F 1,c I’ 1,c +F 2,c I' 2,c +F 3,c I’ 3,c (11) Among them, I' 1,c , I' 2,c , I' 3,c They are the cth channels of I'1, I'2, and I'3 respectively; The input branches of the first MSFF module are I1 and I2. The inputs I1, I2 and I3 of the second MSFF module are replaced by 0, I1 and I2, which are the inputs of the first MSFF module. The output of the first MSFF module is IF1, IF1∈R 2H×2W×(C / 2) .

8. The multi-focus image fusion method based on focus area detection and boundary processing according to claim 7, characterized in that: The two inputs of the coordinate attention upsampling CAU module are IF1 and IF2. The CAU module includes a coordinate attention block CA. The implementation method of the CAU module includes: IF2 is processed by 1×1 convolution, batch normalization, and ReLU activation function to obtain IF'2, which has the same number of channels as IF1. Upsample IF'2 to obtain IF"2, which has the same size as IF1; Concatenate IF'2 and IF2 to obtain X(X∈R 2H×2W×C ); Take X as the input of CA and generate the coordinate attention map T through CA h , T w ; The output of CA is T h , T w Multiplying the three by IF1 gives CA out , C.A. out ∈R 2H×2W×(C / 2) ; CA out Sum it with IF"2 to get the final output CAU of the module out , CAU out ∈R 2H×2W×(C / 2) .

9. The multi-focus image fusion method based on focus area detection and boundary processing according to claim 8, characterized in that: Take X as the input of CA and generate the coordinate attention map T through CA h , T w The methods include: Perform average pooling on X in the vertical and horizontal directions respectively to obtain K h , K h ∈R 2H×1×C , h∈{1,2,…,2H}, K w , K w ∈R 1×2W×C , w∈{1,2,…,2W},K h , K w The Cth channel of The calculation formula is as follows: X c is the Cth channel of X, X c Ability to capture orientation and position across channels; For S w The 2H and 2W dimensions are exchanged and combined with S h Concatenate to get G, G∈R (2H+2W)×1×C ; Then, use 1×1 convolution to reduce the dimension of G and get U, U∈R (2H+2W)×1×D , where D = 8; batch normalization and ReLu6 activation function processing are performed on U to obtain J, J∈R (2H+2W)×1×D ; Split J into two separate tensors J h , J w , where J h ∈R 2H×1×D , J w ∈R 2W×1×D , and J w 2H and 2W are exchanged back to their original dimensions; then, h and J w Perform 1×1 convolution respectively; finally, perform sigmoid activation function processing to obtain the coordinate attention map T h , T w , where T h ∈R 2H×1×(C / 2) , T w ∈R 1×2W×(C / 2) , the specific formula is as follows: T h =σ(E h J h )(14) T w =σ(E w J w )(15) Among them, σ is the sigmoid activation function, E h 、E w It plays the role of 1×1 convolution.

10. The multi-focus image fusion method based on focus area detection and boundary processing according to claim 1, characterized in that: The multi-scale feature interaction Siamese network uses the mean square error loss L MSE and the structural similarity loss L SSIM Combined to construct the loss function L, the specific formula is as follows: L=L SSIM +λL MSE L=L SSIM +λL MSE L SSIM =1-SSIM Among them, O and P represent the network output image and label image respectively, μ O and μ P represents the mean of the image, σ O and σ P represents the image variance, σ O,P is the covariance of the two images, C1, C2, and C3 are constants used to avoid the denominator being 0, where W and H are the width and height of the image, and (i, j) is the pixel position.

Images