Multi-focus image fusion method for generating GCN by using superpixel features and reconstructing CNN by using pixel-level features

Abstract

The invention relates to a multi-focus image fusion method for generating a GCN by using superpixel features and reconstructing a CNN by using pixel-level features, and the method comprises the steps: carrying out the flexible convolution of a large-scale irregular region formed by superpixel clustering in an input image through employing an image convolution neural network, and obtaining block-level coarse segmentation features; performing pixel-level feature extraction on the input image by using a parallel lightweight convolutional neural network; and carrying out fusion processing on the extracted pixel-level features and the block-level coarse segmentation features to obtain a full-focus image. According to the invention, the GCN is used for solving the multi-focus image fusion problem for the first time. An ingenious thought of combining block cutting and pixel optimization is provided in FusionGCN, so that a decision diagram is transited from coarse to fine, the inference difficulty of a network is reduced, and a fused image with higher quality is generated.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, in particular to a multi-focus image fusion method using super-pixel features to generate GCN and pixel-level features to reconstruct CNN Background Art

[0002] Due to the limited depth of field of optical lenses and image sensors, the image recorded by the sensor in the same imaging device at the same time is difficult to contain all the important information in the scene, resulting in objects within the depth of field being usually clear and objects outside the depth of field being blurred, which is not conducive to many industries that require high-precision image information. Multi-focus image fusion aims to use multiple images taken at different focal lengths in the same scene to synthesize a fully focused image with a wider depth, which provides a solid guarantee for the development of medical imaging, microscope image processing, industrial inspection and other fields.

[0003] Traditional multi-focus image fusion methods can be divided into two categories: fusion methods based on the spatial domain and fusion methods based on the change domain. Among them, the fusion method based on the spatial domain measures the activity level of the source image by some method, and directly processes the area selected by the algorithm in the spatial domain. Although the computational complexity based on the spatial domain is low and more in line with computer processing rules, how to accurately determine and divide the focus area is the main difficulty to be solved by this type of method. The method based on the change domain first maps the source image from the spatial domain to the change domain through the change domain algorithm, then fuses the information of the change domain according to the manually designed fusion rules, and finally reconstructs the information of the change domain and remaps it back to the spatial domain to obtain the fused image. Although the fusion method based on the change domain has more advantages in edge processing than the fusion method based on the spatial domain, this type of fusion method is often accompanied by problems such as contrast reduction and image distortion.

[0004] As a deep learning model specially designed for processing graph structure data, graph convolutional neural network was originally designed to solve problems in the fields of social networks and recommendation systems. Images are a special form of expression of graph data. At present, researchers have applied GCN to the field of medical image processing for cell structure processing, posture estimation and other tasks, and have achieved good results. Multi-focus image fusion technology is committed to processing image data about the same target collected from multiple source channels, extracting beneficial information from each channel, and forming high-quality images. Affected by the defocus diffusion effect, objects in the focal plane will appear larger on the imaging device than real objects. This is not friendly to CNN based on local information, which requires CNN to deepen the network to obtain a larger receptive field, resulting in increased computational overhead, and often requires a large number of parameters to fit the task. Although GCN can use the connection relationship between nodes in graph structure data to capture the semantic information between different regions in the image and bring cross-scale feature information to the network, the initial matrix construction and segmentation accuracy of GCN are still problems that need to be solved. Summary of the invention

[0005] The purpose of the present invention is to propose a multi-focus image fusion method that utilizes superpixel features to generate GCN and pixel-level features to reconstruct CNN, so as to solve the problems existing in the above-mentioned prior art.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] The multi-focus image fusion method using superpixel features to generate GCN and pixel-level features to reconstruct CNN includes:

[0008] Using graph convolutional neural networks, soft convolution is performed on large-scale irregular regions formed by superpixel clustering in the input image to obtain block-level coarse segmentation features.

[0009] Using a parallel lightweight convolutional neural network to extract pixel-level features from the input image;

[0010] The extracted pixel-level features and the block-level coarse segmentation features are fused to obtain a fully focused image.

[0011] Optionally, performing soft convolution on a large-scale irregular region formed by superpixel clustering in the input image includes:

[0012] Use the SLIC algorithm to perform preliminary clustering of pixels in the input image;

[0013] The Scharr operator is used to preliminarily process the image after preliminary clustering, and the total gradient value in the centroid of a single superpixel is used as the initial feature to construct the node feature matrix;

[0014] According to the spatial adjacency relationship of pixel nodes in the image after preliminary clustering, an edge matrix is ​​constructed;

[0015] A graph convolution operation is performed on the node feature matrix and the edge matrix to obtain block-level coarse segmentation features.

[0016] Optionally, performing preliminary clustering of pixels in the input image using the SLIC algorithm includes:

[0017] Based on the preset formula, the number of nodes in the input image is divided, and the superpixel area is divided;

[0018] Based on the number of nodes and superpixel areas, pixels in the input image are preliminarily clustered.

[0019] Optionally, the preset formula is:

[0020] Map=ImageProcess(Concat(A,B))

[0021]

[0022] Among them, Concat() represents the concatenation operation, A, B represent the source image A and the source image B, ImageProcess represents the convolution operation with a convolution kernel size of 7 to reduce the number of channels to 1, Split() represents the use of the Split function to split according to the batch size dimension, i represents the position of the image in the same batchsize, H, W represent the height and width of the source image A respectively, Scale represents the scaling of the number of initial superpixel divisions, SegmentsLabelProcess() represents the post-processing function, Batchsize represents the amount of data used each time during the training process, and Map represents a channel input feature map after ImageProcess processing. Represents the i-th feature map in the same batch, dim represents the matrix dimension for the required split operation, represents the initial segmentation map generated by the i-th SLIC partition in the same batch, N init Indicates the number of initial partition nodes, Max(H,W) indicates the maximum value of the length or width of the selected image, Segments i represents the i-th superpixel label generated after post-processing, N i Represents the number of superpixels after post-processing.

[0023] Optionally, the method for constructing the node feature matrix is:

[0024] img i =Split(img,dim=1),img∈(A,B)

[0025] Processed i =Conv 3×3 (img i )

[0026]

[0027] where j=1,2,3,...,N and i=1,2,...,Batchsize

[0028] Among them, Conv 3×3 It means that the source images A and B are reduced to 1 dimension using convolution with a convolution kernel size of 3. Flatten means that the gradient map obtained by the Scharr operator is linearly flattened. Represents the convolution operation, Grad_img i represents the gradient values ​​of all pixels under the jth superpixel, Sum() represents the summation of all pixel gradient values ​​under the same superpixel, img i represents the i-th image of the same input batch, Split() represents the use of the Split function to split according to the batch size dimension, ScharrX represents the use of the Scharr operator to process in the x-axis direction, and Processed represents the image after 3×3 convolution. i , ScharrY means using the Scharr operator to process in the y-axis direction, X[j,_] means the value of the feature matrix at the jth superpixel, It represents the gradient value of the jth superpixel node in the i-th gradient map, N and i represents the number of superpixels and their positions in the same batch, and Batchsize represents the batch size.

[0029] Optionally, the edge matrix is ​​constructed as:

[0030]

[0031] Among them, M i,j Indicates that there is an edge from node i to node j, N i and N j They all represent superpixel nodes.

[0032] Optionally, performing soft convolution on a large-scale irregular region formed by superpixel clustering in the input image further includes:

[0033] The block-level coarse segmentation features are converted from a superpixel scale to a pixel scale using a type conversion matrix;

[0034] The type conversion matrix is:

[0035]

[0036] Among them, T[i,j] represents the type conversion matrix, and i,j represent i pixel and j pixel.

[0037] Optionally, fusing the extracted pixel-level features with the block-level coarse segmentation features to obtain a fully focused image includes:

[0038] Fusing the extracted pixel-level features with the block-level coarse segmentation features to obtain fused features;

[0039] Classify the fused features to generate an initial decision graph;

[0040] Post-processing the initial decision graph to obtain a final decision graph;

[0041] According to the final decision graph, the multi-focus source images are fused into a fully focused image.

[0042] Optionally, the post-processing method is:

[0043]

[0044]

[0045]

[0046] in, and represents the final decision graph after post-processing steps, Small_object_removal() represents small area filtering, I A and I A Represents the multi-focus source image pairs A and B, D that are input to the network pre (x,y) represents the initial decision graph output by the network, and F(x,y) represents the fused image.

[0047] The beneficial effects of the present invention are:

[0048] The present invention uses GCN to solve the multi-focus image fusion problem for the first time. In FusionGCN, a clever idea of ​​combining block cutting with pixel optimization is proposed, which makes the decision graph transition from coarse to fine, reduces the difficulty of network reasoning, and generates higher quality fused images. The present invention designs a superpixel-based graph decoder and a pixel-based CNN basic extraction block, so that GCN and CNN can cooperate with each other in the same network, alleviating the incompatibility between non-Euclidean space data and Euclidean space data. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0050] Figure 1 A diagram comparing the differences between the network structure and the decision graph generation strategy according to an embodiment of the present invention;

[0051] Figure 2 This is a diagram of the overall network structure of FusionGCN according to an embodiment of the present invention;

[0052] Figure 3 It is a graph structure constructed after GCN Encoder in an embodiment of the present invention;

[0053] Figure 4 A detailed design diagram of the GCN structure module in FusionGCN according to an embodiment of the present invention;

[0054] Figure 5 This is an example of a matrix generated after an image composed of four pixels passes through a GCN Encoder and a GCN Decoder in an embodiment of the present invention;

[0055] Figure 6 A detailed design diagram of the CNN structure module in FusionGCN according to an embodiment of the present invention;

[0056] Figure 7 This is an example of a data set constructed by VOC2012 according to an embodiment of the present invention;

[0057] Figure 8 It is a display of some test images from three public data sets of the embodiments of the present invention;

[0058] Fig. 9 A visual display of the comparative experiment of FusionGCN of the embodiment of the present invention and other 10 SOTA methods on Lytro-13;

[0059] Fig.10 It is a visual display of the comparative experiment of FusionGCN of the embodiment of the present invention and other 10 SOTA methods on MFI-HWU-29;

[0060] Fig.11 This is a visualization of the comparative experiment of FusionGCN of the embodiment of the present invention and other 10 SOTA methods on Grayscale-17;

[0061] Fig.12It is a visual display of the comparative experiment of FusionGCN of the embodiment of the present invention and other 10 SOTA methods on MFFW-O7;

[0062] Fig.13 A schematic diagram of the fusion process and result diagram of multiple source images according to an embodiment of the present invention;

[0063] Fig.14 The following is a summary of objective evaluation indicators of four ablation networks according to embodiments of the present invention on the Lytro dataset;

[0064] Fig.15 A comparative display of decision graphs generated by four ablation networks according to an embodiment of the present invention on the Lytro dataset;

[0065] Fig.16 The objective evaluation index of the network with different scale values ​​according to the embodiment of the present invention is displayed on Lytro;

[0066] Fig.17 The results of processing different images with different division scales are shown in the embodiment of the present invention;

[0067] Fig.18 The figure is a comparison of the subjective visual effects of different small area filter thresholds in the MFI-WHU dataset of the embodiment of the present invention;

[0068] Fig.19 This is the difference in strategy between the discriminant network and FusionGCN in focusing on the activity measurement of pixels according to an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0071] As a deep learning model specifically used to process graph structure data, graph convolutional neural network was originally designed to solve problems in the fields of social networks and recommendation systems. Image is a special form of expression of graph data. At present, researchers have applied GCN to the field of medical image processing for cell structure processing, posture estimation and other work, and have achieved good results. Multi-focus image fusion technology is committed to processing image data about the same target collected from multiple source channels, extracting favorable information from each channel, and forming high-quality images. Affected by the defocus diffusion effect, objects in the focal plane will be larger than real objects on the imaging device. This is not friendly to CNN based on local information, which requires CNN to deepen the network to obtain a larger receptive field, resulting in increased computational overhead, and often requires a large amount of parameters to fit the task. Although GCN can use the connection relationship between nodes in graph structure data to capture semantic information between different regions in the image and bring cross-scale feature information to the network, the initial matrix construction and segmentation accuracy of GCN are still problems that need to be solved. To overcome the above problems, this embodiment proposes a lightweight multi-focus image fusion method combining block cutting GCN and pixel optimization CNN. Figure 1 The difference between the decision graphs generated by FusionGCN at the superpixel level and the pixel level is shown, where Figure 1 (a) indicates that only GCN is used for block-level segmentation. Figure 1 (b) indicates that only CNN is used to perform pixel-level segmentation on the source image. Figure 1 (c) is the method proposed in this embodiment.

[0072] The multi-focus image fusion method proposed in this embodiment, which generates GCN using superpixel features and reconstructs CNN using pixel-level features, includes:

[0073] Using graph convolutional neural networks, soft convolution is performed on large-scale irregular regions formed by superpixel clustering in the input image to obtain block-level coarse segmentation features.

[0074] Using a parallel lightweight convolutional neural network to extract pixel-level features from the input image;

[0075] The extracted pixel-level features and the block-level coarse segmentation features are fused to obtain a fully focused image.

[0076] Specifically, in this embodiment, a graph convolutional neural network is first used to perform flexible convolution on the large-scale irregular area formed by superpixel clustering to obtain block-level coarse segmentation. Then a parallel lightweight convolutional neural network is used for pixel-level guidance. Finally, a more accurate decision graph is obtained. In addition, in order to achieve the complementary advantages of the two networks, so that the Euclidean data-oriented convolutional neural network can optimize the graph convolutional neural network of non-Euclidean oriented data for feature generation, a superpixel-based graph decoder and a pixel-based CNN basic extraction block are designed to realize feature acquisition and propagation. Compared with many state-of-the-art methods, this embodiment has good competitiveness in qualitative and quantitative analysis and efficiency comparison.

[0077] The method of this embodiment is implemented using the FusionGCN model. The overall network structure is as follows: Figure 2 As shown. According to the scale difference of the intermediate feature map, it can be divided into two parts: GCN based on block coarse segmentation and CNN based on pixel optimization. In this network, GCN uses the superpixel matrix to calculate large irregular areas, providing the network with large receptive field information and rich feature information, but due to inaccurate division, the edges of the generated fused image have very serious jagged segmentation traces. CNN extracts features from images at the pixel level, but often requires a large number of parameters and complex feature fusion methods to achieve perfect results. For this reason, GCN is effectively combined with CNN to generate higher quality fused images. Fig.19 The differences in strategies between the existing discriminant-based multi-focus image fusion network and FusionGCN in measuring the activity of focused pixels are demonstrated.

[0078] Furthermore, performing soft convolution on the large-scale irregular region formed by superpixel clustering in the input image includes:

[0079] Use the SLIC algorithm to perform preliminary clustering of pixels in the input image;

[0080] The Scharr operator is used to preliminarily process the image after preliminary clustering, and the total gradient value in the centroid of a single superpixel is used as the initial feature to construct the node feature matrix;

[0081] According to the spatial adjacency relationship of pixel nodes in the image after preliminary clustering, an edge matrix is ​​constructed;

[0082] A graph convolution operation is performed on the node feature matrix and the edge matrix to obtain block-level coarse segmentation features.

[0083] Among them, the margin matrix and feature nodes will be used as the input of the GCN network later. Feature extraction will be automatically performed in GCN. Because GCN is based on superpixels, it can be simplified as a clustering method. Several or dozens of pixels with the same similarity will be aggregated into a superpixel, and this superpixel will be used as the feature of these pixels. Therefore, the decision graph generated after simple GCN processing is block-level.

[0084] Furthermore, the SLIC algorithm is used to perform preliminary clustering of pixels in the input image, including:

[0085] Based on the preset formula, the number of nodes in the input image is divided, and the superpixel area is divided;

[0086] Based on the number of nodes and superpixel areas, pixels in the input image are preliminarily clustered.

[0087] Specifically, in this embodiment, the above content is implemented by the GCN part based on block coarse segmentation in the FusionGCN model. The GCN part is described in detail below:

[0088] GCN Encoder

[0089] GCN is a convolutional neural network model for graph data. Its input is usually composed of a node feature matrix (NodeFeature Matrix) and an edge matrix (Edge Matrix). The node feature matrix is ​​a two-dimensional matrix, and each row represents the feature vector of a node. The edge matrix describes the connection relationship between the nodes in the graph. In this embodiment, the edge matrix is ​​also a two-dimensional matrix, but each row represents the starting point and end point of the edge.

[0090] Construction of node feature matrix: For node feature matrix Where N represents the number of nodes, and D represents the number of features of each node. Inspired by previous work, the image can be regarded as a special graph structure, but if each pixel is regarded as an independent node input into the network training, the initial node feature matrix constructed will be very large and almost unusable for network learning. Therefore, this embodiment uses SLIC to perform preliminary clustering of each pixel of the image according to the pixel mean to reduce the computational complexity of graph construction. SLIC refers to a method for clustering by pixel mean. An algorithm that aggregates multiple pixels into a superpixel by similar color features. Among them, SLIC is an algorithm evolved from K-means. It only needs to provide an expected number of cluster center points. The algorithm can quickly cluster adjacent pixels with similar color features and spatial information, and generate a small number of superpixels instead of a large number of pixels to express image information, which greatly reduces the complexity of image post-processing and makes GCN possible in processing large-scale images. However, in the multi-focus image fusion task, due to the physical property limitations of the imaging device, the multi-focus image based on real shooting cannot avoid the influence of defocus diffusion, so that a noise band larger than the original boundary will be formed around the focused object. However, the SLIC algorithm converts the color image into a five-dimensional feature vector in the CIELAB color space and XY coordinates. The image color and contour feature information after noise interference often does not produce a large amount of loss. The SLIC algorithm can still accurately aggregate similar areas and maintain a high robustness in the processing of images interfered by noise. Unlike other tasks, during training, the network processes a batch of source images A and B with complementary focus information each time. Their color information at the same pixel position is similar but not exactly the same. Therefore, when constructing the SILC initial graph node, if the source images A and B are processed separately, the size of the generated divided area and the number of superpixel nodes N are inconsistent, resulting in a great waste of operating efficiency and space. For this reason, under the trade-off between accuracy and time complexity, this embodiment first uses ImageProcess() to reduce the dimension of the result after splicing A and B on the channel to 1 channel, and uses the hyperparameter N init =Max(H, W)×Scale fixes the number of initial division nodes, where H and W represent the height and width of the input image, and Scale is set to 1.5. Since the multi-focus image pairs taken in reality are often of different sizes, if a completely fixed-size hyperparameter is used to guide the formation of superpixel centroids, it will easily lead to inconsistent effects on different image processing at the same scale. In addition, this embodiment carefully designs the following ablation experiments on different GCN initial node numbers and the ablation experiments on the effects of the same Scale division scale on different image processing to explore N init The rationality of the design. Then, the number of division nodes N and the corresponding superpixel division area are obtained according to the following formula.

[0091] Map=ImageProcess(Concat(A,B)) (1)

[0092]

[0093] Among them, Concat() represents the concatenation operation, which concatenates the source images A and B into one image according to the channel dimension. ImageProcess represents the convolution operation with a convolution kernel size of 7 to reduce the number of channels to 1. Split() represents the use of the Split function to split according to the batch size dimension, so that the feature data processed by the GCN Encoder is for a single image. i represents the position of the image in the same batchsize, and H and W represent the height and width of the source image A respectively. Scale represents the scale of the initial superpixel division number. If the value is too large, the conversion matrix in the GCN Decoder will be abnormally large, and the superpixel features will degenerate into pixels, increasing the computational consumption. If the value is too small, the segmentation will be inaccurate and serious block effects will occur. It is set to 1.5 here. SegmentsLabelProcess() represents the post-processing function to avoid discontinuous integer values ​​in the initial image segmentation labels. Batchsize represents the amount of data used each time during the training process. The default setting here is 4. After the above steps, the continuous superpixel segmentation labels Segments and the number of superpixel nodes N are obtained. In addition, for the selection of eigenvalues, compared with other visual downstream tasks, this embodiment also makes corresponding designs for image fusion tasks. In the fields of object recognition, image segmentation, etc., some existing technologies use the color mean of all pixels in the superpixel centroid as the initial feature to be passed into GCN, while other existing technologies use the results of deep pre-trained networks such as ResNet and VGG16 as the initial feature to be passed into GCN. In the field of multi-focus image fusion, affected by the defocus diffusion effect, the focus of the imaging is not completely concentrated on one point, but shows a certain degree of diffusion and blur, which makes the objects that need to be distinguished in the real imaging environment irregular. If the initial features of the node are constructed according to the above method, the network cannot be effectively learned. This embodiment finds that only when the pixel is in focus, there will be a gradient around the pixel, and the gradient around the defocused pixel is very small or almost non-existent. Therefore, in this embodiment, the Scharr operator is used to preliminarily process the source image, and the total gradient value in a single superpixel centroid is used as the initial feature. Node feature matrix Follow the steps below to construct.

[0094] img i =Split(img,dim=1),img∈(A,B) (5)

[0095] Processed i =Conv 3×3 (img i ) (6)

[0096]

[0097]

[0098]

[0099] where j=1,2,3,...,N and i=1,2,...,B (10)

[0100] Among them, Conv 3×3 It means that the source images A and B are reduced to 1 dimension using convolution with a convolution kernel size of 3. Flatten means that the gradient map obtained by the Scharr operator is linearly flattened. Represents the convolution operation. Grad_img i Represents the gradient values ​​of all pixels under the jth superpixel. Sum() represents the summation of all pixel gradient values ​​under the same superpixel. The larger the value, the greater the probability that the target superpixel block is clear. j represents. Algorithm 1 describes the construction process of the node feature matrix.

[0101]

[0102] Construction of edge matrix: For edge matrix Where E represents the number of edges in the graph. It is worth noting that in an image, each pixel only has a spatial coordinate relationship with another pixel, and the information carried by each pixel is equally important.

[0103] Therefore, the edge matrix constructed in this embodiment is It is a binary matrix that is either 0 or 1.

[0104]

[0105] Among them, M i,j Indicates that there is an edge from node i to node j. If superpixel node N i and N j Set to 1 if adjacent in source image space, 0 otherwise.

[0106] The edge matrix is ​​calculated and constructed as described in Algorithm 2.

[0107]

[0108] Figure 3The graph structure constructed after the GCN Encoder is shown, in which each separate yellow area represents that the pixels in the area are clustered into a superpixel node because of similar color features. If two superpixel nodes are adjacent in space, they are connected by an edge. Even though the method of this embodiment shows dense edge connections in the figure, any superpixel node is also aggregated by hundreds of pixels. In addition, it should be noted that in order to make GCN consider its own characteristics during training, this embodiment also adds self-loop connections for each superpixel node.

[0109] GCN network structure

[0110] CNN can make up for the defect of insufficient feature expression by deepening the network depth. However, in GCN, as the number of layers increases, the features of the nodes will gradually become similar, and will tend to be evenly distributed after a certain layer, which often leads to information loss and reduces the differences between different nodes. Therefore, the GCN network of this embodiment is composed of only two basic graph convolution blocks and one graph attention block, and the Mish function is used as the activation function between blocks to increase the nonlinear expression ability of the network. The GCN structure module design is as follows: Figure 4 shown.

[0111] GCN Decoder

[0112] In the GCN Encoder, in order to make it possible to run GCN on large image data, this embodiment uses SLIC to aggregate all pixels of the input source image into many superpixels that are connected in space and similar in focus properties, using the difference in gradient values ​​as the evaluation scale. The adjacency relationship between superpixels is constructed, revealing the spatial structure of multi-focus image fusion pairs. In GCN, each superpixel node interacts with other nodes to learn whether all pixels under the superpixel are focused. However, due to the variability of images, the focused and defocused pixels are not arranged in a regular shape. Therefore, since different superpixels contain different numbers of pixels, it is difficult to perform direct calculations through the network. To this end, this embodiment uses a type conversion matrix Associate the correspondence between superpixels and pixels.

[0113] like Figure 2 As shown in Figure 1, GCN Decoder serves as a bridge between GCN superpixel-based non-Euclidean oriented data and CNN pixel-based Euclidean oriented data. It can be mainly divided into type conversion matrix and the network interaction fusion module. Specifically, the type conversion matrix is ​​used to decode the features from superpixels to pixels and is generated according to the following formula.

[0114]

[0115] The formula indicates that if pixel i belongs to superpixel node j, the value at T[i,j] is set to 1, otherwise it is 0. After constructing the above type conversion matrix T, this embodiment can propagate superpixel node features to pixels.

[0116]

[0117] Among them, × represents matrix multiplication. Reshape() means converting the matrix to the corresponding dimension. Since the number of superpixel nodes generated by each image is not exactly the same, the subsequent node feature matrix, edge matrix and transformation matrix are all different, so GCN cannot process a batch of images like CNN. Previous work fused a batch of data into a single image as the input of GCN, but in this embodiment, each batch is regarded as consisting of multiple images with a batchsize of 1, and Concat() means concatenating the feature matrix obtained by GCN according to the batch dimension.

[0118] For GCN Decoder, its detailed module design is as follows Figure 6 As shown. This embodiment first performs feature dimensionality upgrade on the input feature map. Then, some channels are extracted from it and spliced ​​with GCN so that the network can use global information to correct features. In addition, as mentioned above, the intermediate feature map after the GCN Decoder is based on the superpixel scale. It is unreasonable to simply splice it with the intermediate feature map based on the pixel scale generated by CNN. For this reason, the SE attention mechanism is introduced to break the interdependence between channels and adaptively recalibrate the channel feature response. Finally, information fusion is achieved through a convolution layer with a convolution kernel size of 1. From Figure 6 It can be seen that there are two feature interactions between GCN and CNN during network training, and the processing methods selected in this embodiment are not the same. During the first feature interaction, this embodiment chooses to use pixel-by-pixel feature addition. This is because the nodes based on superpixel GCN often aggregate information of multiple pixels, and directly adding features to them is equivalent to introducing a larger receptive field for CNN, playing a role similar to large kernel convolution. However, when the network information interacts for the second time, this embodiment chooses to use a network interaction fusion module instead of simple pixel-by-pixel feature addition. This is because as the network modules through which the features pass continue to deepen, the information extracted by CNN becomes more and more refined. Direct addition will cause the inaccurate information based on superpixels of GCN to interfere with the generation of the decision graph. Figure 5 An example of the matrix generated by a 4-pixel image after passing through the GCN Encoder and GCN Decoder is shown.

[0119] Further, fusing the extracted pixel-level features with the block-level coarse segmentation features to obtain a fully focused image includes:

[0120] Fusing the extracted pixel-level features with the block-level coarse segmentation features to obtain fused features;

[0121] Classify the fused features to generate an initial decision graph;

[0122] Post-processing the initial decision graph to obtain a final decision graph;

[0123] According to the final decision graph, the multi-focus source images are fused into a fully focused image.

[0124] Specifically, in this embodiment, the CNN part is used to fuse the extracted pixel-level features and the block-level coarse segmentation features to obtain a fully focused image; specifically, as follows:

[0125] CNN part

[0126] according to Figure 2 In FusionGCN, the GCN part is used as the main feature extraction network, which is responsible for the rough segmentation of a large range of focused and defocused areas. However, using GCN alone will cause uncontrollable block segmentation in the decision graph, which will affect the quality of the fused image. In this embodiment, a clever idea of ​​combining block cutting with pixel optimization is proposed, which provides a new solution for image fusion tasks to run in an environment with limited computing resources. Inspired by previous work, the model scales of deep neural networks such as depth, width and resolution are very important for prediction accuracy. Therefore, the CNN body of this embodiment is composed of a basic processing block basicconv, which is composed of an initial convolution layer with an unfixed convolution kernel size to increase the number of channels by 2 times. In addition, in order to obtain feature information under a larger receptive field, this embodiment uses a grouped convolution with a convolution kernel size of 5 and a dilation rate of 2 to process the initial information, and increases the number of channels to 4 times the number of initial feature channels, in order to seek feature interpretation in a higher dimension, to help the network perform feature selection, while retaining the advantages of the lightweight network of this embodiment. To this end, this embodiment introduces a parameter-free attention SimAM in the Basic-Conv layer to enhance the performance of CNN by calculating the local self-similarity of the feature map. Then, to reduce the number of parameters, a group convolution with a group size equal to the initial number of channels is used to reduce the channel dimension of the feature map, and then another convolution is used to obtain the output. The detailed structure of CNN is as follows: Figure 6As shown in Figure 1, the source image pair first passes through a multi-scale feature extraction block to increase the richness of network features. Then, two intermediate convolutional layers with a convolution kernel size of 3 further process the fusion information of the shallow CNN and GCN. Then, the feature maps generated by the two networks are interactively processed by the GCN Decoder. Finally, after passing through the classification layer, the initial decision map is output. The detailed parameter structure of each layer in the FusionGCN network is shown in Table 1.

[0127] Table 1: Summary of detailed parameters of each network module

[0128]

[0129]

[0130] Loss Function

[0131] The loss function plays a vital role in network learning. By minimizing the difference between the model prediction result and the target true value, the model continuously adjusts the parameters in iteration. In this embodiment, the mean absolute error is used as the loss function, and the loss function Loss expression is as follows:

[0132]

[0133] Among them, I f and I gt They represent the predicted image and the true label image generated by the network, respectively, and H and W represent the height and width of the label.

[0134] Fusion Strategy

[0135] FusionGCN is a discriminant-based fusion model. According to previous work, the network initially predicts graph D pre is often unreliable. Therefore, this embodiment first performs the following steps to initialize the decision graph D pre Post-processing is performed to correct the misjudgment information caused by the network's limited expression ability.

[0136]

[0137] in, and It represents the decision graph obtained after the post-processing step. Small_object_removal() represents small area filtering, and the threshold size is set to threshold=0.001. A and I A Representing the multi-focus source image pair A and B input to the network.

[0138] The experimental settings of this embodiment are described in detail below:

[0139] Training dataset:

[0140] Supervised neural networks often require a large number of high-quality training data sets to obtain good results, but there is no large public training data set in the field of multi-focus image fusion. To this end, this embodiment constructs a training data set through Pascal Visual Object Classes 2012 (VOC2012). The production of the data set can be carried out according to the following steps. First, this embodiment performs binary segmentation on the segmentation mask to obtain the initial decision graph label. Then, the clear image is filtered by Gaussian filtering with different filter kernel sizes. Figure 7 The foreground and background of (a) are Gaussian blurred to obtain the initial source image pair. Finally, the initial label and the initial source image pair are cropped into images of the same size for network training. After completing the above operations, 8740 pairs of multi-focus image pairs with a size of 256×256 are finally obtained, and the training set and validation set are divided into a ratio of 9:1. Figure 7 The example of the dataset constructed by VOC2012 in this embodiment is shown. Figure 7 (a) is the original image of VOC2012. Figure 7 (b) is the VOC2012 segmentation mask, Figure 7 (c) is the generated decision graph label, Figure 7 (d) is the generated source image A, Figure 7 (e) is the generated source image B.

[0141] Training implementation details:

[0142] FusionGCN is implemented in Python and runs on the platforms of Intel(R) Core(TM) i9-10900X CPU@3.70GHz and NVIDIA GeForce RTX 3090. In order to prevent overfitting of the model and increase its robustness, this embodiment uses data enhancement techniques during training, including random cropping, random rotation and other means. In addition, this embodiment uses the Adam optimizer for optimization, and the initial learning rate of the network is set to 2×10-4, and the learning rate of each epoch decays exponentially to 2×10-4×0.88epoch-1.

[0143] Test Dataset:

[0144] To further test the network performance and its adaptability to data in real scenarios, in this section we will use three public test datasets to conduct a comprehensive evaluation of FusionGCN. Figure 8 Some test image pairs from the above three public datasets are shown. They include:

[0145] Lytro: Photographed by a light field camera, consisting of 20 pairs of Double Series and 4 pairs of Triple Series color images, with a natural transition between the focused and defocused areas.

[0146] MFI-WHU: It is made by adding masks to the COCO dataset. The focused and defocused areas have obvious boundaries and the transition is unnatural.

[0147] GrayScale: Consists of 10 pairs of Double Series grayscale images, with a natural transition between the focused and defocused areas.

[0148] Multi-focus image fusion method involved in comparison:

[0149] In order to comprehensively evaluate the effectiveness of the proposed algorithm, this embodiment selects the most advanced algorithms in each category as comparison. These include: GFF[ and MWGF based on traditional methods. Deep learning methods based on generative models: U2Fusion, SwinFusion, ZMFF and FusionDiff. Deep learning methods based on discriminative models: SESF, MFIF-GAN and DB-MFIF. In addition, this embodiment also selects a relatively novel deep learning method MFFT that uses generative and discriminative models. The network summary involved in the comparative experiment is shown in Table 2.

[0150] Table 2. Classification, implementation methods and backbone networks of 10 reference SOTA methods

[0151]

[0152] Objective evaluation indicators:

[0153] This embodiment will quantify and measure the quality of the fused image through 8 objective evaluation indicators, including: QNMI based on information theory is used to measure information loss. QSSIM and QCB based on visual perception are used to measure the quality index of structure and color respectively. QSF based on frequency domain is used to evaluate spectral fidelity. QG based on gradient is used to evaluate the gradient similarity between the fused image and the source image. Other QAB / F, QNCIE and QVIFF based on composite are used to comprehensively evaluate the effect of the fusion algorithm.

[0154] The following is an introduction to the comparative experiment performed in this embodiment:

[0155] Subjective evaluation experiment

[0156] This example will use the three public test sets mentioned above to conduct subjective evaluation experiments. It is worth noting that misjudgments in small areas such as the focus and defocus boundaries and potential image distortion are difficult to detect with the naked eye, and the improvements of each SOTA method are relatively small. Therefore, for ease of observation, this example uses the source image B as the reference image and subtracts it pixel by pixel from the fused image, and the results are displayed as grayscale images after each method.

[0157] In the Lytro dataset, this example selects Lytro-13 for subjective visual comparison. The experimental results are shown in the figure below. Fig. 9 As shown in the figure, it can be seen that for the transition area between the foreground and the animal background of the handheld photo, the method based on the generative model performs well, but the animal background has uncontrollable color distortion, among which U2Fuison distortion is the most serious. Although the method based on the discriminant model avoids the problem of color distortion in the fusion strategy, the edge transition is very rigid, among which MFIF-GAN has the most misjudgments on the edge. As can be seen from the magnified box, the FusionGCN proposed in this embodiment has almost no misjudgment of the clear foreground and background areas, and has certain advantages in processing edge details even compared with the novel MFFT.

[0158] In the MFI-WHU dataset, this example selects MFI-29 for subjective visual effect comparison, and the visualization effect can be seen. Fig.10 . It can be seen that for the MFI-WHU dataset that suffers little or almost no impact from the defocus diffusion effect, both the discriminant-based and generative model-based methods have unnatural segmentation boundaries in the segmentation areas of focused and defocused pixels. However, the generative model-based method still has uncontrollable color distortion, among which FusionDiff is the most serious, and almost the entire image has a serious contrast drop problem. For the traditional methods GFF and MWGF, although there is no misjudgment in large blocks of background pixels, the segmentation boundaries are not fine enough. Compared with all methods, FusionGCN retains all the information of the source image and maintains good edge division, achieving the highest quality fused image.

[0159] In addition, to demonstrate that FusionGCN is also highly robust when processing grayscale images, this embodiment selects the public dataset Grayscale as a verification dataset to demonstrate the superiority of the method proposed in this embodiment from many aspects. Fig.11The visual effect comparison of this embodiment and other 10 SOTA methods on Grayscale-17 is shown. It can be concluded that compared with the relatively new discriminant method MFFT, FusionGCN is almost consistent in the edge division of stone and seashore diffusion, but the number of parameters and video memory used by FusionGCN are much smaller than MFFT, which is conducive to completing multi-focus image fusion tasks in some environments with limited computing resources. Compared with the generative model, U2Fusion and SwinFusion have serious distortion in the stone close-up, and almost no clear pixels are generated, and the algorithm robustness is not high.

[0160] Objective evaluation experiment:

[0161] In order to reduce the influence of personal subjective opinions on the evaluation results, this embodiment will use the 8 indicators mentioned above to conduct objective evaluation experiments. Tables 3 to 5 show the objective evaluation index scores of 10 SOTA methods and the method proposed in this embodiment on the Lytro, MFI-WHU and Grayscale datasets. It is worth noting that the 8 indicators selected in this embodiment are all positive indicators, and the larger the value, the better. For the convenience of observation and to distinguish the top three scores of each type, this embodiment marks and emphasizes the data, and the first place of each score is represented by red, bold and underlined. The second place is marked with blue and underlined. The third place is highlighted in green. It can be seen from the three tables that the indicators QNMI and QAB / F based on information theory have achieved the first place in the three datasets, which means that FusionGCN retains more complete information on the source image and has the best processing level of the edge transition part. For the indicators QNICE and QSSIM based on image correlation, the method proposed in this embodiment ranks among the top ten methods, which means that if directly perceived by the human eye, the method of this embodiment has a better visual perception. In addition, although FusionGCN did not achieve the top three results in some indicators, the gap with the SOTA method was only three decimal places, which is very small, and it completed the multi-focus image fusion task well.

[0162] Table 3. Average scores of objective indicators of the tested methods on the Lytro dataset, where the top three of each indicator are highlighted in red, blue, and green respectively

[0163]

[0164] Table 4. Average scores of objective indicators of the tested methods on the MFI-WHU dataset, where the top three of each indicator are highlighted in red, blue, and green respectively

[0165]

[0166] Table 5. Average scores of objective indicators of the tested methods on the Grayscale dataset, where the top three of each indicator are highlighted in red, blue, and green respectively

[0167]

[0168] More experimental results and discussions

[0169] Image fusion experiment under strong defocus diffusion effect

[0170] In the actual imaging process, when the light beam passes through the focusing optical device, due to the diffraction effect, the light beam will have a larger diffusion effect than expected near the focus, so that the photographed object is often larger than the actual object state. However, the phenomenon of misalignment caused by the effect is not taken into account in the production process of many data sets. Therefore, in order to comprehensively evaluate the robustness of the proposed algorithm, this embodiment selects another public data set MFFW in the field of multi-focus image fusion for experiments. The MFFW data set consists of 19 color images of different sizes, and each pair of images suffers from severe defocus diffusion effects. In this data set, this embodiment selects the very difficult MFFW-07 for subjective visual comparison. From the visual comparison Fig.12 It can be seen that FusionGCN, like other multi-focus image fusion methods based on discriminative models, has a serious sense of visual fragmentation, and there is a big gap with the MFFT network that uses the discriminative network to modify the decision graph. However, compared with other homogeneous networks, the edge discrimination of FusionGCN is closer to the area around the petals, and it still has certain advantages in the processing of edge details. From this experiment, it can be seen that the fusion method based on the generative model has better fused image quality and more natural edge transition when dealing with strong defocus diffusion effects. This is also the direction that needs to be overcome in the future based on the discriminative task model.

[0171] Fusion experiment of multiple source images

[0172] In a real imaging environment, due to the existence of depth of field, a single imaging device cannot obtain a completely clear image at the same time. Therefore, in some fields with high-precision requirements for images, such as autonomous driving, fault detection, etc. These fields often use multiple imaging devices to collect data on the same object at the same time, and then use a fusion algorithm to collect all information, and finally obtain a fused image containing all valid information. Here, this embodiment will use 4 pairs of Triple Series color images provided by the Lytro dataset for experiments. Even though the network of this embodiment does not consider the situation of multiple source images during training, the fusion of multiple images can be processed according to the following steps: first, the source images A and B are input into the network to obtain a preliminary fused image, and then the preliminary fused image and source image C are input into the network for the second time to obtain a final fused image containing all information. The fusion experiment of multiple source images is a great test of the robustness of the algorithm. In the Triple Series provided by the Lytro dataset, the information of each source image is complementary. If the preliminary fused image is inaccurate, the network will be non-aligned when performing the second fusion, which will indirectly interfere with the subsequent network judgment and thus affect the quality of the fused image. The above mentioned fusion steps and the schematic diagram of the fusion results produced at each stage can be seen Fig.13 . Fi represents the fused image of the i-th stage, and Di represents the decision diagram of the i-th stage. It can be seen that the algorithm of this embodiment produces almost no misjudgment in each round of fusion, and well completes the fusion experiment of multiple source images.

[0173] Efficiency analysis experiment

[0174] Here, this embodiment will conduct efficiency analysis experiments with the 10 SOTA methods mentioned above, and all methods will run on devices with Intel(R) Core(TM) i9-10900x CPU@3.70GHz and NVIDIA GeForce RTX 3090. To ensure fairness, this embodiment independently conducts experiments with one project corresponding to one virtual environment, and the parameters are consistent with the parameters recommended by the author. It is worth noting that for the traditional methods GFF and MWGF, CUDA acceleration is not considered, so they run on the CPU, and the remaining methods all run on the GPU. The efficiency comparison will be based on the Lytro dataset, and the detailed data analysis is shown in Table 6, where the first place for each indicator is highlighted in red and the second place is highlighted in blue.

[0175] By comparison, this example found that FusionGCN performed well in terms of network parameter quantity and floating point operations per second, and the computational consumption was much less than the existing methods. This provides a new solution for the application of multi-focus image fusion tasks on devices with limited computing resources. Nevertheless, there is still room for improvement in the performance of FusionGCN in terms of running time. The main reason is that the features of each image are quite different. Even if a pair of complementary source images are input into the GCN Encoder, the resulting node feature matrix Sum and edge matrix There are also significant differences between and , which results in the GCN part of FusionGCN being unable to process a batch of data in parallel as efficiently as CNN.

[0176] In order to enable GCN to handle the multi-focus image fusion task with acceptable training and running efficiency, as mentioned above, before performing SLIC processing on each source image, this embodiment uses ImageProcess() to comprehensively consider the information from the same pair of source images A and B to ensure that the margin matrix of the two during training is and type conversion matrix Consistent, so that for a given input feature The computational complexity of graph construction is O(2H 2 W 2 C) is reduced to O(H 2 W 2 C), this design significantly reduces computational time while maintaining computational efficiency and accuracy. Nevertheless, this embodiment recognizes that there is still room for improvement in runtime, especially when processing image pairs with significant feature differences. Future work will explore more efficient parallel processing strategies to further optimize the runtime of FusionGCN and make it more suitable for deployment on resource-constrained devices.

[0177] Table 6. Comparison of 10 SOTA methods and the proposed method in terms of network scale, parameter size, and time to process a single pair of Lytro dataset images. The first and second place of each indicator are highlighted in red and blue, respectively.

[0178] Table 6. Network scale of 10 SOTA methods and the method proposed in this paper

[0179]

[0180] Ablation experiment

[0181] Ablation experiment on network architecture

[0182] This example uses the Lytro dataset as a benchmark and carefully designs four ablation network structures for experiments. The detailed parameter information of each network is shown in Table 7, and the objective evaluation index score is given by Fig.14 As shown in the figure, network (a) and network (b) represent the use of only the GCN branch and the CNN branch to perform the multi-focus image fusion task separately, and network (c) represents the use of Bottleneck as the basic feature extraction block to replace the BasicConv and GCN Decoder proposed in this embodiment. Fig.15 The quality comparison of decision graphs produced by four ablation experimental networks is shown. From the experiment, it can be concluded that although the use of GCN alone for multi-focus image fusion tasks can fit the task with a very small amount of parameters, the edge segmentation is very inaccurate, and a large number of jagged stripes appear. If the CNN branch is used alone for processing, although good results can be achieved in some data graphs, the feature extraction capability is still insufficient, and a large number of misjudgments are generated in the decision graph. Using Bottleneck instead of Basic-Conv has a significant improvement in the task compared to using only GCN or CNN branches, but there is still a big gap from FusionGCN. This shows the effectiveness of the network structure and modules proposed in this embodiment.

[0183] Table 7: Summary of network data of four ablation networks on the Lytro dataset

[0184]

[0185] Ablation experiments on different GCN initial node scales:

[0186] In the above, this embodiment uses the hyperparameter Scale to fix the initial number of superpixel nodes N. init Here, this embodiment will discuss the significance of this parameter in detail. Fig.16 visible, Fig.16 The objective evaluation indicators of networks with different scale values ​​on Lytro are shown. Scale = 1.5 is the method proposed in this paper, and it ranks first with an efficiency score of 44.15. The size of the initial node number and the type conversion matrix and the node feature matrix is directly related to the composition of the edge matrix The generation of is indirectly related to . This parameter runs through the entire GCN part and restricts the accuracy of the fused image. Although a smaller Scale value will reduce the computational time, it will cluster pixels in a larger range into a superpixel center, causing serious information loss in the constructed feature matrix and reducing the quality of the fused image. As the Scale and value increase, the multiple pixels constituting the superpixel center transition to a single pixel, and the network fusion scale also transitions from superpixels to pixels. Although the fusion quality increases, the three constructed matrices grow exponentially, consuming a lot of computing resources while also making the time for processing a single pair of fused images unacceptable. The consumed computing resources are not proportional to the improvement in the quality of the fused image, and the boundary effect is very serious. In order to explore the optimal setting values ​​of the hyperparameters Scale and , this embodiment sets up this experiment to further control the variables in order to intuitively demonstrate the efficiency of each method. This embodiment proposes a novel efficiency evaluation method, which uses the following formula to calculate the score of each method.

[0187]

[0188] Where N represents the total number of objective evaluation indicators, which is 8 here. Time represents the average time spent processing a single pair of Lytro datasets. i Represents the i-th objective evaluation index, the higher the index score, the better. i Represents the score ranking of the method on the corresponding indicator. This experiment was tested on the Lytro dataset. Fig.16 The running time, objective evaluation index summary and efficiency score of different scales on Lytro are shown. Among them, Scale = 1.5 won the first place with an efficiency score of 44.15 points, and completed the multi-focus image fusion task well under the premise of taking into account both accuracy and computational efficiency.

[0189] Ablation experiment on the effect of the same scale division on different image processing:

[0190] As mentioned above, in order to balance accuracy and running time, a convolutional layer is used to comprehensively consider the information from source images A and B, so that the partition labels generated by SLIC are more consistent with the similar features of the two. But this also leads to other problems: the same hyperparameter Scale value may have completely different effects on different images. To this end, this embodiment carefully designs this experiment and uses the MFFW dataset that is severely affected by the defocus diffusion effect as a benchmark to explore the differences in the fusion results of different source images due to the partition size. The experimental results Fig.17 As shown, Fig.17The results of processing different images at various division scales are shown, where Fusion result is the fusion result after SLIC segmentation. Among them, if Scale = 0.75 is used as the standard, the column direction is observed. This embodiment found that for the arms and bird heads that are severely interfered by noise, FusionGCN can correctly aggregate the noise bands into multiple superpixels when processing the arms in MFFW-02, but for MFFW-13, it mistakenly aggregates the bird head and noise bands that should belong to clear pixels into one superpixel. However, if the observation angle is converted to row-by-row, as the Scale continues to increase, the superpixel features gradually transform into pixel features, and the fuzzy boundaries can be correctly aggregated into independent superpixel nodes at the arms and bird heads, which is reflected in the decision diagram as a significant reduction in the misjudgment area. However, when the Scale value increases to 1.5, the boundary effect is very obvious. Even if more superpixel nodes are used to describe the boundary area, the quality of the generated fused image has not been qualitatively improved, but the running time has increased exponentially, greatly reducing the running efficiency. Therefore, it can be concluded that although the same Scale parameter value setting will produce quite different results on different images, when the Scale multiple is set to 1.5, it can also produce good adaptability for the processing of different images.

[0191] Ablation experiment on small area filtering with different thresholds in post-processing technology

[0192] As a common post-processing method, small area filtering improves the final result by eliminating or merging small areas in the image. It is especially important in tasks such as image segmentation and edge detection. In multi-focus image fusion, it is necessary to correct the errors in the initial decision graph output by the network. This example uses the MFI-WHU dataset as a benchmark, and explores the relationship between the threshold size and the fusion effect by changing the threshold used in small area filtering. The experimental results are shown in Figure 2. Fig.18 As shown, Fig.18 This is a comparison of the subjective visual effects of different small area filter thresholds in the MFI-WHU data set, where the threshold value of 0.001 is the method proposed in this embodiment. It can be concluded that without using small area filtering, many small misjudgment spots are generated in the fused image. A larger filter threshold will cause a lot of detail information to be lost. A smaller filter threshold cannot correct the errors in the initial decision graph. Thanks to the excellent performance of the network in this embodiment, compared with other discriminant models with a threshold value set to 0.1, this embodiment sets Threshold = 0.001, which is sufficient to correct network errors.

[0193] This example uses GCN to solve the multi-focus image fusion problem for the first time. In FusionGCN, a clever idea of ​​combining block cutting and pixel optimization is proposed, which makes the decision graph transition from coarse to fine, reduces the difficulty of network reasoning, and generates a higher quality fusion image.

[0194] A superpixel-based graph decoder and a pixel-based CNN base extraction block are carefully designed to enable GCN and CNN to cooperate with each other in the same network, alleviating the incompatibility between non-Euclidean space data to Euclidean space data.

[0195] Thanks to the GCN's automatic feature clustering, even without any means of reducing network parameters, FusionGCN only needs 0.13M parameters and 50.76G floating point operations per second to process a pair of 520×520 multi-focus images, providing a new solution for running multi-focus image fusion tasks on computing-constrained devices.

[0196] In order to effectively evaluate the efficiency of each method, this embodiment proposes a novel evaluation method based on objective evaluation indicators to make the efficiency evaluation results comparable.

[0197] In this embodiment, a novel multi-focus image fusion method combining superpixel feature generation GCN and pixel-level feature refinement CNN is proposed. As far as we know, this is the first time that graph convolutional neural networks have been applied to multi-focus image fusion tasks. Through the idea of ​​block cutting and pixel optimization, the intermediate feature map transitions from coarse to fine, reducing the difficulty of network reasoning. Compared with the other 10 SOTA methods, it has demonstrated excellent performance in both qualitative and quantitative experiments. In addition, thanks to the carefully designed network structure of this embodiment, the FusionGCN training parameters are only 0.13M, which provides a new solution for multi-focus image fusion tasks running on certain devices with limited computing resources.

[0198] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A multi-focus image fusion method using superpixel features to generate GCN and pixel-level features to reconstruct CNN, characterized in that: include: Using graph convolutional neural networks, soft convolution is performed on large-scale irregular regions formed by superpixel clustering in the input image to obtain block-level coarse segmentation features. Using a parallel lightweight convolutional neural network to extract pixel-level features from the input image; The extracted pixel-level features and the block-level coarse segmentation features are fused to obtain a fully focused image.

2. The multi-focus image fusion method of generating GCN using superpixel features and reconstructing CNN using pixel-level features according to claim 1, characterized in that: The soft convolution of large-scale irregular regions formed by superpixel clustering in the input image includes: Use the SLIC algorithm to perform preliminary clustering of pixels in the input image; The Scharr operator is used to preliminarily process the image after preliminary clustering, and the total gradient value in the centroid of a single superpixel is used as the initial feature to construct the node feature matrix; According to the spatial adjacency relationship of pixel nodes in the image after preliminary clustering, an edge matrix is ​​constructed; A graph convolution operation is performed on the node feature matrix and the edge matrix to obtain block-level coarse segmentation features.

3. The multi-focus image fusion method of generating GCN using superpixel features and reconstructing CNN using pixel-level features according to claim 2, characterized in that: The initial clustering of pixels in the input image using the SLIC algorithm includes: Based on the preset formula, the number of nodes in the input image is divided, and the superpixel area is divided; Based on the number of nodes and superpixel areas, pixels in the input image are preliminarily clustered.

4. The multi-focus image fusion method of generating GCN using superpixel features and reconstructing CNN using pixel-level features according to claim 3, characterized in that: The preset formula is: Map=ImageProcess(Concat(A,B)) Among them, Concat() represents the concatenation operation, A, B represent the source image A and the source image B, ImageProcess represents the convolution operation with a convolution kernel size of 7 to reduce the number of channels to 1, Split() represents the use of the Split function to split according to the batch size dimension, i represents the position of the image in the same batchsize, H, W represent the height and width of the source image A respectively, Scale represents the scaling of the number of initial superpixel divisions, SegmentsLabelProcess() represents the post-processing function, Batchsize represents the amount of data used each time during the training process, and Map represents a channel input feature map after ImageProcess processing. Represents the i-th feature map in the same batch, dim represents the matrix dimension for the required split operation, represents the initial segmentation map generated by the i-th SLIC partition in the same batch, N init Indicates the number of initial partition nodes, Max(H,W) indicates the maximum value of the length or width of the selected image, Segments i represents the i-th superpixel label generated after post-processing, N i Represents the number of superpixels after post-processing.

5. The multi-focus image fusion method of generating GCN using superpixel features and reconstructing CNN using pixel-level features according to claim 2, characterized in that: The method for constructing the node feature matrix is: img i =Split(img,dim=1),img∈(A,B) Processed i =Conv 3×3 (img i ) where j=1,2,3,...,N and i=1,2,...,Batchsize Among them, Conv 3×3 It means that the source images A and B are reduced to 1 dimension using convolution with a convolution kernel size of 3. Flatten means that the gradient map obtained by the Scharr operator is linearly flattened. Represents the convolution operation, Grad_img i represents the gradient values ​​of all pixels under the jth superpixel, Sum() represents the summation of all pixel gradient values ​​under the same superpixel, img i represents the i-th image of the same input batch, Split() represents the use of the Split function to split according to the batch size dimension, ScharrX represents the use of the Scharr operator to process in the x-axis direction, and Processed represents the image after 3×3 convolution. i , ScharrY means using the Scharr operator to process in the y-axis direction, X[j,_] means the value of the feature matrix at the jth superpixel, It represents the gradient value of the jth superpixel node in the i-th gradient map, N and i represents the number of superpixels and their positions in the same batch, and Batchsize represents the batch size.

6. The multi-focus image fusion method of generating GCN using superpixel features and reconstructing CNN using pixel-level features according to claim 1, characterized in that: The method to construct the edge matrix is: Among them, M i,j Indicates that there is an edge from node i to node j, N i and N j They all represent superpixel nodes.

7. The multi-focus image fusion method of generating GCN using superpixel features and reconstructing CNN using pixel-level features according to claim 2, characterized in that: Soft convolution of large-scale irregular regions formed by superpixel clustering in the input image also includes: The block-level coarse segmentation features are converted from a superpixel scale to a pixel scale using a type conversion matrix; The type conversion matrix is: Among them, T[i,j] represents the type conversion matrix, and i,j represent i pixel and j pixel.

8. The multi-focus image fusion method of generating GCN using superpixel features and reconstructing CNN using pixel-level features according to claim 1, characterized in that: The extracted pixel-level features and the block-level coarse segmentation features are fused to obtain a fully focused image, including: Fusing the extracted pixel-level features with the block-level coarse segmentation features to obtain fused features; Classify the fused features to generate an initial decision graph; Post-processing the initial decision graph to obtain a final decision graph; According to the final decision graph, the multi-focus source images are fused into a fully focused image.

9. The multi-focus image fusion method of generating GCN using superpixel features and reconstructing CNN using pixel-level features according to claim 8, characterized in that: The method for carrying out the post-processing is: in, and represents the final decision graph after post-processing steps, Small_object_removal() represents small area filtering, I A and I A Represents the multi-focus source image pairs A and B, D that are input to the network pre (x,y) represents the initial decision graph output by the network, and F(x,y) represents the fused image.

Images