A multi-focus image fusion method based on unsupervised learning and its model construction method

Abstract

The present invention provides a multi-focus image fusion method based on unsupervised operation and a model construction method thereof, and belongs to the technical field of computer image processing; the present invention is a method for image fusion based on a decision graph, wherein an encoder is used to extract features, and a decoder is used to reconstruct an image using the features provided by the encoder. In the encoder, a double branch including a detail branch and a semantic branch is used to extract image features. In addition, the present invention proposes a new activity level calculation module that uses a pixel-based Laplace correction and as a processing unit to obtain the initial decision graph, and then obtains the final decision graph through a morphological processing unit, and finally realizes fusion through a fusion module. The model of the present invention can be fused to obtain a clear fused image without supervised information and a large-scale image set. The use of a double-branch feature extraction method can extract features more effectively, and the use of measuring activity levels can greatly improve the fusion effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer image processing, and in particular relates to an unsupervised multi-focus image fusion method and a model construction method thereof. Background Art

[0002] In recent years, image fusion has made great progress in the field of digital image processing, which makes multi-focus image fusion an important topic in image fusion. Due to the limitation of depth-of-field (DOF), the image taken by the optical lens cannot be fully focused, and only the material within the DOF will be focused. Therefore, the purpose of multi-focus image fusion is to generate a fully focused image from images with different focusing conditions.

[0003] Since the proposal of multi-focus image fusion, many methods have been proposed in the past decade. These methods can be roughly divided into three categories: transform domain methods, spatial domain methods and deep learning methods. The transform domain method mainly performs fusion through three steps: transforming the source image to the transform domain using mathematical methods, then fusing and finally inverse transforming. At present, many transform domain-based methods have been proposed, such as sparse representation methods, multi-scale methods, gradient domain-based methods and hybrid methods. Methods based on the spatial domain can be further divided into three categories: pixel-based methods, block-based methods and region-based methods. This method directly processes pixels or regions and uses linear or nonlinear methods to fuse multi-focus images. Methods based on deep learning can be divided into two categories: supervised learning and unsupervised learning. A CNN-based multi-focus image fusion method was first proposed in 2017. They used Gaussian filters to generate synthetic images with different degrees of blur to train a two-class image classification network. By using this supervised learning strategy, the network can distinguish whether the patch is in focus. Later, DenseFuse used an unsupervised encoder-decoder network to extract deep features of the image and designed The norm fusion strategy fuses the two feature maps, and then the decoder uses the fused features to obtain the fused image. Later, SESF-Fuse proposed that in multi-focus tasks, what really matters is the feature gradient, not the feature strength, and used Calculate feature gradients. It is mainly based on the frequency distribution of the image, and measures the clarity of the image through the frequency changes in the horizontal and vertical directions. It reflects the spatial detail of the entire image. The higher the frequency, the more details the image has and the clearer the image is. However, it does not respond well to image blur and contrast reduction. Objects with different depths of field all have some semantic features, and most of the current unsupervised methods use detail branches to extract the detail features of the image separately and then reconstruct the image, thus ignoring the role of semantic features in the field of multi-focus image fusion. Summary of the invention

[0004] In view of the above problems, the first aspect of the present invention provides a method for constructing a multi-focus image fusion model based on unsupervised operation, comprising the following steps:

[0005] Step 1, collect and acquire a large number of original all-focus images;

[0006] Step 2, preprocessing the obtained original all-focus image, wherein the preprocessing includes grayscale conversion, and normalizing the obtained gray image to obtain an image tensor, and constructing a training set and a test set;

[0007] Step 3, based on the encoder-decoder network structure, construct an unsupervised dual-channel multi-focus image fusion model UDC MFIF model; the UDC MFIF model includes an encoder, a decoder, an activity level calculation module and a fusion module;

[0008] The encoder adopts a dual-branch channel encoder network structure for input, the two branches are detail branch and semantic branch, the weights of the two branches are not shared, and only the channel dimension feature splicing is performed when entering the decoder; the detail branch and the semantic branch receive the same input image to extract rich detail information and semantic information of the image respectively; the input of the decoder is the joint feature of splicing the detail information and semantic information in the channel dimension to reconstruct the image; the activity level calculation module includes processing unit and morphological processing unit; said The processing unit calculates the depth features by Laplace correction and obtains the initial decision graph; the morphological processing unit optimizes the decision graph by opening operation, closing operation and removing small holes; the fusion module uses the decision graph obtained after processing by the above-mentioned activity level calculation module and the input image of different focusing conditions under the same viewing angle to perform weighted product summation of corresponding pixels to obtain the final fused image;

[0009] Step 4: Use the training set and the test set to train the encoder and decoder to minimize the difference between the decoder's reconstructed image and the encoder's input image, thereby completing the construction of the overall model.

[0010] Preferably, the grayscale conversion in step 2 is specifically as follows:

[0011] The original image Convert the color image into a gray image , and then the gray image is uniformly adjusted to , then convert the image into a tensor and input it into the network. The formula is as follows:

[0012]

[0013] For gray images Each pixel value in , which is normalized to arrive interval, and finally get the image tensor.

[0014] Preferably, the encoder includes two feature extraction parts, a detail branch and a semantic branch; the detail branch is used to extract detail feature information in the image, and the detail branch includes a shallow feature extraction module C1 and four EC detail feature extraction modules, and the shallow feature extraction module C1 includes a two-dimensional convolution, The activation function and batch normalization layer are used to extract the shallow features of the image without changing the size of the input features, so as to facilitate the subsequent extraction of detail features and speech features; the EC detail feature extraction module is used to extract detail features; the semantic branch includes a shallow feature extraction module C1, three CBEC semantic feature extraction modules and an upsampling module, wherein the shallow feature extraction module is the same as that in the detail branch; the CBEC semantic feature extraction module is used to extract semantic features; the upsampling module is used to make the size of the semantic feature map consistent with the size of the detail feature map, so as to facilitate the final splicing;

[0015] Preferably, the EC detail feature extraction module is composed of a two-dimensional convolution, an activation function, a batch normalization layer and a channel attention mechanism ECA module, and uses dense links to focus on shallow features; the CBEC semantic feature extraction module is composed of a two-dimensional convolution, an activation function, a batch normalization layer and a convolution attention mechanism CBAM module. The step size of the two-dimensional convolution in the CBEC semantic feature extraction module is 2. After passing through this module, the size of the feature map of the image is halved. By continuously downsampling, the image loses a large number of detail features, thereby extracting semantic features; the upsampling module is composed of an upsampling and convolution attention mechanism CBAM module. The scale factor in the upsampling module is 8. Bilinear interpolation is used to align the corner points of the input and output. This module can return the feature map size of the image that has been reduced by 8 times through the CBEC semantic feature extraction module to its original size, and then splice it with the detail features in the channel dimension; after two branches, the detail features are obtained. and semantic features , concatenate these two feature tensors in the channel dimension and After that, we get a new feature tensor , the number of channels is and The sum of the number of channels, while the height and width remain unchanged; the formula is as follows:

[0016]

[0017]

[0018]

[0019] in, and It is a detail feature , semantic features and joint features The height and width of It is a detail feature The number of channels, It is a semantic feature The number of channels, It is a joint feature The number of channels.

[0020] Preferably, the specific data processing process of the activity level calculation module is:

[0021] The input size of n images is Pictures The encoder extracts n feature tensors in turn, and then uses the The processing unit calculates n The value is used as the feature map reflecting the image information, and then n images of size The feature maps are spliced ​​in three dimensions into a size of The mosaic diagram, and then along The best focus pixel point at the same position in the feature map of each layer is selected in the axial direction. All focus infographics in Direction The point with the largest value is the best pixel, and the best focus point is assigned a value of 1, and the corresponding points on the other layers are assigned a value of 0 to obtain n pixels of size Decision diagram ,in Number the source images;

[0022] The morphological processing unit is then used to output the final decision graph ; First, use the open operation of the small disk structure to process To eliminate most of the noise in the background area; secondly, the small hole removal strategy is used to eliminate the remaining noise in the background area; finally, the noise in the background area is removed by closing operation and small hole removal strategy to obtain the final decision graph .

[0023] Preferably, the fusion module performs weighted fusion on each layer of source images according to the decision graph to obtain a final fusion graph F, which is expressed as:

[0024]

[0025] in For the An initial decision diagram, For the The final decision diagram, For the source images, is the pixel coordinate, F is the final fused image.

[0026] Preferably, in step 4, the training target is to minimize the difference between the reconstructed image of the decoder and the input image of the encoder, and the calculation formula is as follows:

[0027]

[0028] in is the joint loss, i.e., the minimum difference between the decoder’s reconstructed image and the encoder’s input image, is the structural similarity loss, is the gradient loss, is the pixel loss; is the parameter weight for balancing the loss.

[0029] The second aspect of the present invention also provides an unsupervised multi-focus image fusion method, comprising the following process:

[0030] Get n images with different focal points in the same field of view;

[0031] Convert n images to grayscale and normalize them to get image tensor;

[0032] Inputting the image tensors of n images into the multi-focus image fusion model constructed by the construction method described in the first aspect;

[0033] The model generates and outputs an all-in-focus image.

[0034] The third aspect of the present invention also provides a multi-focus image fusion device based on unsupervision, the device includes at least one processor and at least one memory, the processor and the memory are coupled; the memory stores a computer execution program of the multi-focus image fusion model constructed by the construction method described in the first aspect; when the processor executes the computer execution program stored in the memory, the processor executes a multi-focus image fusion method based on unsupervision.

[0035] The fourth aspect of the present invention also provides a computer-readable storage medium, which stores a computer execution program of the multi-focus image fusion model constructed by the construction method described in the first aspect. When the computer execution program is executed by a processor, the processor executes a multi-focus image fusion method based on unsupervised operation.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention is based on unsupervised multi-focus image fusion for model training. The model can fuse clear fused images without supervised information and large-scale image sets. Inspired by semantic information, the present invention applies dual channels to more fully extract features and proposes a UDC MFIF model; compared with the latest unsupervised network-based model, the present invention more fully extracts feature detail information and semantic information. The present invention fuses through a decision graph and uses feature gradients to measure activity levels; using the pixel-based calculation proposed by the present invention A more accurate decision diagram can be obtained. Compared with the currently proposed multi-focus image fusion algorithm, the experimental results show that the dual-branch feature extraction method can extract features more effectively. Measuring activity levels can greatly improve fusion results. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a schematic diagram of the overall structure of the focused image fusion model proposed in the present invention.

[0039] Figure 2 This is a structural diagram of the shallow feature extraction module C1.

[0040] Figure 3 Schematic diagram of the structure of the detail feature extraction module EC_block.

[0041] Figure 4 This is a structural diagram of the semantic feature extraction module CBEC_block.

[0042] Figure 5 This is a schematic diagram of the upsampling module structure.

[0043] Figure 6 Schematic diagram of the ECA channel attention module structure.

[0044] Figure 7 Schematic diagram of the CBAM convolutional attention module structure.

[0045] Figure 8 A schematic diagram of the decoder structure.

[0046] Fig. 9 This is a schematic diagram of the activity level calculation module structure proposed in the present invention.

[0047] Fig.10 This is a schematic diagram of the fusion module structure proposed in the present invention.

[0048] Fig.11 This is a flowchart of the fusion model processing of multiple differently focused images.

[0049] Fig.12 This is a simplified structural diagram of the multi-focus image fusion device in Example 2. DETAILED DESCRIPTION

[0050] The present invention designs an unsupervised dual channel fusion method based on an encoder-decoder network structure: Unsupervised dual channel (hereinafter referred to as UDC MFIF model). The present invention is a method for image fusion based on a decision graph, the encoder is used to extract features, and the decoder is used to reconstruct images using the features provided by the encoder. The present invention uses a dual branch in the encoder, including a detail branch and a semantic branch, to extract image features. In addition, the present invention proposes a new activity level calculation module using a pixel-based Laplace correction sum: Sum-modified-Laplacian (hereinafter referred to as ) to act as The processing unit obtains the initial decision graph, and then the morphological processing unit obtains the final decision graph, and finally the fusion module realizes fusion. The UDC MFIF model construction process is as follows:

[0051] Step 1, collecting and acquiring a large number of all-focus original images, wherein the original images are downloaded from the Internet or directly photographed digital color images;

[0052] Step 2: Preprocess the original image obtained in step 1: first convert the original image into grayscale, that is, convert the color image into a grayscale image, which can increase the robustness of the model and enable the model to achieve the fusion of color or gray multi-focus images; then resize the gray image to a uniform size. After preprocessing, the image data is The ratio is divided into training set and validation set.

[0053] Step 3, based on the encoder-decoder network structure, build an unsupervised dual-channel multi-focus image fusion model UDC MFIF model; the UDC MFIF model adopts a dual-branch channel encoder network structure for input, and the two branches are detail branch and semantic branch respectively; the weights of these two branches are not shared, and only the channel dimension feature splicing is performed when entering the decoder;

[0054] The detail branch and semantic branch receive the same input image to extract rich detail information and semantic information of the image respectively. Finally, at the output of the encoder, the detail information and semantic information are spliced ​​in the channel dimension to enrich the diversity of feature extraction, and the features are input to the decoder for image reconstruction.

[0055] Step 4: Use the training set and validation set described in step 2 to train the DC MFIF model built in step 3, and select the minimum difference between the decoder's reconstructed image and the encoder's input image as the final model.

[0056] The invention will be further described below in conjunction with specific embodiments.

[0057] 1. Dataset Construction and Data Preprocessing

[0058] This embodiment conducts experiments on existing data sets, and the present invention can also be applied to color images taken by individuals.

[0059] The obtained data set is preprocessed; first, all color images are converted to grayscale images, and then the grayscale images are uniformly adjusted to Size, no matter the input is a color image or a grayscale image, the present invention adopts a unified fusion step; finally, the image is input into the network model in the form of a tensor for training, and the tensor is calculated as follows:

[0060]

[0061] For gray images Each pixel value in , which is normalized to arrive interval, and finally get the image tensor.

[0062] For example, input a size of A color image with 3 channels is converted to grayscale. A gray image with a channel number of 1, and then resized uniformly to Gray image with 1 channel; after normalization tensor operation, the pixel value is limited to , that is, the original pixel of 0 is converted to a tensor of 0.0, and the original pixel of 255 is converted to a tensor of 1.0.

[0063] 2. UDC MFIF Model Construction

[0064] Based on the encoder-decoder network structure, an unsupervised dual-channel multi-focus image fusion model UDC MFIF model is constructed. The structure is as follows Figure 1 As shown; the UDC MFIF model includes an encoder, a decoder, an activity level calculation module and a fusion module;

[0065] The encoder adopts a dual-branch channel encoder network structure for input. The two branches are detail branch and semantic branch. The weights of the two branches are not shared, and only the channel dimension feature splicing is performed when entering the decoder; the detail branch and the semantic branch receive the same input image to extract rich detail information and semantic information of the image respectively; the input of the decoder is the joint feature of splicing the detail information and semantic information in the channel dimension to reconstruct the image; the activity level calculation module includes processing unit and morphological processing unit; The processing unit calculates the depth features through the Laplace correction and obtains the initial decision graph; the morphological processing unit optimizes the decision graph through opening operation, closing operation and pinhole removal operation; the fusion module uses the decision graph obtained after processing by the above-mentioned activity level calculation module and the input image with different focusing conditions under the same viewing angle to perform weighted product summation of corresponding pixels to obtain the final fused image.

[0066] 1. Encoder and Decoder

[0067] The encoder consists of a detail branch and a semantic branch. The detail branch has a shallow feature extraction module C1, such as Figure 2 As shown in the figure, and the EC detail feature extraction module, the specific structure is as follows Figure 3 As shown; the semantic branch includes the shallow feature extraction module C1, the CBEC semantic feature extraction module and the upsampling module. The specific structure is as follows Figure 4 Shown and Figure 5 As shown in the figure, the shallow feature extraction module C1 consists of a two-dimensional convolution with a convolution kernel of 3, a step size of 1, and a padding of 1. Activation function and batch normalization composition;

[0068] The input image is convolved by the shallow feature extraction module C1 to obtain a shallow feature with 16 channels and a size of 256x256. Then a copy of the feature is sent to the detail branch and the semantic branch respectively.

[0069] In the detail branch, a total of 4 EC detail feature extraction modules are used, and the number of output channels of each EC module is 16. Each convolution block is densely linked together. The number of channels after each EC module changes to 16, 32, 48, 64, and 80, and the feature size does not change. The EC detail feature extraction module inputs the extracted detail information into the ECA channel attention module to obtain the channel detail feature, as shown in the following figure. Figure 6 In addition, the EC detail feature extraction module includes a convolution layer (convolution kernel is 3x3, step size is 1, padding is 1), an activation function ( ), which can be expressed by the following formula:

[0070]

[0071]

[0072] Batch Normalization ( ) is a two-dimensional batch normalization layer, which is used to normalize the output of the convolutional layer, which can speed up the training process and improve the stability of the model. The main idea is to normalize each The data is normalized so that each The data distribution has a mean of 0 and a variance of 1.

[0073] Specifically, the batch normalization process includes the following steps:

[0074] (1) Calculation The mean and variance of :

[0075] For one First calculate the The mean and variance :

[0076]

[0077]

[0078] (2) Normalization:

[0079] Normalize each input data using the calculated mean and variance:

[0080]

[0081] here, is a small constant used to prevent the denominator from being zero.

[0082] (3) Zoom and pan:

[0083] The normalized data is then scaled and translated to restore the expressive power of the data. and , and implement the following transformation:

[0084]

[0085] In the semantic branch, the CBEC semantic feature extraction module with a step size of 2 is used three times to achieve downsampling. The CBEC module includes (convolution kernel is 3x3, step size is 2, padding is 1), activation function ( ) and batch normalization ( ). Because the step size of the convolution is 2, according to the convolution calculation formula:

[0086]

[0087] in: is the size after convolution, is the original size, For filling, is the size of the convolution kernel, is the step size.

[0088] After passing through the CBEC semantic feature extraction module, the size is reduced from the original becomes If the input is an even number, then the output yes , downsampling is achieved. After continuous downsampling operations, the size of the image will become smaller, so a lot of detail information will be lost, and the semantic information will be retained. Finally, the model inputs the extracted semantic information into the CBAM convolutional attention module to obtain fine features. The specific CBAM convolutional attention module structure is as follows Figure 7 As shown. The number of channels changes to: 16, 16, 32, 64, 32. The feature size changes from the input 256 to 128, 64, 32, and then changes to 256 after the upsampling module, in order to splice the channel dimension with the detail feature. At the end of the encoder, the two feature channel dimensions are spliced ​​to obtain a deep joint feature with a channel number of 112 and a size of 256x256; the decoder is used to reduce the dimension of the high-dimensional deep features obtained by the encoder one by one to obtain the reconstructed image. The decoder structure is as follows Figure 8 As shown in Figure 1, the number of channels passing through the decoder changes from 112 of the deep joint feature to 64, 32, 16, and 1, and finally the reconstructed grayscale image is obtained. The specific network parameters are shown in Table 1:

[0089] Table 1 is a diagram of specific network parameters

[0090]

[0091] 2. Activity level calculation module

[0092] The activity level calculation module is calculated by using The processing unit is used to calculate the feature tensor output by the model encoder and the maximum feature map fusion strategy is used to obtain the initial decision map; the morphological processing unit is used to process the initial decision map to obtain the final decision map, as shown in the following figure. Fig. 9 shown.

[0093] It can effectively reflect edge feature information and obtain better visual features. It is specially designed for grayscale images, so the present invention proposes a Calculation method. is the high-dimensional feature obtained by the feature extraction module. In pixels The eigenvector at . It is calculated by the following formula

[0094]

[0095]

[0096] in Determines the calculation The size of the pixel neighborhood considered when , that is, the size of the window. In the present invention, it is set to 2, that is, the window is In order to better adapt to the changes in texture element size, it is generally Set to 1. The difference is that the traditional method is based on block calculation, while the present invention is based on pixel calculation. When processing the feature map boundary, the same filling method as the traditional one is used. Next, the following formula is used to compare the two input feature maps: 1 and 2 Get a rough decision diagram .

[0097]

[0098] For multiple input feature maps 1 , …, , , then the feature map of a single layer In the three-dimensional stacking, the best focus pixel points at the same position in each layer of image are selected along the z-axis direction, and all feature maps in the z-axis direction are selected. The point with the largest value is the best pixel, and the best focus point is assigned a value of 1, and the corresponding points on the other levels are assigned a value of 0 to obtain the initial binary decision map. .because There may be some cases where pixels are misclassified, resulting in noise. Therefore, the present invention uses a morphological processing method to process these noises. First, the opening operation of the smaller disk structure is used to process To eliminate most of the noise in the background area. Secondly, the small hole removal strategy is used to eliminate the remaining noise in the background area. Finally, the noise in the background area is removed by closing operation and small hole removal strategy to obtain the final decision graph. The processing flow of the fusion model of multiple images with different focus is as follows: Fig.11 shown.

[0099] 3. Fusion Module

[0100] Fusion modules such as Fig.10 As shown in the figure, according to the decision graph, each layer of source images is weighted fused to obtain the final fusion graph F, which is expressed as:

[0101]

[0102] in For the An initial decision diagram, For the The final decision diagram, For the source images, is the pixel coordinate, F is the final fused image.

[0103] 3. Model Training

[0104] The present invention designs a joint loss function , including the structural similarity loss , gradient loss And pixel loss The image is reconstructed using the joint loss function constrained network. Ultimately, the closer the decoder reconstructed image is to the input fully focused image, the encoder will learn how to extract features to enable the decoder to reconstruct the fully focused image, and obtain the final multi-focus image fusion model:

[0105]

[0106] in is the joint loss, i.e., the minimum difference between the decoder’s reconstructed image and the encoder’s input image, is the structural similarity loss, is the gradient loss, is the pixel loss; is the parameter weight for balancing the loss.

[0107] Structural Similarity Loss To calculate the output and input The structural difference between them is as follows:

[0108]

[0109] in is the structural similarity, and the formula is as follows:

[0110]

[0111] in and are the means of the input image and output image respectively; and are the variances of the input image and the output image respectively; is the covariance of the input image and the output image; and A small constant added to avoid the denominator being 0. , .

[0112] Gradient loss function It is used to maintain the edges and details of the image and constrain the gradient of the generated calculated image. The formula for gradient loss is:

[0113]

[0114] To calculate the image exist The directional gradient is , The directional gradient is , For the convolution operation:

[0115]

[0116]

[0117] Gradient size The calculation is as follows:

[0118]

[0119] in is the mean square error, and the formula is as follows:

[0120]

[0121] in and are the input image and the output image respectively.

[0122] Pixel loss Used to calculate the difference between the reconstructed image and the input image at the pixel level, here used to calculate the input and output The Euclidean distance between:

[0123]

[0124] in express Norm.

[0125] In this embodiment, an effective multi-focus image fusion method is implemented on a platform based on the Windows operating system, the programming language is Python 3.7, the deep learning framework is Pytorch 1.8.2, the CUDA version is 11.1, the initial learning rate is set to 0.0001, and it decreases by 0.8 times every two cycles. The batch size is set to 24, the cycle is 30, and Adam is used as the optimizer of the network. During the experiment, in order to balance the importance of each in the loss function, the following are set respectively: , , The model training was performed on an RTX2080 Ti GPU with 16GB of memory for 30 rounds, which took approximately 40 hours.

[0126] IV. Experimental Results

[0127] In order to verify the performance of the proposed algorithm, the present invention selected the Lytro color multi-focus image fusion dataset and the GrayScale gray multi-focus image fusion dataset for comprehensive algorithm comparison. A total of 9 algorithms were compared in the present invention, namely, the algorithm based on dual-tree complex wavelet transform (DTCWT), the algorithm based on non-subsampled transform (NSCT), the algorithm based on spatial frequency (SF), and the algorithm based on multi-scale weighted gradient (MWG).

[0128] In the past few years, deep learning has achieved great success in image processing. Some studies have tried to use large-capacity deep learning models to measure brain activity levels. Therefore, many deep learning methods have been proposed: DenseFuse ( 1 norm fusion strategy), regression pair learning DRPL, universal fusion network MUFusion based on memory unit ( 1 norm fusion strategy), generative adversarial network MFIF-GAN, compressed decomposition network SDNet. In the comparative experiments, the public source code or public fusion results provided by the author are used, and the parameters are the default values ​​without any changes.

[0129] In order to objectively evaluate the fusion quality of each algorithm, the present invention uses the following evaluation indicators for comparison: , , , , , , and It is worth noting that in addition to The larger the value of these indicators, the better the fusion effect. is a relative measure that indicates the direction of the fusion error—over-fusion ( ) or under-fusion ( ), the closer it is to 0, the better the fusion effect. The evaluation indicators based on information theory include: and The evaluation indicators based on image features include: , and The evaluation indicators based on image structure similarity include: and Fusion metrics inspired by human perception include: .

[0130] The Lytro dataset contains 20 pairs of images with different focuses. The present invention compares 9 multi-focus image fusion methods. The quantitative results of the Lytro color multi-focus image fusion public dataset are shown in Table 2, where each value represents the average metric value of the 20 pairs of Lytro datasets fused by different fusion methods. In addition, in the corresponding metric column of the table, the best value is shown in bold and the suboptimal value is shown underlined. From the quantitative comparison in Table 2, it can be seen that the present invention has , , , , , , Overall, the proposed method ranks first in 7 out of 8 indicators. Compared with other methods, the proposed method ranks first in comprehensive evaluation indicators on the Lytro dataset.

[0131] Table 2 Average scores of different fusion algorithms on the Lytro dataset

[0132]

[0133] In addition, multi-focus image fusion should not only perform well in color image fusion, but also in gray images. Therefore, the present invention also calculates eight evaluation indicators in the GrayScale gray multi-focus image fusion dataset, and the quantitative results are shown in Table 3. Each value represents the average metric value of 10 pairs of GrayScale gray fusions by different fusion methods. In addition, in the corresponding metric column of the table, the best value is displayed in bold and the suboptimal value is displayed in underline.

[0134] From the quantitative comparison in Table 3, it can be seen that the present invention has , , , , , The evaluation indexes ranked high and achieved excellent results. Overall, among the 8 indicators, 5 indicators of the present invention ranked first and 3 indicators ranked second. Compared with other methods, the comprehensive ranking of the evaluation indicators of the present invention on the GrayScale dataset is the highest.

[0135] According to the above comparison of subjective visual quality and objective evaluation indicators, it can be concluded that the proposed dual-channel fusion method is generally superior to other methods.

[0136] Table 3 Average scores of different fusion algorithms on the grey data set

[0137]

[0138] Ablation experiment:

[0139] In order to prove the effectiveness of the proposed method, different settings were tested. The following control experiments were conducted on the Lytro color multi-focus image fusion dataset at Epoch 20: using a separate detail branch to extract features and Perform deep feature calculation ( ), using dual branches to extract features and Calculate the depth feature ( ), extract features using separate detail branches and Perform deep feature calculation ( ) and dual-branch feature extraction and Perform deep feature calculation ( ). Each value represents the average metric value of 20 pairs of Lytro color multi-focus image fusion datasets fused by different fusion methods. In addition, in the corresponding metric column of the table, the best value is shown in bold red. From the quantitative comparison in Table 4, it can be seen that when the extracted feature branches are the same, The fusion quality is better than ; When the method of measuring activity level is the same, the dual-branch feature extraction is better than the single-branch feature extraction. In summary: the above dual-branch feature extraction and Measuring activity levels all contribute to the performance gains of the present invention.

[0140] Table 4 Ablation experiments under different settings

[0141]

[0142] like Fig.12 As shown, the present invention also provides a multi-focus image fusion device based on unsupervision, the device includes at least one processor and at least one memory, and also includes a communication interface and an internal bus; a computer execution program is stored in the memory; a computer execution program of a multi-focus image fusion model constructed by the construction method described above is stored in the memory; when the processor executes the computer execution program stored in the memory, the processor can execute a multi-focus image fusion method based on unsupervision. The internal bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus. The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a U disk, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0143] The device may be provided as a terminal, a server or other forms of device. In an exemplary embodiment, the electronic device may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0144] The present invention also provides a computer-readable storage medium, which stores a computer execution program of the multi-focus image fusion model constructed by the construction method as described above. When the computer execution program is executed by a processor, the processor can execute a multi-focus image fusion method based on unsupervised operation.

[0145] Specifically, a system, device or equipment equipped with a readable storage medium may be provided, on which a software program code for implementing the functions of any of the above-mentioned embodiments is stored, and a computer or processor of the system, device or equipment reads and executes the instructions stored in the readable storage medium. In this case, the program code read from the readable medium itself can implement the functions of any of the above-mentioned embodiments, so the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of the present invention.

[0146] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0147] Although the above describes the specific implementation methods of the present invention, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for constructing a multi-focus image fusion model based on unsupervised operation, characterized in that: The following steps are involved: Step 1, collect and acquire a large number of original all-focus images; Step 2, preprocessing the obtained original all-focus image, wherein the preprocessing includes grayscale conversion, and normalizing the obtained gray image to obtain an image tensor, and constructing a training set and a test set; Step 3, based on the encoder-decoder network structure, construct an unsupervised dual-channel multi-focus image fusion model UDC MFIF model; the UDC MFIF model includes an encoder, a decoder, an activity level calculation module and a fusion module; The encoder adopts a dual-branch channel encoder network structure for input, the two branches are detail branch and semantic branch, the weights of the two branches are not shared, and only the channel dimension feature splicing is performed when entering the decoder; The detail branch and the semantic branch receive the same input image to extract rich detail information and semantic information of the image respectively; the input of the decoder is the joint feature of splicing the detail information and semantic information in the channel dimension to reconstruct the image; the activity level calculation module includes processing unit and a morphological processing unit; the activity level calculation module is by using The processing unit is used to calculate the feature tensor output by the model encoder, and the maximum feature map fusion strategy is used to obtain the initial decision map; the morphological processing unit is used to process the initial decision map to obtain the final decision map; The processing unit calculates the depth features by Laplace correction and obtains the initial decision graph; the morphological processing unit optimizes the decision graph by opening operation, closing operation and removing small holes; the fusion module uses the decision graph obtained after processing by the above-mentioned activity level calculation module and the input image of different focusing conditions under the same viewing angle to perform weighted product summation of corresponding pixels to obtain the final fused image; Step 4: Use the training set and the test set to train the encoder and decoder to minimize the difference between the decoder's reconstructed image and the encoder's input image, thereby completing the construction of the overall model.

2. The method for constructing an unsupervised multi-focus image fusion model according to claim 1, characterized in that: The grayscale conversion in step 2 is specifically as follows: The original image Convert the color image into a gray image , and then the gray image is uniformly adjusted to , then convert the image into a tensor and input it into the network. The formula is as follows: For gray images Each pixel value in , which is normalized to arrive interval, and finally get the image tensor.

3. The method for constructing an unsupervised multi-focus image fusion model according to claim 1, characterized in that: The encoder includes two feature extraction parts: detail branch and semantic branch; the detail branch is used to extract detail feature information in the image, and the detail branch includes a shallow feature extraction module C1 and 4 EC detail feature extraction modules. The shallow feature extraction module C1 includes two-dimensional convolution, The activation function and batch normalization layer are used to extract the shallow features of the image without changing the size of the input features, so as to facilitate the subsequent extraction of detail features and speech features; the EC detail feature extraction module is used to extract detail features; the semantic branch includes 1 shallow feature extraction module C1, 3 CBEC semantic feature extraction modules and 1 upsampling module, wherein the shallow feature extraction module is the same as that in the detail branch; the CBEC semantic feature extraction module is used to extract semantic features; the upsampling module is used to make the size of the semantic feature map consistent with the size of the detail feature map, so as to facilitate the final splicing.

4. The method for constructing an unsupervised multi-focus image fusion model according to claim 3, characterized in that: The EC detail feature extraction module consists of a two-dimensional convolution, an activation function, a batch normalization layer, and a channel attention mechanism ECA module, which uses dense links to focus on shallow features; The CBEC semantic feature extraction module is composed of a two-dimensional convolution, an activation function, a batch normalization layer, and a convolutional attention mechanism CBAM module. The step size of the two-dimensional convolution in the CBEC semantic feature extraction module is 2. After the module, the size of the feature map of the image is halved. By continuously downsampling, the image loses a large number of detail features, thereby extracting semantic features. The upsampling module is used to make the size of the semantic feature map consistent with the size of the detail feature map to facilitate the final splicing; After two branches, we get the detailed features and semantic features , concatenate these two feature tensors in the channel dimension and After that, we get a new feature tensor , the number of channels is and The sum of the number of channels, while the height and width remain unchanged; the formula is as follows: in, and It is a detail feature , semantic features and joint features The height and width of It is a detail feature The number of channels, It is a semantic feature The number of channels, It is a joint feature The number of channels.

5. The method for constructing an unsupervised multi-focus image fusion model according to claim 1, characterized in that: The specific data processing process of the activity level calculation module is as follows: The input size of n images is Pictures The n feature tensors are extracted through the encoder in sequence, and then the The processing unit calculates n The value is used as the feature map reflecting the image information, and then n images of size The feature maps are spliced ​​in three dimensions into a size of The mosaic diagram, and then along The best focus pixel point at the same position in the feature map of each layer is selected in the axial direction. All focus infographics in Direction The point with the largest value is the best pixel, and the best focus point is assigned a value of 1, and the corresponding points on the other layers are assigned a value of 0 to obtain n pixels of size Decision diagram ,in Number the source images; The morphological processing unit is then used to output the final decision graph ; First, use the open operation of the small disk structure to process To eliminate most of the noise in the background area; secondly, the small hole removal strategy is used to eliminate the remaining noise in the background area; finally, the noise in the background area is removed by closing operation and small hole removal strategy to obtain the final decision graph .

6. The method for constructing an unsupervised multi-focus image fusion model according to claim 1, characterized in that: The fusion module performs weighted fusion on each layer of source images according to the decision graph to obtain the final fusion graph F, which is expressed as: in For the An initial decision diagram, For the The final decision diagram, For the source images, is the pixel coordinate, F is the final fused image.

7. The method for constructing an unsupervised multi-focus image fusion model according to claim 1, characterized in that: In step 4, the training target is to minimize the difference between the decoder's reconstructed image and the encoder's input image. The calculation formula is as follows: in is the joint loss, i.e., the minimum difference between the decoder’s reconstructed image and the encoder’s input image, is the structural similarity loss, is the gradient loss, is the pixel loss; is the parameter weight for balancing the loss.

8. An unsupervised multi-focus image fusion method, characterized in that: The process includes: Get n images with different focal points in the same field of view; Convert n images to grayscale and normalize them to get image tensor; Inputting the image tensors of n images into the multi-focus image fusion model constructed by the construction method according to any one of claims 1 to 7; The model generates and outputs an all-in-focus image.

9. An unsupervised multi-focus image fusion device, characterized in that: The device includes at least one processor and at least one memory, the processor and the memory are coupled; the memory stores a computer execution program of a multi-focus image fusion model constructed by the construction method according to any one of claims 1 to 7; when the processor executes the computer execution program stored in the memory, the processor executes a multi-focus image fusion method based on unsupervised operation.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer execution program of the multi-focus image fusion model constructed by the construction method according to any one of claims 1 to 7. When the computer execution program is executed by the processor, the processor executes a multi-focus image fusion method based on unsupervised operation.

Images