A self-supervised multi-focus image fusion method and model construction method

Abstract

The present invention provides a self-supervised multi-focus image fusion method and a model construction method thereof, and belongs to the field of computer image processing technology. First, the original image data pair is obtained, and a noise image is randomly sampled from a uniform distribution; the gradient information of the original image data pair is calculated and an initial focus score map is obtained; based on the U-Net structure, a multi-focus image fusion model jointly optimized by a D-Net network and an F-Net network is designed; the noise image data is input, and an initial decision map is generated through a D-Net network under the guidance of the initial focus score map; the original image data pair is input, and a final fused image is generated through an F-Net network under the guidance of the initial decision map; at the same time, an overall loss function is designed, model training is performed, a final model is obtained, and the model is used for image fusion output. The model can predict a clear fused image without supervised information and a large-scale image set.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer image processing, and in particular relates to a self-supervised multi-focus image fusion method and a model construction method thereof. Background Art

[0002] In the process of image processing, a fully focused image is very necessary. However, due to the limitation of camera depth of field, a fully focused image is often not obtained. Multi-focus image fusion aims to solve the problem that the image cannot be fully focused due to the limited depth of field (DOF) of the camera, and it plays a vital role in various image tasks. For decades, researchers have proposed various methods for multi-focus image fusion. Traditional multi-focus image fusion methods can be roughly divided into two categories: based on the spatial domain and based on the transform domain. Regarding transform domain-based algorithms, the general idea is to transform the source image into another feature domain. Generally speaking, the fusion process includes three steps: first, the source image is decomposed into multi-scale sub-images; then, these different levels of decomposition coefficients are fused according to certain fusion criteria; finally, the fusion result can be generated by using the inverse transform. The choice of transform domain and the fusion criterion are two important factors for transform domain-based algorithms. Transform domain-based methods usually cannot obtain fully focused images by generating decision maps, and there is still some color difference compared with the focused area in the source image. The spatial domain-based method is to fuse images in the spatial domain, which can be further divided into three categories: pixel-based, block-based, and region-based. The patch-based strategy widely used in spatial domain based methods leads to detail loss near boundaries, which can easily blur both small in-focus and out-of-focus regions as well as boundaries.

[0003] Although these recent advanced algorithms have achieved attractive performance, they use hand-crafted features and fusion criteria to complete the fusion task, which limits the fusion effect because it is almost impossible to combine all the necessary factors for an ideal design from a certain perspective. In recent years, with the development of deep learning in the field of computer vision, many researchers have used deep learning methods to solve the problem of multi-focus image fusion. The mainstream frameworks based on deep learning methods include convolutional neural networks and generative adversarial networks. Among them, the framework based on convolutional neural networks can generate preliminary decision graphs by learning with training datasets, or train convolutional neural networks with multi-scale input methods. Although the performance of these methods is impressive, the results also show some misjudgment near the boundary between the focused and defocused areas. The main reason behind this phenomenon is that it does not consider the matching relationship between the decision graph and the original image. Later, multi-level deep supervised convolutional neural network (ML-CNN) was proposed, which combines multi-level visual features for multi-focus image fusion. However, due to the lack of accurate focus area detection, they may produce some blurry effects, and the final fused image cannot meet people's actual needs, which will hinder its wide application in subsequent image processing tasks. Summary of the invention

[0004] In view of the above problems, the first aspect of the present invention provides a method for constructing a self-supervised multi-focus image fusion model, comprising the following steps:

[0005] Step 1: Get the source images containing different focus areas, construct them into raw image data pairs, and randomly sample the input noise images from a uniform distribution. ;

[0006] Step 2: Calculate the gradient information of the acquired original image data pair and obtain the initial focus score map , used to guide the D-Net network to generate an initial decision graph;

[0007] Step 3: Based on the traditional U-Net network structure, a multi-focus image fusion model is designed which is jointly optimized by a D-Net network and an F-Net network. Input the noisy image , in the initial focus score map The initial decision graph is generated through the D-Net network under the guidance of ; Input the original image data pair, in the initial decision graph The final fusion image is generated through the F-Net network under the guidance of ;

[0008] Among them, at the output end of the F-Net network, an early-stop strategy block based on the variance trend of the detection-generated sequence is adopted to achieve the goal of stopping when the image recovers to the best performance;

[0009] Step 4: Design the overall loss function , including the introduction of a guided loss function in the D-Net network And introduce the reconstruction loss function into the overall network architecture , train the multi-focus image fusion model and obtain the final multi-focus image fusion model.

[0010] Preferably, in the multi-focus image fusion model, the upper layer D-Net network adopts a three-layer basic U-Net model, and the lower layer F-Net network adopts a five-layer basic U-Net model; the basic U-Net model includes an encoder, a jump connection layer, and a decoder; the encoder includes a convolution layer, a normalization layer, and a linear rectification unit for feature extraction of multi-focus images; the jump connection layer includes a convolution layer, a normalization layer, and a linear rectification unit, which are used to connect the feature map of the corresponding position in the encoder to the layer at the same position in the decoder to retain rich feature information; the decoder includes a normalization layer, a convolution layer, a linear rectification unit, and an upsampling layer to convert the output of the encoder part into a target output.

[0011] Preferably, the F-Net network adds a feature extraction module SEM-Block in the encoder part to better extract the global and detail features of the image; the D-Net network and the F-Net network use the upsampled signal to convolve with the interpolation filter using transposed convolution to perform Gaussian filtering, control the network's ability to smooth multi-focus images, and achieve the purpose of balancing performance and convergence.

[0012] Preferably, the feature extraction module SEM-Block is composed of a simple parameter-free attention module and a compression and excitation module; the feature extraction module SEM-Block is located in the downsampling of the encoder of the F-Net network;

[0013] The simple parameter-free attention module is an attention module that infers the 3-D attention weights of the feature maps in the layer by optimizing the energy function and finding the importance of each neuron without adding parameters to the original network; the energy function is represented by the following formula:

[0014]

[0015]

[0016]

[0017] in, They are the input feature maps target neurons and other neurons on a single channel in; are the values ​​of the target neuron and other neurons, respectively; are the weight and bias of the linear transformation respectively; is the total number of neurons in a single channel; The target neurons are and other neurons Linear transformation of ;

[0018] The compression and excitation module is added after the simple parameter-free attention. In the compression stage, the module averages all the features of the input original image and extracts the global features. The process is shown as follows:

[0019]

[0020] in, represents the input feature map, represents the extrusion operation, Indicates output;

[0021] In the excitation stage, two fully connected layers are applied to realize the feature information mapping between channels. Then the channel weight is obtained by the activation function Sigmoid. Finally, the weights obtained above are weighted and multiplied by the original feature map to obtain the final feature map.

[0022] Preferably, the D-Net network and the F-Net network perform Gaussian filtering by convolving the upsampled signal with an interpolation filter using transposed convolution, specifically:

[0023] First, upsampling is decomposed into filtering and expansion steps, where for the input No. Channels Fill with zero padding, that is, fill along its rows and columns in Interpolate between pixels zeros, expanding and creating a high-frequency replica of the original input image; to smooth the noisy high frequencies, the upsampled signal is combined with The filtering is performed by convolving the sampled interpolation Gaussian filter, and the Gaussian upsampling is expressed by the following formula:

[0024]

[0025] in, Representative Factor Filling Extension , represents Gaussian filter;

[0026] The ability to learn high- and low-frequency information and the convergence of the network are balanced by controlling the up-sampling Fourier spectrum in the frequency domain. The Fourier spectrum formula is as follows:

[0027]

[0028]

[0029] in, represents Fourier transform, Stands for element-wise product.

[0030] Preferably, the early-stop strategy block based on detecting the running variance trend of the generated sequence is adopted at the output end of the F-Net network. In the process of generating the fused image, the running variance trend of the generated sequence is detected to determine whether the best peak performance of the output image is achieved. The running variance curve will follow the mean square error curve trend of the generated image. The performance of the generated image is determined by determining the running variance trend. The running variance formula is as follows:

[0031]

[0032] in is the initial image, For a given window size; for each step , calculate the variance of the current window and update the variance curve to check whether the variance decreases. If it decreases, reset the stagnation counter; if it stagnates, increase the stagnation counter; when the stagnation counter exceeds the tolerance parameter P, stop the iteration and output the optimal iteration point and the corresponding reconstructed image.

[0033] Preferably, in step 4, the overall loss function is designed Specifically:

[0034]

[0035] in, represents the reconstruction loss function, represents the guidance loss function, To guide the loss weights;

[0036] Introducing a guided loss function in the D-Net network This function helps the D-Net network generate an effective initial decision map through the initial focus score map of the source image , the guiding loss function The formula is as follows:

[0037]

[0038] in represents the D-Net network, , They represent the initial focus score map of the source image, , represents the input noise randomly sampled from a uniform distribution, is 1 norm;

[0039] Introducing reconstruction loss function into the overall network architecture To ensure that the information generated by the network comes from the source image pair, the reconstruction loss function The formula is as follows:

[0040]

[0041] in represents the F-Net network, represents a multi-focus source image pair, represents the splicing operation along the channel, is the initial decision graph; the two loss functions guide the network to generate the final fused image by guiding the loss function.

[0042] The second aspect of the present invention also provides a self-supervised multi-focus image fusion method, comprising the following process:

[0043] Acquire a multi-focus image data pair including different focus areas;

[0044] The multi-focus image pairs and the randomly sampled input noise during training Input into the multi-focus image fusion model constructed by the construction method described in the first aspect;

[0045] Output the fused clear all-focused fused image.

[0046] The third aspect of the present invention also provides a self-supervised multi-focus image fusion device, which includes at least one processor and at least one memory, and the processor and the memory are coupled; the memory stores a computer execution program of the multi-focus image fusion model constructed by the construction method described in the first aspect; when the processor executes the computer execution program stored in the memory, the processor executes a self-supervised multi-focus image fusion method.

[0047] The fourth aspect of the present invention also provides a computer-readable storage medium, which stores a computer execution program of the multi-focus image fusion model constructed by the construction method described in the first aspect. When the computer execution program is executed by a processor, the processor executes a self-supervised multi-focus image fusion method.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] The present invention uses a self-supervised multi-focus image fusion model to implement the multi-focus image fusion process. The model can predict a clear fused image without supervised information and a large-scale image set. Inspired by DIP, two generative networks are applied to simultaneously estimate the clear fused image and the target image prior of the decision graph. The method based on decision graph estimation can well retain the information of the source image, and the method based on fused image generation can provide good visual effects. The invented SEM-Block attention module is used for feature extraction. Compared with the latest U-Net network-based model, the present invention more fully extracts features and retains the detailed information of the features, and gives full play to the learning ability and convergence of the Gaussian filter upsampling balanced network for high and low frequency information. Compared with the currently best multi-focus image fusion model based on the U-Net network, the present invention achieves better fusion effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is the original image data pair in Example 1.

[0051] Figure 2 is the noise image data in Example 1.

[0052] Figure 3 This is a schematic diagram of the overall structure of the multi-focus image fusion model proposed in the present invention.

[0053] Figure 4 This is the structure diagram of the feature extraction module SEM-Block.

[0054] Figure 5 This is a simple parameter-free attention structure diagram.

[0055] Figure 6 This is the structure diagram of the compression and excitation module Squeeze-and-Excitation.

[0056] Figure 7 This is a simplified structural diagram of the multi-focus image fusion device in Example 2. DETAILED DESCRIPTION

[0057] The invention will be further described below in conjunction with specific embodiments.

[0058] Embodiment 1:

[0059] The present invention designs an efficient self-supervised multi-focus image fusion method, which includes the following steps:

[0060] Step 1: Get the source images containing different focus areas, construct them into raw image data pairs, and randomly sample the input noise images from a uniform distribution. ;

[0061] Step 2: Calculate the gradient information of the acquired original image data pair and obtain the initial focus score map , used to guide the D-Net network to generate an initial decision graph;

[0062] Step 3: Based on the traditional U-Net network structure, a multi-focus image fusion model is designed which is jointly optimized by a D-Net network and an F-Net network. Input the noisy image data , in the initial focus score map The initial decision graph is generated through the D-Net network under the guidance of ; Input the original image data pair, in the initial decision graph The final fusion image is generated through the F-Net network under the guidance of ;

[0063] Among them, at the output end of the F-Net network, an early-stop strategy block based on the variance trend of the detection-generated sequence is adopted to achieve the goal of stopping when the image recovers to the best performance;

[0064] Step 4: Design the overall loss function , including the introduction of a guided loss function in the D-Net network And introduce the reconstruction loss function into the overall network architecture , train the multi-focus image fusion model to obtain the final multi-focus image fusion model;

[0065] Step 5: Use the trained multi-focus image fusion model to complete the transmission from the source image to the fused image.

[0066] This embodiment conducts experiments on the Lytro and Real_MFF datasets. The present invention can also be applied to color images taken by individuals, which need to be manually segmented to obtain segmented images.

[0067] 1. Data Acquisition

[0068] Get the source images containing different focus areas, construct them into raw image data pairs, and randomly sample the input noise images from a uniform distribution Different focus areas generally refer to the distant focus and the near focus, which are constructed as the original image data pairs of distant and near focus, such as Figure 1 shown.

[0069] The experiment of this embodiment is carried out by downloading the public multi-focus image fusion test sets Lytro and Real_MFF. The Lytro dataset is taken by a light field camera and is widely used in the field of multi-focus image fusion. It contains 20 pairs of images with different focuses. The Real_MFF dataset is generated by light field images, which contains 710 pairs of source images and corresponding ground truth images. Among them, the input noise refers to generating two random numbers uniformly distributed between 0 and 1, and then using these random numbers as the pixel values ​​of the image to realize two noisy images. The input noise of this embodiment is as follows Figure 2 shown.

[0070] 2. Gradient calculation

[0071] The gradient of the acquired original image data is calculated to obtain the initial focus score map, which is used to guide the generation of the initial decision map of the D-Net network. The specific formula for generating the initial focus score map is as follows:

[0072]

[0073] in represents the gradient operation, Represents the initial focus score map of the source image, such as Figure 3 As shown in FIG. , this embodiment includes and , , is a pair of multi-focus original image data. The specific operation is to perform a two-dimensional convolution operation with a step size of 1 and a padding of 1 on the input and the Laplacian filter. This is the prior art and will not be described in detail.

[0074] 3. Constructing a multi-focus image fusion model

[0075] In this invention, based on the idea of ​​DIP network structure, a network structure jointly optimized by D-Net network and F-Net network based on U-Net network model is used to input multi-focus source image pairs and noise. The input random noise image is in the initial focus score map. The initial decision graph is generated through the D-Net network under the guidance of ; Input multi-focus source image pairs in the initial decision graph generated by the D-Net network The final fusion image is generated through the F-Net network under the guidance of The obtained fully focused image The specific formula is:

[0076]

[0077] in, represents element-wise product, is a multi-focus source image pair, is the initial decision diagram.

[0078] The two networks guide the generation of fused images through guiding loss functions and reconstruction loss functions. Each network can receive different input images to enrich the diversity of feature extraction. A feature extraction module (SEM-Block) is added to the encoder of the F-Net network. This module can retain more detailed features while extracting global feature information, so that it can distinguish the boundaries of focus and defocus, thereby better solving the problem of misjudgment of tiny focus areas. The Gaussian upsampling method is used in the decoders of the two networks to give full play to the learning ability and convergence of the Gaussian filter upsampling balance network for high and low frequency information. The early-stop strategy block (early-stop) is added at the output end of the F-Net network to detect the running variance trend and determine when the network reaches the optimal fusion effect image. The specific structure of the network model is as follows: Figure 3 As shown, it includes D-Net network and F-Net network.

[0079] The D-Net network includes an encoder, a jump connection layer and a decoder; the encoder includes a convolution layer, a normalization layer and a linear rectifier unit, which is used for feature extraction of multi-focus images, including a two-dimensional convolution with a convolution kernel size of 3, a step size of 2, a padding of 1, an input channel number of 8, and an output channel number of 128, and its corresponding group normalization and leaky linear rectifier unit (Leaky ReLU), two two-dimensional convolutions with a convolution kernel size of 3, a step size of 2, a padding of 1, an input channel number of 128, and an output channel number of 128, and their corresponding group normalization and leaky linear rectifier unit (Leaky ReLU);

[0080] The jump connection layer includes a convolution layer, a normalization layer, and a linear rectifier unit, which is used to connect the feature map at the corresponding position in the encoder to the layer at the same position in the decoder to retain rich feature information. It includes a two-dimensional convolution with a convolution kernel size of 1, a padding of 1, an input channel number of 2, and an output channel number of 16, and its corresponding group normalization and leaky linear rectifier unit (Leaky ReLU), two two-dimensional convolutions with a convolution kernel size of 1, a padding of 1, an input channel number of 16, and an output channel number of 16, and their corresponding group normalization and leaky linear rectifier unit (Leaky ReLU);

[0081] The decoder includes a normalization layer, a convolution layer, a linear rectifier unit, and an upsampling layer to convert the output of the encoder part into a target output. It includes a two-dimensional convolution with a kernel size of 3, a stride of 2, a padding of 1, an input channel number of 144, and an output channel number of 1, and its corresponding group normalization and leaky linear rectifier unit (Leaky ReLU), two two-dimensional convolutions with a kernel size of 3, a stride of 2, a padding of 1, an input channel number of 144, and an output channel number of 128, and their corresponding group normalization and leaky linear rectifier unit (Leaky ReLU), and the upsampling layer is composed of a transposed convolution with a kernel size of 5, a stride of 2, a padding of 2, an output padding of 1, an input channel number of 128, and an output channel number of 128;

[0082] The F-Net network includes an encoder, a feature extraction module (SEM-Block), a jump connection layer and a decoder; the encoder includes a convolution layer, a normalization layer and a linear rectifier unit for feature extraction of multi-focus images, including a two-dimensional convolution with a convolution kernel size of 3, a step size of 2, a padding of 1, an input channel number of 2, and an output channel number of 128, and its corresponding group normalization and leaky linear rectifier unit (Leaky ReLU), four convolution kernels with a size of 3, a step size of 2, a padding of 1, an input channel number of 128, and an output channel number of 128, and their corresponding group normalization and leaky linear rectifier unit (LeakyReLU);

[0083] The jump connection layer includes a convolution layer, a normalization layer, and a linear rectifier unit, which is used to connect the feature map at the corresponding position in the encoder to the layer at the same position in the decoder to retain rich feature information. It includes a two-dimensional convolution with a convolution kernel size of 1, a padding of 1, an input channel number of 2, and an output channel number of 16, and its corresponding group normalization and leaky linear rectifier unit (Leaky ReLU), four two-dimensional convolutions with a convolution kernel size of 1, a padding of 1, an input channel number of 2, and an output channel number of 16, and their corresponding group normalization and leaky linear rectifier unit (Leaky ReLU);

[0084] The decoder includes a normalization layer, a convolution layer, a linear rectifier unit, and an upsampling layer to convert the output of the encoder part into a target output. It includes a two-dimensional convolution with a kernel size of 3, a stride of 2, a padding of 1, an input channel number of 144, and an output channel number of 1, and its corresponding group normalization and leaky linear rectifier unit (Leaky ReLU), four two-dimensional convolutions with a kernel size of 3, a stride of 2, a padding of 1, an input channel number of 144, and an output channel number of 128, and their corresponding group normalization and leaky linear rectifier unit (Leaky ReLU), and the upsampling layer is composed of a transposed convolution with a kernel size of 5, a stride of 2, a padding of 2, an output padding of 1, an input channel number of 128, and an output channel number of 128;

[0085] The overall structure of the feature extraction module (SEM-Block) is as follows Figure 4 As shown in the figure, it consists of a simple parameter-free attention and a squeeze and excitation module. The simple parameter-free attention uses the mean and variance of the input features to calculate the attention weights, and uses these weights to adjust the amplitude of the input features, and finally multiplies the input features by the attention weights. The attention weights are normalized by the activation function (Sigmoid). This mechanism can help the network better focus on important features and improve the ability to express features. The squeeze and excitation module (Squeeze-and-Excitation) first uses adaptive average pooling to reduce the input feature map to , and then the weight of each channel is calculated through a sequence of fully connected layers containing two linear layers and two leaky linear rectifier units (Leaky ReLU), and the input feature map is reweighted using the calculated channel weights to highlight important features and suppress unimportant features.

[0086] The simple parameter-free attention structure is as follows Figure 5 As shown in the figure, different from the traditional 1-D and 2-D attention modules, it is based on optimizing the energy function and finding the importance of each neuron to infer the 3-D attention weight of the feature map in the layer, without adding parameters to the original network, so as to efficiently extract the edge information and global information of the multi-focus image. The energy function can be expressed by the following formula:

[0087]

[0088]

[0089]

[0090] The input feature maps are target neurons and other neurons on a single channel in; are the values ​​of the target neuron and other neurons, respectively; are the weight and bias of the linear transformation respectively; is the total number of neurons in a single channel; The target neurons are and other neurons Linear transformation of .

[0091] A Squeeze-and-Excitation module is added after the simple parameter-free attention, such as Figure 6 As shown in the figure, in the compression stage, this module averages all the features of the original input image and extracts global features. The process can be expressed as follows:

[0092]

[0093] in, represents the input feature map, represents the extrusion operation, Indicates output.

[0094] In the excitation stage, two fully connected layers are applied to realize the feature information mapping between channels. Then, the channel weights are obtained through the activation function (Sigmoid). Finally, the weights obtained above are weighted and multiplied by the original feature map to obtain the final feature map.

[0095] The early-stop strategy block determines whether the best peak performance of the output image is achieved by detecting the variance trend of the generated sequence during the generation of the fused image. It first calculates the mean image of the image set, then calculates and stores the variance between each image and the mean image, checks whether the verification loss is optimal based on the current variance, and determines whether to stop training based on the tolerance. This method can dynamically monitor the changes in verification loss during training and decide whether to stop training based on the tolerance set in advance. Specifically:

[0096] The early-stop strategy block at the output of the F-Net network determines whether the best peak performance of the output image is achieved by detecting the running variance trend of the generated sequence during the process of generating the fused image. Since the network only restores the clear image through a single degraded image, if the number of iterations is not limited, the network will eventually fit the degraded image. The network itself has the ability to learn low frequencies first and then high frequencies. Therefore, if the iteration is stopped when the image is detected to be restored to a good degree during the network iteration process, a clear fused image will be obtained, and the calculation time will be greatly reduced and the operation speed will be improved. The early-stop strategy block is a strategy for detecting when the network restores a clear image and stopping the iteration. The initial model quickly learns the ideal visual content, resulting in a rapid decrease in the mean square error curve, so the running variance of the expected image will also decrease rapidly. In short, the running variance curve will follow the trend of the mean square error curve of the generated image. Therefore, the performance of the generated image is judged by judging the running variance trend. The running variance formula is as follows:

[0097]

[0098] in is the initial image, is a given window size. For each step , calculate the variance of the current window and update the variance curve to check whether the variance decreases. If it decreases, reset the stagnation counter. If it stagnates, increase the stagnation counter. When the stagnation counter exceeds the tolerance parameter P, stop the iteration and output the optimal iteration point and the corresponding reconstructed image.

[0099] 4. Loss function design and model training

[0100] In this embodiment, an efficient self-supervised multi-focus image fusion method is implemented on a Linux operating system, with Python 3.8 as the programming language, Pytorch 1.11.0 as the deep learning framework, and CUDA version 11.1.1. Adam is used as the optimizer, and the following settings are used: is 20. After a certain number of iterations (e.g. 800), Set to 0, mainly because it is assumed that in the early stage of fusion, the guidance information of the initial focus score map can enable DIP to find clear areas, while in the later stage, the guidance loss will limit the further optimization of the D-Net network. Set the learning rate to 0.01, the batch size to 1, and the system is trained on an NVIDIA RTX 2080Ti GPU with 11GB memory.

[0101] The self-supervised multi-focus image fusion model built takes minimizing the guidance loss, reconstruction loss and total loss as the training goal during the training process. The calculation formula is as follows:

[0102]

[0103]

[0104]

[0105] in Represents the input source images respectively and ,in Respectively represent D-Net and F-Net networks, To guide the loss weight, 、 is the input noise randomly sampled from a uniform distribution, Indicates splicing operation along the channel represents the initial focus score map of the source image, is 1 norm.

[0106] 5. Experimental Results

[0107] In this embodiment, the fusion effect of the multi-focus image fusion model based on self-supervision of the present invention is compared with the multi-focus image fusion model that has performed well recently on the public multi-focus image test sets Lytro and Real_MFF. The Lytro dataset is shot with a light field camera and is widely used in the field of multi-focus image fusion. It contains 20 pairs of images with different focuses. The Real_MFF dataset is generated by light field images, which contains 710 pairs of source images and corresponding ground truth images. The present invention also conducts an ablation experiment to compare the role of each sub-module in the feature extraction module (SEM-Block) of the present invention to verify the effectiveness of the scheme described in the present invention. The present invention provides 7 commonly used evaluation indicators for image fusion, which are based on image features: , , ; Based on information theory: , ; Based on image structure similarity: , and inspiration from human perception: The larger the value, the higher the score of the evaluation indicator.

[0108] The Lytro dataset contains 20 pairs of images with different focuses. The present invention compares 10 multi-focus image fusion methods. As can be seen from Table 1 (bold indicates the best, single underline is the second best, and double underline is the third best), the network of the present invention is , , The index has reached the optimal level. In general, among the 7 indicators, 3 indicators of the present invention ranked first, 1 ranked second, and 2 ranked third. Compared with other methods, the comprehensive ranking of evaluation indicators on the Lytro test set of the present invention is the highest.

[0109] Table 1 Scores of different multi-focus image fusion algorithms on the Lytro test set

[0110]

[0111] The Real_MFF dataset contains 710 pairs of images with different focuses. The present invention and 10 other multi-focus image fusion methods are tested on the Real_MFF dataset. The scores of the fusion results under 7 evaluation indicators are compared quantitatively, as shown in Table 2. Bold indicates the best, single underline is the second, and double underline is the third. The present invention is , The index has reached the optimal level. Overall, among the 7 indicators, the present invention ranks first in 2 indicators, second in 1 indicator, and third in 3 indicators. Compared with other methods, the present invention has the best comprehensive ranking of evaluation indicators on the Real_MFF test set.

[0112] Table 2 Scores of different multi-focus image fusion algorithms on the Real_MFF test set

[0113]

[0114] In summary, compared with other state-of-the-art models, our proposed method outperforms recent multi-focus image fusion methods on both public test sets.

[0115] Ablation experiment:

[0116] In order to prove the rationality of each part of the detail feature extraction module of the present invention, an ablation experiment comparison was carried out, and the detail feature extraction module was ablated into three components to verify the robustness of the module proposed by the present invention, which are: All of them only undergo simple parameter-free attention; All of them have only been subjected to the Squeeze-and-Excitation module; After simple parameter-free attention and squeeze and excitation modules, as shown in Table 3 below, these three combinations are verified on two public datasets. It can be seen that the fusion effect is not good when there is only simple parameter-free attention or only squeeze and excitation modules. Net1 represents the network with only simple parameter-free attention, Net2 represents the network with only squeeze and excitation modules, and the bold represents the best

[0117] Table 3 Ablation comparison of the scores of seven evaluation indicators of different modules on two test sets

[0118]

[0119] Comparative test:

[0120] In order to prove the rationality of the Gaussian upsampling module of the present invention, a comparative experiment was conducted to compare the Gaussian upsampling method with the common nearest neighbor interpolation upsampling and bilinear interpolation upsampling. As shown in Table 4 below, the three upsampling methods are verified on two public data sets. It can be seen that the fusion effect in the case of nearest neighbor upsampling and bilinear interpolation upsampling is not as good as the effect under the Gaussian upsampling method. Gaussian represents a network using Gaussian upsampling, Nearest represents a network using nearest neighbor interpolation upsampling, Bilinear represents a network using bilinear interpolation upsampling, and bold represents the best.

[0121] Table 4 Comparison of the scores of seven evaluation indicators of different modules on the two test sets

[0122]

[0123] It can be seen that each of the above components contributes to the performance gain of the present invention.

[0124] Embodiment 2:

[0125] like Figure 7As shown, the present invention also provides a self-supervised multi-focus image fusion device, the device includes at least one processor and at least one memory, and also includes a communication interface and an internal bus; a computer execution program is stored in the memory; a computer execution program of a multi-focus image fusion model constructed by the construction method described in Example 1 is stored in the memory; when the processor executes the computer execution program stored in the memory, the processor can execute a self-supervised multi-focus image fusion method. The internal bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus. The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a U disk, a mobile hard disk, a read-only memory, a disk or an optical disk.

[0126] The device may be provided as a terminal, a server or other forms of devices.

[0127] Figure 7 is a block diagram of a device shown as an example. The device may include one or more of the following components: a processing component, a memory, a power component, a multimedia component, an audio component, an input / output (I / O) interface, a sensor component, and a communication component. The processing component generally controls the overall operation of the electronic device, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component may include one or more processors to execute instructions to complete all or part of the steps of the above method. In addition, the processing component may include one or more modules to facilitate interaction between the processing component and other components. For example, the processing component may include a multimedia module to facilitate interaction between the multimedia component and the processing component.

[0128] The memory is configured to store various types of data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0129] The communication component is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0130] In an exemplary embodiment, the electronic device may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above methods.

[0131] Embodiment 3:

[0132] The present invention also provides a computer-readable storage medium, which stores a computer execution program of the multi-focus image fusion model constructed by the construction method described in Example 1. When the computer execution program is executed by a processor, the processor can execute a self-supervised multi-focus image fusion method.

[0133] Specifically, a system, device or equipment equipped with a readable storage medium may be provided, on which a software program code for implementing the functions of any of the above-mentioned embodiments is stored, and a computer or processor of the system, device or equipment reads and executes the instructions stored in the readable storage medium. In this case, the program code read from the readable medium itself can implement the functions of any of the above-mentioned embodiments, so the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of the present invention.

[0134] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk (such as CD-ROM, CD-R, CD-RW, DVD-20 ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tape, etc. The storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0135] It should be understood that the storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in a terminal or server.

[0136] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0137] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0138] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0139] Although the above describes the specific implementation methods of the present invention, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A self-supervised multi-focus image fusion model construction method, characterized in that: The following steps are involved: Step 1: Get the source images containing different focus areas, construct them into raw image data pairs, and randomly sample the input noise images from a uniform distribution. ; Step 2: Calculate the gradient information of the acquired original image data pair and obtain the initial focus score map , used to guide the D-Net network to generate an initial decision graph; Step 3: Based on the traditional U-Net network structure, a multi-focus image fusion model is designed which is jointly optimized by a D-Net network and an F-Net network. Input the noisy image data , in the initial focus score map The initial decision graph is generated through the D-Net network under the guidance of ; Input the original image data pair, in the initial decision graph The final fusion image is generated through the F-Net network under the guidance of ; Among them, at the output end of the F-Net network, an early-stop strategy block based on the variance trend of the detection-generated sequence is adopted to achieve the goal of stopping when the image recovers to the best performance; Step 4: Design the overall loss function , including the introduction of a guided loss function in the D-Net network And introduce the reconstruction loss function into the overall network architecture , train the multi-focus image fusion model to obtain the final multi-focus image fusion model; Designing the overall loss function Specifically: in, represents the reconstruction loss function, represents the guidance loss function, To guide the loss weights; Introducing a guided loss function in the D-Net network This function helps the D-Net network generate an effective initial decision map through the initial focus score map of the source image , the guiding loss function The formula is as follows: in represents the D-Net network, , They represent the initial focus score map of the source image, , represents the input noise randomly sampled from a uniform distribution, is 1 norm; Introducing reconstruction loss function into the overall network architecture To ensure that the information generated by the network comes from the source image pair, the reconstruction loss function The formula is as follows: in represents the F-Net network, represents a multi-focus source image pair, represents the splicing operation along the channel, is the initial decision graph; reconstruct the loss function And the guidance loss function By guiding the overall loss function To guide the network to generate the final fused image.

2. The method for constructing a self-supervised multi-focus image fusion model according to claim 1, characterized in that: The multi-focus image fusion model has an upper D-Net network that adopts a three-layer basic U-Net model, and a lower F-Net network that adopts a five-layer basic U-Net model; the basic U-Net model includes an encoder, a jump connection layer, and a decoder; the encoder includes a convolution layer, a normalization layer, and a linear rectification unit for feature extraction of multi-focus images; the jump connection layer includes a convolution layer, a normalization layer, and a linear rectification unit for connecting feature maps at corresponding positions in the encoder to layers at the same position in the decoder to retain rich feature information; the decoder includes a normalization layer, a convolution layer, a linear rectification unit, and an upsampling layer to convert the output of the encoder part into a target output.

3. A self-supervised multi-focus image fusion model construction method as claimed in claim 2, characterized in that: The F-Net network adds a feature extraction module SEM-Block in the encoder part to better extract the global and detail features of the image; the D-Net network and the F-Net network use the upsampled signal to convolve with the interpolation filter using transposed convolution to perform Gaussian filtering, control the network's ability to smooth multi-focus images, and achieve the purpose of balancing performance and convergence.

4. The method for constructing a self-supervised multi-focus image fusion model according to claim 3, characterized in that: The feature extraction module SEM-Block is composed of a simple parameter-free attention module and a compression and excitation module; the feature extraction module SEM-Block is located in the downsampling of the encoder of the F-Net network; The simple parameter-free attention module is an attention module that infers the 3-D attention weights of the feature maps in the layer by optimizing the energy function and finding the importance of each neuron without adding parameters to the original network; the energy function is represented by the following formula: in, They are the input feature maps target neurons and other neurons on a single channel in; are the values ​​of the target neuron and other neurons, respectively; are the weight and bias of the linear transformation respectively; is the total number of neurons in a single channel; The target neurons are and other neurons Linear transformation of ; The compression and excitation module is added after the simple parameter-free attention. In the compression stage, the module averages all the features of the input original image and extracts the global features. The process is shown as follows: in, represents the input feature map, represents the extrusion operation, Indicates output; In the excitation stage, two fully connected layers are applied to realize the feature information mapping between channels. Then the channel weight is obtained by the activation function Sigmoid. Finally, the weights obtained above are weighted and multiplied by the original feature map to obtain the final feature map.

5. The method for constructing a self-supervised multi-focus image fusion model according to claim 3, characterized in that: The D-Net network and the F-Net network perform Gaussian filtering by convolving the upsampled signal with an interpolation filter using transposed convolution, specifically: First, upsampling is decomposed into filtering and expansion steps, where for the input No. Channels Fill with zero padding, that is, fill along its rows and columns in Interpolate between pixels zeros, expanding and creating a high-frequency replica of the original input image; to smooth the noisy high frequencies, the upsampled signal is combined with The filtering is performed by convolving the sampled interpolation Gaussian filter, and the Gaussian upsampling is expressed by the following formula: in, Representative Factor Filling Extension , represents Gaussian filter; The ability to learn high- and low-frequency information and the convergence of the network are balanced by controlling the up-sampling Fourier spectrum in the frequency domain. The Fourier spectrum formula is as follows: in, represents Fourier transform, Stands for element-wise product.

6. The method for constructing a self-supervised multi-focus image fusion model according to claim 1, characterized in that: The early-stop strategy block based on detecting the running variance trend of the generated sequence is adopted at the output end of the F-Net network. In the process of generating the fused image, the running variance trend of the generated sequence is detected to determine whether the best peak performance of the output image is achieved. The running variance curve will follow the mean square error curve trend of the generated image. The performance of the generated image is determined by judging the running variance trend. The running variance formula is as follows: in is the initial image, For a given window size; for each step , calculate the variance of the current window and update the variance curve to check whether the variance decreases. If it decreases, reset the stagnation counter. If it stagnates, increase the stagnation counter; When the stagnation counter exceeds the tolerance parameter P, the iteration is stopped and the optimal iteration point and the corresponding reconstructed image are output.

7. A self-supervised multi-focus image fusion method, characterized in that: The process includes: Acquire a multi-focus image data pair including different focus areas; The multi-focus image pairs and the randomly sampled input noise during training Input into the multi-focus image fusion model constructed by the construction method according to any one of claims 1 to 6; Output the fused clear all-focused fused image.

8. A self-supervised multi-focus image fusion device, characterized in that: The device includes at least one processor and at least one memory, the processor and the memory are coupled; the memory stores a computer execution program of a multi-focus image fusion model constructed by the construction method according to any one of claims 1 to 6; when the processor executes the computer execution program stored in the memory, the processor executes a self-supervised multi-focus image fusion method.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer execution program of the multi-focus image fusion model constructed by the construction method according to any one of claims 1 to 6. When the computer execution program is executed by the processor, the processor executes a self-supervised multi-focus image fusion method.

Images