Unsupervised multi-focus image fusion method and system based on attention mechanism
The unsupervised multi-focus image fusion method addresses blurred boundaries and information loss by using an attention mechanism, enhancing image fusion with clear edges and improved texture details through iterative training and gradient map analysis.
Patent Information
- Application Number
- CN202411211367.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-12-31
AI Technical Summary
Existing deep learning-based multi-focus image fusion methods face challenges in accurately classifying focused and defocused areas, leading to blurred boundaries and information loss, requiring manual decision graph construction and post-processing, which increases complexity and workload.
An unsupervised multi-focus image fusion method using an attention mechanism, involving a generator and discriminator, with Gaussian Laplace transform to generate gradient maps and a joint gradient map, and iterative training to enhance image fusion with clear edges.
The method effectively extracts global and local image information, focusing on different features at the same scale, resulting in a fused image with clear edges and improved texture details, outperforming existing methods in distortion and detail retention.
Smart Images
Figure CN119228658A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and more specifically, to an unsupervised multi-focus image fusion method and system based on an attention mechanism. Background Art
[0002] Due to the limitation of the depth of field of optical lenses, imaging devices cannot obtain clear images of all objects in the same scene. To solve the above problem, multi-focus image fusion (MFIF) technology has become an effective means as an image enhancement method. It can fuse different focus areas of the image to obtain a fully focused image, which is more suitable for human observation or computer processing. It has wide application value in digital imaging, microscopic imaging, holographic imaging and other fields.
[0003] At present, MFIF algorithms can be mainly divided into three categories: transform domain-based, spatial domain-based, and deep learning-based. The first two categories belong to traditional methods. The disadvantage is that the existing methods usually require manual design of activity level measurement and fusion rules, but the deep learning-based method can avoid these problems. The trained network can obtain the relationship between the source image and the fused image, automatically extract feature information from the image and fuse it, so there is no need for manual transformation and activity level measurement. Although the existing deep learning methods have the advantages mentioned above, there are still three shortcomings: First, it is necessary to generate a decision graph to perform MFIF, but it is usually not possible to accurately classify the focused and defocused areas, resulting in blurred boundaries and information loss. Second, post-processing is required when generating the decision graph, such as consistency check, which increases the complexity of the algorithm. Third, it is necessary to manually construct a decision graph as a true label to train the network, which undoubtedly increases the workload. Summary of the invention
[0004] In order to overcome the above-mentioned defects of blurred boundaries and information loss in focused and defocused areas, the present invention provides an unsupervised multi-focus image fusion method and system based on an attention mechanism.
[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:
[0006] The present invention provides an unsupervised multi-focus image fusion method based on an attention mechanism, comprising:
[0007] S1: Acquire a source image data set, including a corresponding first source image and a second source image;
[0008] S2: construct a multi-focus image fusion model, the model including a generator and a discriminator;
[0009] S3: inputting the first source image and the second source image into a generator to obtain a fused image;
[0010] S4: performing Gaussian Laplace transform on the first source image, the second source image and the fused image to obtain a first gradient map, a second gradient map and a fused gradient map; and obtaining a joint gradient map based on the first gradient map and the second gradient map;
[0011] S5: inputting the joint gradient map and the fused gradient map into a discriminator to obtain a predicted fused image;
[0012] S6: setting a discrimination loss function, calculating the discrimination loss function value based on the fused gradient map and the predicted fused image, and performing back propagation until a preset discrimination loss threshold is reached, and transmitting the corresponding discriminator parameters to the generator;
[0013] S7: setting a generation loss function, repeating steps S3-S6 until a preset generation loss function value is reached, and obtaining a trained multi-focus image fusion model;
[0014] S8: Obtain the image to be fused and input it into the trained multi-focus image fusion model to obtain a fused image.
[0015] Preferably, the generator includes a first convolution layer, a second convolution layer, a third convolution layer, a fourth convolution layer, a fifth convolution layer, a first activation layer, a second activation layer, a third activation layer, a fourth activation layer, a first CAM layer, a second CAM layer, a third CAM layer, a fourth CAM layer, a fifth CAM layer, a sixth CAM layer, a seventh CAM layer, an eighth CAM layer, a first connection layer, a second connection layer, a third connection layer, a fourth connection layer, a fifth connection layer, a sixth connection layer and a seventh connection layer;
[0016] The output end of the first convolutional layer is connected to the input end of the first activation layer; the output end of the first activation layer is connected to the input end of the second convolutional layer; the output end of the second convolutional layer is connected to the input end of the second activation layer; the output end of the second activation layer is connected to the input end of the first CAM layer; the output end of the first CAM layer is connected to the input end of the second connection layer; the output end of the second connection layer is connected to the input end of the fourth CAM layer; the output end of the fourth CAM layer is connected to the input end of the fourth connection layer; the output end of the fourth connection layer is connected to the input end of the seventh CAM layer; the output end of the seventh CAM layer is connected to the input end of the seventh connection layer;
[0017] The output end of the third convolutional layer is connected to the input end of the third activation layer; the output end of the third activation layer is connected to the input end of the fourth convolutional layer; the output end of the fourth convolutional layer is connected to the input end of the fourth activation layer; the output end of the fourth activation layer is connected to the input end of the second CAM layer; the output end of the second CAM layer is connected to the input end of the third connection layer; the output end of the third connection layer is connected to the input end of the fifth CAM layer; the output end of the fifth CAM layer is connected to the input end of the sixth connection layer; the output end of the sixth connection layer is connected to the input end of the eighth CAM layer; the output end of the eighth CAM layer is connected to the input end of the seventh connection layer;
[0018] The output end of the second activation layer is connected to the input end of the second connection layer; the output end of the second activation layer is connected to the input end of the fifth connection layer; the output end of the fourth activation layer is connected to the input end of the third connection layer; the output end of the fourth activation layer is connected to the input end of the sixth connection layer; the output ends of the second activation layer and the fourth activation layer are both connected to the input end of the seventh connection layer;
[0019] The output ends of the first CAM layer and the second CAM layer are both connected to the input end of the first connection layer; the output end of the first connection layer is connected to the input end of the third CAM layer; the output end of the first CAM layer is connected to the input end of the fifth connection layer; the output end of the second CAM layer is connected to the input end of the sixth connection layer; the output ends of the first CAM layer and the second CAM layer are both connected to the input end of the seventh connection layer; the output end of the third CAM layer is connected to the input ends of the second connection layer and the third connection layer;
[0020] The output ends of the fourth CAM layer and the fifth CAM layer are both connected to the input end of the seventh connection layer; the output ends of the fourth CAM layer and the fifth CAM layer are both connected to the input end of the fourth connection layer; the output end of the fourth connection layer is connected to the input end of the sixth CAM layer; the output end of the sixth CAM layer is connected to the input ends of the fifth connection layer and the sixth connection layer;
[0021] The output end of the seventh connection layer is connected to the input end of the fifth convolutional layer; the fifth convolutional layer outputs a fused image.
[0022] Preferably, the first CAM layer, the second CAM layer, the third CAM layer, the fourth CAM layer, the fifth CAM layer, the sixth CAM layer, the seventh CAM layer, and the eighth CAM layer have the same structure, and all include a fifth activation layer, a pixel attention layer, a channel attention layer, a spatial attention layer, an eighth connection layer, and a sixth convolutional layer;
[0023] The output end of the fifth activation layer is connected to the input end of the pixel attention layer; the output end of the pixel attention layer is connected to the eighth connection layer; the output end of the fifth activation layer is connected to the input end of the channel attention layer; the output end of the channel attention layer is connected to the input end of the eighth connection layer; the output end of the fifth activation layer is connected to the input end of the spatial attention layer; the output end of the spatial attention layer is connected to the input end of the eighth connection layer;
[0024] The output of the eighth connection layer is connected to the input of the sixth convolutional layer.
[0025] Preferably, the formula for obtaining the joint gradient map is:
[0026] G joint =max(abs(LoG(I a )),abs(LoG(I b )))
[0027] The formula for obtaining the fused gradient map is:
[0028] G f =abs(LoG(I f ))
[0029] Among them, abs(.) is the absolute value function; max() is the maximum value function; I a I is the first source image; b is the second source image; I f To fuse the images.
[0030] Preferably, the identification loss function is:
[0031]
[0032] Among them, D is the discriminator; b is the first parameter; c is the second parameter; N is the batch size of the input model.
[0033] Preferably, the generation loss function includes structure perception loss, content loss and adversarial loss, specifically:
[0034] L G =αL ps +βL con +γL adv
[0035] Among them, L ps is the structure-aware loss; L con is the content loss; L adv is the adversarial loss; α is the structure-aware loss weight; β is the content loss weight; γ is the adversarial loss weight.
[0036] Preferably, the structural loss function is:
[0037] L ps =L SSIM ·L PSNR
[0038]
[0039] Wherein, H is the height of the source image; W is the width of the source image; i is the i-th row of the source image; j is the j-th column of the source image; S1 is the first adaptive weight; S2 is the second adaptive weighting; ω is the third parameter; μ is the mean; σ is the standard deviation; is the maximum pixel value of the image; SSIM is the structural similarity; PSNR is the peak signal-to-noise ratio; L SSIM is the structural similarity loss; L PSNR is the peak signal-to-noise ratio loss; C1 is the fourth parameter; C2 is the fifth parameter; C3 is the sixth parameter.
[0040] Preferably, the content loss includes intensity loss and gradient loss, specifically:
[0041] L con =η1L int +η2L grad
[0042] Among them, L int is the strength loss; L grad is the gradient loss; η1 is the strength loss weight; η2 is the gradient loss weight.
[0043] Preferably, the adversarial loss is:
[0044]
[0045] Among them, D is the discriminator; a is the seventh parameter; N is the batch size of the input model.
[0046] The present invention also provides an unsupervised multi-focus image fusion system based on an attention mechanism, which is used to implement the above method. The system comprises:
[0047] A data acquisition module, which acquires a source image data set, including a corresponding first source image and a second source image;
[0048] A model building module, constructing a multi-focus image fusion model, wherein the model includes a generator and a discriminator;
[0049] A generating module, inputting the first source image and the second source image into a generator to obtain a fused image;
[0050] A gradient map acquisition module performs Gaussian Laplace transform on the first source image, the second source image and the fused image to obtain a first gradient map, a second gradient map and a fused gradient map; and obtains a joint gradient map based on the first gradient map and the second gradient map;
[0051] A discriminator module, inputting the joint gradient map and the fused gradient map into a discriminator to obtain a predicted fused image;
[0052] The parameter adjustment module sets the identification loss function, calculates the identification loss function value based on the fused gradient map and the predicted fused image, and performs back propagation until the preset identification loss threshold is reached, and transmits the corresponding discriminator parameters to the generator;
[0053] The update module is fed back to set the generation loss function and fed back to the generation module until the preset generation loss function value is reached to obtain the trained multi-focus image fusion model;
[0054] The image fusion module obtains the image to be fused and inputs it into the trained multi-focus image fusion model to obtain the fused image.
[0055] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:
[0056] The present invention proposes an unsupervised multi-focus image fusion method and system based on attention mechanism, which fully extracts the global and local information of the source image by using the multi-focus image fusion model; focuses on different features at the same scale; and in order to enhance the texture details of the image, the gradient map corresponding to the source image is sent to the discriminator training, thereby obtaining a fused image with clear edges. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] Figure 1 This is a flow chart of the unsupervised multi-focus image fusion method based on the attention mechanism described in Example 1;
[0058] Figure 2 is a schematic diagram of the structure of the multi-focus image fusion model described in Example 2;
[0059] Figure 3 is a schematic diagram of the structure of the generator described in Example 2;
[0060] Figure 4 is a schematic structural diagram of the CAM layer described in Example 2;
[0061] Figure 5 This is a schematic diagram of the structure of the unsupervised multi-focus image fusion system based on the attention mechanism described in Example 3. DETAILED DESCRIPTION
[0062] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;
[0063] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;
[0064] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.
[0065] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0066] Example 1
[0067] This embodiment provides an unsupervised multi-focus image fusion method based on attention mechanism. Figure 1 As shown, including:
[0068] S1: Acquire a source image data set, including a corresponding first source image and a second source image;
[0069] S2: construct a multi-focus image fusion model, the model including a generator and a discriminator;
[0070] S3: inputting the first source image and the second source image into a generator to obtain a fused image;
[0071] S4: performing Gaussian Laplace transform on the first source image, the second source image and the fused image to obtain a first gradient map, a second gradient map and a fused gradient map; and obtaining a joint gradient map based on the first gradient map and the second gradient map;
[0072] S5: inputting the joint gradient map and the fused gradient map into a discriminator to obtain a predicted fused image;
[0073] S6: setting a discrimination loss function, calculating the discrimination loss function value based on the fused gradient map and the predicted fused image, and performing back propagation until a preset discrimination loss threshold is reached, and transmitting the corresponding discriminator parameters to the generator;
[0074] S7: setting a generation loss function, repeating steps S3-S6 until a preset generation loss function value is reached, and obtaining a trained multi-focus image fusion model;
[0075] S8: Obtain the image to be fused and input it into the trained multi-focus image fusion model to obtain a fused image.
[0076] Example 2
[0077] This embodiment provides an unsupervised multi-focus image fusion method based on an attention mechanism, including:
[0078] S1: Acquire a source image data set, including a corresponding first source image and a second source image;
[0079] MFI-WHU is used as the training dataset, which contains 120 pairs of images. The original images are from the COCO dataset. In order to obtain more training data, the images of the MFI-WHU dataset are cropped into 202246 image blocks of size 60*60 using the expansion strategy of cropping and decomposition. The test dataset uses the Lytro dataset;
[0080] S2: construct a multi-focus image fusion model, the model including a generator and a discriminator;
[0081] Because the twin network can learn the similar features of two data, the generator is designed as two paths to extract information; first, the source image is passed through 5*5 and 3*3 convolutional layers to extract low-level semantic features. Then, the feature map is input into the CAM module to extract channel, pixel, and spatial features respectively. In order to extract high-level semantic features, the two paths complete information fusion through cascading and convolution, and input to the next convolutional layer. At the same time, in order to prevent information loss during feature extraction, dense connections are used. Finally, the features extracted from the two paths are stacked on the channel, and the fused image is obtained through 1*1 convolution.
[0082] like Figure 2 As shown in the figure, first, the first source image and the second source image are passed through LoG to obtain their respective gradient maps, and the joint gradient map is obtained through the maximum selection principle. The fused image is passed through LoG to obtain the fused gradient map; then the joint gradient map is used as the real data and the fused gradient map is used as the generated data. It is sequentially sent to the discriminator, passes through four 3*3 convolutional layers and the LeakyReLU activation function, and finally average pooling is performed to obtain the probability value.
[0083] like Figure 3 As shown, the generator includes a first convolution layer, a second convolution layer, a third convolution layer, a fourth convolution layer, a fifth convolution layer, a first activation layer, a second activation layer, a third activation layer, a fourth activation layer, a first CAM layer, a second CAM layer, a third CAM layer, a fourth CAM layer, a fifth CAM layer, a sixth CAM layer, a seventh CAM layer, an eighth CAM layer, a first connection layer, a second connection layer, a third connection layer, a fourth connection layer, a fifth connection layer, a sixth connection layer and a seventh connection layer;
[0084] The output end of the first convolutional layer is connected to the input end of the first activation layer; the output end of the first activation layer is connected to the input end of the second convolutional layer; the output end of the second convolutional layer is connected to the input end of the second activation layer; the output end of the second activation layer is connected to the input end of the first CAM layer; the output end of the first CAM layer is connected to the input end of the second connection layer; the output end of the second connection layer is connected to the input end of the fourth CAM layer; the output end of the fourth CAM layer is connected to the input end of the fourth connection layer; the output end of the fourth connection layer is connected to the input end of the seventh CAM layer; the output end of the seventh CAM layer is connected to the input end of the seventh connection layer;
[0085] The output end of the third convolutional layer is connected to the input end of the third activation layer; the output end of the third activation layer is connected to the input end of the fourth convolutional layer; the output end of the fourth convolutional layer is connected to the input end of the fourth activation layer; the output end of the fourth activation layer is connected to the input end of the second CAM layer; the output end of the second CAM layer is connected to the input end of the third connection layer; the output end of the third connection layer is connected to the input end of the fifth CAM layer; the output end of the fifth CAM layer is connected to the input end of the sixth connection layer; the output end of the sixth connection layer is connected to the input end of the eighth CAM layer; the output end of the eighth CAM layer is connected to the input end of the seventh connection layer;
[0086] The output end of the second activation layer is connected to the input end of the second connection layer; the output end of the second activation layer is connected to the input end of the fifth connection layer; the output end of the fourth activation layer is connected to the input end of the third connection layer; the output end of the fourth activation layer is connected to the input end of the sixth connection layer; the output ends of the second activation layer and the fourth activation layer are both connected to the input end of the seventh connection layer;
[0087] The output ends of the first CAM layer and the second CAM layer are both connected to the input end of the first connection layer; the output end of the first connection layer is connected to the input end of the third CAM layer; the output end of the first CAM layer is connected to the input end of the fifth connection layer; the output end of the second CAM layer is connected to the input end of the sixth connection layer; the output ends of the first CAM layer and the second CAM layer are both connected to the input end of the seventh connection layer; the output end of the third CAM layer is connected to the input ends of the second connection layer and the third connection layer;
[0088] The output ends of the fourth CAM layer and the fifth CAM layer are both connected to the input end of the seventh connection layer; the output ends of the fourth CAM layer and the fifth CAM layer are both connected to the input end of the fourth connection layer; the output end of the fourth connection layer is connected to the input end of the sixth CAM layer; the output end of the sixth CAM layer is connected to the input ends of the fifth connection layer and the sixth connection layer;
[0089] The output end of the seventh connection layer is connected to the input end of the fifth convolutional layer; the fifth convolutional layer outputs a fused image.
[0090] This paper proposes a CAM layer based on the attention mechanism, which aims to fully extract the global and local information of the source image and strengthen the learning of the color information of the image. The CAM layer combines three attention layers, namely the pixel attention block (PAB), the channel attention block (CAB), and the spatial attention block (SAB). The advantage of PAB is that it can obtain local information and is sensitive to the color information and edges of the image. The disadvantage is that it lacks attention to global information. CAB and SAB can enhance the network's learning of global information. CAB can enhance the network's understanding of the correlation between different channels and be sensitive to the brightness information of the image; SAB can enhance the understanding of the spatial position relationship in the feature map and be sensitive to the color information of the image. The deep separable large kernel convolution (5*5) is used to increase the receptive field while reducing the parameters. Pixel normalization (LayerNorm) is used to improve the stability of training, acting on pixels, and the output features have local information. The global average pooling operation is used to extract the information of the global receptive field, and each feature channel is abstracted as a point with global information. The channel information is integrated, and weights are generated for each spatial feature through parameter learning. The input features are subjected to maximum pooling and global average pooling so that the output features have global information.
[0091] like Figure 4 As shown, the first CAM layer, the second CAM layer, the third CAM layer, the fourth CAM layer, the fifth CAM layer, the sixth CAM layer, the seventh CAM layer, and the eighth CAM layer have the same structure, and all include a fifth activation layer, a pixel attention layer, a channel attention layer, a spatial attention layer, an eighth connection layer, and a sixth convolutional layer;
[0092] The output end of the fifth activation layer is connected to the input end of the pixel attention layer; the output end of the pixel attention layer is connected to the eighth connection layer; the output end of the fifth activation layer is connected to the input end of the channel attention layer; the output end of the channel attention layer is connected to the input end of the eighth connection layer; the output end of the fifth activation layer is connected to the input end of the spatial attention layer; the output end of the spatial attention layer is connected to the input end of the eighth connection layer;
[0093] The output of the eighth connection layer is connected to the input of the sixth convolutional layer.
[0094] The source image is passed through a 5×5 convolutional layer and a 3×3 convolutional layer to extract low-level semantic features and obtain a feature map.
[0095] The feature map is input to the CAM module. In the CAM module, the feature map first passes through a 3×3 convolution layer, and then is sent to PAB, where local information is extracted through 1×1 convolution, residual block, and layer normalization to obtain a pixel feature map; sent to CAB, where global information is extracted through global pooling, 1×1 convolution, and sigmoid activation function to obtain a channel feature map; sent to SAB, where spatial information is extracted through global pooling and maximum pooling, 1×1 convolution, and sigmoid activation function to obtain a spatial feature map. Finally, the feature maps obtained are concatenated through the concat function, and the scale of the merged feature map is ensured to be the same as the input through 1×1 convolution.
[0096] The generator and discriminator are trained iteratively in sequence, with the batch size set to k, m epochs are trained, each epoch iterates n steps, and the ratio of the discriminator training number to the generator training number is q. In the experiment, k = 32, m = 30, n = 9032, q = 2. The optimizers of the generator and discriminator are both AdamOptimize, and the learning rates of the generator and discriminator are set to 0.0001 and 0.000001 respectively. The entire training process is as follows. Convert the image from RGB to the color space YCbCr. Since the Y channel (brightness) can represent structural details and brightness changes, only the Y channel is fused, and the Cb and Cr channels (chrominance) are fused in the traditional way. The final result converts the fused image to the RGB color space;
[0097] S3: inputting the first source image and the second source image into a generator to obtain a fused image;
[0098] S4: performing Gaussian Laplace transform on the first source image, the second source image and the fused image to obtain a first gradient map, a second gradient map and a fused gradient map; and obtaining a joint gradient map based on the first gradient map and the second gradient map;
[0099] The formula for obtaining the joint gradient map is:
[0100] G joint =max(abs(LoG(I a )),abs(LoG(I b )))
[0101] The formula for obtaining the fused gradient map is:
[0102] G f =abs(LoG(I f ))
[0103] Among them, abs(.) is the absolute value function; max() is the maximum value function; I a I is the first source image; b is the second source image; If To fuse the images.
[0104] S5: inputting the joint gradient map and the fused gradient map into a discriminator to obtain a predicted fused image;
[0105] S6: setting a discrimination loss function, calculating the discrimination loss function value based on the fused gradient map and the predicted fused image, and performing back propagation until a preset discrimination loss threshold is reached, and transmitting the corresponding discriminator parameters to the generator;
[0106] The purpose of the identification loss is to enable the discriminator to accurately distinguish true and false data. The identification loss function is:
[0107]
[0108] Among them, D is the discriminator; b is the first parameter; c is the second parameter; N is the batch size of the input model.
[0109] S7: setting a generation loss function, repeating steps S3-S6 until a preset generation loss function value is reached, and obtaining a trained multi-focus image fusion model;
[0110] In order to better preserve the structural information of the source image and reduce the distortion in the fusion process, we propose a structure-aware loss L ps , by L SSIM and L PSNR Composition. SSIM Reflects the structural similarity between images
[18] , used to model image distortion. Here, 1-SSIM is used as the loss term. The smaller its value, the better the fusion effect. PSNR Represents the ratio of peak power to noise power in the fused image
[19] , can measure the distortion in the image fusion process, and take 1 / PSNR as the loss term. The smaller its value, the better the fusion performance. In order to balance the relationship between the two and the impact on the fusion performance, we set L SSIM With L PSNR Multiplication is done instead of simple weighting, because the former only needs one weight term, while the latter needs two, which saves computational cost.
[0111] The generation loss function includes structure perception loss, content loss and adversarial loss, specifically:
[0112] L G =αL ps +βL con +γL adv
[0113] Among them, L ps is the structure-aware loss; L con is the content loss; Ladv is the adversarial loss; α is the structure-aware loss weight; β is the content loss weight; γ is the adversarial loss weight.
[0114] The structural loss function is:
[0115] L ps =L SSIM ·L PSNR
[0116]
[0117]
[0118] Wherein, H is the height of the source image; W is the width of the source image; i is the i-th row of the source image; j is the j-th column of the source image; S1 is the first adaptive weight; S2 is the second adaptive weighting; ω is the third parameter, ω=0.005; μ is the mean; σ is the standard deviation; is the maximum pixel value of the image, SSIM is the structural similarity, which reflects the structural similarity between images and is used to model image distortion; PSNR is the peak signal-to-noise ratio; L SSIM is the structural similarity loss, the ratio of the peak power to the noise power in the fused image, which can measure the distortion in the image fusion process; L PSNR is the peak signal-to-noise ratio loss; C1 is the fourth parameter; C2 is the fifth parameter; C3 is the sixth parameter.
[0119] The content loss includes intensity loss and gradient loss, the purpose of which is to make the fused image close to the intensity distribution and gradient distribution of the source image, specifically:
[0120] L con =η1L int +η2L grad
[0121] Among them, L int is the strength loss; L grad is the gradient loss; η1 is the strength loss weight; η2 is the gradient loss weight.
[0122] The purpose of the adversarial loss is to enhance the texture details of the fusion image and make the edges clearer. The adversarial loss is:
[0123]
[0124] Among them, D is the discriminator; a is the seventh parameter; N is the batch size of the input model.
[0125] S8: Obtain the image to be fused and input it into the trained multi-focus image fusion model to obtain a fused image.
[0126] In order to verify the feasibility of the model, it is compared with 7 representative methods. These methods include methods based on convolutional neural networks (CNN), methods based on regression models (IFCNN), methods based on unsupervised networks (U2Fusion), methods based on generative adversarial networks (MFF-GAN, ACGAN, MFIF-GAN), and methods based on decision graphs (SESF). In addition, the algorithm in this paper is evaluated qualitatively and quantitatively on the Lytro dataset and grayscale dataset. Subjective evaluation depends on human subjective visual perception, and objective evaluation uses indicators to quantitatively analyze the fusion results. In order to reduce the one-sidedness of a single indicator, multiple indicators such as information entropy-based metrics (PSNR, AG), image feature-based metrics (SF, EI), structural similarity-based metrics (SSIM), and human perception-based metrics (VIF) are used to evaluate the performance of the fused image.
[0127] As shown in Table 1, the comparison results between the method proposed in this embodiment and other image fusion methods;
[0128] Algorithm PSNR AG SF EI SSIM VIF U2Fusion 47.55 5.20 13.20 55.36 0.836 0.70 IFCNN 48.38 7.13 19.58 74.17 0.801 0.94 MFIF-GAN 49.10 7.14 19.57 74.18 <![CDATA[ 0.841 ]]> 0.95 ACGAN <![CDATA[ 49.47 ]]> 6.82 19.13 71.23 0.801 0.99 MFF-GAN 46.73 <![CDATA[ 8.07 ]]> <![CDATA[ 22.42 ]]> <![CDATA[ 84.18 ]]> 0.759 1.12 CNN 49.34 7.04 19.30 73.24 0.845 0.94 SESF 48.58 7.13 19.59 74.13 <![CDATA[ 0.841 ]]> 0.94 The method proposed in this embodiment 52.38 8.25 22.74 85.96 0.813 <![CDATA[ 1.06 ]]>
[0129] As shown in Table 1, for the PSNR index, the proposed method has the least distortion and the highest fusion quality, which is 5.5% higher than the suboptimal ACGAN algorithm. For the AG, SF, and EI indexes, the proposed method can retain the most details and textures and better handle boundaries, which are 2.2%, 1.4%, and 2.1% higher than the suboptimal MFF-GAN algorithm, respectively. For the SSIM index, the CNN algorithm is 3.5% better than the proposed method. For the VIF index, the MFF-GAN algorithm is 5.3% better than the proposed method. In general, the proposed method performs better than other comparison algorithms in objective evaluation.
[0130] Example 3
[0131] This embodiment also provides an unsupervised multi-focus image fusion system based on an attention mechanism, which is used to implement the method of embodiment 1 or 2, such as Figure 5 As shown, the system comprises:
[0132] A data acquisition module acquires a source image data set, including a corresponding first source image and a second source image;
[0133] A model building module, constructing a multi-focus image fusion model, wherein the model includes a generator and a discriminator;
[0134] A generating module, inputting the first source image and the second source image into a generator to obtain a fused image;
[0135] A gradient map acquisition module performs Gaussian Laplace transform on the first source image, the second source image and the fused image to obtain a first gradient map, a second gradient map and a fused gradient map; and obtains a joint gradient map based on the first gradient map and the second gradient map;
[0136] A discriminator module, inputting the joint gradient map and the fused gradient map into a discriminator to obtain a predicted fused image;
[0137] The parameter adjustment module sets the identification loss function, calculates the identification loss function value based on the fused gradient map and the predicted fused image, and performs back propagation until the preset identification loss threshold is reached, and transmits the corresponding discriminator parameters to the generator;
[0138] The update module is fed back to set the generation loss function and fed back to the generation module until the preset generation loss function value is reached to obtain the trained multi-focus image fusion model;
[0139] The image fusion module obtains the image to be fused and inputs it into the trained multi-focus image fusion model to obtain the fused image.
[0140] The same or similar reference numerals correspond to the same or similar components;
[0141] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;
[0142] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.
Claims
1. An unsupervised multi-focus image fusion method based on attention mechanism, characterized in that: include: S1: Acquire a source image data set, including a corresponding first source image and a second source image; S2: construct a multi-focus image fusion model, the model including a generator and a discriminator; S3: inputting the first source image and the second source image into a generator to obtain a fused image; S4: performing Gaussian Laplace transform on the first source image, the second source image and the fused image to obtain a first gradient map, a second gradient map and a fused gradient map; and obtaining a joint gradient map based on the first gradient map and the second gradient map; S5: inputting the joint gradient map and the fused gradient map into a discriminator to obtain a predicted fused image; S6: setting a discrimination loss function, calculating the discrimination loss function value based on the fused gradient map and the predicted fused image, and performing back propagation until a preset discrimination loss threshold is reached, and transmitting the corresponding discriminator parameters to the generator; S7: setting a generation loss function, repeating steps S3-S6 until a preset generation loss function value is reached, and obtaining a trained multi-focus image fusion model; S8: Obtain the image to be fused and input it into the trained multi-focus image fusion model to obtain a fused image.
2. The unsupervised multi-focus image fusion method based on the attention mechanism according to claim 1, characterized in that: The generator includes a first convolution layer, a second convolution layer, a third convolution layer, a fourth convolution layer, a fifth convolution layer, a first activation layer, a second activation layer, a third activation layer, a fourth activation layer, a first CAM layer, a second CAM layer, a third CAM layer, a fourth CAM layer, a fifth CAM layer, a sixth CAM layer, a seventh CAM layer, an eighth CAM layer, a first connection layer, a second connection layer, a third connection layer, a fourth connection layer, a fifth connection layer, a sixth connection layer and a seventh connection layer; The output of the first convolutional layer is connected to the input of the first activation layer; the output of the first activation layer is connected to the input of the second convolutional layer; The output end of the second convolutional layer is connected to the input end of the second activation layer; the output end of the second activation layer is connected to the input end of the first CAM layer; the output end of the first CAM layer is connected to the input end of the second connection layer; The output end of the second connection layer is connected to the input end of the fourth CAM layer; the output end of the fourth CAM layer is connected to the input end of the fifth connection layer; The output end of the fifth connection layer is connected to the input end of the seventh CAM layer; the output end of the seventh CAM layer is connected to the input end of the seventh connection layer; The output end of the third convolutional layer is connected to the input end of the third activation layer; the output end of the third activation layer is connected to the input end of the fourth convolutional layer; the output end of the fourth convolutional layer is connected to the input end of the fourth activation layer; the output end of the fourth activation layer is connected to the input end of the second CAM layer; the output end of the second CAM layer is connected to the input end of the third connection layer; the output end of the third connection layer is connected to the input end of the fifth CAM layer; the output end of the fifth CAM layer is connected to the input end of the sixth connection layer; the output end of the sixth connection layer is connected to the input end of the eighth CAM layer; the output end of the eighth CAM layer is connected to the input end of the seventh connection layer; The output end of the second activation layer is connected to the input end of the second connection layer; The output terminal of the second activation layer is connected to the input terminal of the fifth connection layer; The output end of the fourth activation layer is connected to the input end of the third connection layer; An output terminal of the fourth activation layer is connected to an input terminal of the sixth connection layer; The output ends of the second activation layer and the fourth activation layer are both connected to the input end of the seventh connection layer; The output ends of the first CAM layer and the second CAM layer are both connected to the input end of the first connection layer; the output end of the first connection layer is connected to the input end of the third CAM layer; the output end of the first CAM layer is connected to the input end of the fifth connection layer; the output end of the second CAM layer is connected to the input end of the sixth connection layer; the output ends of the first CAM layer and the second CAM layer are both connected to the input end of the seventh connection layer; the output end of the third CAM layer is connected to the input ends of the second connection layer and the third connection layer; The output ends of the fourth CAM layer and the fifth CAM layer are both connected to the input end of the seventh connection layer; the output ends of the fourth CAM layer and the fifth CAM layer are both connected to the input end of the fourth connection layer; the output end of the fourth connection layer is connected to the input end of the sixth CAM layer; the output end of the sixth CAM layer is connected to the input ends of the fifth connection layer and the sixth connection layer; The output end of the seventh connection layer is connected to the input end of the fifth convolutional layer; the fifth convolutional layer outputs a fused image.
3. The unsupervised multi-focus image fusion method based on attention mechanism according to claim 2, characterized in that: The first CAM layer, the second CAM layer, the third CAM layer, the fourth CAM layer, the fifth CAM layer, the sixth CAM layer, the seventh CAM layer, and the eighth CAM layer have the same structure, and all include a fifth activation layer, a pixel attention layer, a channel attention layer, a spatial attention layer, an eighth connection layer, and a sixth convolutional layer; The output end of the fifth activation layer is connected to the input end of the pixel attention layer; the output end of the pixel attention layer is connected to the eighth connection layer; the output end of the fifth activation layer is connected to the input end of the channel attention layer; the output end of the channel attention layer is connected to the input end of the eighth connection layer; The output of the fifth activation layer is connected to the input of the spatial attention layer; the output of the spatial attention layer is connected to the input of the eighth connection layer; The output of the eighth connection layer is connected to the input of the sixth convolutional layer.
4. The unsupervised multi-focus image fusion method based on attention mechanism according to claim 1, characterized in that: The formula for obtaining the joint gradient map is: Gd joint =max(abs(LoG(I a )),abs(LoG(I b ))) The formula for obtaining the fused gradient map is: Gd f =abs(LoG(I f )) Among them, abs(.) is the absolute value function; max() is the maximum value function; I a I is the first source image; b is the second source image; I f is the fused image; LoG() is the Gaussian Laplace transform operation.
5. The unsupervised multi-focus image fusion method based on attention mechanism according to claim 4, characterized in that: The identification loss function is: Among them, D is the discriminator; b is the first parameter; c is the second parameter; N is the batch size of the input model.
6. The unsupervised multi-focus image fusion method based on attention mechanism according to claim 4, characterized in that: The generation loss function includes structure perception loss, content loss and adversarial loss, specifically: L G =αL ps +βL con +γL adv Among them, L ps is the structure-aware loss; L con is the content loss; L adv is the adversarial loss; α is the structure-aware loss weight; β is the content loss weight; γ is the adversarial loss weight.
7. The unsupervised multi-focus image fusion method based on attention mechanism according to claim 6, characterized in that: The structural loss is: L ps =L SSIM ·L PSNR Wherein, H is the height of the source image; W is the width of the source image; i is the i-th row of the source image; j is the j-th column of the source image; S1 is the first adaptive weight; S2 is the second adaptive weighting; ω is the third parameter; μ is the mean; σ is the standard deviation; is the maximum pixel value of the image; SSIM is the structural similarity; PSNR is the peak signal-to-noise ratio; L SSIM is the structural similarity loss; L PSNR is the peak signal-to-noise ratio loss; C1 is the fourth parameter; C2 is the fifth parameter; C3 is the sixth parameter.
8. The unsupervised multi-focus image fusion method based on attention mechanism according to claim 6, characterized in that: The content loss includes intensity loss and gradient loss, specifically: L con =η1L int +η2L grad Among them, L int is the strength loss; L grad is the gradient loss; η1 is the strength loss weight; η2 is the gradient loss weight.
9. The unsupervised multi-focus image fusion method based on attention mechanism according to claim 6, characterized in that: The adversarial loss is: Among them, D is the discriminator; a is the seventh parameter; N is the batch size of the input model.
10. An unsupervised multi-focus image fusion system based on an attention mechanism, used to implement the method of claims 1-9, characterized in that: The system comprises: A data acquisition module acquires a source image data set, including a corresponding first source image and a second source image; A model building module, constructing a multi-focus image fusion model, wherein the model includes a generator and a discriminator; A generating module, inputting the first source image and the second source image into a generator to obtain a fused image; A gradient map acquisition module performs Gaussian Laplace transform on the first source image, the second source image and the fused image to obtain a first gradient map, a second gradient map and a fused gradient map; and obtains a joint gradient map based on the first gradient map and the second gradient map; A discriminator module, inputting the joint gradient map and the fused gradient map into a discriminator to obtain a predicted fused image; The parameter adjustment module sets the identification loss function, calculates the identification loss function value based on the fused gradient map and the predicted fused image, and performs back propagation until the preset identification loss threshold is reached, and transmits the corresponding discriminator parameters to the generator; The update module is fed back to set the generation loss function and fed back to the generation module until the preset generation loss function value is reached to obtain the trained multi-focus image fusion model; The image fusion module obtains the image to be fused and inputs it into the trained multi-focus image fusion model to obtain the fused image.