Infrared and visible light fusion method based on spectrum attention generative adversarial network

Abstract

The invention belongs to the technical field of image fusion, and discloses an infrared and visible light fusion method based on a spectrum attention generative adversarial network, which comprises the following steps: carrying out data cleaning and preprocessing on a public infrared and visible light fusion data set; constructing an infrared and visible light fusion model based on the spectrum attention generative adversarial network, wherein the infrared and visible light fusion model comprises a double-branch feature extraction sub-network, a fusion sub-network and a discriminator sub-network; constructing the total loss of the infrared and visible light fusion network; sending the processed data into an infrared and visible light fusion model based on a spectrum attention generative adversarial network for model training; and testing the performance of the trained infrared and visible light fusion model based on the spectrum attention generative adversarial network to obtain a fusion image. According to the infrared and visible light fusion method based on the spectrum attention generative adversarial network, the visual effect of the fusion image and the robustness of the fusion algorithm can be improved, the performance of the model is improved, and downstream advanced tasks can be better adapted.

Description

Technical Field

[0001] The present invention relates to the field of image fusion technology, and in particular to an infrared and visible light fusion method based on a spectrum attention generative adversarial network. Background Art

[0002] Due to the limitations and differences of a single sensor, different imaging methods produce different modal images in the same scene at the same time, which leads to fundamental difficulties in the registration and fusion technology of multimodal images. Infrared sensors use radiation information from different targets for imaging, and their target features are particularly obvious; visible light sensors use the reflection of light from objects for imaging, and their detailed texture information is relatively rich. In severe weather conditions, such as strong light noise, rain, snow, fog and haze, multimodal fusion technology can be used to provide target and scene information from different imaging systems, such as the fusion of infrared images and visible light images, to make up for the lack of target features in the visible light band imaging system and obtain a more complete information image.

[0003] Multimodal image fusion technology is a technology that utilizes the differences and complementarities between image information of different modalities, fuses them through some algorithms and strategies, and reconstructs a fused image with high signal-to-noise ratio and high visual fidelity. Compared with single-modal images, multimodal images can obtain information of different modalities for interactive fusion, and are gradually widely used in the fields of transportation, autonomous driving, forest fire prevention, environmental monitoring, aerospace, etc. However, with the widespread application of fusion technology, while improving image quality and image information, there are also some problems, such as low resolution and contrast of infrared images, serious damage to the image quality of visible light images in harsh environments, and the influence of noise on the imaging system. Therefore, the importance of developing a fusion algorithm that can achieve detail enhancement and visual fidelity and improve the quality while retaining the effective information of multimodal images is self-evident.

[0004] Therefore, in the field of image fusion, the development of infrared and visible light image fusion technology is particularly important. The present invention is based on a generative adversarial network to fuse infrared and visible light images, effectively improving the image quality of the fused image, and enhancing the detail information of the fused image and the robustness of the fusion algorithm. Summary of the invention

[0005] The purpose of the present invention is to provide an infrared and visible light fusion method based on a spectral attention generative adversarial network. By designing feature extraction and feature fusion for infrared and visible light images, designing spatial domain and frequency domain converters and introducing a self-attention mechanism, different levels of information enhancement and fusion constraints in multimodal images are obtained, and finally the visual effect of the fused image and the robustness of the fusion algorithm are improved.

[0006] To achieve the above object, the present invention provides an infrared and visible light fusion method based on a spectrum attention generative adversarial network, comprising the following steps: Step S1, cleaning and preprocessing the public infrared and visible light fusion data set; Step S2, constructing an infrared and visible light fusion model based on a spectrum attention generative adversarial network, including a dual-branch feature extraction subnetwork, a fusion subnetwork, and a discriminator subnetwork; Step S3, constructing the overall loss of the infrared and visible light fusion network; Step S4: sending the processed data to the infrared and visible light fusion model based on the spectrum attention generative adversarial network for model training; Step S5: test the performance of the trained infrared and visible light fusion model based on the spectral attention generative adversarial network to obtain a fused image.

[0007] Preferably, in step S1, the data cleaning operation includes: selecting and collecting fully registered infrared image and visible light image pairs to form target data; The preprocessing operation includes: extracting image block data, scaling and segmenting the image into region image pairs, and performing data enhancement on the region image pairs.

[0008] Preferably, infrared images and visible light images of different sizes are collected for the target image to form a target image pair, which is then segmented into sub-image pairs, and the sub-image pairs are reassembled to process pixels; For the target image pair (I ir , I vis ) is down-sampled and segmented, and the segmentation parameters of the sub-image pair are set to 80 pixels in width and height and 14 pixels in step length, forming a number of 80 pixel × 80 pixel sub-image pairs; where I ir is the infrared image, I vis is a visible light image; Image blocks are extracted and pixels are reassembled for each sub-image in the sub-image pair to obtain a processed 80 pixel×80 pixel sub-image pair.

[0009] Preferably, in step S2, an infrared and visible light fusion model based on a spectrum attention generative adversarial network is constructed, and the specific process is as follows: Step S21, constructing a dual-branch feature extraction sub-network, including an infrared branch network and a visible light branch network; The infrared branch network extracts infrared features and enhances targets for infrared images, including three 3*3 convolutional layers, two intensity attention modules, and two multi-head frequency domain attention conversion modules. Among them, the three convolutional layers are cascaded and the last two convolutional layers are connected in parallel with the intensity attention module and the multi-head frequency domain attention conversion module respectively. The visible light branch network mainly extracts high-frequency information features from visible light images, including three 3*3 convolutional layers and two semantic conversion modules. Among them, the three convolutional layers are cascaded and the last two convolutional layers are connected in parallel with the semantic conversion modules respectively. Step S22, construct a fusion sub-network, the fusion sub-network receives the output feature map of the double-branch sub-network, splices the double-branch feature map, and obtains an iterative fusion image after passing through one 5*5 convolution layer and one 1*1 convolution layer; S23. Construct a discriminator subnetwork. The discriminator subnetwork receives the iterative fusion image, passes through two 3*3 convolutional layers, one 5*5 convolutional layer and three fully connected layers, performs a normalization operation, and outputs a discrimination loss vector.

[0010] Preferably, the intensity attention module comprises three 3*3 convolutional layers and one 1*1 convolutional layer; wherein the three convolutional layers are connected in parallel and the output features of each convolutional layer are multiplied and normalized with the output features of the previous layer, and finally concatenated with the output features of the 1*1 convolutional layer, and the output features of the final module are IF; The multi-head frequency domain attention conversion module includes 2 layers of normalization, Fourier transform layer, inverse Fourier transform layer and multi-layer perceptron module; After the input features are normalized by the layer, they are transformed into frequency domain features through the Fourier transform layer. The frequency domain features are related to the weight matrix W. c The spatial domain features are obtained after multiplication, inverse Fourier transform layer and normalization, and finally concatenated with the initial input spatial domain features through a multi-layer perceptron module to obtain the output features of the multi-head frequency domain attention conversion module.

[0011] Preferably, in step S3, the overall loss of the infrared and visible light fusion network, specifically including the generator loss function and the discriminator loss function, is as follows: ; in, represents the generator loss; represents the discriminator loss.

[0012] Preferably, the generator loss aims to measure whether the generator achieves the expected performance according to the generator loss function; the generator loss includes content loss, structural loss and adversarial loss, as shown below: ; in, For content loss; For structural losses; To combat losses; It represents the measurement of the contrast distance and texture distance between the fused image and the original image. The definition is as follows: ; in, ; in, represents the fused image; Indicates infrared image; represents the visible image; ▽ represents the image gradient; SD represents the image standard deviation, and ‖*‖2 represents the L2 norm; G Indicates image gradient changes; C Indicates image contrast; It is the fusion balance parameter, which is calculated according to the IQA module; It represents the generator network based on The structural similarity between the source image and the fused image is used to constrain the fused image to maintain its structural information. The definition is as follows: ; in, SSIM Indicates structural similarity; is the fusion balance parameter.

[0013] Preferably, the loss function of the discriminator network aims to accurately classify the source image, and the discriminator loss function is as follows: ; Where D represents the discriminator; Represents the prediction result of the classification of infrared images; Represents the prediction result of the classification of the visible light image; is the similarity parameter; is the difference parameter.

[0014] Preferably, in step S4, the Adam function is used to optimize the loss function, and the optimizer optimizes the average loss of logistic regression to perform the training process.

[0015] Therefore, the present invention adopts the above-mentioned infrared and visible light fusion method based on spectrum attention generative adversarial network, and the beneficial effects are as follows: (1) The image fusion algorithm proposed in this invention can efficiently fuse infrared and visible light images, extract and fuse features of infrared and visible light images, design spatial domain and frequency domain converters, and introduce self-attention mechanism to obtain different levels of information enhancement and fusion constraints in multimodal images; (2) Compared with other image fusion algorithms, the algorithm of the present invention improves the visual effect of the fused image and improves the robustness of the algorithm.

[0016] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of the infrared and visible light fusion method based on the spectrum attention generative adversarial network of the present invention; Figure 2 It is a structural schematic diagram of the infrared and visible light fusion method based on the spectrum attention generative adversarial network of the present invention; Figure 3 This is an infrared and visible light fusion image in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0019] like Figure 1 As shown, the infrared and visible light fusion method based on the spectrum attention generation adversarial network of the present invention includes the following steps: Step S1, cleaning and preprocessing the public infrared and visible light fusion data set; Step S2, constructing an infrared and visible light fusion model based on a spectrum attention generative adversarial network, including a dual-branch feature extraction subnetwork, a fusion subnetwork, and a discriminator subnetwork; Step S3, constructing the overall loss of the infrared and visible light fusion network; Step S4: sending the processed data to the infrared and visible light fusion model based on the spectrum attention generative adversarial network for model training; Step S5: test the performance of the trained infrared and visible light fusion model based on the spectral attention generative adversarial network to obtain a fused image.

[0020] Example Step S1: Clean and preprocess the public infrared and visible light fusion data set.

[0021] The public infrared and visible light fusion dataset TNO is used to clean and preprocess the dataset. The data cleaning operation includes: selecting and collecting fully registered infrared images and visible light image pairs to form the target data; the preprocessing operation includes: extracting image block data, scaling and segmenting them into regional image pairs, and performing data enhancement on the regional image pairs.

[0022] For the target image, infrared images and visible light images of different sizes are collected to form a target image pair, which is then segmented into sub-image pairs, and the sub-image pairs are processed by pixel recombination.

[0023] For the target image pair (I ir , I vis) is down-sampled and segmented, and the segmentation parameters of the sub-image pair are set to 80 pixels in width and height and 14 pixels in step length, forming a number of 80 pixel × 80 pixel sub-image pairs; where I ir is the infrared image, I vis It is a visible light image.

[0024] Image blocks are extracted and pixels are reassembled for each sub-image in the sub-image pair to obtain a processed 80 pixel×80 pixel sub-image pair.

[0025] Step S2: construct an infrared and visible light fusion model based on spectrum attention generative adversarial network, which mainly includes a dual-branch feature extraction sub-network, a fusion sub-network, and a discriminator sub-network, such as Figure 2 shown.

[0026] Step S21, constructing a dual-branch feature extraction sub-network, including an infrared branch network and a visible light branch network.

[0027] The infrared branch network mainly performs infrared feature extraction and target enhancement on infrared images, including three 3*3 convolutional layers, two intensity attention modules, and two multi-head frequency domain attention conversion modules; among them, three convolutional layers are cascaded and the last two convolutional layers are connected in parallel with the intensity attention module and the multi-head frequency domain attention conversion module respectively.

[0028] The visible light branch network mainly extracts high-frequency information features of visible light images, including three 3*3 convolutional layers and two semantic conversion modules; among them, the three convolutional layers are cascaded and the last two convolutional layers are connected in parallel with the semantic conversion modules respectively.

[0029] The intensity attention module contains three 3*3 convolutional layers and one 1*1 convolutional layer; among them, the three convolutional layers are connected in parallel and the output features of each convolutional layer are multiplied and normalized with the output features of the previous layer, and finally concatenated with the output features of the 1*1 convolutional layer, and the output features of the final module are IF.

[0030] The multi-head frequency-domain attention transformation module consists of 2 normalization layers, a Fourier transform layer, an inverse Fourier transform layer, and a multi-layer perceptron module.

[0031] After the input features are normalized by the layer, they are transformed into frequency domain features through the Fourier transform layer. The frequency domain features are related to the weight matrix W. c The spatial domain features are obtained after multiplication, inverse Fourier transform layer and normalization, and finally concatenated with the initial input spatial domain features through a multi-layer perceptron module to obtain the output features of the multi-head frequency domain attention conversion module.

[0032] Step S22: construct a fusion subnetwork, which receives the output feature map of the dual-branch subnetwork, concatenates the dual-branch feature map, and obtains an iterative fused image after passing through one 5*5 convolution layer and one 1*1 convolution layer.

[0033] S23. Construct a discriminator subnetwork. The discriminator subnetwork receives the iterative fusion image, passes through two 3*3 convolutional layers, one 5*5 convolutional layer and three fully connected layers, performs a normalization operation, and outputs a discrimination loss vector.

[0034] Step S3, constructing the overall loss of the infrared and visible light fusion network; The overall loss of the infrared and visible light fusion network, including the generator loss function and the discriminator loss function, is as follows: ; in, represents the generator loss; represents the discriminator loss.

[0035] (1) The generator loss aims to measure whether the generator achieves the expected performance according to the generator loss function; the generator loss includes content loss, structural loss and adversarial loss, as shown below: ; in, For content loss; For structural losses; To combat losses.

[0036] It represents the contrast distance and texture distance between the fused image and the original image. The definition is as follows: ; in, ; in, represents the fused image; Indicates infrared image; represents the visible image; ▽ represents the image gradient; SD represents the image standard deviation, and ‖*‖2 represents the L2 norm; G Indicates image gradient changes; C Indicates image contrast; It is the fusion balance parameter, which is calculated according to the IQA module.

[0037] It represents the generator network based on The structural similarity with the source image is used to constrain the fused image to maintain its structural information. The definition is as follows: ; in, SSIM Indicates structural similarity; is the fusion balance parameter.

[0038] (2) The loss function of the discriminator network aims to accurately classify the source image. The discriminator loss function is as follows: ; Where D represents the discriminator; Represents the prediction result of the classification of infrared images; Represents the prediction result of the classification of the visible light image; is the similarity parameter; is the difference parameter.

[0039] Step S4: Send the processed infrared image and visible light image pair data to the infrared and visible light fusion model based on the spectral attention generative adversarial network for model training.

[0040] NVIDIA GTX 3090 GPU and i7-10700k CPU were used for training. During the deep learning image fusion network training process, the Adam function was used to optimize the loss function L, and the optimizer optimized the average loss of logistic regression to perform the training process.

[0041] Step S5: Test the performance of the trained infrared and visible light fusion model based on the spectrum attention generative adversarial network to obtain a fused image, such as Figure 3 shown.

[0042] Therefore, the present invention adopts the above-mentioned infrared and visible light fusion method based on spectral attention generative adversarial network, which effectively improves the image quality of the fused image, enhances the detail information of the fused image and the robustness of the fusion algorithm to adapt to more application scenarios.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. Infrared and visible light fusion method based on spectrum attention generative adversarial network, characterized by: The following steps are involved: Step S1, cleaning and preprocessing the public infrared and visible light fusion data set; Step S2, constructing an infrared and visible light fusion model based on a spectrum attention generative adversarial network, including a dual-branch feature extraction subnetwork, a fusion subnetwork, and a discriminator subnetwork; Step S3, constructing the overall loss of the infrared and visible light fusion network; Step S4: sending the processed data to the infrared and visible light fusion model based on the spectrum attention generative adversarial network for model training; Step S5: test the performance of the trained infrared and visible light fusion model based on the spectral attention generative adversarial network to obtain a fused image.

2. According to claim 1, the infrared and visible light fusion method based on spectrum attention generative adversarial network is characterized by: In step S1, the data cleaning operation includes: selecting and collecting fully registered infrared image and visible light image pairs to form target data; The preprocessing operation includes: extracting image block data, scaling and segmenting the image into region image pairs, and performing data enhancement on the region image pairs.

3. The infrared and visible light fusion method based on spectrum attention generative adversarial network according to claim 2 is characterized in that: For the target image, infrared images and visible light images of different sizes are collected to form a target image pair, which is then segmented into sub-image pairs, and pixels are reassembled to process the sub-image pairs; For the target image pair (I ir , I vis ) is down-sampled and segmented, and the segmentation parameters of the sub-image pair are set to 80 pixels in width and height and 14 pixels in step length, forming a number of 80 pixel × 80 pixel sub-image pairs; where I ir is the infrared image, I vis is a visible light image; Image blocks are extracted and pixels are reassembled for each sub-image in the sub-image pair to obtain a processed 80 pixel×80 pixel sub-image pair.

4. The infrared and visible light fusion method based on spectrum attention generative adversarial network according to claim 1 is characterized in that: In step S2, an infrared and visible light fusion model based on spectral attention generative adversarial network is constructed. The specific process is as follows: Step S21, constructing a dual-branch feature extraction sub-network, including an infrared branch network and a visible light branch network; The infrared branch network extracts infrared features and enhances targets for infrared images, including three 3*3 convolutional layers, two intensity attention modules, and two multi-head frequency domain attention conversion modules. Among them, the three convolutional layers are cascaded and the last two convolutional layers are connected in parallel with the intensity attention module and the multi-head frequency domain attention conversion module respectively. The visible light branch network mainly extracts high-frequency information features from visible light images, including three 3*3 convolutional layers and two semantic conversion modules. Among them, the three convolutional layers are cascaded and the last two convolutional layers are connected in parallel with the semantic conversion modules respectively. Step S22, construct a fusion sub-network, the fusion sub-network receives the output feature map of the double-branch sub-network, splices the double-branch feature map, and obtains an iterative fusion image after passing through one 5*5 convolution layer and one 1*1 convolution layer; S23. Construct a discriminator subnetwork. The discriminator subnetwork receives the iterative fusion image, passes through two 3*3 convolutional layers, one 5*5 convolutional layer and three fully connected layers, performs a normalization operation, and outputs a discrimination loss vector.

5. The infrared and visible light fusion method based on spectrum attention generative adversarial network according to claim 4 is characterized in that: The intensity attention module contains three 3*3 convolutional layers and one 1*1 convolutional layer. The three convolutional layers are connected in parallel and the output features of each convolutional layer are multiplied and normalized with the output features of the previous layer, and finally concatenated with the output features of the 1*1 convolutional layer. The output features of the final module are IF. The multi-head frequency domain attention conversion module includes 2 layers of normalization, Fourier transform layer, inverse Fourier transform layer and multi-layer perceptron module; After the input features are normalized by the layer, they are transformed into frequency domain features through the Fourier transform layer. The frequency domain features are related to the weight matrix W. c The spatial domain features are obtained after multiplication, inverse Fourier transform layer and normalization, and finally concatenated with the initial input spatial domain features through a multi-layer perceptron module to obtain the output features of the multi-head frequency domain attention conversion module.

6. The infrared and visible light fusion method based on spectrum attention generative adversarial network according to claim 1 is characterized in that: In step S3, the overall loss of the infrared and visible light fusion network, specifically including the generator loss function and the discriminator loss function, is as follows: ； in, represents the generator loss; represents the discriminator loss.

7. The infrared and visible light fusion method based on spectrum attention generative adversarial network according to claim 6 is characterized in that: The generator loss aims to measure whether the generator achieves the expected performance according to the generator loss function; the generator loss includes content loss, structure loss and adversarial loss, as shown below: ； in, For content loss; For structural losses; To combat losses; It represents the measurement of the contrast distance and texture distance between the fused image and the original image. The definition is as follows: ； in, ; in, represents the fused image; Indicates infrared image; represents the visible image; ▽ represents the image gradient; SD represents the image standard deviation, and ‖*‖2 represents the L2 norm; G Indicates image gradient changes; C Indicates image contrast; It is the fusion balance parameter, which is calculated according to the IQA module; It represents the generator network based on The structural similarity between the source image and the fused image is used to constrain the fused image to maintain its structural information. The definition is as follows: ； in, SSIM Indicates structural similarity; a is the fusion balance parameter.

8. The infrared and visible light fusion method based on spectrum attention generative adversarial network according to claim 6 is characterized in that: The loss function of the discriminator network aims to accurately classify the source image. The discriminator loss function is as follows: ； Where D represents the discriminator; Represents the prediction result of the classification of infrared images; Represents the prediction result of the classification of the visible light image; is the similarity parameter; is the difference parameter.

9. The infrared and visible light fusion method based on spectrum attention generative adversarial network according to claim 1 is characterized by: In step S4, the Adam function is used to optimize the loss function, and the optimizer optimizes the average loss of logistic regression to perform the training process.

Images