Infrared and visible light image fusion method based on fusion-semantic feature classifier

Abstract

The present invention relates to an infrared and visible light image fusion method based on a fusion-semantic feature classifier. The present invention comprises: obtaining fusion features and fusion results; calculating fusion loss, optimizing fusion sub-network parameters, and freezing fusion sub-network parameters; obtaining semantic features and segmentation results of infrared and visible light images; calculating semantic segmentation loss, optimizing semantic segmentation sub-network parameters, and freezing semantic segmentation sub-network parameters; sending the extracted fusion features and semantic features to a fusion-semantic feature classifier; calculating classification loss, optimizing classifier parameters, and freezing classifier parameters; extracting features containing rich semantic and fusion information; calculating classification loss, fine-tuning fusion sub-network encoder parameters, and freezing encoder parameters in the fusion sub-network; obtaining fusion results; calculating fusion loss, and fine-tuning decoder parameters; the present invention enables the encoder in the fusion sub-network to extract features containing rich fusion and semantic information for fusion result reconstruction.

Description

Technical Field

[0001] The invention relates to an infrared and visible light image fusion method based on a fusion-semantic feature classifier, and belongs to the technical field of image fusion. Background Art

[0002] Under different imaging principles, there are huge differences in the images obtained by infrared and visible light sensors. Infrared sensors use the thermal radiation of objects to image, which is almost not limited by light intensity. Even in low light conditions, they can clearly highlight the target object, but lack texture detail information. Visible light sensors are based on the reflected light of objects. They are easily affected by light intensity, but can obtain images with rich texture details under sufficient light conditions. Given the complementarity of infrared and visible light images, the goal of infrared and visible light image fusion is to integrate the complementary information of the two to obtain an image with rich spatial texture details and highlight the target object, so as to facilitate human visual recognition and better support downstream high-level visual tasks such as semantic segmentation, target detection and target tracking. However, most of the existing infrared and visible light fusion methods only focus on the visual effect of image fusion and ignore the support for downstream tasks. In order to better serve downstream tasks, some methods simply mechanically connect the high-level task network with the fusion network in series to obtain features containing rich fusion information and semantic information for reconstruction. However, the fusion performance of these methods is limited by the high-level task network used. In order to more flexibly obtain features containing rich fusion information and semantic information for reconstruction, a method for fusion of infrared and visible light images based on fusion-semantic feature classifier is proposed. This method inputs fusion features and semantic segmentation features into the classifier, and through the discriminative ability of the classifier, it can clearly distinguish the type of features. By using the classifier to fine-tune the encoder of the fusion sub-network, it can extract features containing rich fusion information and semantic information, that is, the encoder cannot clearly distinguish the feature type, and then use it for subsequent reconstruction tasks, thereby improving the fusion performance and better meeting the needs of downstream advanced tasks. Summary of the invention

[0003] The technical problem to be solved by the present invention is: a method for fusion of infrared and visible light images based on a fusion-semantic feature classifier is proposed to improve the performance of fusion of infrared and visible light images, and to reconstruct a higher quality fused image by utilizing features containing rich fusion information and semantic information.

[0004] The technical solution of the present invention is: a method for fusion of infrared and visible light images based on a fusion-semantic feature classifier, and the specific steps of the method are as follows:

[0005] Step 1: Obtain a training dataset containing registered infrared and visible light image pairs and semantic segmentation labels;

[0006] Step 2: Preprocess all data in the training dataset;

[0007] Step 3: Input the preprocessed infrared image and visible light image into the fusion sub-network to obtain fusion features and fusion results;

[0008] Step 4: Calculate the fusion loss, optimize the fusion sub-network parameters, and then freeze the fusion sub-network parameters;

[0009] Step 5: Send the preprocessed infrared image and visible light image to the semantic segmentation sub-network respectively to obtain the semantic features and segmentation results of the infrared and visible light images respectively;

[0010] Step 6: Calculate the semantic segmentation loss, optimize the semantic segmentation sub-network parameters, and freeze the semantic segmentation sub-network parameters;

[0011] Step 7: Send the infrared and visible light images to the fusion sub-network and the semantic segmentation sub-network, and send the extracted fusion features and semantic features to the fusion-semantic feature classifier to obtain the classification results;

[0012] Step 8: Calculate classification loss, optimize classifier parameters, and freeze classifier parameters;

[0013] Step 9: Unfreeze the fusion sub-network parameters, send the fusion sub-network features to the classifier, and extract features containing rich semantics and fusion information;

[0014] Step 10: Calculate the classification loss, optimize the encoder parameters of the fusion sub-network, and freeze the encoder parameters in the fusion sub-network;

[0015] Step 11: Send the frozen encoder features to the decoder for reconstruction to obtain the fusion result;

[0016] Step 12: Calculate the fusion loss and optimize the decoder parameters;

[0017] Furthermore, in step 1, a training data set is obtained, and each image group in the training data set is composed of an infrared image, a visible light image, and a corresponding semantic segmentation label. The number of categories of the semantic segmentation label is 9.

[0018] Furthermore, in step 2, the size of all images in the training data set is unified to 256×256, and then the data is randomly inverted and randomly rotated, and the processed images are normalized.

[0019] Furthermore, in step 3, the specific operation of acquiring the fusion feature is as follows:

[0020] The preprocessed infrared image I ir and visible light image I visInput the progressive feature extraction encoder E and obtain the infrared light feature F ir and visible light feature F vis , as follows:

[0021]

[0022] in, They represent the fusion sub-network feature extraction encoder E f Mid-infrared feature encoder E ir and visible light feature code E vis The i-th (i=1,2,3,4) layer output features, E ir and E vis The structure is the same but the parameters are not shared. Each layer of the encoder E is composed of multi-scale convolution blocks and convolution layers. The output features of the first three layers are converged in the fourth layer through residual connections. Multi-scale convolution blocks can be used to extract features under different receptive fields and enhance the feature representation ability of the model. The convergence of features through jump connections can enable the information contained in shallow features to be retained in the deep network, avoiding information loss. The multi-scale convolution block consists of three branches with different convolution kernel sizes, each of which consists of a convolution layer, a ReLU activation layer, and a BatchNorm layer.

[0023] Furthermore, in step 4, the fusion loss is calculated to optimize the network parameters. Loss of texture detail and content loss Composition can be defined as:

[0024]

[0025] Loss of texture detail Defined as:

[0026]

[0027] Among them, ||·||1 represents the l1 norm; Represents the gradient map extracted by the Sobel operator. Content loss Defined as:

[0028]

[0029] Furthermore, in step 5, the specific operation of semantic feature extraction is as follows:

[0030] The preprocessed infrared image I ir and visible light image I vis Input semantic segmentation subnetwork semantic feature extraction encoder E s , and obtain the semantic features of infrared images and semantic features of visible light images As shown below:

[0031]

[0032] in, Represents the infrared feature encoder in the semantic segmentation subnetwork and visible light feature encoder The output features of the semantic segmentation subnetwork are as follows. The encoder structure of the semantic segmentation subnetwork is the same as that of the fusion segmentation subnetwork. After extracting the semantic features of the infrared and visible light images respectively, and Separately fed into the semantic decoder and Get the semantic segmentation results of infrared and visible light images respectively:

[0033]

[0034] Among them, I s,ir Represents the semantic segmentation result of infrared image, I s,ir Represents the semantic segmentation result of the visible light image.

[0035] Furthermore, in step 6, the segmentation loss is calculated and the network parameters are optimized. The infrared image segmentation loss and visible light image segmentation loss Composition can be defined as:

[0036]

[0037] Infrared image segmentation loss Defined as:

[0038]

[0039] Infrared image segmentation loss Defined as:

[0040]

[0041] in, Represents the semantic segmentation label; and Represents the prediction result of semantic segmentation, H, W, and C represent the height, width, and number of channels of the output result, respectively.

[0042] Furthermore, in step 7, the infrared and visible light images are sent to the fusion sub-network and the semantic segmentation sub-network, and the extracted fusion features and semantic features are shuffled to obtain the classification features F cla , Fcla Send to the fusion-semantic feature classifier D cla , get the classification results:

[0043] I cla =D cla (F cla )

[0044] Among them, I cla Represents the classification result, I cla It consists of three values: 0, 1 / 2, and 1. 0 indicates that the feature is a fusion feature, 1 indicates a semantic feature, and 1 / 2 indicates that the feature is a mixture of the above two types of features and cannot be classified.

[0045] Furthermore, in step 8, the classification parameters are calculated and the classifier parameters are optimized. It can be defined as:

[0046]

[0047] in, Represents the classification label, which is generated based on the feature source before the feature order is disrupted. When the feature comes from the fusion feature, The values ​​in are all 0; when the feature source is a semantic feature, The values ​​in are all 1.

[0048] Furthermore, in step 9, the features extracted from the fusion sub-network are and After shuffling the order, send it to the frozen D cla .

[0049] Furthermore, in step 10, the classification loss in step 8 is used to calculate and fine-tune the encoder and Make its classification result close to 1 / 2. Enable the fusion encoder to extract features F containing rich fusion information and semantic segmentation information ir and F vis .

[0050] Furthermore, in step 11, the fine-tuned encoder and Output result F ir and F vis After splicing, it is sent to the decoder D f The fusion result is obtained by decoding and reconstruction.

[0051] Furthermore, in step 12, the fusion loss described in step 7 is used to calculate and fine-tune the decoder parameters in the fusion sub-network.

[0052] The beneficial effects of the present invention are:

[0053] 1. The present invention uses a fusion-semantic feature classifier to extract features containing rich fusion information and semantic information;

[0054] 2. The present invention utilizes features containing rich fusion information and semantic information to reconstruct a fused image of higher quality;

[0055] 3. The present invention utilizes features containing rich fusion information and semantic information to reconstruct high-quality fusion images that can better serve downstream advanced tasks; BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The present invention will be described in detail below in conjunction with the accompanying drawings and detailed embodiments. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative labor.

[0057] Figure 1 It is a schematic diagram of the process structure of the present invention;

[0058] Figure 2 The overall network structure diagram of the present invention; DETAILED DESCRIPTION

[0059] Example 1: Figure 1-Figure 2 As shown, the infrared and visible light image fusion method based on the fusion-semantic feature classifier, the specific steps of the method are as follows:

[0060] Step 1: Get the training data set. Each image group in the training data set consists of an infrared image, a visible light image, and corresponding semantic segmentation labels. The number of categories of the semantic segmentation label is 9.

[0061] Step 2: Unify the size of all images in the training dataset to 256×256, then randomly invert and randomly rotate the data, and normalize the processed images.

[0062] Step 3: Input the preprocessed infrared image and visible light image into the fusion subnetwork to obtain the fusion features and fusion results; the specific operation of obtaining the fusion features is as follows:

[0063] The preprocessed infrared image I ir and visible light image I vis Input the progressive feature extraction encoder E and obtain the infrared light feature F ir and visible light feature F vis , as follows:

[0064]

[0065] in, They represent the fusion sub-network feature extraction encoder E f Mid-infrared feature encoder and visible light feature encoder The output features of and They have the same structure, but the parameters are not shared; encoder and They are composed of 4 densely connected layers, each of which consists of a 3×3 convolution, a ReLU activation layer and a BatchNorm layer. and After concatenation along the channel dimension, it is fed into the decoder D f The fusion result is obtained by decoding and reconstruction. The specific process is as follows:

[0066]

[0067] Among them, I f represents the fusion result, cat[,] represents the concatenation operation along the channel dimension, and the decoder D f It consists of five convolutional layers, each of which consists of a 3×3 convolution, a ReLU activation layer, and a BatchNorm layer.

[0068] Step 4: Calculate the fusion loss, optimize the fusion sub-network parameters, and then freeze the fusion sub-network parameters;

[0069] Fusion Loss Loss of texture detail and content loss Composition can be defined as:

[0070]

[0071] Loss of texture detail Defined as:

[0072]

[0073] Among them, ||·||1 represents the l1 norm; Represents the gradient map extracted by the Sobel operator. Content loss Defined as:

[0074]

[0075] Step 5: Send the preprocessed infrared image and visible light image to the semantic segmentation sub-network respectively to obtain the semantic features and segmentation results of the infrared and visible light images respectively;

[0076] The specific operations of semantic feature acquisition are as follows:

[0077] The preprocessed infrared image I ir and visible light image I vis Input semantic segmentation subnetwork semantic feature extraction encoder E s , and obtain the semantic features of infrared images and semantic features of visible light images As shown below:

[0078]

[0079] in, Represents the infrared feature encoder in the semantic segmentation subnetwork and visible light feature encoder The output features of the semantic segmentation subnetwork are as follows. The encoder structure of the semantic segmentation subnetwork is the same as that of the fusion segmentation subnetwork. After extracting the semantic features of the infrared and visible light images respectively, and Separately fed into the semantic decoder and Get the semantic segmentation results of infrared and visible light images respectively:

[0080]

[0081] Among them, I s,ir Represents the semantic segmentation result of infrared image, I s,ir Represents the semantic segmentation result of the visible light image.

[0082] Step 6: Calculate the semantic segmentation loss, optimize the semantic segmentation sub-network parameters, and freeze the semantic segmentation sub-network parameters;

[0083] Fusion Loss The infrared image segmentation loss and visible light image segmentation loss Composition can be defined as:

[0084]

[0085] Infrared image segmentation loss Defined as:

[0086]

[0087] Infrared image segmentation loss Defined as:

[0088]

[0089] in, Represents the semantic segmentation label; and Represents the prediction result of semantic segmentation, H, W, and C represent the height, width, and number of channels of the output result, respectively.

[0090] Step 7: Send the infrared and visible light images to the fusion sub-network and the semantic segmentation sub-network, and send the extracted fusion features and semantic features to the fusion-semantic feature classifier to obtain the classification results;

[0091] The infrared and visible light images are sent to the fusion sub-network and the semantic segmentation sub-network, and the extracted fusion features and semantic features are shuffled to obtain the classification features F cla , F cla Send to the fusion-semantic feature classifier D cla , get the classification results:

[0092] I cla =D cla (F cla )

[0093] Among them, I cla Represents the classification result, I cla It consists of three values: 0, 1 / 2, and 1. 0 indicates that the feature is a fusion feature, 1 indicates a semantic feature, and 1 / 2 indicates that the feature is a mixture of the above two types of features and cannot be classified.

[0094] Step 8: Calculate classification loss, optimize classifier parameters, and freeze classifier parameters. Classification loss It can be defined as:

[0095]

[0096] in, Represents the classification label, which is generated based on the feature source before the feature order is disrupted. When the feature comes from the fusion feature, The values ​​in are all 0; when the feature source is a semantic feature, The values ​​in are all 1.

[0097] Step 9: Unfreeze the fusion sub-network parameters, send the fusion sub-network features to the classifier, and extract features containing rich semantics and fusion information;

[0098] Specifically, the features extracted from the fusion sub-network are and After shuffling the order, send it to the frozen D cla .

[0099] Step 10: Calculate the classification loss, optimize the encoder parameters of the fusion sub-network, and freeze the encoder parameters in the fusion sub-network;

[0100] Specifically, we use the classification loss in step 8 to calculate and fine-tune the encoder and Make its classification result close to 1 / 2. Enable the fusion encoder to extract features F containing rich fusion information and semantic segmentation information ir and F vis .

[0101] Step 11: Send the frozen encoder features to the decoder for reconstruction to obtain the fusion result;

[0102] Specifically, the fine-tuned encoder and Output result F ir and F vis After splicing, it is sent to the decoder D f The fusion result is obtained by decoding and reconstruction.

[0103] Step 12: Calculate the fusion loss and optimize the decoder parameters.

[0104] Specifically, the fusion loss described in step 7 is used to calculate and fine-tune the decoder parameters in the fusion sub-network.

[0105] Furthermore, in order to verify the effectiveness of the method of the present invention, the present invention performed a performance evaluation on the proposed method and conducted experiments on the public data set MSRS. During the experiment, the model of the present invention was trained on the training set of the MSRS data set and tested on the validation set. The training set contains 1083 image groups, and the validation set contains 361 image groups. Each image group includes an infrared image, a visible light image, and a corresponding semantic segmentation label. The algorithm of the present invention is developed based on the PyTorch1.13.1 framework and is trained on an NVIDIA GTX 3090 graphics card (with a video memory of 24G). During the training process, the Adam optimizer is used to optimize the model parameters. The batch size is set to 4, the initial learning rate is 0.0001, and the learning rate is adjusted using the cosine annealing algorithm. The entire training process was iterated for a total of 2500 rounds.

[0106] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A method for fusion of infrared and visible light images based on a fusion-semantic feature classifier, characterized by: The specific steps of the method are as follows: Step 1: Obtain a training dataset containing registered infrared and visible light image pairs and semantic segmentation labels; Step 2: Preprocess all data in the training dataset; Step 3: Input the preprocessed infrared image and visible light image into the fusion sub-network to obtain fusion features and fusion results; Step 4: Calculate the fusion loss, optimize the fusion sub-network parameters, and then freeze the fusion sub-network parameters; Step 5: Send the preprocessed infrared image and visible light image to the semantic segmentation sub-network respectively to obtain the semantic features and segmentation results of the infrared and visible light images respectively; Step 6: Calculate the semantic segmentation loss, optimize the semantic segmentation sub-network parameters, and freeze the semantic segmentation sub-network parameters; Step 7: Send the fusion features and semantic features corresponding to the frozen fusion sub-network and semantic segmentation sub-network to the fusion-semantic feature classifier to obtain the classification results; In step 7, the fusion features and semantic features corresponding to the frozen fusion sub-network and the semantic segmentation sub-network are shuffled to obtain the classification feature F cla , F cla Send to the fusion-semantic feature classifier D cla , get the classification results: I cla =D cla (F cla ) Among them, I cla Represents the classification result, I cla It consists of three values: 0, 1 / 2, and 1, where 0 represents the classification feature F cla is the fusion feature, 1 represents the classification feature F cla is a semantic feature, and 1 / 2 represents a classification feature F cla The classification cannot be determined by mixing the fusion features and semantic features; Step 8: Calculate classification loss, optimize classifier parameters, and freeze classifier parameters; Step 9: Unfreeze the fusion sub-network parameters and send the corresponding fusion features to the classifier; Step 10: Calculate the classification loss, optimize the encoder parameters of the fusion sub-network, and freeze the encoder parameters in the fusion sub-network; Step 11: Send the frozen encoder features to the decoder for reconstruction to obtain the fusion result; Step 12: Calculate the fusion loss and optimize the decoder parameters.

2. The infrared and visible light image fusion method based on fusion-semantic feature classifier according to claim 1 is characterized in that: In step 1, a training data set is obtained, and each image group in the training data set consists of an infrared image, a visible light image, and a corresponding semantic segmentation label; wherein the number of categories of the semantic segmentation label is 9.

3. The infrared and visible light image fusion method based on fusion-semantic feature classifier according to claim 1 is characterized in that: In step 2, the size of all images in the training data set is unified to 256×256, and then the data is randomly inverted and randomly rotated, and the processed images are normalized.

4. The infrared and visible light image fusion method based on fusion-semantic feature classifier according to claim 1, characterized in that: In step 3, the specific operation of fusion feature acquisition is as follows: The preprocessed infrared image I ir and visible light image I vis Input fusion sub-network feature extraction encoder E f , and obtain infrared characteristics and visible light characteristics As shown below: in, They represent the fusion sub-network feature extraction encoder E f Mid-infrared feature encoder and visible light feature encoder The output features of and They have the same structure, but the parameters are not shared; encoder and They are composed of 4 densely connected layers, each of which consists of a 3×3 convolution, a ReLU activation layer and a BatchNorm layer. and After concatenation along the channel dimension, it is fed into the decoder D f The fusion result is obtained by decoding and reconstruction. The specific process is as follows: Among them, I f represents the fusion result, cat[,] represents the concatenation operation along the channel dimension, and the decoder D f It consists of five convolutional layers, each of which consists of a 3×3 convolution, a ReLU activation layer, and a BatchNorm layer.

5. The infrared and visible light image fusion method based on fusion-semantic feature classifier according to claim 1, characterized in that: In step 4, the fusion loss is calculated and the network parameters are optimized; the fusion loss Loss of texture detail and content loss Composition, defined as: Loss of texture detail Defined as: Among them, ||·||1 represents the l1 norm; Indicates that the Sobel operator extracts the gradient map and the content loss Defined as:

6. The infrared and visible light image fusion method based on fusion-semantic feature classifier according to claim 1, characterized in that: In step 5, the specific operation of acquiring semantic features is as follows: The preprocessed infrared image I ir and visible light image I vis Input semantic segmentation sub-network semantic feature extraction encoder E respectively s , and obtain the semantic features of infrared images and semantic features of visible light images As shown below: in, Represents the infrared feature encoder in the semantic segmentation subnetwork and visible light feature encoder After extracting the semantic features of infrared and visible light images, and Separately fed into the semantic decoder and Get the semantic segmentation results of infrared and visible light images respectively: Among them, I s,ir Represents the semantic segmentation result of infrared image, I s,vis Represents the semantic segmentation result of the visible light image.

7. The infrared and visible light image fusion method based on fusion-semantic feature classifier according to claim 1, characterized in that: In step 6, the segmentation loss is calculated, the network parameters are optimized, and the segmentation loss The infrared image segmentation loss and visible light image segmentation loss Composition, defined as: Infrared image segmentation loss Defined as: Visible light image segmentation loss Defined as: in, Represents the semantic segmentation label; and Represents the infrared image I ir and visible light image I vis The prediction result of semantic segmentation, H, W, C represent the height, width and number of channels of the output result respectively.

8. The infrared and visible light image fusion method based on fusion-semantic feature classifier according to claim 1, characterized in that: In step 8, the classification loss is calculated, the classifier parameters are optimized, and the classification loss Defined as: in, Represents the classification label, which is generated based on the feature source before the feature order is disrupted. When the feature comes from the fusion feature, The values ​​in are all 0; when the features are derived from semantic features, The values ​​in are all 1.

9. The infrared and visible light image fusion method based on fusion-semantic feature classifier according to claim 1, characterized in that: In step 9, the features extracted from the fusion sub-network are and After shuffling the order, send it to the frozen D cla ; In step 10, the classification loss in step 8 is used to calculate and fine-tune the encoder and Make its classification result close to 1 / 2; enable the fusion encoder to extract features F containing rich fusion information and semantic segmentation information ir and F vis ; In step 11, the fine-tuned encoder and Output result F ir and F vis After splicing, it is sent to the decoder D f Decode and reconstruct to obtain the fusion result; In step 12, the fusion loss is calculated and the decoder parameters in the fusion sub-network are fine-tuned.

Images