Multi-band image description generation method based on feature fusion

Abstract

This invention relates to image description and image fusion methods, and more particularly to a method for generating multi-band image descriptions, specifically a method based on feature fusion. By introducing infrared detection imaging into the field of image description and establishing a visible-light-infrared image description dataset, this method first uses a multi-layer convolutional neural network to extract features from visible-light and infrared images, respectively. Based on the complementarity of different detection bands, a feature fusion enhancement module is designed to achieve feature fusion enhancement at the spatial and channel levels. Finally, an attention mechanism enhancement module is constructed to establish a deep relationship between image and text, eliminating noise generated by traditional additive attention mechanisms. This method enables multi-band image description generation, which can be used for understanding complex scenarios such as security monitoring and military reconnaissance.

Description

Technical Field

[0001] The present invention relates to an image description method and an image fusion method, and in particular to a multi-band image description generation method, specifically a multi-band image description generation method based on feature fusion. Background Art

[0002] Image captioning is a cross-modal research task involving computer vision and natural language processing. Its goal is to automatically generate natural language descriptions for input images. Image captioning technology has enormous application value and research significance, with applications in scenarios such as human-computer interaction, intelligent information fusion, and medical imaging report generation.

[0003] Image description generation methods follow an "encoder-decoder" architecture. Early methods used convolutional neural networks as encoders, fixing the image into a set of vectors that were then input into a recurrent neural network for decoding. Subsequently, most image description research adopted an encoder-decoder architecture based on the Transformer paradigm. The Transformer has a powerful ability to encode long-sequence relationships, thereby capturing global image features. Although deep learning-based methods can generate image descriptions of variable length, highly relevant to image content, and rich in semantics, their objects are all visually distinguishable. Existing high-precision intelligent detection systems are often used in fields such as security monitoring and military reconnaissance confrontation, where stealth and anti-stealth competition exist. The image understanding targets here are mainly multi-band detection in complex scenes such as night scenes, deep space, and the ocean. Therefore, the existing image description research paradigm based on visible light detection has long been far behind.

[0004] Infrared imaging technology is less susceptible to environmental influences and has excellent visibility through dust, smoke, snow, and rain, capturing clear thermal radiation from targets. For example, in military reconnaissance scenarios, infrared imaging technology can easily identify targets hidden or camouflaged in bushes and jungle, effectively avoiding potential dangers. In security surveillance scenarios, infrared thermal imagers passively receive the target's own thermal infrared radiation, enabling monitoring of various targets, including people and vehicles.

[0005] To this end, it is necessary to propose a multi-band image description generation method that can solve the problem of inaccurate description results in night scenes, occluded scenes and low-resolution image scenes. Summary of the Invention

[0006] In order to solve the problem that existing image description generation methods are not suitable for night scenes, scenes with obscured objects and low-resolution images, the present invention proposes a multi-band image description generation method based on feature fusion.

[0007] The present invention is implemented by adopting the following technical solution: a multi-band image description generation method based on feature fusion, comprising the following steps:

[0008] We designed and constructed an image description generation network consisting of an image feature extraction module, a feature fusion enhancement module, and an attention-enhanced decoder. Given the varying feature distributions of images in different bands, different convolutional neural networks (CNNs) were used to extract features from the dual-band images. Leveraging the complementarity of different detection bands, a cross-attention module was designed to enhance the features of visible and infrared images at the spatial level. At the channel level, a channel-wise attention mechanism was used to redistribute weights for feature fusion enhancement. Finally, an attention mechanism enhancement module was constructed, extending the traditional additive attention mechanism to establish a deep relationship between word embeddings and fused feature maps. Long Short-Term Memory (LSTM) networks were then used as language models for text decoding.

[0009] The image feature extraction module consists of two Residual Networks (ResNets) of different depths. ResNets have a short-connection structure, which alleviates the vanishing gradient and network degradation problems associated with increasing network depth. ResNet152 is used to extract features for visible light images, and ResNet50 for infrared images. Specifically, the final average pooling layer and classification layer are removed, and a maximum adaptive pooling layer is added.

[0010] The feature fusion enhancement module consists of two submodules: spatial attention and channel attention. The spatial attention module is composed of a cross-attention module (CAM) and a feedforward neural network (FFN). In an image, the self-attention mechanism learns the relationship between a pixel and other pixels, capturing long-term dependencies and achieving a larger receptive field. Its core idea is to first map image features into three different spaces: the query vector Q, the key vector K, and the value vector V. Secondly, the similarity between the query vector Q and the key vector K is calculated to obtain a weight map. Finally, the weighted sum of the weight map and the corresponding value vector is performed to obtain the attention feature map. The core idea of ​​the multi-head attention mechanism is to combine different subspace representations of the query vector, key vector, and value vector to learn the behavior of different spaces. In other words, each independent set of Q, K, and V represents a "head." Designing the CAM based on the multi-head attention mechanism achieves feature fusion enhancement at the spatial level. Relevant research has shown that channel information is closely related to the generation of function words. In order to further enhance the dual-band image features, the channel attention mechanism is used to aggregate information in the spatial domain, redistribute weights for the fused feature map, and achieve feature fusion enhancement at the channel level.

[0011] In the field of natural language processing, LSTM, due to its unique gating mechanism, alleviates the vanishing gradient problem of recurrent neural networks to a certain extent. Therefore, this paper uses LSTM as a language model. To mitigate the impact of the image-text correlation weight map on the description results and eliminate the interference of irrelevant variables, an attention enhancement module is designed to expand on traditional additive attention. This module establishes a relationship between word vectors and the image-text correlation weight map, thereby guiding the LSTM to correctly decode.

[0012] The model is represented by a special symbol <start>Starting symbol, special symbol <end>End symbol, special symbol <pad>To fill in the symbol, the probability distribution P(y i |y 1:i-1 ,X), X represents the input image, y 1:i-1 Represents given 1 to i-1 words, y i Represents the i-th word to be predicted. The training of the network structure follows the general training method of the image description model and is divided into a cross-entropy loss function training phase and a reinforcement learning optimization phase. The goal of cross-entropy training is to minimize the negative log-likelihood of a word. Since subsequent words are affected by previous words when predicting a word, this can lead to error accumulation. At the same time, cross-entropy training focuses on reducing loss during the training phase and on evaluation indicators during the testing phase, which can lead to exposure bias. Therefore, using the evaluation indicator CIDEr (Consensus-based Image Description Evaluation) as the reward score, a policy gradient algorithm is used for optimization to minimize the negative expected reward. The dropout algorithm is used during training to prevent model overfitting, and the parameter is set to 0.5. The experiment is based on the PyTorch framework and uses the Adam gradient optimization algorithm for parameter update.

[0013] The present invention realizes end-to-end image description of multi-band images, solving the problem that traditional image description methods cannot generate accurate descriptions for night scenes, obscured scenes and low-resolution images, and provides technical support for high-precision intelligent detection systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is the overall structure diagram of the network.

[0015] Figure 2 This is the structural diagram of the feature fusion enhancement module.

[0016] Figure 3 Module structure diagram for attention mechanism enhancement.

[0017] Figure 4 Schematic diagram of a dual-band image description result example a.

[0018] Figure 5 Schematic diagram of example b of dual-band image description results.

[0019] Figure 6 Schematic diagram of sample c of dual-band image description results. DETAILED DESCRIPTION

[0020] A multi-band image description generation method based on feature fusion, comprising the following steps:

[0021] 1. Design and build feature extraction module

[0022] The feature extraction module of the present invention is composed of two residual networks of different depths. ResNet152 is used to extract features of the visible light image I1, the last average pooling layer and classification layer are removed, and the maximum adaptive pooling layer is added. The formula is as follows: v =ResNet152(I1), where I v Represents the extracted visible light image features. ResNet50 is used to extract features from the infrared image I2, removing the last average pooling layer and classification layer, and adding a maximum adaptive pooling layer. The formula is as follows: i =ResNet50(I2), where I i Represents the extracted infrared image features.

[0023] 2. Design and build feature fusion enhancement module

[0024] The feature fusion enhancement module of the present invention is composed of two submodules: spatial attention and channel attention. At the feature space level, the source image feature I v (I i ) Fusion target image features I i (I v ) to enhance its own features; then, the two types of features are spliced ​​along the channel dimension, and the channel attention mechanism is used to redistribute the weights of the fused feature map.

[0025] (1) Spatial Attention Module

[0026] The spatial attention module primarily fuses image features from different bands at the feature space level. It consists of a camera (CAM) and a FFN (Full Focus Network). The self-attention mechanism captures global information to achieve a larger receptive field and contextual information. The multi-headed self-attention (MHSA) mechanism has the ability to represent multiple subspaces. Therefore, based on the complementary nature of different detection bands, a CAM is designed with the MHSA as the main component to enhance the spatial features of each image band.

[0027] ① Enhanced visible light characteristics

[0028] In order to enhance the spatial features of visible light images, the visible light features I v Mapped into key vectors and value vectors, the infrared feature I i Mapped to query vector, Q i =W Q I i ,K v =W K I v ,V v =W V I v , where W Q 、W K and W V are all learnable weight matrices, Q i , K v and V v They are all obtained through linear mapping, representing the query vector, key vector and value vector; then, the scaled dot product attention is used to calculate the similarity between the query vector and the key vector, and the weighted value vector is used to obtain the "one-head" enhanced visible light image feature. in, represents the visible light image features after feature enhancement, subscript · v Indicates visible light characteristics, superscript · 1 Means "one end", d k Represents the scaling factor. Repeat the calculation 8 times to get 8 "heads". Concatenate the "head" matrices. The multi-head attention mechanism is formulated as follows: Where concate represents the concatenation operation and W1 represents the learnable weight matrix. The visible light feature is enhanced by CAM and FFN. The residual structure and layer normalization (Layer Normalization, LayerNorm) are both applied to MHSA and FFN, which can be expressed as I' v =LayerNorm(I v +MHSA(Q i ,K v ,V v )), Represents the visible light image features with enhanced spatial features; FFN consists of two linear layers and an activation function, and FFN can be expressed as FFN(I' v )=(I' v W2+b1)W3+b2, where W2, b1, W3 and b2 are all learnable weight parameters.

[0029] ②Infrared feature enhancement

[0030] In order to enhance the spatial features of infrared images, the infrared features I i Mapped into key vector and value vector, visible light feature I v Mapped to query vector, Q v =W q I v ,K i =W k I i ,V i =W v I i , where W q 、W k and W v are all learnable weight matrices, Q v , K i and V i They are all obtained through linear mapping, representing the query vector, key vector and value vector; then, the scaled dot product attention is used to calculate the similarity between the query vector and the key vector, and the weighted value vector is used to obtain the "one-head" enhanced infrared image feature. in, represents the infrared image features after feature enhancement, subscript · i Indicates infrared characteristics, superscript · 1 Means "one end", d k Represents the scaling factor, and the calculation is repeated 8 times. The multi-head attention mechanism is formulated as follows: Where concate represents the concatenation operation, and W4 represents the learnable weight matrix. The infrared feature is enhanced by CAM and FFN. i =LayerNorm(I i +MHSA(Q v ,K i ,V i ))FFN(I' i )=(I' i W5+b3)W6+b4, Among them, W5, b3, W6 and b4 are all learnable weight parameters. Indicates infrared image features with enhanced spatial features.

[0031] (2) Channel Attention Module

[0032] In order to enhance the channel features of the fusion feature map, the visible light image features and infrared image features The fused feature map F is spliced ​​in the channel dimension, and the channel attention mechanism is applied to redistribute the weights of the image features of different bands. The global average pooling (GAP) and global maximum pooling (GMP) are applied to each channel of F to aggregate the information in the spatial domain. The aggregated features are then fed into two fully connected layers, mapped to a 0-1 vector by the sigmoid function, and multiplied with the original fused feature map F to obtain the fused feature enhancement map F. * . It can be expressed as Among them, W7 and W8 are both learnable weight matrices. represents matrix cross product, ⊙ represents dot product, F * Represents the fused feature enhancement map.

[0033] 3. Design and build an attention-enhanced decoder

[0034] This paper adopts a variant of the paper "Attention on Attention for Image Captioning", expands the traditional additive attention mechanism, and fuses features to enhance the image F * And the text gets the attention result through the additive attention mechanism To measure attention outcomes The correlation between the attention result and the word vector is eliminated, and the interference of irrelevant variables is eliminated. The attention result and the word vector are subjected to two linear mappings to obtain an "information vector" and an "information gate". and in, b i and b g are all learnable parameters, x t Represents the word vector at the tth time step, i represents the "information vector", and g represents the "information gate". Apply the "information gate" to the "information vector", Get the fusion enhanced feature map after attention enhancement The fused enhanced feature map after attention enhancement With word vector x t After concatenation, it is input to the next time step of LSTM for decoding.

[0035] 4. Neural Network Training

[0036] In the cross entropy loss function training phase, the goal is to minimize the negative log likelihood of generating each word. in, Indicates the words that have been generated. Indicates the word to be predicted, T represents the sentence length, and θ represents the model parameters. In the reinforcement learning optimization phase, the model is optimized using the CIDEr score as the reward, and the mean reward of the beam search sampling is used as the reward baseline. The goal is to minimize the negative expected reward. Among them, w i represents the i-th sentence of beam search, r(·) represents the reward function, k represents the width of beam search, which is set to 5, Represents the reward baseline. Similarly, in the test phase, beam search is used for decoding, and the sequence with the highest probability in the beam search is retained as the description result.

[0037] 5. Multi-band image description based on feature fusion

[0038] (1) Input the multi-band image into the neural network, and ResNet152 extracts the visible light image features, and ResNet50 extracts the infrared image features;

[0039] (2) The visible light image features and infrared image features are enhanced by the feature fusion module to achieve feature enhancement at the spatial and channel levels, and a fusion enhanced feature map is obtained;

[0040] (3) The additive attention mechanism is used to calculate the weights of the fused enhanced feature map and the text, and the attention enhancement module is used to establish the deep relationship between the fused enhanced feature map and the text to eliminate the interference of irrelevant variables;

[0041] (4) Input into LSTM to generate description text.

[0042] The above-mentioned multi-band image description generation method based on feature fusion is based on the lack of open source visible light image and infrared image description datasets. We use the visible light and infrared tracking datasets CAMEL Dataset, CSR_GTOT and RGB-T234 datasets. We manually select 2000 image pairs and manually configure 5 descriptions for each image pair. The data is preprocessed and enhanced. All description texts are converted to uppercase and lowercase, and special characters are used. <unk>The words with less than 5 occurrences are marked and replaced, and the final dictionary size is 3038.

[0043] In the above-mentioned multi-band image description generation method based on feature fusion, in the cross-entropy training stage, the initial learning rate of the encoder and decoder is 1e-4, and the training is carried out for 30 epochs, with a decay of 0.8 every 5 epochs, and the batch size is set to 32; in the reinforcement learning optimization stage, the initial learning rate of the encoder and decoder is 5e-5, and the training is carried out for 5 epochs with a batch size of 32.

[0044] The present invention relates to an image fusion and image description method, specifically a multi-band image description generation method based on feature fusion. The present invention is carried out according to the following steps: first, a visible light image-infrared image description dataset is constructed; second, a multi-band image description generation model based on feature fusion is constructed, the main content of which is composed of an image feature extractor, a feature fusion enhancement module and an attention-enhanced decoder; finally, cross-entropy training is used as the basic training method, and a self-critical training strategy is used to optimize the model. The model process is as follows: a dual-band image pair is input into the model to extract their respective features; then feature fusion enhancement is performed at the spatial and channel levels; finally, the correlation coefficient between the fused feature map and the word vector is calculated, the relationship between the correlation coefficient and the word vector is established, the interference of irrelevant variables is eliminated, and the image description is generated by inputting it into the LSTM. The present invention realizes end-to-end image description of multi-band images, solves the problem that traditional image description methods cannot generate accurate descriptions for night scenes, occluded scenes and low-resolution images, and provides technical support for high-precision intelligent detection systems.< / unk> < / pad> < / end> < / start>

Claims

1. A multi-band image description generation method based on feature fusion, characterized by: The following steps are involved: Design and build a multi-band image description generation model: The image description generation model is based on an encoder-decoder architecture. The encoder consists of an image feature extraction module and a feature fusion enhancement module, and the decoder consists of a long short-term memory network with an attention mechanism enhancement module. Among them, the image feature extraction module is divided into a visible light image feature extractor and an infrared image feature extractor. The visible light feature extractor removes the average pooling layer and classification layer of ResNet152 and adds maximum adaptive pooling to obtain visible light image features; the infrared image feature extractor removes the average pooling layer and classification layer of ResNet50 and adds maximum adaptive pooling to obtain infrared image features; Among them, the feature fusion enhancement module is divided into two sub-modules: spatial attention and channel attention. In the spatial attention module, when visible light features are enhanced, the infrared image feature map is used as the query vector, and the visible light image feature map is used as the key vector and value vector. The multi-head attention mechanism is used to fuse features at the spatial level. When infrared features are enhanced, the infrared image feature map is used as the key vector and value vector, and the visible light image feature map is used as the query vector. The multi-head attention mechanism is used to fuse features at the spatial level. In the channel attention module, the dual-band image features are first spliced ​​along the channel dimension, and the global average pooling and global maximum pooling are used to aggregate the information in the spatial domain. After two linear layers and the sigmoid activation function, it is mapped to a 0-1 vector and applied to the fusion feature map. The features are fused at the channel level to obtain a fusion enhanced feature map. Among them, the attention mechanism enhancement module, based on the calculation of the additive attention of the fused enhanced feature map and the word vector, maps the word vector and the additive attention result map to a 0-1 vector through two linear layers and a sigmoid activation function, and applies it to the attention result map to obtain the fused enhanced feature map after attention enhancement. The fused enhanced feature map after attention enhancement is connected in series with the word vector and input into the long short-term memory network to generate a description text.

2. The method for generating multi-band image description based on feature fusion according to claim 1, characterized in that: The multi-band image dataset contains registered visible light images and infrared images. Five description texts are manually configured for each image pair. The dataset is used to train a multi-band image description generation model.

3. The method for generating multi-band image description based on feature fusion according to claim 2, characterized in that: The training of the multi-band image description generation model is divided into the cross-entropy loss function training stage and the reinforcement learning optimization stage. The goal of the cross-entropy loss function training is to minimize the negative log-likelihood of words. Since the following words will be affected by the previous words when predicting words, it will lead to the problem of error accumulation. At the same time, the cross-entropy loss function training focuses on the reduction of loss in the training stage and the evaluation index in the testing stage, which will lead to the exposure bias problem. Therefore, the reinforcement learning optimization stage uses the evaluation index CIDEr as the reward score and adopts the policy gradient algorithm for optimization to minimize the negative expected reward. The dropout algorithm is used during training to prevent the model from overfitting. The parameter is set to 0.

5. The experiment is based on the pytorch framework and the Adam gradient optimization algorithm is used for parameter update.

4. The method for generating multi-band image description based on feature fusion according to claim 3, characterized in that: In the image feature extraction module, the feature channel dimension of the visible light image is 2048, and that of the infrared image is 512. The multi-head attention mechanism of the feature fusion enhancement module is set to 8 heads, and d_model is 512. In the cross-entropy loss function training phase, the initial learning rate of the encoder and decoder is 1e-4, the training epoch is set to 30, and the decay is 0.8 every 5 epochs, and the batch size is set to 32. In the reinforcement learning optimization phase, the initial learning rate of the encoder and decoder is 5e-5, and 5 epochs are trained, with a batch size of 32.

Images