Dual-band image description generation method based on feature alignment fusion

Abstract

The invention provides a dual-band image description generation method based on feature alignment fusion, aiming at solving the problems that effective features are difficult to extract only depending on a visible light image under a complex background and are easily influenced by surrounding environments and interferents and the like. According to the network model, firstly, input visible area and infrared grid features are transmitted into a fusion position attention module and an FFN to respectively obtain fusion position features; then, attention calculation is carried out on the fused position features by using a hybrid cross module, and the output features comprise information of different wavebands; and finally, outputting the text description of the image pair through a decoder module and an adaptive module of the traditional Transform so as to realize the image description under the complex background. According to the method, complex scenes (such as special scenes of target shielding, dark environment, rain and fog and the like) can be comprehensively and accurately described.

Description

Technical Field

[0001] The present invention belongs to the field of infrared visible image description, that is, a deep network model is used to input the features of two images of different bands and output a description sentence containing the contents of the two images, specifically a dual-band image description generation method based on feature alignment and fusion. Background Art

[0002] Image description technology is mainly used in artificial intelligence related fields such as human-computer interaction, blind assistance, and autonomous driving. Its task is to enable the computer to automatically generate a sentence describing the main content of a given image. As an emerging field that intersects computer vision and natural language processing, image description research with a single visible light image as the object and deep neural network as the main technical feature has achieved fruitful results. However, these results cannot meet the actual needs of the current widespread use of multi-band synchronous imaging technology for complex scene detection. Except for the initial exploration of the research team, no relevant research has been retrieved, which seriously affects the intelligent process of high-precision detection systems. Therefore, it is urgent and necessary to carry out multi-band image description tasks.

[0003] At present, the existing image description research mainly adopts the "encoder-decoder" structure, among which the convolutional neural network and the recurrent neural network as the encoder-decoder are the most widely used structures in the early image description, but this type of method cannot capture long-distance contextual information, which causes this type of research to stagnate. Thanks to the powerful long-distance dependency modeling ability of the self-attention mechanism, the Transformer framework was introduced into the field of image description and gradually became the mainstream method in the field of image description. The research basis based on the Transformer architecture is to extract regional-level features, wherein the regional-level features include image salient area features (abbreviated as regional features) and image grid area features (abbreviated as grid features). Since the regional features and the grid features have their own pertinence, the present invention considers the complementary advantages of the two features to achieve improved description effects. In addition, the pre-trained language models in the field of natural language processing have developed rapidly, and these models contain rich text information. Consider introducing them into the field of image description to enhance the richness of the text description content and the fluency of the logic. Summary of the invention

[0004] In order to solve the problem that only visible light images are used under complex background conditions and the detected scene information cannot be fully and accurately described, the present invention proposes a dual-band image description generation method based on feature alignment and fusion. The method is suitable for image description under complex backgrounds such as night and occlusion, and can achieve a comprehensive and accurate description of the current scene.

[0005] The present invention adopts the following technical solution: a dual-band image description generation method based on feature alignment and fusion, the steps of which are as follows:

[0006] Step 1: Extract visible light image I v Visible area feature R V = {r i} M , extract infrared image I r The infrared grid feature G I = {g j} N ,The proposed features all contain appearance features and location information, where M and N represent the total number of visible area features and infrared grid features extracted from each image, respectively;

[0007] Step 2: Input the obtained visible area features and infrared grid features into the multi-head fusion position attention module to combine the appearance features with the position information, and then pass through the first feedforward neural network FFN respectively. This process is repeated T times. After T times, the output of the first feedforward neural network FFN is input into the multi-head hybrid cross attention module;

[0008] Step 3: The visible area features and infrared grid features are fused and enhanced through a multi-head hybrid cross attention module. The fused and enhanced results are input into the second feedforward neural network FFN. This process is repeated T times. After T times, the output of the second feedforward neural network FFN is concatenated to obtain visual information U.

[0009] Step 4: Input the visual information U into the Transformer decoder and decode it to get the hidden state h t ;

[0010] Step 5: Decode the hidden state h t The visual information U encoded by the encoder and the language information s generated by the language model t The two are input into the adaptation module together to obtain a text description.

[0011] This method requires the construction of an encoding part and a decoding part. The core of the encoding part is the Multi-head Fusion Positional Attention (MHFPA) and the Multi-head Mixed Cross Attention (MHMCA); the decoding part consists of a Transformer decoder and an adaptive module.

[0012] In the method, the multi-head fusion positional attention module realizes the combination of appearance features and position information, and is composed of h fusion positional attentions (FPA). The appearance features and position information are input into the fusion positional attention. The appearance features are transformed into vectors Q, K, and V through different linear projections. The position information is encoded into pos through absolute position encoding and into matrix Ω through relative position encoding. Pos is integrated into Q and K. The integrated vectors Q+pos and K+pos are used as the query vector and key vector in FPA. The scaled dot product is used to calculate the similarity score between the query vector and the key vector. The matrix Ω is then used to adjust the attention weight. Finally, it is normalized by softmax and mapped into a probability distribution and weighted with the value vector V to obtain the feature representation of one of the heads. Repeat the calculation h times to obtain the feature representation of h heads. Each feature representation is recorded as head i , and then concatenate the feature representations of h heads to achieve multi-head fusion position attention, where i represents 1~h.

[0013] The multi-head mixed cross attention module is composed of i mixed cross attention (MCA). Taking the visible area feature as the query, the infrared grid feature as the key and value, the visible area feature and infrared grid feature information are embedded in the infrared grid feature and visible area feature respectively and the original feature representation is strengthened. As shown in the formula As shown, Q V Represents C′ r The output of linear projection, K I and V I Represents C′ g The output of linear projection, pos V and pos I They represent the absolute position coding of the current infrared visible region feature and the absolute position coding of the current infrared grid feature, respectively. The vector Q after integrating the absolute position coding information of each V +pos V With K I +pos I As the query vector and key vector in MCA, the scaled dot product is used to calculate the similarity score between the two, and the relative position information matrix Ω is used to adjust the attention weight. Finally, it is normalized by softmax and mapped into a probability distribution and value vector V I The weighted feature representation of one of the heads is obtained. The logarithm of the relative position encoding matrix Ω is taken for numerical stability. G is the constructed visible area-infrared grid alignment map. Repeat the calculation h times to obtain the feature representation of h heads. Each feature representation is recorded as head i, and then concatenate the feature representations of h heads to obtain the output of multi-head mixed cross attention.

[0014] The adaptive module is similar to the masked attention module of the Transformer decoder, except that the input K and V are the visual information U output by the encoder and the language information s output by the language model. t Concatenate the vectors after different linear transformations. Among them, the language information s t The Bert-based language model is used to obtain the information in the Transformer decoder. This is to solve the problem of severe language deviation caused by the high coupling between memory information and hidden information. The input in this part of the training process is a complete real description sentence, and the test process only receives some words generated by the model. The language model is trained only using cross entropy loss. After the training is completed, all parameters are frozen, and the output s after mask attention is t As the language information of the current moment.

[0015] In order to solve the problems that it is difficult to extract effective features by visible light images alone under complex backgrounds and that they are easily affected by the surrounding environment and interferences, the present invention proposes a dual-band image description generation method based on feature alignment and fusion. This method can provide a comprehensive and accurate description of complex scenes (such as special scenes such as obscured targets, dim environments, rain and fog). BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is the overall framework diagram of the present invention.

[0017] Figure 2 The encoding part of the present invention and its main module structure diagram, where (a) is the overall structure of the encoding part, (b) is the fused position attention structure, and (c) is the hybrid cross attention structure.

[0018] Figure 3 Schematic diagram of the adaptive module of the present invention.

[0019] Figure 4 It is a qualitative comparison diagram of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be further described below in conjunction with the accompanying drawings:

[0021] Figure 1 The overall framework diagram of the present invention is shown in FIG. First, the input visible light image I v Faster-RCNN extracts the visible area features of the image to get R V = {r i} M , infrared image I rThe infrared grid feature G of the image is extracted by Faster-RCNN I = {g j} N , the proposed features all contain appearance features and position information, where M and N represent the total number of visible area features and infrared grid features extracted from each image, respectively; secondly, the obtained visible area features and infrared grid features are input into the VIIAF encoder for encoding, and the encoded visual information U fuses and enhances the information of the visible light image and the infrared image; finally, the visual information U is input into each layer of the Transformer decoder, and the decoded hidden state h t The visual information U encoded by the encoder and the language information s generated by the language model t The adaptive module is jointly input to establish a deep relationship between image and text to achieve visible light and infrared dual-band image description.

[0022] Figure 2 It is the encoder part. The core of the encoder is the multi-head fusion position attention module and the multi-head mixed cross attention module. First, the visible area features and infrared grid features are fused with position attention respectively, and then pass through the first feedforward neural network FFN to realize the combination of appearance features and position information; then, the visible area features and infrared grid features are fused and enhanced through mixed cross attention and the first feedforward neural network FFN, and finally the fused and enhanced results are sent to the second feedforward neural network FFN, and the output of the second feedforward neural network FFN is concatenated to obtain visual information. The specific structure is as follows: Figure 2 (a) shown.

[0023] Figure 3 is an adaptive module. After the Transformer decoder, the hidden state h is obtained. t , the adaptation module is similar to the Transformer decoder layer, h t After a masked multi-head attention, the processed visual information U and language information s t After multi-head attention, the results of the multi-head attention are spliced ​​together, and finally the text output of the image pair can be obtained.

[0024] Establishment of the data set: The present invention uses a self-made data set made by the research group. The data set images come from the open source data sets CAMELDataset, CSR_GTOT and RGB-T234 data sets, with a total of 2000 pairs of images. Each pair of images is manually configured with 5 description sentences, and then the 2000 pairs of images are expanded to 4000 pairs through data enhancement. The image expansion method is to obtain the original image by rotating and cropping, and the text expansion method is back translation, that is, English-German-English. The expanded images and texts constitute image-text pairs and are supplemented in the data set.

[0025] Dataset division: The present invention uses a 6:2:2 random division of the training, validation, and test data sets, including 2423 pairs of training images, 801 pairs of validation images, and 776 pairs of test images.

[0026] Model training process: The dual-band image description generation method based on feature alignment and fusion proposed in the present invention inputs the visible light image area features and infrared image grid features extracted by Faster-RCNN and the real text labels corresponding to the image pairs, and the image feature part enters the model training. The loss of the trained text and the real text labels is calculated and the gradient is derived to optimize the network parameters; the training is divided into two stages, the first stage is the cross entropy training stage, and the second stage is the reinforcement learning stage.

[0027] Model training parameter setting: A Linux server with a graphics card model of V100 Tensor Core GPU and a video memory of 32GB was used for model training. The Python version used was 3.7.12 and the PyTorch version was 1.10.0. The infrared grid feature size was 7×7. Dropout was used to prevent the model from overfitting, and the parameter was set to 0.1. The Adam algorithm was used to update the parameters, and the number of heads was set to 8. The training was divided into two stages. The first stage was the cross entropy training stage, with batch_size set to 50, the initial value of the learning rate to 1e-4, and 20 epochs of training; the second stage was the reinforcement learning stage, with batch_size set to 100, the initial value of the learning rate to 5e-6, the beam search width to 5, and 15 epochs of training.

Claims

1. A dual-band image description generation method based on feature alignment and fusion, characterized by: The following steps are involved: Step 1: Extract visible light image I v Visible area feature R V = {r i } M , extract infrared image I r The infrared grid feature G I = {g j } N ,The proposed features all contain appearance features and location information, where M and N represent the total number of visible area features and infrared grid features extracted from each image, respectively; Step 2: Input the obtained visible area features and infrared grid features into the multi-head fusion position attention module to combine the appearance features with the position information, and then pass through the first feedforward neural network FFN respectively. This process is repeated T times. After T times, the output of the first feedforward neural network FFN is input into the multi-head hybrid cross attention module; Step 3: The visible area features and infrared grid features are fused and enhanced through a multi-head hybrid cross attention module. The fused and enhanced results are input into the second feedforward neural network FFN. This process is repeated T times. After T times, the output of the second feedforward neural network FFN is concatenated to obtain visual information U. Step 4: Input the visual information U into the Transformer decoder and decode it to get the hidden state h t ; Step 5: Decode the hidden state h t The visual information U encoded by the encoder and the language information s generated by the language model t The two are input into the adaptation module together to obtain a text description.

2. The method for generating dual-band image description based on feature alignment and fusion according to claim 1, characterized in that: The multi-head fusion position attention module combines appearance features with position information. The appearance features and position information are input into the multi-head fusion position attention. The appearance features are subjected to different linear projections, and the position information is subjected to absolute position encoding and relative position encoding. The absolute position encoding is integrated into the appearance features. The integrated vectors are used as the query vector and key vector in the multi-head fusion position attention. The scaled dot product is used to calculate the similarity score between the query vector and the key vector. Then, the relative position information is used to adjust the attention weight. Finally, it is normalized and mapped to a probability distribution and a value vector for weighting to obtain one of the feature representations. Multiple feature representations are spliced ​​to achieve multi-head fusion position attention. The multi-head hybrid cross-attention module uses the visible area feature as the query, and the infrared grid feature as the key and value, so as to embed the visible area feature and infrared grid feature information into the infrared grid feature and the visible area feature respectively and strengthen the original feature representation. The current infrared visible area feature and the current infrared grid feature are respectively encoded with the absolute position, and the vectors after integrating their absolute position encoding information are used as the query vector and key vector in the hybrid cross-attention. The similarity scores of the two are calculated, and the relative position information is used to adjust the attention weight. Finally, it is normalized and mapped into a probability distribution and value vector to obtain the feature representation. Multiple feature representations are spliced ​​to obtain the output of multi-head hybrid cross-attention.

3. The method for generating dual-band image description based on feature alignment and fusion according to claim 2, characterized in that: The key and value vectors input to the adaptive module are visual information U and language information s t Concatenate the vectors after different linear transformations, where the language information s t The Bert-based language model is used to obtain the hidden state h t After a masked multi-head attention, it becomes a query vector. The key, value vector, and query vector are multi-headed, and the results of the multi-head attention are spliced ​​together to finally get the text description of the image pair.

Images