An Image Captioning Method Based on Dual Attention Model

Abstract

This paper uses methods from the field of network security to implement an image description method based on a dual-attention model. The image description research task is completed using an "encoder-decoder" framework structure, combining a convolutional neural network with a long-short-term memory network and an attention mechanism. First, an encoder is constructed, using the ResNeXt-101 network and the target detector Faster R-CNN to construct a new encoder model to obtain the target area of the image to be tested and perform feature extraction. Then, a decoder is constructed, using the dual-attention mechanism and a two-layer LSTM to construct a new decoder to generate a description statement for the image. Finally, the model is optimized using the REINFORCE reinforcement learning algorithm. The method provided by the present invention can effectively improve the accuracy of the image description model and the richness of the image description content, thereby improving the overall performance of the image description model.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to an image description method based on a dual attention model. Background Art

[0002] Image captioning involves using computers to automatically generate textual descriptions of image content. This requires not only identifying the types of objects in an image but also analyzing the relationships between them. This allows the machine to understand the image's deeper meaning and, after further analysis, return a user-friendly description. Image captioning is a fusion of computer vision and natural language processing, a comprehensive and interdisciplinary research topic spanning vision and text.

[0003] Most of the current mainstream research on image description tasks has made significant changes to the traditional "decoder-decoder" architecture, using convolutional neural networks (CNN) as encoders to collect image feature data and long short-term memory networks (LSTM) as decoders to implement image description. This method uses a loop to generate description words one by one, obtains all the words after the loop, and then combines them into a description sentence of the input image. However, this method may cause information loss over time during the loop, resulting in errors in the content of the final generated image description sentence and the content of the image to be tested.

[0004] Currently, popular mobile internet apps, such as Xiaohongshu, Dianping, Weibo, and Zhihu, are all social products based on the integration of multimedia information. Taking Xiaohongshu as an example, each article is illustrated with text and pictures, and most pictures have category labels, but they need to be added manually by the publisher. Image annotation in this scenario has limitations. First, the editing time cost is high for the publisher, and second, it is not convenient and flexible enough for the searcher. In the field of artificial intelligence, human-computer interaction research is still prone to understanding barriers. Typical AI assistants such as Xiaoai, Siri, and Xiaodu do not currently support operations related to inputting images. Human-computer dialogue robots are not truly intelligent in terms of images and dialogue. The main reason for this communication barrier is the poor performance of image description task technology. People believe that computers should be able to obtain more information from images, rather than just simple information such as classification and positioning. We hope that computers can generate natural image description sentences like humans, which has accelerated the pace of image description task research [2]. Text generation and image recognition technologies are based on the natural language processing and deep learning fields in computer learning, respectively. Combining the two has become the current image description task. Currently, many studies on image description tasks are being carried out in full swing. The most common one is that in our daily exams or the English and Chinese papers of various large-scale exams, there are more or less some picture-based writing-related questions. Summary of the Invention

[0005] To this end, this paper first proposes an image captioning method based on a dual-attention model. An image is fed into the system via external input and an "encoder-decoder" framework is employed to accomplish the image captioning task. This framework combines a convolutional neural network (CNN), a long short-term memory (LSTM), and an attention mechanism. First, an encoder is constructed using the ResNeXt-101 network and the object detector Faster R-CNN to construct a new encoder model to capture the target region of the image and perform feature extraction. Next, a decoder is constructed using the dual-attention mechanism and a two-layer LSTM to generate a description of the image. Finally, the model is optimized using the REINFORCE reinforcement learning algorithm. The output structure is a computer-generated description of the image content. This technology can significantly help people with visual impairments better understand image information online and in the real world. It can also be applied to fields such as early childhood education, intelligent annotation of news headlines, and human-computer interaction. Currently, this technology is most widely used in the automatic generation of news articles. For example, for the 2022 Winter Olympics, some media outlets used this technology to convert images into textual descriptions, enabling the real-time release of competition results.

[0006] During network pre-training, the ResNeXt network structure is first fine-tuned to remove the useless network layer in the last layer of the ResNeXt network structure, and then ResNeXt-101 is pre-trained on the ImageNet dataset.

[0007] The encoder first uses the ResNeXt network to extract the basic features of the image, and then uses the Faster R-CNN algorithm to perform object detection and further extract the target features.

[0008] The object detection is divided into two stages. In the first stage, the features output by the last layer of the ResNeXt network are injected into the candidate region network of Faster R-CNN to generate candidate boxes, and the bounding box of the region of interest is corrected for the first time. In the second stage, the generated candidate boxes and feature maps are injected into the pooling layer of the region of interest to obtain feature graphics with a certain scale of the candidate boxes. Finally, the softmax and fully connected layers are used to determine its category and precise information, and the second correction of the bounding box is performed. The resulting visual feature set is recorded as V = {v1,v2,...,v n};

[0009] The features obtained by average pooling are recorded as The calculation formula is It is used as a global visual feature input to the decoding end to build a visual attention model;

[0010] The visual LSTM model is denoted as LSTM 1v , LSTM 1v The 1 in the first layer represents the LSTM; the average vector Previous word embedding w t-1 and the previous output state vector from the language LSTM Fusion is used as input to compute visual attention queries Where E represents the identity matrix transformation, θ v is a parameter. The visual attention query vector is used as the guidance vector of the visual attention mechanism to obtain a visual LSTM based on the visual attention mechanism, whose output is expressed as

[0011] The global visual features extracted by the convolutional neural network are used to predict the description sentence using the recurrent neural network. The attention model is added in the encoder stage, and the semantic features of the object are extracted using the scene graph parser, which is recorded as the set S = {S obj ,S attr ,S rel}, where the target object word is represented by O = {obj1, obj2, ..., obj n}, the object attribute word list is A={attr1,attr2,...,attr n}, object relationship word representation is R = {rel1,rel2,...,rel n}, these semantic words are encoded using the One-Hot method and mapped into the Visual-Semantic space using the word embedding matrix. The vocabulary feature of the target entity is denoted as S obj ={s obj1 ,s obj2 ,...,s objn}, the attribute vocabulary feature is recorded as S attr ={s attr1 ,s attr2 ,...,s attrn}, the relational vocabulary feature is recorded as S rel ={s rel1 ,s rel2 ,...,s reln}, and then according to the formula: Get the average feature of semantic global features

[0012] The semantic LSTM model is denoted as LSTM 1s , where 1 represents the first layer LSTM; the average vector Previous word embedding w t-1 and the state vector from the language LSTM The previous output of is fused as input to compute the semantic attention query where E represents the identity matrix transformation, θ v is a parameter. The semantic LSTM based on the semantic attention mechanism is obtained, and its output is expressed as

[0013] The decoder adopts a two-layer LSTM structure, including attention LSTM and language LSTM, combining semantic attention mechanism and visual attention mechanism, and placing them in the attention model of the first layer LSTM. The image features detected by Faster R-CNN are defined as set V. The image features are average-pooled to obtain the mean of visual and semantic values. The word embedding vector of the current time step is used as the input of the second layer language LSTM. This model is denoted as attRNeXt model. A standard LSTM operates as follows on a single time step: h t =LSTM(h t-1 ,x t ), the encoder obtained and The attention image features are generated by the attention LSTM layer “Attend”, and then the language LSTM is input and To generate words sequentially, where θ l , W p , b p is a parameter, y t is the generated word;

[0014] Then the image description is optimized, and its objective function is: P (θ)=-∑log p(y t |I;y 1:t-1 ), where y t is the word generated at time t, I represents the input image, y 1:t-1 Represents the generated subtitles.

[0015] Then using reinforcement learning algorithm, Solve the problem of mismatch between model training and testing, where r is the standard score of the automatic evaluation indicator BLEU, CIDEr or METEOR used to judge whether the predicted statement is similar to the evaluation standard.

[0016] The technical effects to be achieved by the present invention are:

[0017] The image description method based on the dual attention model provided by the present invention effectively improves the accuracy of the image description model and the richness of the image description content, thereby improving the overall performance of the image description model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 ResNeXt101 model image description model;

[0019] Figure 2 Faster R-CNN network structure target detection technology roadmap;

[0020] Figure 3 The structure diagram of single-layer LSTM and double-layer LSTM at each moment;

[0021] Figure 4 Based on a double-layer LSTM and dual-attention mechanism model structure;

[0022] Figure 5 Example of a graphical representation of experimental results. DETAILED DESCRIPTION

[0023] The following is a preferred embodiment of the present invention and combines the accompanying drawings to further describe the technical solution of the present invention, but the present invention is not limited to this embodiment.

[0024] This paper proposes an image description method based on a dual attention model.

[0025] Image description is the collision and fusion of computer vision and natural language processing; it is a cross-disciplinary research topic from vision to text. However, most image captioning models are intentionally uncontrollable, that is, they passively generate image descriptions from a coarse-grained layer. For this reason, the present invention conducts research at the existing fine-grained level to make the generated image description sentences more accurate and richer in content. Computer automatic description of image content technology can greatly help people with visual impairments to better understand image information on the Internet and even in the real world. At the same time, we can also apply it to early childhood education, intelligent annotation of news headlines, and human-computer interaction. Currently, the most widely used technology is the automatic generation of news articles. For example, for the 2022 Winter Olympics, some media used this technology to convert pictures into textual descriptions, achieving the same effect as the real-time release of competition results.

[0026] This paper improves the computational mechanism of deep learning-based computer vision image description to enhance its performance. It uses an encoder-decoder framework to complete image description research tasks. By utilizing multiple methods to address some of the problems encountered in image description tasks, the paper improves model performance, increases the accuracy of image descriptions, and enhances the richness of sentences. The research focuses on:

[0027] (1) When we build a new image description model, in the encoding part, we use the ResNeXt-101 network and the target detector Faster R-CNN to build a new encoder model to obtain the target area of the image to be tested and perform feature extraction.

[0028] (2) In the decoder part, we use the dual attention mechanism and double-layer LSTM to construct a new decoder to generate the description sentence of the image

[0029] (3) Optimization research on the model. In view of the characteristics of the REINFORCE algorithm that can optimize non-differentiable and discrete indicators during the model training process, we decided to use this reinforcement learning algorithm to optimize the model in this invention.

[0030] The present invention adopts a framework that combines convolutional neural network + long short-term memory network + attention mechanism.

[0031] When the present invention uses the ResNext network to build a model, the RESNET101 network model is replaced with the ResNeXt-101 network structure model in the encoding part, which is called the ResNeXt101 model in the present invention, while the decoding part still uses the LSTM network. The model diagram is as follows Figure 1 shown.

[0032] When we perform network pre-training, we need to fine-tune the ResNeXt network structure first, remove the useless network layer in the last layer of the ResNeXt network structure, and then pre-train ResNeXt-101 on the ImageNet dataset.

[0033] At the encoding end, the ResNeXt network is first used to extract the basic features of the image, and then the Faster R-CNN algorithm is used to perform object detection to extract the target features. Object detection is divided into two stages. The first stage is to inject the features output by the last layer of the ResNeXt network into the Region Proposal Networks (RPN) of Faster R-CNN to generate candidate boxes and perform the first correction on the bounding box of the region of interest. The second stage is to inject the generated candidate boxes and feature maps into the region of interest pooling (ROI) layer to obtain feature graphics with a certain scale to the candidate box. Finally, the softmax and fully connected layers are used to determine its category and precise information, and the second correction of the bounding box is performed, and the resulting visual feature set is recorded as V = {v1,v2,...,v n The technical route of object detection is as follows: Figure 2 shown.

[0034] The features obtained by average pooling are recorded as The calculation formula is It is used as a global visual feature input to the decoding end to build a visual attention model.

[0035] The visual LSTM based on the visual attention mechanism is obtained from formula (2), and its output is expressed as The visual LSTM model is denoted as LSTM 1v , LSTM 1v The 1 in the first layer represents the LSTM; the average vector Previous word embedding w t-1 and the state vector from the previous language LSTM Fusion as a computational visual attention query The visual attention query vector is used as the guidance vector of the visual attention mechanism.

[0036]

[0037] On the other hand, we are inspired by the visual attention mechanism and use the recurrent neural network to predict the description sentence based on the global visual features extracted by the convolutional neural network. We also add the attention model in the encoder stage to achieve semantic attention feature extraction. This paper uses the scene graph parser to extract the semantic features of the object, which is recorded as the set S obj ={s obj1 ,s obj2 ,...,s objn}, where the target object word is represented by O = {obj1, obj2, ..., obj n}, the object attribute word list is A={attr1,attr2,...,attr n}, object relationship word representation is R = {rel1,rel2,...,rel n These semantic words are encoded using the One-Hot method and mapped into the Visual-Semantic space using the word embedding matrix. The vocabulary feature of the target entity is denoted as S obj ={s obj1 ,s obj2 ,...,s objn}, the attribute vocabulary feature is recorded as S attr ={s attr1 ,s attr2 ,...,s attrn}, the relational vocabulary feature is recorded as S rel ={s rel1 ,s rel2 ,...,s reln According to formula (3), the average feature of the semantic global feature is obtained.

[0038]

[0039] The semantic LSTM based on the semantic attention mechanism is obtained from formula (4), and its output is expressed as The semantic LSTM model is denoted as LSTM 1s , where 1 represents the first layer LSTM; the average vector Previous word embedding w t-1 vector and the state vector from the previous language LSTM Fusion as Computational Semantic Attention Query

[0040]

[0041] Semantic attention takes the semantic features of the target entities, relations, and attributes of the scene graph as attention query elements, and the query vector as the guidance vector of the semantic attention mechanism.

[0042] When using the language LSTM, the ResNeXt101 model uses the image visual signal transmitted by the encoder, the current image words, and the text information manually annotated by the image as the input of the language layer LSTM model. The structure diagram of the single-layer LSTM and the double-layer LSTM at each moment is as follows: Figure 3 As shown, the left side is a single-layer LSTM single time step structure, and the right side is a double-layer LSTM single time step structure.

[0043] The present invention adopts a two-layer LSTM structure in the decoder part, including attention LSTM and language LSTM. It combines the semantic attention mechanism and visual attention mechanism, and puts them into the attention model of the first layer LSTM. The image features detected by Faster R-CNN are defined as set V. The image features are average pooled to obtain the mean of visual and semantic values. The word embedding vector of the current time step is used as the input of the second layer language LSTM. This model is denoted as attRNeXt model, and the decoder model structure is as follows: Figure 4 The LSTM used in each layer is a standard long short-term memory network structure. A standard LSTM operates on a single time step as shown in the following formula (5).

[0044] h t =LSTM(h t-1 ,x t )......(5)

[0045] We can get from formula (2) and formula (4) and The attention image features are generated by the attention LSTM layer “Attend”. Then the language LSTM is input and To generate words sequentially, as shown in formula (6) and formula (7), where θ l , W p , b p is a parameter, y t is the generated word.

[0046]

[0047]

[0048] Next, we optimize the image description, and the objective function is calculated as shown in the following formula (8).

[0049] H P (θ)=-∑log p(y t |I;y 1:t-1 )......(8)

[0050] In addition, because the objective function returns a standard answer at every moment during training, the evaluation criteria used in this invention are not continuous, which will cause a mismatch between the model during training and testing. To address this, we use a reinforcement learning algorithm to improve it, as shown in the following formula (9).

[0051]

[0052] Where r is the standard score of the automatic evaluation indicators used, such as BLEU, CIDEr, and METEOR, which is used to judge whether the predicted sentence is similar to the evaluation criteria.

[0053] Table 1 shows the performance comparison of the proposed model and the benchmark model on the MSCOO dataset.

[0054]

[0055] It can be seen that there are some similarities and differences between each model. The similarity is that these models all use the encoder-decoder deep learning model architecture. The difference is that there are slight differences in the use of the encoder and decoder. The difference in the encoder is mainly reflected in the different network models for image feature extraction; the difference in the decoder is mainly reflected in the different attention mechanisms. In this invention, we use the ResNeXt network as the encoder and a two-layer LSTM and two-layer attention mechanism model in the decoder part, which makes our image description model have better performance. At the same time, it can be seen from the above table that our model is 0.08 higher than the lowest Deep VS and 0.06 higher than the highest Hard-Att in its coherence and accuracy indicators B@4; and in terms of semantic richness CIDEr, its specific value is 1.103, which is 0.443 higher than the lowest Deep VS and 0.247 higher than the highest Hard-Att. The results show that the overall performance of the model has been improved in both accuracy and semantic richness.

[0056] In the validation experiments, to evaluate the performance and effectiveness of our model, we selected the MSCOCO2014 dataset for evaluation. Each image includes a manually annotated image description (ground truth). We then used Karpathy's partitioning method to classify the MSCOCO2014 dataset, with 113,287 images in the training set, 5,000 in the validation set, and 5,000 in the test set. The model counts all manually annotated description words in the dataset, and the output description sentence consists of words that appear more than five times and start / end markers. <star> / <end>constitute.

[0057] The experimental results are as follows: Figure 5 shown.

[0058] Figure 5 In (a), the baseline description is an adult wearing a gray T-shirt with red sleeves sleeping on the sofa. Our model is: a (woman) wearing (black pants) and a gray T-shirt sleeping on the sofa in her (room).

[0059] Figure 5 In (b), our model's description is more accurate, semantically rich, and complete. It also includes the color of the other dog (black and white) and clarifies their location (on the road). These results demonstrate that the proposed attRNeXt image description model closely matches the original description while improving on both richness and accuracy, providing a more comprehensive and accurate overall image description.< / end> < / star>

Claims

1. An image description method based on a dual-attention model, characterized by: Image information is input through the input unit, and the "encoder-decoder" framework is used to complete the image description research task. The framework combines a convolutional neural network, a long short-term memory network, and an attention mechanism to ultimately output a textual description of the image. First, the encoder is constructed, using the ResNeXt-101 network and the object detector Faster R-CNN to build a new encoder model to obtain the target area of the test image and perform feature extraction. Then, the decoder is constructed, using the dual attention mechanism and a two-layer LSTM to build a new decoder to generate a description sentence for the image. Finally, the model is optimized using the REINFORCE reinforcement learning algorithm. During network pre-training, the ResNeXt-101 network structure was fine-tuned to remove the useless network layer at the end of the ResNeXt-101 network structure. Then, ResNeXt-101 was pre-trained on the ImageNet dataset. The pre-trained ResNeXt-101 network was combined with the Faster R-CNN algorithm to obtain target features, namely visual features and semantic features. Then, a dual-attention mechanism image description model was constructed by using the attention mechanism method. The encoder first uses the ResNeXt-101 network to extract the basic features of the image, and then uses the Faster R-CNN algorithm to perform object detection and further extract the target features; The object detection is divided into two stages. In the first stage, the features output by the last layer of the ResNeXt-101 network are injected into the candidate region network of Faster R-CNN to generate candidate boxes, and the bounding box of the region of interest is corrected for the first time. In the second stage, the generated candidate boxes and feature maps are injected into the region of interest pooling layer to obtain feature graphics consistent with the scale of the candidate boxes. Finally, the softmax and fully connected layers are used to determine its category and precise information, and the second correction of the bounding box is performed. The resulting visual feature set is recorded as V = {v1,v2,...,v n }; The features obtained by average pooling are recorded as The calculation formula is It is used as a global visual feature input to the decoding end to build a visual attention model; The visual LSTM model is denoted as LSTM 1v , LSTM 1v The 1 in represents the first layer of LSTM; Global visual features Previous word embedding w t-1 and the previous output state vector from the language LSTM Fusion is used as input to compute visual attention queries Where E represents the identity matrix transformation, θ v is a parameter, and the visual attention query vector is used as the guidance vector of the visual attention mechanism to obtain a visual LSTM based on the visual attention mechanism, whose output is expressed as The global visual features extracted by the convolutional neural network are used to predict the description sentence using the recurrent neural network. The attention model is added in the encoder stage, and the semantic features of the object are extracted using the scene graph parser, which is recorded as the set S = {S obj ,S attr ,S rel }, where the target object word is represented by O = {obj1, obj2, ..., obj n }, the object attribute word list is A={attr1,attr2,...,attr n }, object relationship word representation is R = {rel1,rel2,...,rel n }, these semantic words are encoded using the One-Hot method and mapped into the Visual-Semantic space using the word embedding matrix; the vocabulary feature of the target entity is denoted as S obj ={s obj1 ,s obj2 ,...,s objn }, the attribute vocabulary feature is recorded as S attr ={s attr1 ,s attr2 ,...,s attrn }, the relational vocabulary feature is recorded as S rel ={s rel1 ,s rel2 ,...,s reln }, and then according to the formula: Get the average feature of semantic global features The semantic LSTM model is denoted as LSTM 1s , where 1 represents the first layer LSTM; the average feature of the semantic global feature Previous word embedding w t-1 vector and the state vector from the previous language LSTM Fusion as Computational Semantic Attention Query Where E represents the identity matrix transformation, θ s is a parameter; the semantic LSTM based on the semantic attention mechanism is obtained, and its output is expressed as 2. The image description method based on the dual attention model according to claim 1, characterized in that: The decoder adopts a two-layer LSTM structure, including an attention LSTM and a language LSTM, combining the semantic LSTM of the semantic attention mechanism and the visual LSTM of the visual attention mechanism, and placing them in the attention model of the first layer LSTM. The image features detected by Faster R-CNN are defined as a set V. The image features are average-pooled to obtain the mean of the visual and semantic values. The word embedding vector of the current time step is used as the input of the second layer language LSTM. This model is denoted as the attRNeXt model. A standard LSTM operates as follows on a single time step: h t =LSTM(h t-1 ,x t ), the decoder gets and The attention image features are generated by the attention LSTM layer "Attend" and then the language LSTM is input into the language LSTM. and To generate words sequentially, p(y t |y <t)= where θ l , W p , b p is a parameter, y t is the generated word; The image description is then optimized, and its objective function is: H P (θ)=-∑log p(y t |I;y 1:t-1 ), where y t is the word generated at time t, I represents the input image, y 1:t-1 represents the generated subtitles; Then using reinforcement learning algorithm, Solve the problem of mismatch between model training and testing, where r is the standard score of the automatic evaluation indicator BLEU, CIDEr or METEOR used to judge whether the predicted statement is similar to the evaluation standard.

Images