Image description method based on extended layer normalization and enhanced self-attention mechanism

Abstract

The application discloses an image description method based on an extended layer normalization and enhanced self-attention mechanism, and comprises the following steps: extracting preset image region features; constructing an image description network structure formed by an extended layer normalization encoder and an enhanced self-attention mechanism decoder; designing a training method, first performing model training on the image description network structure through a cross-entropy loss function, then performing model optimization on the image description network structure through a training method based on reinforcement learning, and forming a trained image description model; and inputting an image to be searched, and outputting a detection result through inference of the trained image description model. The application solves the problem of insufficient model performance in the current image description method.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image description, and particularly relates to an image description method based on extended layer normalization and enhanced self-attention mechanism. BACKGROUND

[0002] Image description is a very challenging task because it requires a machine to perform some complex computer vision tasks, such as object recognition, scene classification, attribute and relationship detection, and then summarize them in one sentence. In recent years, with the rapid development of deep learning, image description models have made great progress. For example, improved convolutional neural networks (CNN) and object detection architectures help improve the image description task. More complex sequence models, such as attention-based recurrent neural networks (RNN), can produce more accurate captions. Reinforcement learning techniques can solve the problem of exposure bias and the measurement of indistinguishable tasks. In particular, the Transformer-based architecture represents the state-of-the-art technology for image captioning. SUMMARY

[0003] The purpose of the application is to provide an image description method based on extended layer normalization and enhanced self-attention mechanism, which solves the problem of insufficient model performance in current image description methods.

[0004] To solve the above technical problems, the technical scheme of the application is as follows: an image description method based on extended layer normalization and enhanced self-attention mechanism, comprising the following steps:

[0005] S1, extracting a preset image region feature;

[0006] S2, constructing an image description network structure formed by an extended layer normalization encoder and an enhanced self-attention mechanism decoder;

[0007] S3, designing a training method, first training the image description network structure through a cross-entropy loss function, then optimizing the image description network structure through a reinforcement learning-based training method, and forming a trained image description model;

[0008] S4, inputting an image to be searched, and outputting a detection result through the trained image description model.

[0009] S1 specifically comprises: extracting an image region feature similar to human interest based on a target detection method.

[0010] The target detection method is specifically a Faster-RCNN target detection algorithm.

[0011] The Transformer encoder is used in S2 to optimize the normalization of the extended layer, which considers the correlation between the input sequence and the attention result of the previous stage, guides the model to focus on the preset target precision improvement, and corrects the attention result of the previous stage to some extent, and deviates to the memory module about the input sequence.

[0012] S3 is specifically:

[0013] First, use the cross-entropy loss function to train the model, and then use the SCST method to train it again;

[0014] Given a real data set of word sequence The length of the word sequence is represented, and the cross-entropy loss is minimized:

[0015]

[0016] In the formula, θ represents the model parameters, L CE (θ) represents the cross-entropy loss with respect to θ, P θ represents the real distribution with respect to θ.

[0017] After training the model with cross-entropy loss, the model is trained with reinforcement learning according to the SCST method to minimize the negative expectation, and the corresponding formula is:

[0018]

[0019] In the formula, Y s =(y 1,s ,y 2,s ,…,y t,s ,…,y T,s ) y t,s is the word sampled from the model at time t, and r represents the CIDEr evaluation index as the reward function of reinforcement learning, represents the expectation of Y s , which can be further approximated as

[0020]

[0021] In the formula, Y s =(y 1,s ,y 2,s ,…,y T,s ) is obtained by sampling according to p θ using the idea of Monte-Carlo, and represents the sampled word set. is the word set obtained by greedy selection according to the cross-entropy loss of the model, L SCST (θ) is obtained according to Y s and A reinforcement learning method for model optimization.

[0022] S4 is specifically: input the training set into the trained image description model, specify the number of iterations of training, train in batches, and save the optimal model result.

[0023] A computer device is also provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of the preceding embodiments when executing the computer program.

[0024] A computer readable storage medium is also provided, which stores a computer program, wherein the computer program is executable on a processor to implement the steps of the method according to any one of the preceding embodiments.

[0025] Compared with the prior art, the present application has the following advantages:

[0026] The present application provides an image description method based on an extended layer normalization and enhanced self-attention mechanism Transformer, wherein the encoder of the extended layer normalization enhances the generalization performance and feature expression ability of the model. The decoder of the extended standard self-attention mechanism improves the deficiencies of the existing Transformer applied in the image description field, i.e. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 The flowchart of the embodiment of the present application is shown in the figure;

[0028] Figure 2 The network structure diagram of the embodiment of the present application is shown in the figure;

[0029] Figure 3 The system flowchart of the embodiment of the present application is shown in the figure;

[0030] Figure 4 The result diagram of the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with the figures and embodiments.

[0032] The image description method based on the extended layer normalization and enhanced self-attention mechanism Transformer of the embodiment of the present application is as follows:Figure 1 As shown, comprising the following steps:

[0033] S1, extract image region features, use target detection based method to extract image region features of human interest type; when extracting image region features, use Faster-RCNN based target detection method, which is different from the traditional CNN extracted regular, size same rectangular box, the image region features extracted by this method are irregular in size, more similar to human observation of things, different regions have different attention weights.

[0034] S2, design a Transformer based image description network structure, including an extended layer normalization encoder and an enhanced self-attention mechanism decoder; the network structure is divided into two models, which are an extended layer normalization encoder and an enhanced self-attention mechanism decoder. The extended layer normalization encoder can improve the generalization performance of the model to a certain extent, and we add the structure symmetrically to the feedforward neural network and self-attention network of the Transformer encoder, which can make the model obtain better generalization performance. The decoder based on enhanced self-attention mechanism, the existing Transformer applied in the field of image description has the following shortcomings: unrelated and non-adjacent words will obtain additional attention score because of the adjacent relationship with the same word, thereby affecting the model to focus on the words that really need to be focused on. Based on this, we propose an enhanced self-attention mechanism decoder, which considers the correlation between the input sequence and the attention result of the previous stage, guides the model to focus on the words that really need to be focused on, and corrects the attention result of the previous stage to a certain extent, so that it is biased towards the memory module we maintain about the input sequence.

[0035] S3, design a training method, first use the traditional cross entropy loss function to train the model, and then use the method based on reinforcement learning to train the model; as most image description models do, first use the cross entropy loss (Cross Entropy Loss, CE) to train the model, and then use the self-critical (SCST) method to train it again. Given a real data set description word sequence Wherein represents the length of the word sequence, and the minimum cross entropy loss is

[0036]

[0037] Where, θ represents the model parameters. After training the model with cross entropy loss, the model is trained with SCST reinforcement learning. The goal of training the model with SCST is to minimize the negative expectation. The corresponding formula is

[0038]

[0039] Where, Y s =(y 1,s ,y 2,s ,…,y t,s ,…,y T,s ) of y t,s The word sampled from the model at time t is the word in the formula. The loss can be further approximated as...

[0040]

[0041] Where, Y s =(y 1,s ,y 2,s ,…,y T,s It utilizes the Monte Carlo concept based on p θ Obtained through sampling. It is obtained by greedy decoding based on the model.

[0042] S4. Input the image to be retrieved, use the trained image description model to perform reasoning, and return the retrieval results.

[0043] Input the training set into the above network structure, specify the number of training iterations, train in batches, and save the best model result.

[0044] This invention also provides an image description visual assistance system for implementing the above-described Transformer image description method based on extended layer normalization and enhanced self-attention mechanism, such as... Figure 3 As shown, it includes:

[0045] Data preprocessing module 101 selects a portion of the data to train the feature extraction network;

[0046] Training module 102 takes training images as input to train the model and obtains the optimal model after a given number of training iterations.

[0047] Inference module 103 takes the image to be retrieved as input and outputs the image of the retrieval result.

[0048] Test Implementation Example:

[0049] We report results using the MSCOCO captioning evaluation toolkit, which reports the widely used automatic evaluation metrics SPICE

[18] , CIDEr

[32] , BLEU

[17] , METEOR

[33] and ROUGE

[34] . We compare our method with 7 other methods, LSTM, SCST, LSTM-A, RFNET, up-down, GCN-LSTM, SGAE, under two training methods, Cross-Entropy Loss and CIDEr Score Optimization. Table 1 is the result of Cross-Entropy Loss, and Table 2 is the result of CIDEr Score Optimization.

[0050] Table 1

[0051]

[0052] Table 2

[0053]

[0054]

[0055] From Table 1 and Table 2, we can see that our method achieves higher scores on multiple evaluation metrics compared to the other seven methods.

[0056] Those skilled in the art will readily understand that the above is only the preferred embodiment of the present application, and is not intended to limit the present application, any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0057] Those skilled in the art will readily understand that the above is only the preferred embodiment of the present application, and is not intended to limit the present application, any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image description method based on extended layer normalization and enhanced self-attention mechanism, characterized in that, The method comprises the following steps: S1, extracting a preset image region feature; S2, constructing an image description network structure formed by an extended layer normalization encoder and an enhanced self-attention mechanism decoder; S3, designing a training method, first training the image description network structure by using a cross-entropy loss function, then optimizing the image description network structure by using a training method based on reinforcement learning to form a trained image description model; specifically: First, the model is trained using a cross-entropy loss function, and then the SCST method is used to train it again; Given a sequence of description words for a real dataset The length of the sequence of words is denoted, and the cross-entropy loss is minimized as: wherein denotes the model parameters, denotes the cross-entropy loss with respect to ;​ After training the model using the cross-entropy loss, the model is trained using the SCST method for reinforcement learning to minimize the negative expectation to the greatest extent, and the corresponding formula is: wherein is the is the word sampled from the model at time t, r denotes the CIDEr evaluation metric as the reward function for reinforcement learning, is the , which can be further approximated as In the formula, is obtained by sampling according to the Monte-Carlo idea, , is a word set obtained by greedy selection according to the cross-entropy loss of the model, is a reinforcement learning method for optimizing the model according to and .​ S4, inputting an image to be searched, and outputting a detection result by inference of the trained image description model.

2. The image description method based on extended layer normalization and enhanced self-attention mechanism according to claim 1, characterized in that, S1 specifically comprises: extracting an image region feature similar to human interest by using a target detection method.

3. The image description method based on extended layer normalization and enhanced self-attention mechanism according to claim 2, characterized in that, The target detection method specifically comprises a Faster-RCNN target detection algorithm.

4. The image description method based on extended layer normalization and enhanced self-attention mechanism according to claim 1, characterized in that, In S2, the Transformer encoder is used to optimize the extended layer normalization, which considers the correlation between the input sequence and the attention result of the previous stage to guide the model to pay attention to the preset target precision improvement, correct the attention result of the previous stage to a certain extent, and deviate to the input sequence.

5. The image description method based on extended layer normalization and enhanced self-attention mechanism according to claim 1, characterized in that, S4 specifically comprises: inputting a training set into the trained image description model, specifying the number of iterations for training, training in batches, and saving the optimal model result.

6. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-5.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method according to any one of claims 1-5.

Images