A method for generating Chinese description of images based on encoder-dual decoder

By using the encoder-dual decoder model in image subtitle generation, and using the combination of Transformer encoder and LSTM decoder, the problem of insufficient information expression in the prior art is solved, and a richer and more accurate image description generation is achieved.

CN114139011BActive Publication Date: 2024-11-08SHANGHAI NORMAL UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111419735.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-26
Publication Date
2024-11-08
Estimated Expiration
2041-11-26

AI Technical Summary

Technical Problem

The prior art cannot fully utilize the feature information obtained by the encoder in image subtitles generation, resulting in the generated description text being single and the information in the image cannot be fully expressed, especially the implicit information.

Method used

Using the Chinese image description generation method based on encoder-dual decoder, the Transformer encoder and LSTM dual decoder model is constructed, and the image feature information is extracted using a multi-level self-attention mechanism, and Title-Decoder and Text are integrated under the Attention mechanism. -Decoder output, generate rich description text.

Benefits of technology

A more complete description of image information is achieved, the generated text is richer and more accurate, and the implicit information in the image can be effectively expressed, improving the quality and relevance of the description.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114139011B_ABST
    Figure CN114139011B_ABST
Patent Text Reader

Abstract

The present invention relates to a method for generating Chinese description of an image based on an encoder-dual decoder, comprising: obtaining a data set, including an image description data set and a corresponding background corpus data set; performing data preprocessing on the data set, and dividing the preprocessed data set into a training set, a validation set, and a test set; constructing an encoder-dual decoder model, and setting model parameters and hyperparameters, and training the encoder-dual decoder model based on the training set and the validation set; testing the trained encoder-dual decoder model based on the test set to obtain an image description generation model; inputting an image to be described into the image description generation model, and outputting a title field and a description text corresponding to the image to be described. Compared with the prior art, the present invention can make full use of the feature information obtained by the encoder, can enrich the description of the information contained in the image, and has the advantages of high accuracy, rich and fluent sentence meaning, and strong correlation between image and text.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image description generation, and in particular to an image Chinese description generation method based on an encoder-dual decoder. Background Art

[0002] Image caption is a comprehensive problem that integrates computer vision, natural language processing, and machine learning. It is similar to translating a picture into a description text. This task is very challenging for machines. It not only requires the use of models to understand the content of the picture but also needs to use natural language to express the relationship between them. In addition, the model needs to be able to grasp the semantic information of the image and generate human-readable sentences. Early description generation used simple template-based generation, which was filled with content by the output of object detectors or attribute predictors. With the emergence of deep neural networks, most image description technologies use RNN as a language model and use the output of one or more layers of CNN to encode visual information and conditional language generation. In the training stage, although the initial method was based on temporal cross-entropy training, the introduction of reinforcement learning has achieved remarkable results, which makes the use of non-differentiable caption indicators as optimization targets. In contrast, in terms of image encoding, a single-layer attention mechanism is used to incorporate spatial knowledge, initially from the CNN feature grid, and then using the image region extracted by the object detector. In order to further improve the encoding of objects and their relationships, some studies have proposed using graph convolutional neural networks to integrate the semantic and spatial relationships between objects in the image encoding stage; at the same time, multimodal graph convolutional networks are used to modulate scene graphs into visual representations, but RNN-based models are susceptible to their limited representation capabilities and sequentiality.

[0003] After the emergence of convolutional language models, they were also explored for subtitles. A new full-attention paradigm was proposed and achieved state-of-the-art results in machine translation and language understanding tasks. Therefore, the existing technology applies the Transformer model to the image subtitle task. The Transformer model includes an encoder consisting of a stack of self-attention and feedforward layers, and a decoder that uses self-attention on words and cross-attention on the output of the last encoder layer. At present, the use of the Transformer architecture for image subtitle generation mostly pairs a Transformer-like encoder with an LSTM decoder, but there are still two problems: 1) It is impossible to fully utilize the feature information obtained by the encoder for subsequent description text generation; 2) The generated description text is single and cannot express the information contained in the image very richly, let alone express the implicit information in the image. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a method for generating Chinese description of an image based on an encoder-dual decoder, so as to make full use of the feature information obtained by the encoder and to enrich the description of the information contained in the image.

[0005] The object of the present invention can be achieved by the following technical solution: A method for generating Chinese description of an image based on an encoder-dual decoder, comprising the following steps:

[0006] S1. Obtain a data set, wherein the data set includes an image description data set and a corresponding background corpus data set;

[0007] S2, preprocessing the data set, dividing the preprocessed data set into a training set, a validation set, and a test set;

[0008] S3, construct an encoder-dual decoder model, set corresponding model parameters and hyperparameters, and then train the encoder-dual decoder model based on the training set and the validation set;

[0009] The trained encoder-dual decoder model is tested based on the test set to obtain an image description generation model;

[0010] S4. Input the image to be described into the image description generation model, and output the title field and description text corresponding to the image to be described.

[0011] Furthermore, the image description dataset in step S1 is specifically an AI-Challenger image description dataset, and the background corpus dataset in step S1 is specifically wiki2019zh.

[0012] Furthermore, the step S2 specifically includes the following steps:

[0013] S21, performing image format processing and image enhancement processing on the image description dataset; performing cleaning and word segmentation and tagging processing on the background corpus dataset;

[0014] S22. Divide the preprocessed data set into a training set, a validation set, and a test set according to a set ratio.

[0015] Furthermore, the specific process of cleaning and segmenting the background corpus data set in step S21 is as follows: first, the sentences are cleaned, and stop words such as periods, commas, and blank characters are removed using a self-built stop word list;

[0016] Then, a segmentation word graph is established to obtain the largest segmentation combination;

[0017] Then Chinese, English and numbers are distinguished and processed separately, and finally the segmented words are output and marked.

[0018] Furthermore, the ratio set in step S22 is specifically 8:1:1.

[0019] Furthermore, the step S3 specifically includes the following steps:

[0020] S31, constructing an encoder-dual decoder model, wherein the encoder is used to extract information in the image and the relationship between the information, and the dual decoder is used to read the information output from the encoder and output the corresponding description;

[0021] S32, training the encoder-dual decoder model based on the training set and the validation set;

[0022] S33. Test the trained encoder-dual decoder model based on the test set, and evaluate the test results according to the set evaluation indicators. If the evaluation passes, the currently trained encoder-dual decoder model is the image description generation model, otherwise return to step S32.

[0023] Furthermore, in the step S31, the encoder adopts a Transformer structure, the dual decoder includes a Title-Decoder and a Text-Decoder connected in sequence, the encoder is connected to the Title-Decoder and the Text-Decoder respectively, and the Title-Decoder and the Text-Decoder are both implemented using an LSTM (Long-Short Term Memory) network.

[0024] Furthermore, when the trained encoder-dual decoder model is tested in step S33, the input of the Text-Decoder includes the encoder output information, the keyword information extracted from the Title-Decoder output, and the information from the background corpus for the overall expansion of the description text under the Attention mechanism.

[0025] Furthermore, the evaluation indicators set in step S33 include but are not limited to BLEU, METEOR, ROUGE, CIDEr, and SPICE.

[0026] Furthermore, the step S4 specifically includes the following steps:

[0027] S41, input the image to be described, and the encoder uses a multi-level self-attention mechanism to extract image feature information;

[0028] S42, the encoder inputs all the extracted image feature information into the Title-Decoder, and outputs the title field corresponding to the image to be described;

[0029] S43, extracting keyword information from the title field output by Title-Decoder, combining it with the image feature information extracted by the encoder, and inputting them into Text-Decoder, and outputting text information corresponding to the image to be described;

[0030] S44, integrating the outputs of Title-Decoder and Text-Decoder to serve as a description of the information contained in the image to be described.

[0031] Compared with the prior art, the present invention adopts an encoder-dual decoder architecture, wherein the encoder introduces the attention idea in Transformer for improvement, and the feature information extracted by the encoder from the image will flow to the dual decoders at the same time. On the one hand, the encoder can obtain richer image feature information, and on the other hand, the dual encoder can make more full use of the information obtained from the encoder for subsequent description text generation, thereby ensuring the richness and accuracy of description generation;

[0032] The present invention uses Title-Decoder and Text-Decoder connected in sequence to construct a dual encoder, and tests the trained encoder-dual decoder model in combination with a background corpus, so that the Text-Decoder can correct the original image information and the generated text keywords under the guidance of the Attention mechanism, thereby ensuring that the output text information is richer to fully express the implicit information in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0034] Figure 2 Schematic diagram of the encoder-dual decoder model structure in the present invention;

[0035] Figure 3 Schematic diagram of the application process in the embodiment. DETAILED DESCRIPTION

[0036] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] Example

[0038] like Figure 1 As shown, a method for generating Chinese description of an image based on an encoder-dual decoder comprises the following steps:

[0039] S1. Obtain a data set, where the data set includes an image description data set and a corresponding background corpus data set. In this embodiment, the image description data set is specifically an AI-Challenger image description data set, and the background corpus data set in step S1 is specifically wiki2019zh;

[0040] S2. Preprocess the data set and divide the preprocessed data set into a training set, a validation set, and a test set. Specifically:

[0041] Perform image format processing and image enhancement processing on the image description dataset; perform cleaning and word segmentation and annotation processing on the background corpus dataset (first clean the sentences and use the self-built stop word list to remove stop words such as periods, commas, and blank characters; then build a word segmentation graph to obtain the largest segmentation combination; then distinguish and process Chinese, English, and numbers separately, and finally output and annotate the word segmentations);

[0042] Divide the preprocessed data set into a training set, a validation set, and a test set according to a set ratio. In this embodiment, the set ratio is specifically 8:1:1;

[0043] S3, construct an encoder-dual decoder model, set corresponding model parameters and hyperparameters, and then train the encoder-dual decoder model based on the training set and the validation set;

[0044] The trained encoder-dual decoder model is tested based on the test set to obtain an image description generation model;

[0045] Specific:

[0046] S31, build an encoder-dual decoder model, such as Figure 2 As shown, the encoder is used to extract information from the image and the relationship between the information, and the dual decoder is used to read the information output from the encoder and output the corresponding description. In this embodiment, the encoder adopts a Transformer structure, and the dual decoder includes a Title-Decoder and a Text-Decoder connected in sequence. The encoder is connected to the Title-Decoder and the Text-Decoder respectively, and the Title-Decoder and the Text-Decoder are both implemented using an LSTM network;

[0047] S32, training the encoder-dual decoder model based on the training set and the validation set;

[0048] S33, testing the trained encoder-dual decoder model based on the test set, and evaluating the test result according to the set evaluation index. If the evaluation passes, the currently trained encoder-dual decoder model is the image description generation model, otherwise, returning to step S32;

[0049] When testing the trained encoder-dual decoder model, the input of Text-Decoder includes the encoder output information, the keyword information extracted from the Title-Decoder output, and the information from the background corpus for the overall expansion of the description text under the Attention mechanism;

[0050] S4. Input the image to be described into the image description generation model, and output the title field and description text corresponding to the image to be described. Specifically:

[0051] S41, input the image to be described, and the encoder uses a multi-level self-attention mechanism to extract image feature information;

[0052] S42, the encoder inputs all the extracted image feature information into the Title-Decoder, and outputs the title field corresponding to the image to be described;

[0053] S43, extracting keyword information from the title field output by Title-Decoder, combining it with the image feature information extracted by the encoder, and inputting them into Text-Decoder, and outputting text information corresponding to the image to be described;

[0054] S44, integrating the outputs of Title-Decoder and Text-Decoder to serve as a description of the information contained in the image to be described.

[0055] This embodiment applies the above technical solution, such as Figure 3 As shown, it mainly includes the following processes:

[0056] Step A: prepare a data set, including an image description data set and a corresponding background corpus data set, which are the AI-Challenger image description data set (the image-caption data set is selected in this embodiment) and the Chinese corpus Wikipedia json version (wiki2019zh). Considering that the technical solution generates short descriptions for images in Chinese, the image description data set uses the AI-Challenger image description data set; the corresponding background corpus uses the general Chinese corpus Wikipedia json version (wiki2019zh) as the pre-trained corpus. By selecting different background corpora, the obtained text description will have the style of the corresponding corpus language. Therefore, if there is a desired text description style, the corresponding corpus can be used as the background corpus here. The corpus corresponding to the image description data set used in the present invention;

[0057] Step B: Divide the data set into training set, validation set and test set, and perform necessary data preprocessing, specifically:

[0058] B1: The preprocessing included in the image description dataset includes image format processing and image enhancement processing;

[0059] The processing of the background prediction dataset used for pre-training includes: first, cleaning the sentences, using the self-built stop word list to remove stop words such as periods, commas, and blank characters, then building a word segmentation graph to obtain the largest segmentation combination, then distinguishing and processing Chinese, English, and numbers separately, and finally outputting and annotating the word segmentation;

[0060] B2: Divide the data set into training set, validation set, and test set in a ratio of 8:1:1. This allows the model to first receive a large amount of training on the training set, then adjust the parameters on the validation set, and finally perform testing and evaluation on the test set.

[0061] Step C: setting the parameters and hyperparameters of the model, and training the encoder-dual decoder model using the training set and the validation set; the encoder-dual decoder model constructed by the present invention is generally divided into two parts: the encoder is used to extract information in the image and the relationship between the information, and the decoder reads the information from the encoder and outputs the description sentence word by word;

[0062] C1: The encoder is implemented using the Transformer structure. Given a set of image features I extracted from the input image, the permutation-invariant encoding of X can be obtained through the self-attention used in the Transformer:

[0063]

[0064] Q=W qI, K = W k I, V = W v I

[0065] Among them, W q , W k , W v are all weight matrices that need to be learned, and Q is the query vector;<K,V> is a set of corresponding data pairs; d is the scaling factor; the output of self-attention is applied to a position feed-forward layer, which consists of two affine transformations with a single nonlinearity, which are applied independently to each element in the image set, which can be expressed as:

[0066] FFN(I) i =Uσ(VI i +b)+c

[0067] Among them, I i Represents the i-th vector of the input set; FFN(I) i It represents the output of the i-th vector; σ(·) represents the ReLU function; U and V represent the weight matrices to be learned; b and c represent bias terms.

[0068] The self-attention and position feed-forward layers are encapsulated as a component in a residual connection and a normalization layer, so the complete definition of an encoding layer can be written as:

[0069] Z=AddNorm(self_attention(I))

[0070]

[0071] Among them, AddNorm represents the operation of residual connection and normalization layer. Then for the entire encoder, it is composed of multiple above-mentioned coding layers stacked in sequence, so the input of the i-th layer is the output calculated by the i-1-th layer, which is equivalent to creating a multi-level encoding of the relationship between image features, in which the higher-level coding layer can utilize and improve the relationship that has been identified by the previous layer, which is ultimately equivalent to using prior knowledge to more fully synthesize the feature information of the image. Therefore, the multi-level output of N coding layers stacked can be expressed as:

[0072] C2: Title-Decoder and Text-Decoder are both implemented using LSTM networks, where the state of Title-Decoder at each time step t can be expressed as:

[0073]

[0074]

[0075] Among them, Y title Represents the output of Title-Decoder;

[0076] The state of Text-Decoder at each time step t can be expressed as:

[0077]

[0078]

[0079] Among them, T t-1 It is the keyword information extracted from the Title description output by Title-Decoder in the previous state; C t-1 is the weight information vector obtained from the pre-training background corpus in the previous time state; Y title Represents the output of Text-Decoder. Therefore, the output of the model decoder It is the output of integrating Title-Decoder and Text-Decoder, that is

[0080]

[0081] C3: This example uses a word-level cross entropy loss (XE) function to pre-train the model and uses reinforcement learning to fine-tune sequence generation. When trained with XE, the model is trained to predict the next token given the previous true word; in this case, the input sequence to the decoder is immediately available, and the calculation of the entire output sequence can be completed at once, parallelizing all operations over time;

[0082] C4: A variant of the self-critical sequence training method on sequences sampled using beam-search when training with reinforcement learning: For decoding, sample the top k words from the decoder probability distribution at each time step and always keep the top k sequences with the highest probability. Since sequence decoding is iterative in this step, the above-mentioned parallelism over time cannot be exploited. However, the intermediate keys and values ​​used to compute the output token at time t can be reused in the next iteration;

[0083] C5: Greedy decoding using the average of the rewards is used as the baseline for rewards, and experiments show that it slightly improves the final performance. Therefore, the gradient expression of the sample update is:

[0084]

[0085] Among them, w i is the i-th sentence in beam; r(·) is the reward function; b=(∑ ir(w i )) / k is the baseline, which refers to the average reward calculated for the sample sequence;

[0086] C6: Parameter settings in the model experiment: set the dimension of each layer to 512 and the number of heads to 8; use a dropout layer with a probability of 0.8 after each self-attention and position feedforward layer to avoid model overfitting; use the learning rate scheduling strategy in the following formula when using XE for pre-training:

[0087]

[0088] The initial iteration of warmup_steps is set to 10000; the batch-size in the model is 50, the beam-size is 5, and the optimizer is Adam.

[0089] Step D: Use the test set to evaluate the trained model; specifically:

[0090] D1: After the model training is completed, it will be verified on the test set, where the output of the Text-Decoder model will be affected by the background corpus dataset;

[0091] D2: The prediction sources of the Text-Decoder model are three parts: 1) the structure of the encoder output; 2) the keyword information extracted from the output of the Title-Decoder model; 3) the overall expansion of the description text by the information from the background corpus under the Attention mechanism. The update of the Attention mechanism here is determined by the parameter This parameter can be regarded as the word-level importance weight of the current keyword. Here, the input of the semantic control unit is a word coverage vector C t , initialized to a K-dimensional vector, K is the number of output keywords of the Title-Decoder layer, each value is 1.0, that is, C0 = [1.0, ..., 1.0], at time step t, C t The calculation method is as follows:

[0092]

[0093] Among them, α t,j is the weight of the current keyword at time step t;

[0094] D3: The evaluation indicators used for model evaluation are mainly BLEU, METEOR, ROUGE, CIDEr, SPICE, etc. When conducting model evaluation or model comparison experiments, you can choose one or more of them as reference standards. This example uses the CIDEr-D score as the evaluation indicator, and during the optimization period, a fixed value of 5×10 -6 The learning rate.

[0095] Step E: Input any image into the model encoder structure, output the title field of the image through Title-Decoder, and output the descriptive short text through Text-Decoder. Combine the outputs of the two as a short text description of the information contained in the image. In detail:

[0096] E1: Input image, the encoder uses a multi-level self-attention mechanism to extract image feature information;

[0097] E2: All the extracted feature information is input into Title-Decoder, and then the title field of the image is output;

[0098] E3: Then extract keyword information from Title-Decoder again, combine it with the image feature information extracted from the encoder, and input it into Text-Decoder to generate a short text;

[0099] E4: Integrate the outputs of Title-Decoder and Text-Decoder as the title + description text of the information contained in the image.

[0100] In summary, this technical solution designs a dual-layer decoder architecture, improves the design of the image encoder and language decoder, introduces the attention idea in the Transformer to improve the encoder, and the acquired feature information flows to the two-layer encoder at the same time, which has the following advantages:

[0101] (1) The encoder module uses the attention idea image in Transformer to obtain richer information than the traditional CNN network;

[0102] (2) The model uses a dual decoder module. First, it can make more effective use of the information obtained from the encoder for subsequent description text generation. Second, after pre-training the model with a background corpus, the model's original image information and generated text keywords are corrected under the guidance of the Attention mechanism, ultimately generating a relatively long description text.

[0103] (3) Experiments have shown that the model in the present invention can achieve better results in terms of evaluation indicators, and has the advantages of high accuracy, rich and smooth sentence meaning, and strong correlation between image and text.

Claims

1. A method for generating Chinese description of an image based on encoder-dual decoder, characterized in that: The following steps are involved: S1. Obtain a data set, wherein the data set includes an image description data set and a corresponding background corpus data set; S2, preprocessing the data set, dividing the preprocessed data set into a training set, a validation set, and a test set; S3, construct an encoder-dual decoder model, set corresponding model parameters and hyperparameters, and then train the encoder-dual decoder model based on the training set and the validation set; The trained encoder-dual decoder model is tested based on the test set to obtain an image description generation model; S4, inputting the image to be described into the image description generation model, and outputting a title field and a description text corresponding to the image to be described; Step S3 specifically includes the following steps: S31, constructing an encoder-dual decoder model, wherein the encoder is used to extract information in the image and the relationship between the information, and the dual decoder is used to read the information output from the encoder and output the corresponding description; S32, training the encoder-dual decoder model based on the training set and the validation set; S33, testing the trained encoder-dual decoder model based on the test set, and evaluating the test result according to the set evaluation index. If the evaluation passes, the currently trained encoder-dual decoder model is the image description generation model, otherwise, returning to step S32; In step S31, the encoder adopts a Transformer structure, the dual decoder includes a Title-Decoder and a Text-Decoder connected in sequence, the encoder is connected to the Title-Decoder and the Text-Decoder respectively, and the Title-Decoder and the Text-Decoder are both implemented by an LSTM network; When testing the trained encoder-dual decoder model in step S33, the input of the Text-Decoder includes the encoder output information, the keyword information extracted from the Title-Decoder output, and the overall expansion of the description text from the background corpus under the Attention mechanism; Step S4 specifically includes the following steps: S41, input the image to be described, and the encoder uses a multi-level self-attention mechanism to extract image feature information; S42, the encoder inputs all the extracted image feature information into the Title-Decoder, and outputs the title field corresponding to the image to be described; S43, extracting keyword information from the title field output by Title-Decoder, combining it with the image feature information extracted by the encoder, and inputting them into Text-Decoder, and outputting text information corresponding to the image to be described; S44, integrating the outputs of Title-Decoder and Text-Decoder to serve as a description of the information contained in the image to be described.

2. The method for generating Chinese description of an image based on encoder-dual decoder according to claim 1, characterized in that: The image description dataset in step S1 is specifically the AI-Challenger image description dataset, and the background corpus dataset in step S1 is specifically wiki2019zh.

3. The method for generating Chinese description of an image based on encoder-dual decoder according to claim 1, characterized in that: The step S2 specifically includes the following steps: S21, performing image format processing and image enhancement processing on the image description dataset; performing cleaning and word segmentation and tagging processing on the background corpus dataset; S22. Divide the preprocessed data set into a training set, a validation set, and a test set according to a set ratio.

4. The method for generating Chinese description of an image based on encoder-dual decoder according to claim 3, characterized in that: The specific process of cleaning and segmenting the background corpus data set in step S21 is as follows: first, the sentences are cleaned and stop words are removed using a self-built stop word list; Then, a segmentation word graph is established to obtain the largest segmentation combination; Then distinguish and process Chinese, English and numbers separately, and finally output and mark the segmented words.

5. The method for generating Chinese description of an image based on encoder-dual decoder according to claim 3, characterized in that: The ratio set in step S22 is specifically 8:1:

1.

6. The method for generating Chinese description of an image based on encoder-dual decoder according to claim 1, characterized in that: The evaluation indicators set in step S33 include but are not limited to BLEU, METEOR, ROUGE, CIDEr, and SPICE.