A method for generating a stylized image description based on transfer learning

Abstract

The present application relates to a kind of style image description generation method based on transfer learning, belong to natural language processing and computer vision image description generation technical field.This method uses the method of step-by-step training, uses two mapping networks, respectively learns the semantic knowledge and style knowledge of letter.Using multimodal pre-training model extracts image and text information, and further reduces the semantic distance of image and text by mapping network.Using the method of transfer learning, the text style knowledge is transferred to the image description model, and the style image semantics is effectively generated.The prompt training mode is used to train on large-scale pre-training language model GPT-2, and the training cost is reduced.This method is aimed at the lack of style image description dataset, uses style text corpus to train the model, to obtain the mapping network containing style knowledge, and is transferred to the image description model.Greatly reduce the cost of data acquisition and model training, also achieve good results in the case of unsupervised.

Description

TECHNICAL FIELD

[0001] The present application relates to a style image description generation method based on transfer learning, in particular to a method for generating style image description by using prompt control language model, and a method for transferring language style knowledge learned on text to image description generation field by using transfer learning, and belongs to the technical field of multi-modal image description generation. BACKGROUND

[0002] Since entering the 21st century, with the rapid development of the Internet, smart phones and social networks, image description has been paid more and more attention by people. Image description can be effectively applied in people's daily life, helping preschool and visually impaired users to understand images, and helping people to provide text materials, tags and the like when publishing photos on social networks. For example, in late 2018, Instagram launched a social software containing an AI picture description function. When users click on the pictures they see, the software can automatically describe the pictures in detail, which can effectively help visually impaired users to easily use the software.

[0003] However, the current image description technology mainly stays on the objective description of the image, and the language style is neutral. It cannot meet the needs of people for style and personalized image description generation. However, due to the high cost of obtaining a high-quality style image description dataset, whether it is from a large number of style-rich image text pairs or manually writing descriptions for images, it requires a lot of manpower and time, and the quality is difficult to control. Therefore, the number of large-scale style image description datasets is relatively small, which greatly hinders the development of style image description work.

[0004] Therefore, it is necessary to conduct more in-depth research on image description, so that the image description model can generate style-rich and readable descriptions under the condition of insufficient style image description data, to meet the needs of different users. SUMMARY

[0005] The purpose of the present application is to solve the problem that the style of the sentence generated by the current image description work is single, and the model is difficult to effectively train under the condition that the number and scale of style image description dataset are insufficient, and a style image description generation method based on transfer learning is proposed.

[0006] The innovation of the method is that, in view of the lack of a stylized image description dataset, a stylized text corpus is used to train the model to obtain stylized description knowledge, and the knowledge is migrated to the image description work to generate stylized image descriptions. The prompt paradigm is used to fully utilize the knowledge contained in the large-scale pre-training model, so as to generate high-quality stylized image descriptions without fine-tuning the language model, thereby greatly reducing the training cost.

[0007] The application adopts the following technical solutions.

[0008] Firstly, the related concepts are explained:

[0009] Definition 1: Stylized image description dataset

[0010] Refers to a set of image text pairs that provide corresponding specific pictures and corresponding descriptions for the image description task. Among them, the same picture may correspond to multiple different texts. And different texts may correspond to different styles, such as positive or negative style.

[0011] Definition 2: Multimodal pre-training model

[0012] Refers to a model that is trained on a large-scale dataset and has good results at the end of training. Researchers share the model for others to use. Because the model has been trained on a large amount of data, the model has learned a wealth of knowledge and can be migrated to other fields of research through fine-tuning. The multimodal pre-training model used in this work is CLIP.

[0013] Definition 3: Pre-training autoregressive language model

[0014] Refers to a model that is trained on a large-scale dataset and predicts the next word based on the previous words in the sentence by inputting a prefix prompt. Because it contains a lot of knowledge, it is often used in few-shot learning or zero-shot learning. The pre-training autoregressive language model used in this work is GPT-2.

[0015] Definition 4: Prompt

[0016] Refers to a template or paradigm designed by researchers for downstream tasks. This paradigm can help the pre-training model recall the knowledge learned during pre-training. The prompt can be artificially designed or automatically learned by the model, and can include discrete or continuous, prefix or fill-in-the-blank, etc. The prompt used in this work is a continuous prefix form.

[0017] Definition 5: Autoencoder

[0018] A type of artificial neural network used in semi-supervised and unsupervised learning, whose function is to learn the representation of input information by taking the input information as the learning target, commonly used for feature compression and extraction.

[0019] The purpose of the present application is achieved by the following technical solutions.

[0020] A style image description generation method based on transfer learning, comprising the following steps:

[0021] Step 1: learn the self-representation of text in GPT-2 space using text semantic mapping network

[0022] Specifically, text data (T) without language style tendency is used for training, and a self-supervised method is used to learn the self-representation of text T from the CLIP feature space to the GPT-2 feature space. First, the CLIP text encoder is used to extract the features T-clip of the text T, and a text semantic mapping network Text Semantic Mapping Network is used to map T-clip to the GPT-2 feature space to obtain the feature vector T-prefix. Then, the text T is embedded using GPT-2 to obtain the vector representation T-embedding of the text T in the GPT-2 feature space. T-prefix is taken as the source and T-embedding is taken as the target, and they are combined in the form of prompt=[source][target] as the input of GPT-2. During training, the parameters of GPT-2 are frozen, and only the text semantic prefix mapping network Text Semantic Mapping Network is trained.

[0023] Step 2: learn the style representation of text in GPT-2 space using text style mapping network

[0024] The pair of stylized text data (T, S) is used for training to learn the style representation of text from the CLIP feature space to the GPT-2 feature space. On the basis of the step 1 model, the pre-trained autoencoder is used to compress the information of T-prefix to obtain the high-density semantic information representation T-prefix' of T-prefix and input the text style mapping network to map the T-prefix' vector to obtain the style vector representation TS-prefix. Then, T-prefix is taken as source1, TS-prefix is taken as source2, T-embedding is taken as target, and prompt = [source2] [source1] [target] is combined in the form of GPT-2 input. In the training process, the parameters of GPT-2 and the text semantic mapping network are frozen, and only the text style mapping network is trained.

[0025] Step 3: Learning the semantic representation of the image in the GPT-2 space using the image semantic network

[0026] The image description data (I, T) without language style bias is used for training to learn the semantic representation of the image I from the CLIP feature space to the GPT-2 feature space. The CLIP image encoder is used to extract the features I-clip of the image I, and the image semantic mapping network is used to map the image features I-clip to the GPT-2 feature space to obtain the feature vector I-prefix. The GPT-2 is used to embed T to obtain the vector representation T-embedding of the text. I-prefix is taken as source1, T-embedding is taken as target, and prompt = [source1] [traget] is combined in the form of GPT-2 input. In the training process, the parameters of GPT-2 are frozen, and only the image semantic mapping network is trained. Step 3 and steps 1 and 2 are in parallel relationship, which can be performed simultaneously to further improve the efficiency.

[0027] Step 4: Migrate the text style knowledge to the image description model to generate stylized image description

[0028] Based on the model in step 3, the generated I-prefix vector is input into the text style mapping network trained in step 2 to obtain a style vector prefix-IS, which is spliced with I-prefix as input of GPT-2, so as to obtain an image description containing a specific style.

[0029] Advantages

[0030] Compared with the prior art, the present application has the following advantages:

[0031] 1. In the face of the situation that the number of stylized image description data sets is insufficient, the method of transfer learning can effectively generate stylized image descriptions.

[0032] 2. The prompt paradigm is used to guide large-scale pre-training language models, fully utilizing the language knowledge contained therein, and the parameters thereof are frozen, only the mapping network is trained, greatly reducing the number of training parameters, and the training speed is obviously improved. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The flowchart of the method of the present application;

[0034] Figure 2 The accompanying drawings illustrate the embodiments. DETAILED DESCRIPTION

[0035] The method of the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments.

[0036] EMBODIMENTS

[0037] Taking the stylized text data in MS COCO and collected by the network as an example, the specific operation steps of the method described in the present application will be described in detail in this embodiment;

[0038] As shown in Figure 1 , a stylized image description generation method based on transfer learning comprises the following steps:

[0039] Step 1: Training a text semantic mapping network

[0040] In this example, style text data without language bias is used for training, for example:

[0041] Neutral style text: "A plate filled with a melted cheese covered sandwich and fries."

[0042] Train with self-supervised method, map text features with Text Semantic Mapping Network, learn semantic representation of text from CLIP space to language model GPT-2 space. After training, the effect is as follows:

[0043] Input: "A plate filled with a melted cheese covered sandwich and fries."

[0044] Output: "The plate has a sandwich with many large french fries."

[0045] Step 2: Train Text Style Mapping Network

[0046] Train with paired (neutral, positive) data, for example:

[0047] Neutral style text: "A plate filled with a melted cheese covered sandwich and fries."

[0048] Positive style text: "a plate of delicious food including French fries."

[0049] Fix the parameters of the Text Semantic Mapping Network trained in step 1, and use it to map text features to get text semantic prefix I_prefix. Use the Text Style Mapping Network to map the text semantic prefix I_prefix to get the text style prefix IS_prefix. Concatenate I_prefix and IS_prefix, input the language model for training, learn the style representation of text from CLIP space to language model GPT-2 space.

[0050] After training, the effect is as follows:

[0051] Input: "A plate filled with a melted cheese covered sandwich and fries."

[0052] Output: "This is a delicious plate of french fries."

[0053] Step 3: Train the image-text mapping network

[0054] Train using image description data without language style bias (MS COCO), for example:

[0055] Picture: as shown in Figure 2 .

[0056] Text: "A plate with food and french fries on it"

[0057] Map image features with the text semantic mapping network Image Semantic Mapping Network, learn the semantic representation of the image from the CLIP space to the language model GPT-2 space.

[0058] After training, the effect is as follows:

[0059] Input: as shown in Figure 2 .

[0060] Output: "Some fries and sandwiches on the plate."

[0061] Step 4: Transfer text style knowledge to image description model

[0062] Based on step 3, input the image semantic vector I-prefix into the text style mapping network Text Style Mapping Network in step 2 and fix the parameters, get the image style vector IS-prefix, splice it with I-prefix as the input of GPT-2, so as to obtain the image description containing specific style.

[0063] After training, the effect is as follows:

[0064] Input picture: as shown in Figure 2 .

[0065] Output text: "The plate has one of my favorite foods on it, French fries."

[0066] As shown in this example, the generated text accurately describes the theme content of the image, and has a specified style bias, with good readability. And because the parameters of the pre-trained language model are fixed, only the prompt mapping network is trained, the parameters are reduced by more than 90% compared with similar methods using pre-trained language models, greatly reducing the training cost.

Claims

1. A method for generating stylized image descriptions based on transfer learning, characterized in that, Includes the following steps: Step 1: Use a text semantic mapping network to learn the text's self-representation in the GPT-2 space. The text data (T) without linguistic style bias is used for training, and a self-supervised method is used to learn the self-representation of the text T from the CLIP feature space to the GPT-2 feature space. First, the CLIP text encoder is used to extract the feature T-clip of the text T, and a Text Semantic Mapping Network is used to map the T-clip to the feature space of GPT-2 to obtain the feature vector T-prefix; Then, the text T is embedded using GPT-2 to obtain the vector representation T-embedding of the text T in the GPT-2 feature space; T-prefix is ​​used as the source and T-embedding is used as the target, and they are combined in the form of prompt = [source][target] as the input of GPT-2. During training, the parameters of GPT-2 are frozen, and only the TextSemantic Mapping Network is trained. Step 2: Use a text style mapping network to learn the style representation of text in the GPT-2 space. The model is trained using paired stylized text data (T, S) to learn the style representation of text from the CLIP feature space to the GPT-2 feature space. Based on the model in step 1, a pre-trained autoencoder is used to compress the information of T-prefix, resulting in a high-density semantic information representation T-prefix', which is then input into the Text Style Mapping Network to map the T-prefix' vector to obtain the style vector representation TS-prefix. Then, T-prefix is ​​used as source1, TS-prefix as source2, and T-embedding as target, and they are combined in the form prompt = [source2][source1][target] as input to GPT-2. During training, the parameters of GPT-2 and the Text Semantic Mapping Network are frozen, and only the Text Style Mapping Network is trained. Step 3: Use an image semantic network to learn the semantic representation of the image in the GPT-2 space. The system trains on non-stylistic image description data (I, T) to learn the semantic representation of image I from the CLIP feature space to the GPT-2 feature space. The CLIP image encoder extracts the I-clip feature from image I, and the Image Semantic Mapping Network maps the I-clip feature to the GPT-2 feature space to obtain the feature vector I-prefix. GPT-2 is used to embed T, resulting in the text vector representation T-embedding. I-prefix is ​​used as source1, and T-embedding as target, combined in the form prompt = [source1][target], as input to GPT-2. During training, the parameters of GPT-2 are frozen, and only the Image Semantic Mapping Network is trained. Step 3 is parallel to steps 1 and 2 and can be performed simultaneously, further improving efficiency. Step 4: Transfer the text style knowledge to the image description model to generate stylized image descriptions. Based on the model in step 3, the generated I-prefix vector is input into the Text Style Mapping Network trained in step 2 to obtain the style vector prefix-IS, which is concatenated with the I-prefix and used as input to GPT-2 to obtain an image description containing a specific style.

Images