Weakly supervised phrase localization method based on dual reinforcement learning
By employing dual reinforcement learning, and leveraging the duality of phrase localization and generation tasks, the model is trained alternately, thus addressing the lack of labeled data in weakly supervised phrase localization and achieving more efficient model optimization and improved localization accuracy.
Patent Information
- Application Number
- CN202211552224.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-05
- Publication Date
- 2025-12-12
- Estimated Expiration
- 2042-12-05
AI Technical Summary
Existing weakly supervised phrase localization methods cannot effectively guide the learning of phrase localization models in the absence of labeled data, resulting in limited performance improvement and high costs due to reliance on manual annotation.
We design a dual reinforcement learning-based method that leverages the duality of phrase localization and generation tasks. We utilize a warm-start model based on pseudo-image region-phrase pairs and optimize it by alternately training the phrase localization and generation models and providing direct feedback signals.
In the absence of labeled data, explicit optimization of the phrase localization model improved localization accuracy, reduced manual annotation costs, and enhanced model performance.
Smart Images
Figure CN115830427B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of multimedia, in particular to a weakly supervised phrase localization method based on dual reinforcement learning. BACKGROUND
[0002] Phrase localization is an important research content in the field of multimedia, and also has very important application in the field of human-computer interaction. A user inputs a phrase description, and requires a robot to locate the target described by the user in the scene. At present, although the fully supervised method has achieved satisfactory results, these methods greatly depend on a large amount of image-region-text pair data, which not only brings great manual annotation cost, but also is limited by the size and quality of the annotation data.
[0003] In order to reduce the labor cost, the weakly supervised method aims to train a phrase localization model given an image-text pair (without region-level annotation). Due to the lack of annotation data, some existing methods optimize through constructing additional proxy tasks, such as reconstructing the input phrase or aligning the phrase features and image-level visual features in the alignment space. However, in these methods, the phrase localization task can be regarded as an intermediate task of the proxy task, and is optimized in an implicit way. The huge gap between the optimization objectives of the proxy task and the target task makes the existing methods unable to guarantee that the loss function of the proxy task can effectively guide the learning of the intermediate task. This inefficient optimization method hinders the further improvement of the performance of the weakly supervised phrase localization model. Therefore, how to design a method to directly and effectively guide the learning of the phrase localization model in the absence of annotation data is an important research direction in the field of weakly supervised phrase localization. SUMMARY
[0004] In view of this, the purpose of the present application is to provide a method capable of directly and effectively optimizing a weakly supervised phrase localization model, which can directly judge the pros and cons of the phrase localization model in the absence of annotation data, so as to optimize.
[0005] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows: a weakly supervised visual localization method based on dual reinforcement learning, comprising the following steps:
[0006] S1, a phrase localization model is designed to locate the image region described by the phrase in the image; a phrase generation model is designed to generate a natural language level phrase description for a certain region in the image; S2, a pre-trained object detector is used to extract the candidate frame of the image and the class and attribute name of each candidate frame. The semantic similarity of each candidate frame and the phrase description is calculated to obtain the pseudo-candidate frame of each phrase description, i.e. pseudo-image region-phrase pair data. The phrase localization and phrase generation models are hot started by using these pseudo data;
[0007] S3, using the duality between the phrase localization and phrase generation tasks, providing feedback signals for each other to measure the pros and cons of the model, and designing two reward functions to quantify the feedback signals and collaboratively train the phrase localization and phrase generation models;
[0008] S4, using the trained phrase localization model to complete the phrase localization task.
[0009] In step S1, the phrase localization model inputs an image I and a phrase description q, and outputs the target region r described by the phrase description in the image. The phrase localization model is a general visual-linguistic dual-branch model. The model maps visual features and language features to a common semantic space, so that the similarity between the two modalities can be directly calculated. The phrase generation model inputs an image I and an image region r, and outputs a natural language level phrase description. The phrase generation model is a classic encoder-decoder structure. The model uses a convolutional neural network CNN to encode the image region features, and then uses a recurrent neural network LSTM to decode the encoded features, outputting a natural language level phrase description.
[0010] In step S2, an image I is input into the pre-trained object detector Faster-RCNN to obtain N image regions (i represents the index of the image region), and the class name c i and the attribute name a i of each image region. By calculating the semantic similarity of the phrase query q and each candidate box r i , the total matching score is obtained by summation, and the candidate box r pseudo with the highest score is selected as the matching result of the input phrase description q, and then the phrase localization and phrase generation models are hot started using these pseudo data to provide a better model initialization parameter.
[0011] In step S3, the duality between the phrase localization and phrase generation tasks is used to collaboratively train the phrase localization and phrase generation models, so that the two models can progress together and improve together. The core idea of this method is:
[0012] There is a bidirectional correspondence between image regions and their corresponding phrase descriptions. Specifically, if a phrase is correctly located, the input phrase can be regenerated based on the located image region. Similarly, if a specific object is accurately described by a text phrase, the spatial location of that specific object can be located based on the generated text phrase. Therefore, the phrase base task and the phrase generation task can act as surrogate tasks, providing direct feedback signals (i.e., the correctness of the located region and the accuracy of the generated phrase) to measure each other's performance. In this way, even in a weakly supervised environment, the phrase localization model can be explicitly optimized through region-level supervision. This invention uses an alternating training method to train the phrase localization and phrase generation models, so that the two models can be trained collaboratively and improve together. For training the phrase localization model, given an image I and a phrase description q, the phrase localization model locates the image region r that q refers to in I. mid Then r mid The input phrase generation model obtains its corresponding natural language description q'. By measuring the semantic consistency between the input phrase description q and the generated phrase description q', the result r located by the phrase localization model can be determined. mid The accuracy of the localization is ensured. Therefore, even in the case of missing labeled data, the quality of the localization results of the phrase localization model can still be judged, allowing for direct optimization of the localization results. For training the phrase generation model, given an image I and an image region r, the phrase generation model generates a natural language description q describing the image region r. mid , then q mid Locating q in the input phrase localization model mid In image region r' of image I, by measuring the spatial consistency between the input image region r and the located image region r', the phrase generation model can be used to determine the generated phrase q. mid The accuracy of the data is ensured, thereby optimizing the generative model accordingly. Through this collaborative, alternating training, both models can obtain instance-level supervision signals even in the absence of labeled data, enabling collaborative optimization and mutual improvement.
[0013] In step S4, during the inference phase, no phrase generation model is needed; only a phrase localization model is used for visual localization, which helps reduce inference time.
[0014] The beneficial effects achieved by this invention include: The method utilizes the duality of phrase localization and phrase generation tasks, forming a closed loop. Even with missing labeled data, these two tasks can provide feedback signals to each other, directly measuring the quality of each model and thus directly optimizing the model results. Compared to state-of-the-art weakly supervised phrase localization methods, this invention provides more direct and effective supervision signals for optimization. Furthermore, this invention benefits from large-scale, readily available image-text pair data to improve the performance of weakly supervised phrase localization and reduce the manual costs associated with labeled data. Attached Figure Description
[0015] Figure 1 For phrase localization models and phrase generation models.
[0016] Figure 2 This is a schematic diagram of pseudo-image region-phrase pair generation.
[0017] Figure 3 This is a block diagram illustrating the principle of weakly supervised phrase localization based on dual reinforcement learning.
[0018] Figure 4 This is a flowchart of the present invention. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the following detailed embodiments are described in conjunction with the appendix. Figures 1 to 4 The present invention will be further described in detail below. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the invention; obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.
[0020] Step 1, as follows Figure 3 As shown, the weakly supervised phrase localization method based on dual reinforcement learning proposed in this invention consists of two models: a phrase localization model and a phrase generation model. Step 1 designs these two models. Figure 1 As shown in (a), the phrase localization model takes an image I and a phrase description q as input, and outputs the localized image region r. The phrase localization model designed in this invention is a classic two-branch embedding model. The visual branch utilizes a pre-trained object detector, Faster R-CNN, to extract all image regions of image I. And by using a convolutional neural network to extract the visual representations of each image region, we obtain... Figure 1Region features in (a) are denoted as The semantic embedding module is used to map the region features from the visual semantic space to the text semantic space, so that the similarity can be directly calculated with the text features of the phrase description. The semantic embedding module is composed of two layers of multilayer perceptron, and the number of neurons in each layer is 512. For a given phrase description where w represents a word, t represents the index subscript of the word, and T represents the number of words contained in the phrase description. The text branch first encodes each word into a 0-1 vector, and then further converts the 0-1 vector into a word vector using the word embedding layer. Finally, the input is input into the recurrent neural network for encoding to obtain the phrase feature shown in the figure. The length of the word vector is 512, and the size of the hidden layer in the recurrent neural network is also 512. The invention uses the hidden feature of the last time step T of the recurrent neural network as the feature h of the phrase description. Finally, the similarity between the region feature and the phrase feature can be directly calculated using the dot product of the vectors, so as to obtain the matching degree of each candidate box and the phrase description, and the candidate box with the highest matching degree is taken as the output result of the phrase localization model. The calculation formula is as follows:
[0021]
[0022] where j represents the index of the target region with the highest similarity, i represents the index of the candidate target region, v i represents the visual feature of the target region r i , h represents the text feature of the phrase description q, and T is the vector transpose symbol. The input of the phrase generation model is an image I and a region r in the image, and the output is a natural language level phrase description q. The phrase generation model is a classic encoder-decoder structure. Among them, the encoder part adopts the convolutional neural network ResNet-101 to encode the visual feature. The image I and the image region r are input into the ResNet-101 network to extract the visual feature, and the two features are spliced and then input into the recurrent neural network for decoding, wherein the number of hidden layer units of the recurrent neural network is 512. In the decoding process, the special text mark <start>a start symbol denoting the beginning of the phrase description, and when the decoder outputs <end>The time represents the end of decoding. As can be seen from the model structure, both the phrase localization model and the phrase generation model are very light, and thus have the advantages of fast training convergence and rapid inference. In addition, the two models also have universality, which is embodied in the fact that the method is not limited to the two models designed by the present application, and the phrase localization or phrase generation models proposed by other inventions can also be applied to the present application.
[0023] The simplicity and universality of the phrase localization model and the phrase generation model; the simplicity is reflected in the simple structure of the two models designed by the present application, thus having the advantages of light model and rapid convergence. The universality is reflected in the fact that the method is not limited to the two models designed by the present application, and the phrase localization or phrase generation models proposed by other inventions can also be applied to the present application.
[0024] Step two, since the phrase localization and phrase generation models with random initialization of parameters cannot provide effective feedback signals for each other in the initial training stage of the model, the present application proposes a heuristic algorithm to generate pseudo image region-phrase description pairs to warm up the phrase localization and phrase generation models, without relying on any manually annotated data, thereby reducing the cost of manual annotation. Considering that a phrase description is usually composed of the class, attribute and location information of the target, for example, "the man (class) in blue (attribute) on the right (location)". Therefore, the present application proposes to obtain the pseudo image region-phrase description pair by calculating the matching degree of the three key elements of the image region and the phrase description. The specific technical solution is shown in Figure (2): given an image I, first use the pre-trained target detector Faster R-CNN (see Shaoqing
[0025] Ren, Kaiming He, Ross Girshick and Jian Sun, "Faster R-CNN: Towards
[0026] Real-Time Object Detection with Region Proposal Networks", in NIPS, 2015.) to obtain N image regions and the class name c i and the attribute name a i of each image region. And a series of spatial position words are defined, such as up, down, left and right, and the corresponding spatial position words are assigned according to the spatial position of the image region in the image. Then, for a given phrase description q, the language converter is used to decompose and convert it to obtain the information words corresponding to the three aspects in the phrase description. Finally, the matching degree SIM x , and the candidate box r with the highest score is selected pseudo The matching result of the input phrase description q. The calculation formula is as follows:
[0027] SIM =∑SIM x , x e (cat, attr, loc)
[0028] r pseudo = argmax (SIM)
[0029] Where cat, attr, loc represent category, attribute and location information; SIM cat , SIM attr , SIM loc represent category matching score, attribute matching score, and location matching score respectively; pseudo represents the index of the candidate target region with higher similarity. The calculation method of the category matching score SIM cat is the cosine similarity (Cos) of the word embedding vectors of the two category names; where the word vector is selected as Glove word embedding, and the calculation method is as follows:
[0030] emb n = Glove (W n )
[0031] emb c = Glove (C i )
[0032] SIM cat (q, r i ) = maxcos (emb n , emb c )
[0033] Where W n represents the category name in the phrase description q, emb n represents the word embedding of W n , and C i represents the category name of the candidate box r i ; emb c represents the word embedding of C i ; and cos represents the cosine similarity. The calculation of the attribute matching score and the location matching score is to determine whether the attribute word or the location word appears in the phrase description, and the calculation formula is as follows:
[0034]
[0035] Among them, attr and loc represent category, attribute and location information respectively; q represents phrase description and X represents attribute word or location word. Then, these pseudo-data are used to warm up the phrase localization and phrase generation model to provide a better model initialization parameter.
[0036] Step 3, as shown in Figure (3), this invention utilizes the duality between phrase localization and phrase generation tasks to collaboratively train the localization and generation models. For training the phrase localization model, the phrase generation model provides feedback signals. Even in the absence of annotation information, the model can obtain feedback on the quality of the output results, which helps reduce the dependence of deep learning models on label data. Given an image I and a phrase description q, r mid The location result is given by the phrase localization model, and q' is the phrase generation model. mid The generated natural language description. The reward value for the phrase localization model is calculated based on the input phrase description q and the generated phrase description q'. l This reward value represents the phrase generation model's performance on r. mid Accuracy assessment. The optimization objective of training the phrase localization model is to maximize the reward value of the localization model under its parameter distribution. l The expectation, that is:
[0037]
[0038] Where the subscript l represents the phrase localization model; J l This represents the expectation of the phrase localization model; r mid This represents the output of the phrase localization model; p represents the probability; the corresponding gradient can be calculated using the policy gradient algorithm, and the calculation formula is as follows:
[0039]
[0040] Where θ represents the parameters of the phrase localization model; The gradient of the phrase localization model is represented by p; the probability is represented by log. p (r mid The output of the phrase localization model is the exponentially normalized output (softmax). The reward function of the localization model is... l The calculation is performed by measuring the semantic consistency between the input phrase q and the generated phrase q'. Higher rewards are awarded based on this consistency. l The positioning result representing the positioning model is more accurate. The present application adopts a simple and common BLEU score (for details, see Papineni, Kishore, et al. "Bleu: a method for automatic evaluation of machine translation." Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 2002.) as the reward value of the phrase positioning model, as follows:
[0041] reward l = BLEU(q, q'), l,
[0042] wherein the subscript l represents the phrase positioning model; and BLEU represents the score name.
[0043] For training the phrase generation model, the phrase positioning model is used to provide feedback signals for the phrase generation model. Given an image I, and an image region r, q mid is the natural language description describing the image region r generated by the phrase generation model, and r' is the image region positioned by the phrase positioning model according to q mid . The reward value reward s of the phrase generation model is calculated according to the input image region r and the positioned region r', which represents the evaluation of the phrase positioning model on the q mid generated by the phrase generation model. The optimization goal of training the phrase generation model is to maximize the expectation of the reward value reward s of the phrase generation model under the parameter distribution of the phrase generation model, that is:
[0044]
[0045] wherein the subscript s represents the phrase generation model; J s represents the expectation of the phrase generation model; q mid is the generation result of the phrase generation model; p represents the probability; and according to the policy gradient algorithm, the corresponding gradient can also be calculated, and the calculation formula is as follows:
[0046]
[0047] wherein, is the parameter of the phrase generation model; represents the gradient of the phrase positioning model; p represents the probability; q mid is the generation result of the phrase generation model; log q (q mid ) is the exponential normalization output (softmax) of the phrase generation model. The reward of the phrase generation model reward s is calculated by measuring the spatial consistency between the input image region r and the localized image region r'. Intuitively, the reward of the phrase generation model reward s encourages the speaker to generate a textual phrase that accurately describes the given object and can be understood by the phrase localization model. Therefore, the intersection over union (IoU) between the two regions is used as the reward reward s . A higher IoU means that the phrase generation model generates a more accurate textual phrase, and the calculation formula is as follows:
[0048] reward s = IoU(r, r'),
[0049] where the subscript s represents the phrase generation model; and IoU represents the intersection over union between the two regions.
[0050] It can be easily seen that although the labeled information between the textual phrase and the image region is not used in the training process, feedback about the quality of the localized image region can still be obtained, so that it can be directly optimized at the region level.
[0051] Step four, although the training phase requires the collaborative training of the phrase localization model and the phrase generation model, the training cost is very low due to the simple structure of the two models. In addition, in the inference phase, the method does not require the phrase generation model, and only the phrase localization model is used.
[0052] So far, the weakly supervised phrase localization method based on dual reinforcement learning proposed by the application has been described. The method first uses the duality of the phrase localization and phrase generation tasks to design an explicit optimization method for the weakly supervised phrase localization task, which can provide a region-level feedback signal for the weakly supervised phrase localization model in the absence of labeled data.
[0053] In the simulation experiment, the application uses two classic data sets: ReferItGame and Flickr30KEntities for experiments, and the evaluation index used in the experiment is the accuracy (%). The following table shows the experimental results of the method of the application and the prior art. The experimental results show that the application proposed in the application achieves the best effect at present.
[0054]
[0055]
[0056] Note: The higher the evaluation index, the better. Among them, GroundR, MATN, KAC Net, Contr. Dist. and ReIR are the best methods in the field of weakly supervised phrase localization before;
[0057] GroundR is found in (Rohrbach A, Rohrbach M, Hu R, et al. Grounding of textual phrases in images by reconstruction [C] / / European Conference on Computer Vision. Springer, Cham, 2016:
[0058] 817-834.);
[0059] MATN is found in [Zhao F, Li J, Zhao J, et al. Weakly supervised phrase localization with multi-scale anchored transformer network [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 5696-5705.];
[0060] KAC Net is found in (Chen K, Gao J, Nevatia R. Knowledge aided consistency for weakly supervised phrase grounding [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 4042-4050.);
[0061] Contr. Dist. is found in (Wang L, Huang J, Li Y, et al. Improving weakly supervised visual grounding by contrastive knowledge distillation [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 14090-14100.), using image-phrase alignment as a proxy task, these methods are in an implicit way to optimize the phrase localization model, unable to ensure that the loss function of the proxy task can effectively guide the learning of the phrase localization model;
[0062] ReIR is found in (Liu Y, Wan B, Ma L, et al. Relation-aware instance refinement for weakly supervised visual grounding [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 5612-5621.), using phrase reconstruction as a proxy task for optimization.
[0063] The method proposed in the application utilizes the duality of the phrase localization and phrase generation tasks, and explicitly provides a supervision signal for the phrase localization model, so better results are achieved.< / end> < / start>
Claims
1. A weakly supervised phrase localization method based on dual reinforcement learning, characterized in that, The method comprises the following steps: Step S1, designing a phrase localization model for locating the image region described by the phrase description in the image; and designing a phrase generation model for generating a natural language level phrase description for a certain region in the image; The phrase localization model is a visual-linguistic dual-branch model; the model maps visual features and linguistic features to a common semantic space, so that the features between the two modalities can be directly compared; the phrase localization model finally outputs the target region r with the highest similarity to the phrase description q j The calculation formula is as follows: wherein r denotes the target region, j denotes the index of the target region with the highest similarity, i denotes the index of the candidate target region, v i ′ denotes the visual feature of the target region r i , h denotes the text feature of the phrase description q, and T is the vector transposition symbol; The phrase generation model adopts an encoder-decoder structure; the model uses a convolutional neural network (CNN) to encode the image region features, and then uses a long short-term memory (LSTM) to perform decoding operation on the encoded features, and outputs a natural language level phrase description; Step S2, using a pre-trained target detector to extract the candidate frame of the image and the class and attribute name of each candidate frame; calculating the semantic similarity between each candidate frame and the phrase description to obtain the pseudo-candidate frame of each phrase description, i.e. pseudo-image region-phrase pair data; and using the data to hot start the phrase localization model and the phrase generation model; Step S3, using the duality between the phrase localization and phrase generation tasks to provide feedback signals for each other to measure the pros and cons of the models, and designing two reward functions to quantify the feedback signals and use reinforcement learning to collaboratively train the phrase localization model and the phrase generation model; Step S4, using the trained phrase localization model to complete the phrase localization task.
2. The weakly supervised phrase localization method based on dual reinforcement learning according to claim 1, characterized in that: In step S2, a pre-trained object detector is used to extract candidate boxes of the image where i represents the index of the candidate target region, N represents the total number of extracted image candidate boxes, and the class name c of each candidate box is extracted i and the attribute name a i and some predefined spatial information words are assigned to each candidate box according to the spatial position of the candidate box in the image; by calculating the similarity of the class, attribute and position described by the candidate box and the phrase, and summing the three similarities to obtain the total matching score, and selecting the candidate box r with the highest score pseudo as the matching result of the input phrase description q; the calculation formula is as follows: SIM =∑SIM x x∈(cat,attr,loc) r pseudo = argmax(SIM) Wherein, cat, attr, loc are category, attribute and location information respectively; SIM cat , SIM attr , SIM loc represent category matching score, attribute matching score and location matching score respectively; pseudo represents the index of the candidate target region with higher similarity; the calculation method of the category matching score SIM cat is the cosine similarity (Cos) of the word embedding vectors of the two category names; wherein the word vector selected is Glove word embedding, and the calculation method is as follows: emb n = Glove(W n ) emb c = Glove(C i ) SIM cat (q,r i ) = maxcos(emb n , emb c ) where W n represents the class name in the phrase description q, emb n represents the word embedding of W n ; C i represents the class name of the candidate box r i ; emb c represents the word embedding of C i ; and cos represents the cosine similarity. The attribute matching score and the location matching score are calculated by judging whether the attribute word or the location word appears in the phrase description, and the calculation formula is as follows: wherein attr, loc are divided into class, attribute and location information; q represents the phrase description; and X represents the attribute word or the location word, and then the pseudo-data are used to hot start the phrase localization and phrase generation models to provide model initialization parameters.
3. The weakly supervised phrase localization method based on dual reinforcement learning according to claim 1, characterized in that: In step S3, the duality of the phrase localization and phrase generation tasks is exploited to provide explicit supervision signals for each other; the phrase localization model reward function reward l The semantic consistency of the input phrase q and the output phrase q ‘ is considered, which is calculated as follows: reward l = BLEU(q, q ‘ ) wherein subscript l denotes a phrase localization model; BLEU denotes a score name that measures semantic consistency of an input phrase q and an output phrase q ‘ ; a phrase generation model reward function reward s considers spatial consistency of an input region r and a localization region r ‘ , and the calculation formula is as follows: reward s = IoU(r, r ‘ ) where subscript s denotes the phrase generation model; IoU denotes the intersection over union between the two regions; the optimization objective of training the phrase localization model is to maximize the reward value reward of the localization model under its parameter distribution l the expectation that: wherein subscript l represents the phrase positioning model; J l represents the expectation of the phrase positioning model; r mid is the output result of the phrase positioning model; p represents a probability; according to a policy gradient algorithm, a corresponding gradient can be calculated, and a calculation formula is as follows: where θ denotes the parameters of the phrase localization model; denotes the gradient of the phrase localization model; p denotes the probability; log p (r mid ) is the exponential normalized output of the phrase localization; the optimization objective of training the phrase generation model is to maximize the expected value of the reward of the phrase generation model under its parameter distribution, i.e.: s reward wherein subscript s represents the phrase generation model; J s represents the expectation of the phrase generation model; q mid is the generation result of the phrase generation model; p represents the probability; according to the policy gradient algorithm, the corresponding gradient can also be calculated, and the calculation formula is as follows: wherein, parameters of the phrase generation model; denotes a gradient of the phrase positioning model; p denotes a probability; q mid is a generation result of the phrase generation model; log p (q mid ) is an exponentially normalized output of the phrase generation model.