A keyword generation method based on optimal transmission theory

Abstract

The application discloses a keyword generation method based on optimal transmission theory, as shown in figure 1, the method comprises the following steps: step 1, data preprocessing; step 2, loss function construction; step 3, model training; step 4, model testing and evaluation. The application constructs a loss function based on optimal transmission theory, strengthens the ability of the model to capture text semantic information, and then enables the model to generate more text semantic keywords, reduces the occurrence of missing keywords irrelevant or even contrary to the text semantics, and improves the quality of the keywords.

Description

TECHNICAL FIELD

[0001] The present application relates to a keyword generation method based on optimal transmission theory, belonging to the technical field of Internet and artificial intelligence. BACKGROUND

[0002] With the development of the Internet, people are more and more used to using keywords to retrieve the required text data in the network, so the quality of the keywords of the text data has a huge impact on the retrieval results. In addition, as a high-level summary of text content, people can understand the text content according to the keywords without reading the whole text, thereby saving a lot of time. Therefore, it has important practical significance to extract or generate keywords from text information through machine learning, deep learning and other artificial intelligence technologies.

[0003] Keywords can be divided into existing keywords and missing keywords, the former refers to keywords that completely appear in the text, and the latter refers to keywords that do not completely appear in the text but are highly related to the text content. At present, the way to obtain keywords mainly includes two kinds: extraction and generation. The extraction method first extracts the noun phrases that may be keywords from the text, and then sorts and filters these noun phrases through some scoring mechanism, and the high-score phrases left are considered as keywords. However, the extraction method can only get existing keywords, and cannot obtain missing keywords that are highly related to the semantic information of the text. In order to alleviate the above problems, the keyword generation method based on Sequence-to-Sequence (Seq2Seq) framework has become more and more popular.

[0004] The keyword generation method based on Sequence-to-Sequence (Seq2Seq) framework usually uses a connector to connect multiple keywords as the final target output. Specifically, the Seq2Seq model first converts the text into a vector representing semantic information using a decoder, and then the decoder uses the Attention mechanism and the Copy mechanism to generate the target output sequence step by step. However, the existing keyword generation method focuses too much on the maximum likelihood estimation (MLE) loss function during training, and pays too much attention to the difference between the predicted keywords and the current output, ignoring the connection between the predicted keywords and the text information, which leads to some keywords generated by the method being irrelevant or even contradictory to the text semantic information, especially for missing keywords. SUMMARY

[0005] In order to solve the problems and deficiencies in the prior art, the application provides a keyword generation method based on optimal transport theory, which introduces the optimal transport theory in the process of model training and combines with the Attention mechanism to strengthen the ability of the model to capture the semantic information of the text, so that the model can generate more keywords that conform to the semantic information of the text, and reduce the occurrence of missing keywords that are irrelevant or even contrary to the semantic information of the text.

[0006] In order to achieve the above-mentioned purpose, the technical scheme of the application is as follows: a keyword generation method based on optimal transport theory, comprising the following steps:

[0007] Step 1: data preprocessing. The text preprocessing is performed on all samples in the data set, and the BART tokenizer is used for word segmentation to form a token sequence. A plurality of keywords are arranged in sequence and connected by a connector; connection, finally forming a binary tuple training sample of <text, keyword sequence>.

[0008] Step 2: loss function construction. The loss function is classified into two parts, one part is the traditional maximum likelihood estimation (MLE) loss function The other part is the loss function designed based on the optimal transport theory First, the weight of each token in the text is dynamically calculated according to the Attention weight of the model, then the cosine similarity between the text token and the predicted keyword token is calculated by using the word vector, and finally the optimal transport distance between the text and the predicted keyword sequence is calculated by using the IPOT algorithm, and the distance is used as the loss function of the second part

[0009] Step 3: model training. First, the preprocessed text token sequence is sent into the BART encoder, and the vector representation of the text is obtained through the word vector layer and the multi-layer Self-Attention, and then the BART decoder decodes and outputs according to the vector representation of the text. The loss function designed based on the optimal transport theory in step 2 is used as the training target.

[0010] Step 4: model testing and evaluation. The model that performs best on the validation set during the training process is selected as the prediction model for the keyword prediction task in actual use. First, the text that needs to generate keywords is preprocessed as the input of the model, and then the keywords are generated. The F1@5 and F1@M values are used as evaluation indexes to measure the quality of the generated keywords.

[0011] Compared with the prior art, the application has the following advantages:

[0012] 1) The application uses BART as the basis, compared with the RNN and the Transformer initialized randomly, the BART pre-trained in the paper has better text representation and performance.

[0013] 2) The application designs a loss function based on the optimal transport theory, which is the first time to apply the optimal transport theory to the keyword generation field. In the process of model training, the loss function can effectively curb the model to generate irrelevant or even contradictory missing keywords with the text, and improve the quality of the generated keywords.

[0014] 3) The application combines the Attention mechanism in the process of constructing the loss function based on the optimal transport theory, and then dynamically generates the weight of each token, that is, μ and v, so that the calculation of the optimal transport distance is more reasonable.

[0015] 4) The method proposed in the application can be applied to other Attention-based Seq2Seq keyword generation models without changing the model structure, which has wide applicability and strong compatibility. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 The overall flowchart of the embodiment of the application.

[0017] Figure 2 The model diagram of the embodiment of the application.

[0018] Figure 3 The design calculation diagram of the embodiment of the application. DETAILED DESCRIPTION

[0019] In order to deepen the understanding and understanding of the application, the following will be further illustrated in combination with specific embodiments.

[0020] Embodiment 1: A keyword generation method based on the optimal transport theory, which first pre-processes and divides the data set; then, referring to Figure 2 , adopts the BART model as the backbone network, then abstracts the keyword generation problem into the optimal transport problem, and constructs the loss function based on the optimal transport distance according to the optimal transport theory The calculation process of refers to Figure 3 , and finally combines to form the final loss function Then the BART is trained under the supervision of the above designed loss function, to suppress the tendency of the model to generate irrelevant or even contradictory missing keywords with the text semantics, and to improve the quality of the generated keywords. The detailed implementation steps are as follows:

[0021] ​Step 1: Data Preprocessing. Given a sample, pre-process the sample by first using regular expressions to remove some illegal characters in the sample, changing all numbers to <digit>Identifiers. Keywords are joined together using the semicolon (;), with existing keywords ordered according to their first appearance in the text, and missing keywords following them, thus forming a keyword sequence. Then, BARTtokenizer is used to segment the text keyword sequence, with the segmentation result denoted as X = {x1, x2, ..., x...}. m }, Y = {y1, y2, ..., y n }, where m and n are the number of text segments and the number of keyword sequence tokens, respectively. Finally<X,Y> This constitutes a final training sample. Finally, the dataset is divided into a training set, a validation set, and a test set in an 8:1:1 ratio.

[0022] Step 2: Loss Function Construction. The loss function constructed in this step serves as the training objective for the next step of the model. Its implementation can be divided into the following sub-steps:

[0023] Sub-step 2-1: Constructing the optimal transmission model. The optimal transmission problem is used to compute the optimal transmission model for a set of locations P = {p1, p2, ..., p...}. m The divisible "goods" on a set Q are transported to another location set Q = {q1,q2,…,q}. n The minimum transportation cost on X is denoted as μ = {μ1, μ2, ..., μ...}. The quantity of "goods" that need to be transported at each location in X is denoted as μ = {μ1, μ2, ..., μ...}. m Let ν = {ν1, ν2, ..., v} be the number of "goods" that can be accepted at each position in Y. n Let C(x) be an integer. i ,y j ) is from position x i Transfer a unit mass of "cargo" to location y. j The expenses, and Given μ, ν, and C, the allocation matrix with minimum cost can be calculated: T * =argmin∑ ij T ij C ij stT1 n =μ,T T 1 m =ν.

[0024] In this invention, the keyword generation problem is abstracted into an optimal transmission problem, X = {x1, x2, ..., x}. m } represents a sequence of text. The keyword sequences predicted by the model correspond to P and Q respectively. Therefore, the semantics of each token in X need to be transferred... The more consistent the predicted keywords are with the semantics of the text, the lower the optimal transmission cost of the transfer.

[0025] Sub-step 2-2: Dynamic generation of μ and v combined with Attention. When X is fed into the encoder of BART, we can get the intermediate vector representation:

[0026] H = BART enc (X) ∈ R m×d ,(1)

[0027] where d is the dimension of the vector. Then the decoder of BART generates the predicted output p t the probability of the token at the t-th step based on H and the previous predicted output

[0028]

[0029] where Z is the number of Attention Heads.

[0030] Then we collect all the Attention weights of the decoding steps, denoted as:

[0031]

[0032] where A can be seen as the attention between each token, and the values of μ and v are dynamically assigned according to A:

[0033]

[0034]

[0035] Sub-step 2-3: Definition of the unit transmission cost matrix C. In this invention, the cosine similarity between the word vectors of tokens is used to calculate the unit cost:

[0036]

[0037] where e i represents the word vector representation of token x i , and the predicted word vector representation of is e j = E T p j , E represents all the word vector matrices. Finally, the unit transmission cost matrix C = {c ij |i = 1, … m; j = 1, …, n} can be obtained:

[0038] c ij = 1 - Sim(e i , e j ).#(7)

[0039] Sub-step 2-4: Calculate the optimal transport distance. After obtaining μ,ν and the cost matrix C, the present application uses the IPOT algorithm to obtain the optimal transport matrix T * and the corresponding loss function

[0040]

[0041] Sub-step 2-5: Calculation of the traditional maximum likelihood estimation loss function. Calculate the maximum likelihood estimation (MLE) loss function

[0042]

[0043] Sub-step 2-6: Integration of the loss function. After obtaining and , they are weighted and added to obtain the final loss function

[0044]

[0045] Step 3: Model training. Use the loss function based on optimal transport theory designed in step 2 as the training target, the model can better capture the semantic information of the text and suppress the generation of missing keywords that are irrelevant or even contradictory to the text under the constraint of . During training, use cosine learning rate decay, and every time the model is trained for a fixed step, it is put on the validation set for verification. The verification index is the loss function value, the model on the validation set value does not decrease for four consecutive times, the model training ends early.

[0046] Step 4: Model keyword generation. After model training in step 3, the best-performing model is obtained. Predict the test set. First, preprocess the text in the test set according to step 1, and use the processed data as the input of the model to obtain the final keyword output represents the i-th keyword, and M is the number of keywords.

[0047] Use F1@5 and F1@M to detect the quality of the generated keywords, and the F1 value is calculated as follows:

[0048]

[0049]

[0050]

[0051] wherein Y represents the true keyword, F1@5 refers to comparing the first five generated keywords, and if there are less than five, adding false keywords randomly until the number reaches five; and F1@M refers to comparing all generated keywords.

[0052] Based on the same inventive concept, the application also discloses a keyword generation method based on optimal transmission theory, which comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the computer program realizes the keyword generation method based on optimal transmission theory when loaded into the processor.

[0053] Those skilled in the art will appreciate that the embodiments described herein are for the purpose of helping the reader to understand the principles of the application, and it should be understood that the embodiments are only used to illustrate the application and not to limit the scope of the application, and after reading the application, those skilled in the art can make various equivalent modifications to the application, which all fall within the scope defined by the claims of the application.< / digit>

Claims

1. A keyword generation method based on optimal transport theory, characterized in that, The method comprises the following steps: Step 1: data preprocessing, Step 2: loss function construction, Step 3: model training, Step 4: model testing and evaluation, Wherein, step 2: loss function construction, as follows: its implementation is divided into the following sub steps: Sub-step 2-1: Optimal transport model construction. The optimal transport problem is used to compute the minimum cost of transporting a set of splittable "goods" from a set of locations to a set of locations , where is the amount of "goods" that needs to be transported from location , is the amount of "goods" that is received at location , is the cost of transporting one unit mass of "goods" from location to location , and is the cost matrix that can be computed given , , and : , The keyword generation problem is abstracted as an optimal transport problem, for a sequence of texts, The predicted keyword sequence for a model corresponds to P and Q respectively, and the semantic of each token in needs to be transferred to the token in If the predicted keyword is more consistent with the text semantically, the optimal transport cost of the transfer will be smaller. Sub-step 2-2: Dynamic generation of and is fed into the encoder of BART, resulting in an intermediate vector representation:​ where is the dimension of the vector, then the decoder of BART computes the probability of the token at step t given the previous predicted output the weights of the Attention of the last layer :​ wherein is the number of AttentionHeads, Then collect all the decoding steps of the attention weight, denoted as: wherein , , A can be seen as the attention between each token, which is dynamically assigned according to and values:​ Substep 2-3: definition of unit transmission cost matrix C, using the cosine similarity between the word vectors of token to calculate the unit cost: wherein the word vector representation of the token , the word vector representation of the predicted token , representing all word vector matrices, finally, the unit transmission cost matrix may be obtained: Sub-step 2-4: Calculate the optimal transmission distance, after obtaining , and the cost matrix , the optimal transmission matrix is obtained by IPOT algorithm and the corresponding loss function is obtained. Sub-step 2-5: Calculation of conventional maximum likelihood estimation loss function, calculating the maximum likelihood estimation (MLE) loss function : Sub-step 2-6: integrate the loss function, after obtaining and then, weight them and add them to obtain the final loss function : 2.The keyword generation method based on optimal transport theory of claim 1, wherein, Step 1: data preprocessing, all samples in the data set are preprocessed and segmented by using BART tokenizer to form token sequence, multiple keywords are arranged in sequence and connected by using connector; Connection, finally constitute the binary training sample of <text, keyword sequence>. 3.The keyword generation method based on optimal transport theory of claim 1, wherein, Step 2: Loss function construction, the loss function is divided into two parts, one part is the traditional maximum likelihood estimation (MLE) loss function , the other part is the loss function we designed based on optimal transport theory . First, according to the attention weight of the model, the weight of each token in the text is dynamically calculated, then the cosine similarity between the text token and the predicted keyword token is calculated using the word vector, and finally the optimal transport distance between the text and the predicted keyword sequence is calculated using the IPOT algorithm, which is used as the loss function of the second part . 4.The keyword generation method based on optimal transport theory of claim 1, wherein, Step 3: model training, first put the preprocessed text token sequence into the BART encoder, get the vector representation of the text through the word vector layer and multiple layers of self-attention, and then the BART decoder decodes and outputs according to the vector representation of the text, using the loss function based on optimal transport theory designed in step 2 as the training target. 5.The keyword generation method based on optimal transport theory of claim 1, wherein, Step 4: Model testing and evaluation, after step 3 model training, the best performing model is obtained, and the test set is predicted, first the text in the test set is preprocessed in step 1, and the processed data is used as the input of the model to get the final keyword output , represents the i-th keyword, is the number of keywords, F1@5 and F1@M are used to detect the quality of the generated keywords, and the calculation of F1 value is as follows: wherein F1@5 refers to comparing the top five generated keywords, and if there are less than five, then adding false keywords at random until the number reaches five; and F1@M refers to comparing all generated keywords.

Images