A semantic understanding method, computer device and storage medium

Abstract

The present invention discloses a semantic understanding method based on triple task combination and a computer device and storage medium for executing the semantic understanding method. The current user input sentence is annotated with intent category, entity, and complete semantic expression, and a data set including annotation, the current user input sentence, and historical conversation data before the current user input sentence is constructed; the data set is feature vectorized and the vector representation of the intent category identifier of the current user input sentence is input into a Transformer model, and multi-task joint training is performed to jointly train the three subtasks of intent recognition, entity recognition, and conversation query rewriting to obtain a natural semantic understanding prediction model; the new user input sentence and the historical conversation data before the new user input sentence are feature vectorized and input into the natural semantic understanding prediction model to obtain the semantic output corresponding to the new user input sentence.

Description

Technical Field

[0001] The present invention relates to the field of intelligent dialogue systems, and in particular to a semantic understanding method based on triple task combination, a computer device and a storage medium. Background Art

[0002] Current natural language understanding models mostly use intent recognition and entity recognition as two subtasks for joint training, but the semantic understanding function of the model is still relatively limited. Since these models usually only perform semantic analysis on the current user input, they cannot consider the impact of session history information on semantic understanding, and therefore cannot solve the problem caused by reference omission in user input sentences.

[0003] Based on the above situation, natural language understanding models urgently need the ability to learn historical conversation information. As a semantic understanding task that rewrites user expressions using conversation history information, the conversation query rewriting task can effectively solve the problem of reference omission in multi-round conversations.

[0004] The patent specification with publication number CN 112364665 A discloses a semantic extraction method, including marking historical question and answer sentences and current question and answer sentences respectively to obtain historical marking vectors and current marking vectors; inputting the historical marking vectors and current marking vectors into a pre-trained encoding network respectively to obtain historical sentence encoding and current sentence encoding; inputting the historical sentence encoding and current sentence encoding into a pre-trained decoder to obtain a rewritten sentence.

[0005] In terms of model training, the Transformer model architecture can be used to jointly expand tasks, making it easier to perform multi-task model training.

[0006] Based on this, the inventors of this case focused on how to jointly learn the three subtasks of intent recognition, entity recognition, and conversation query rewriting to train and recognize user semantics, thereby training a semantic understanding model with higher practicality. The complete sentence representation based on the output of conversation history rewriting can be directly used for retrieval and matching, expanding the multi-round question-and-answer function of the dialogue system architecture. Summary of the invention

[0007] In view of the shortcomings in the art, the present invention provides a semantic understanding method based on triple task combination. Based on the traditional joint task model, the method proposes a natural semantic understanding model with three subtasks joint learning by jointly learning the conversation query rewriting task. The model simultaneously learns the intent classification, entity recognition, and conversation query rewriting tasks, so that it can not only predict the intent category and entity list of the user input sentence, but also rewrite the user's newly input sentence expression according to the conversation history for subsequent question-answer retrieval matching in the dialogue system.

[0008] A semantic understanding method based on triple task combination includes the following steps:

[0009] (1) annotating the current user input sentence with the intent category, entity, and complete semantic expression, and constructing a data set including the annotation, the current user input sentence, and n rounds of historical conversation data before the current user input sentence (each round of conversation contains two sentences); n is a positive integer;

[0010] (2) The feature vectorization of the dataset and the vector representation of the intent category identifier of the current user input sentence are input into the Transformer model, and multi-task joint training is performed to jointly train the three subtasks of intent recognition, entity recognition, and conversation query rewriting to obtain a natural semantic understanding prediction model;

[0011] (3) The new user input sentence and the historical conversation data of n rounds before the new user input sentence are feature vectorized according to the method of step (2) and input into the natural semantic understanding prediction model to obtain the intent category, entity list, and complete semantic expression vector corresponding to the new user input sentence.

[0012] In step (2), the one-hot (Shen J, Zhang J, et al., Proc Natl Acad, USA, 2007, 104(11): 4337-4341.) method is used to perform feature vectorization for the annotation of the intent category and entity.

[0013] In step (2), for the annotation of the complete semantic expression, its feature vector is represented as the concatenation of the word embeddings of all the tokens that constitute it.

[0014] In step (2), for the current user input sentence, its feature vector is represented as the concatenation of the feature embeddings of all the tokens that constitute it, and the feature embedding of the token is the sum of the word embedding and the position embedding corresponding to the token.

[0015] In step (2), the historical session data H can be expressed as H=(H 1 ,H2 ,...,H 2n ), where H 1 ,H 2 ,...,H 2n Represents each statement, and each statement is separated by <seq>As an identifier of the end of a round, H is regarded as a sequence of m tokens, that is, H = (w 1 ,w 2 ,...,w m );

[0016] For the historical session data, its feature vector is represented as the concatenation of feature embeddings of all the tokens that constitute it, and the feature embedding of the token is the sum of the word embedding, position embedding, and round embedding corresponding to the token.

[0017] In step (2), the vector representation of the intent category identifier of the current user input sentence is its corresponding word embedding.

[0018] In step (2), the multi-task joint training is performed to jointly train the three subtasks of intent recognition, entity recognition, and conversation query rewriting to obtain a natural semantic understanding prediction model, specifically including:

[0019] The Transformer model is used to encode and train the vector representation of the intent category identifier of the current user input sentence, the feature vector of the historical session data, the feature vector of the current user input sentence, and the feature vector of the i-th Token of the current user input sentence, and output three embedding sequence representation layers, specifically, a session rewriting embedding layer, an intent category embedding layer, and a current sentence entity tag sequence embedding layer; the feature vector of the i-th Token of the current user input sentence is represented as the sum of the word embedding and position embedding of the Token, where i is a positive integer;

[0020] The loss of each subtask is calculated based on the vector representation of the embedding sequence representation layer and the corresponding task annotation (i.e., the annotation of the intent category, entity, and complete semantic expression in step (1)). The sum of the losses of each subtask is the total loss. By adjusting the Transformer model parameters, the total loss is minimized to obtain a natural semantic understanding prediction model.

[0021] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the semantic understanding method based on triple task combination when executing the computer program.

[0022] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the semantic understanding method based on the combination of triple tasks are implemented.

[0023] Compared with the prior art, the present invention has the following main advantages:

[0024] (1) A semantic understanding method based on triple task union is used to jointly learn the three subtasks of intent recognition, entity recognition, and conversation query rewriting, and conduct joint training. The subtasks influence and work together with each other, which can effectively enhance the performance of the natural language processing model and achieve higher accuracy.

[0025] (2) The natural semantic understanding model trained by the present invention simultaneously inputs the current user sentence and the current historical conversation data, and can learn the historical conversation information, so as to have a more comprehensive and sufficient analysis of the semantic understanding of the current user sentence, thereby achieving a better semantic understanding effect.

[0026] (3) The semantic understanding model obtained through training has more and richer semantic output results, and can be applied to different types of dialogue robots at the same time, such as task-based robots and question-answering robots. Especially in the case of multi-round question-answering, the rewritten semantic representation output by the model can be directly matched with the question and answer, which effectively solves the difficulty of reference resolution in multi-round question-answering. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a partial example of a conversation dataset in the field of genetic counseling in an embodiment;

[0028] Figure 2 A schematic diagram of a natural semantic understanding prediction model based on triple task combination in an embodiment;

[0029] Figure 3 It is a flowchart of a semantic understanding method based on triple task combination according to an embodiment. DETAILED DESCRIPTION

[0030] The present invention will be further described below in conjunction with the accompanying drawings and specific examples. It should be understood that these examples are only intended to illustrate the present invention and are not intended to limit the scope of the present invention. The operating methods in the following examples without specifying specific conditions are usually carried out under conventional conditions or under conditions recommended by the manufacturer.

[0031] The semantic understanding method based on triple task combination of this embodiment is shown in the accompanying drawings, and specifically includes the steps of:

[0032] (1) Annotate the current user input sentence with intent category, entity, and complete semantic expression, and construct a dataset including the annotation, the current user input sentence, and n rounds of historical conversation data before the current user input sentence (each round of conversation contains two sentences); n is a positive integer.

[0033] This example uses genetic counseling field dialogue corpus data, and some examples of the constructed data set are as follows: Figure 1 As shown, data is a data point in the data set, history is the historical conversation data of n rounds before the current user input sentence (n=2 is selected in this embodiment), text is the text of the current user input sentence, intent is the intention category label of the current user input sentence, rewrite is the complete semantic expression label of the current user input sentence, and entities is the entity label of the current user input sentence.

[0034] (2) Figure 2 As shown, the dataset is feature vectorized and the vector representation of __CLS__ is input into the Transformer model for multi-task joint training, and the three sub-tasks of intent recognition, entity recognition, and conversation query rewriting are trained together to obtain a natural semantic understanding prediction model; __CLS__ represents the intent category identifier of the current user input sentence.

[0035] It mainly involves two aspects: feature vectorization of data and joint training of three tasks. The specific steps are as follows:

[0036] The feature vectorization of data includes the feature vectorization of independent variables and dependent variables, among which the current user input statement, the historical conversation data of n rounds before the current user input statement, and the _CLS_ intent identifier are independent variables, and the intent category, entity, and complete semantic expression of the current user input statement are dependent variables.

[0037] For the current user input statement U=(w 1 , w 2 , ..., w j ), whose eigenvector E U It is calculated as follows:

[0038] U(w i )=WE(w i )+PE(w i )

[0039] E U =[U(w 1 ), U(w 2 ), ..., U(w j )]

[0040] Among them, w 1 , w 2 , ..., w j represents the j tokens that make up the current user input sentence, U(w i ) represents the feature vector of the ith Token of the current user input sentence, WE(w i ) represents the word embedding of the ith Token of the current user input sentence, PE(w i ) represents the position embedding of the ith Token of the current user input sentence. The word embedding of the Token in this embodiment is directly obtained from the Bert pre-training model.

[0041] The historical session data H is represented by H=(H 1 , H 2 , ..., H 2n ), where H 1 , H 2 , ..., H 2n Represents each statement, and each statement is separated by <seq>As an identifier of the end of a round, H is regarded as a sequence of m tokens, that is, H = (w 1 , w 2 , ..., w m );

[0042] For the historical session data, its feature vector EH is calculated as follows:

[0043] I(w i )=WE(w i )+PE(w i )+TE(w i )

[0044] E H =[I(w 1 ), I(w 2 ), ..., I(w m )]

[0045] I(w i ) represents the feature vector of the ith Token of the historical session data, WE(w i ) represents the word embedding of the ith Token of the historical session data, PE(w i ) represents the position embedding of the ith Token of the historical session data, TE(w i ) represents the conversation turn embedding of the historical conversation data.

[0046] The vector representation of the _CLS_ intent identifier is its corresponding word embedding. The word embedding of the _CLS_ identifier used in this embodiment is directly obtained from the Bert pre-training model, taking the weighted average vector of all words.

[0047] For the dependent variable part, the one-hot method is used to vectorize the features for the annotation of the intent categories and entities; for the annotation of the complete semantic expression, the method of feature vectorization is: the annotation vector of the complete semantic expression is represented as the sum of the word embeddings of all the tokens that constitute it.

[0048] For the triple-task joint training, we mainly conduct multi-task joint training to jointly train the three sub-tasks of intent recognition, entity recognition, and conversation query rewriting: the vector representation of __CLS__, the feature vector of historical conversation data, the feature vector of the current user input sentence, and the feature vector U(w i ) is used for encoding training, and three embedding sequence representation layers are output, namely the session rewriting embedding layer, the CLS token embedding layer, and the current sentence entity tag sequence embedding layer. Among them: the feature vector of the historical session data and the feature vector of the current user input sentence are respectively passed through the Transformer to obtain the historical embedding representation and the current input representation, and then these two representation vectors are concatenated and input into a fully connected layer of size 512*256 to obtain a vector embedding representation of size 256, namely the session rewriting embedding layer. The CLS token embedding layer is directly obtained by inputting the vector representation of __CLS__ into the Transformer. The current sentence entity tag sequence embedding layer is directly obtained by inputting the feature vector sequence of the token of the current user input sentence into the Transformer.

[0049] The loss of each subtask is calculated based on the vector representation of the embedding sequence representation layer and the corresponding task annotation (i.e., the annotation of the intent category, entity, and complete semantic expression in step (1)). The sum of the losses of each subtask is the total loss. The specific subtask loss is:

[0050] For the intent recognition task, the feature vector representation of the intent classification annotation and the CLS token embedding layer output by the Transformer module are similarly calculated, and then the dot product loss is used as the loss of the intent recognition task.

[0051] For the entity recognition task, the probability score between the current sentence entity tag sequence embedding layer and the entity annotation feature vector representation described in CRF training is used, and the log likelihood is taken as the loss of the entity recognition task.

[0052] For the conversation query rewriting task, the task training target is the complete expression of the manually modified input sentence, which no longer contains references and omissions. The similarity between the feature vectors of the annotations of the conversation rewriting embedding layer and the complete semantic expression is calculated as the loss of the conversation query rewriting task. This embodiment uses the cosine similarity of the above two feature vectors for calculation.

[0053] By adjusting the Transformer model parameters and minimizing the total loss, a natural semantic understanding prediction model is obtained.

[0054] (3) The data set is divided into 10 small portions, each of which has a separate training and test set, and 10 models are trained separately, once for each small portion, and the average value is taken as the final score. This embodiment uses two classic joint models, the Bert-CRF joint model (Chen Q, Zhuo Z, Wang W. Bert for joint intent classification and slot filling [J]. arXiv preprint arXiv: 1902.10909, 2019.) and the DIET joint model (Bunk T, Varshneya D, Vlasov V, et al. Diet: Lightweight language understanding for dialogue systems [J]. arXiv preprint arXiv: 2004.09936, 2020.) to conduct comparative experiments with the triple-task joint model trained by the method of the present invention, and uses precision Precision, recall Recall and F1 score as result indicators. The results of the comparative experiment show that the overall performance of the triple-task joint model in intent understanding is better than other comparative models, and the results of each indicator are more than 2% higher than the second-best model DIET. This shows that the expansion of the session query rewriting task can effectively use the session history information, thereby improving the natural semantic understanding model's ability to understand user semantics. In terms of entity recognition, the F1 score (87.65%) obtained by the triple-task joint model is also higher than that of the DIET method (86.41%). Therefore, through comparative experiments, it can be proved that the triple joint multi-task model of this embodiment is indeed superior to other existing dual-task joint models in performance.

[0055] (4) Figure 3 As shown, the new user input sentence and the historical conversation data of n rounds before the new user input sentence are feature-vectorized according to the method of step (2) and then input into the natural semantic understanding prediction model to obtain the intent category, entity list, and complete semantic expression vector corresponding to the new user input sentence. The intent category and entity list output by the model can be used for skill triggering and word slot filling in task-based dialogues, thereby completing the dialogue task information filling. In the case of multiple rounds of question and answer, the rewritten complete semantic expression vector output by the model can be directly used for question and answer matching, which effectively solves the difficulty of reference resolution in multiple rounds of question and answer.

[0056] In addition, it should be understood that after reading the above description of the present invention, those skilled in the art may make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the claims attached to this application.< / seq> < / seq>

Claims

1. A semantic understanding method based on triple task combination, characterized in that: Includes steps: (1) annotating the current user input sentence with the intent category, entity, and complete semantic expression, and constructing a data set including the annotation, the current user input sentence, and n rounds of historical conversation data before the current user input sentence, where n is a positive integer; (2) The feature vectorization of the dataset and the vector representation of the intent category identifier of the current user input sentence are input into the Transformer model, and multi-task joint training is performed to jointly train the three subtasks of intent recognition, entity recognition, and conversation query rewriting to obtain a natural semantic understanding prediction model; The process of performing multi-task joint training to jointly train three subtasks of intent recognition, entity recognition, and conversation query rewriting to obtain a natural semantic understanding prediction model specifically includes: The Transformer model is used to encode and train the vector representation of the intent category identifier of the current user input sentence, the feature vector of the historical session data, the feature vector of the current user input sentence, and the feature vector of the i-th Token of the current user input sentence, and output three embedding sequence representation layers, specifically, a session rewriting embedding layer, an intent category embedding layer, and a current sentence entity tag sequence embedding layer; the feature vector of the i-th Token of the current user input sentence is represented as the sum of the word embedding and position embedding of the Token, where i is a positive integer; The loss of each subtask is calculated according to the vector representation of the embedding sequence representation layer and the corresponding task annotation. The sum of the losses of each subtask is the total loss. The total loss is minimized by adjusting the Transformer model parameters to obtain a natural semantic understanding prediction model. (3) The new user input sentence and the historical conversation data of n rounds before the new user input sentence are feature vectorized according to the method of step (2) and input into the natural semantic understanding prediction model to obtain the intent category, entity list, and complete semantic expression vector corresponding to the new user input sentence.

2. The semantic understanding method based on triple task combination according to claim 1 is characterized in that: In step (2), the one-hot method is used to perform feature vectorization for the labeling of the intent category and entity.

3. The semantic understanding method based on triple task combination according to claim 1 is characterized in that: In step (2), for the annotation of the complete semantic expression, its feature vector is represented as the concatenation of the word embeddings of all the tokens that constitute it.

4. The semantic understanding method based on triple task combination according to claim 1 is characterized in that: In step (2), for the current user input sentence, its feature vector is represented as the concatenation of the feature embeddings of all the tokens that constitute it, and the feature embedding of the token is the sum of the word embedding and the position embedding corresponding to the token.

5. The semantic understanding method based on triple task combination according to claim 1 is characterized in that: In step (2), the historical session data H is represented by H=(H1, H2, ..., H 2n ), where H1, H2, ..., H 2n Represents each statement, and each statement is separated by <seq>As an identifier of the end of a round, H is regarded as a sequence of m tokens, that is, H = (w1, w2, ..., w m );< / seq> For the historical session data, its feature vector is represented as the concatenation of feature embeddings of all the tokens that constitute it, and the feature embedding of the token is the sum of the word embedding, position embedding, and round embedding corresponding to the token.

6. The semantic understanding method based on triple task combination according to claim 1 is characterized in that: In step (2), the vector representation of the intent category identifier of the current user input sentence is its corresponding word embedding.

7. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the semantic understanding method based on triple task combination as described in any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the semantic understanding method based on triple task combination as described in any one of claims 1 to 6 are implemented.

Images