Multi-round natural language understanding algorithm based on historical dialogues and external knowledge

By introducing the historical context coding layer, external knowledge attention layer and global-local attention mechanism into the multi-round dialogue system, combined with the RCGT algorithm, the problems of insufficient utilization and ambiguity of historical context information in multiple rounds of dialogue are solved, and the dialogue system is improved Accuracy and understanding.

CN119025634APending Publication Date: 2024-11-26UNIV OF ELECTRONICS SCI & TECH OF CHINA

Patent Information

Application Number
CN202410907713.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-08
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In multiple rounds of dialogue scenarios, traditional task-based dialogue systems are difficult to effectively utilize historical context information, resulting in error accumulation and ambiguity problems. The lack of background knowledge may cause professional vocabulary ambiguity, affecting the accuracy and efficiency of the dialogue system.

Method used

A multi-round natural language understanding algorithm based on historical dialogue and external knowledge was designed, and through the historical context encoding layer, external knowledge attention layer and global-local attention mechanism, combined with RCGT algorithm, the historical context and professional knowledge are integrated to optimize intentions. and slot label prediction, and adopt adaptive selective loss function and conditional random field decoder to deal with the problem of positive and negative sample imbalance.

Benefits of technology

It effectively reduces the accumulation and ambiguity of errors in multiple rounds of dialogue, improves the accuracy and robustness of the dialogue system, can better understand complex dialogue needs, and enhances the understanding and responsiveness of the dialogue system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025634A_ABST
    Figure CN119025634A_ABST
Patent Text Reader

Abstract

The invention aims to solve the problems and challenges of the existing method in processing multiple rounds of dialogue scenes. In order to realize a better multi-round dialogue scene, the invention performs lightweight improvement on RCGT, and provides an RCGT-MT (Multi-Turn) model scheme, and the specific technical scheme comprises: in order to alleviate the error accumulation problem of a downstream dialogue management module and to solve the problem that historical dialogue context information is not fully utilized, an RCGT-MT (Multi-Turn) model scheme is provided; according to the method, the context information of the historical dialogues is coded, the context attention mechanism of the historical dialogues is introduced, the historical dialogues and dialogue behaviors are coded, and the information in the historical dialogues is aggregated through the context attention mechanism, so that richer information is introduced for natural language understanding tasks; the problems of error accumulation and ambiguity in multiple rounds of dialogues are relieved. In order to solve the problems that ambiguity exists and background knowledge is lacked in contexts of different fields input by a user, an external knowledge base is adopted for introducing professional field knowledge, and particularly, when a question proposed by the user can be correctly understood by specific background information, a model can be helped to better understand the intention of the user, so that the user experience is improved. And the ambiguity problem is relieved. In order to solve the problem of insufficient utilization of historical context information caused by increase of dialogue rounds, a global attention mechanism is introduced to calculate the weight of each round of historical dialogue information, so that the utilization of the historical dialogue information is realized more sufficiently and efficiently.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of natural language understanding in deep learning, and proposes a natural language understanding algorithm based on historical context, aiming to model historical context information to enhance the effect of RCGT algorithm in multi-round dialogues. Background Art

[0002] With the widespread use of intelligent voice assistants such as Apple's Siri and Microsoft's Cortana, task-based dialogue systems have attracted more and more attention. As the first module of a task-based dialogue system, the quality of the natural language understanding task, which is mainly used to map user speech into machine-understandable natural language, will lay a solid foundation for the effect of the entire task-based dialogue system. The RCGT algorithm explicitly models the information between intent and slots in a bidirectional manner, and introduces richer semantic information through TOD-BERT encoding. It regards the intent recognition task as a multi-label classification task, and uses fine-grained intent labels to guide slot filling to achieve a bidirectional improvement effect. In a single-round natural language understanding task, multi-intent recognition is achieved, which is more in line with the dialogue in real scenarios. However, in real dialogue scenarios, as the complexity of the task increases, a single-round dialogue is increasingly difficult to meet the needs of real dialogue scenarios. In order to confirm the real needs of the user, the system often needs to confirm and ask repeatedly, which leads to an increase in dialogue rounds.

[0003] The following problems and challenges exist in multi-round dialogue scenarios:

[0004] (1) Traditional task-based dialogue systems record and save historical dialogues in the dialogue management module. As the number of dialogue rounds increases, errors will gradually accumulate.

[0005] (2) User sentences are ambiguous. Traditional natural language understanding algorithms mostly only consider the scenario of a single round of dialogue. Since they only consider the user input in the current round, these natural language understanding algorithms ignore the historical contextual information in the dialogue with the user and find it difficult to solve the problem of ambiguous user input.

[0006] (3) As the number of conversation rounds increases, the complexity of historical information will also increase, and the proportion of useless information will also increase. Invalid historical conversation context information may even mislead the model. How to reasonably use historical conversation context information is also an unavoidable problem that needs to be solved in multi-round natural language understanding.

[0007] (4) Conversations often contain a lot of professional vocabulary and information, which may cause ambiguity in the absence of background knowledge. Summary of the invention

[0008] In order to overcome the shortcomings of the above-mentioned existing methods, the present invention mainly makes lightweight improvements to the RCGT algorithm for application scenarios existing in multi-round dialogue scenarios.

[0009] The present invention aims to design an improved algorithm for multi-round dialogue scenarios, and the technical solution adopted is:

[0010] Step 1: The historical context encoding layer uses TOD-BERT to encode the historical dialogue. The input of the historical context encoding layer is the historical dialogue text U prev = {U 1 ,U 2 ,…,U t-1}, for any round of historical dialogue U i ∈U prev , [CLS], [SYS], [USR] tags are introduced in the word segmentation stage. The purpose of this step is to introduce more semantic information. After encoding, the contextual information of the historical rounds will be integrated into the embedding of the [CLS] tag. In the example, the embeddings of the [CLS] tags are concatenated. To avoid over-large input, the number of historical dialogue turns is determined by the hyperparameter t H Controlled, each time only the latest t H Encoding of historical conversations;

[0011] Step 2: Historical dialogue actions also contain information, which can be used to guide natural language understanding tasks to improve the effect of natural language understanding in the current round. In order to encode dialogue actions and their corresponding slots, the present invention introduces two embedding matrices and in the dialogue action encoding layer, where and represent the number of labels of the dialogue action and the number of labels of the slot, respectively. Through the dialogue action encoding layer, the embedding of the dialogue action and the corresponding slot is obtained. The present invention uses GloVe 300d as the external knowledge base to initialize the embedding matrix. Finally, the embedding vectors of the dialogue action and the corresponding slot are summed to obtain the dialogue action embedding;

[0012] Step 3: In order to simulate the ability to query professional background knowledge, the present invention proposes a method to enhance the understanding ability of the dialogue system by using an external knowledge base. Specifically, DBPedia is selected as the external knowledge base, which aggregates structured information from Wikipedia, covers a wide range of fields and topics, and can provide a rich and extensive knowledge base for the system;

[0013] Step 4: In order to effectively integrate historical context information, the present invention designs an innovative attention mechanism framework that combines the historical context attention mechanism and the dialogue action attention mechanism. In the traditional historical context attention mechanism, each round of dialogue information is often treated equally, which leads to potential problems: as the number of dialogue rounds increases, historical information irrelevant to the current task is introduced, which may cause parsing ambiguity and reduce the accuracy and efficiency of the system;

[0014] Step 5: After the input passes through the word embedding encoding layer, the external knowledge attention layer, and the global-local attention layer, this step concatenates the context vectors obtained. In order to obtain the temporal features in the sentence and capture the longer-distance dependencies in the sentence, the context information of the sentence is modeled, and the context vector is captured through Bi-LSTM to capture the temporal features. This step reuses the RCGT algorithm framework, which explicitly constructs the intent and slot label context vectors through the slot-intent interaction layer through the full attention mechanism, and fuses the slot information, historical context information, and external knowledge based on the dynamic context in the conversation and the bidirectional dependency between the slot and the intent, and then uses the decoder to infer the intent label. After obtaining the intent label, the algorithm further combines it with the original input containing historical context information and external knowledge, and sends it to the intent-slot interaction layer. While capturing the semantic details of the current input, it also integrates the intent context information provided by the intent label inferred by the intent recognition layer, and combines the historical context information, thereby optimizing the prediction of the slot label;

[0015] Step 6: Connect the above modules. In view of the imbalance problem of positive and negative samples often encountered in actual dialogue scenarios, that is, the number of positive samples and negative samples is significantly different, the adaptive selective loss function ASL is selected as the optimization target. By introducing learning weights γ_+ and γ_-, ASL can increase the attention to difficult-to-classify samples while not reducing the attention to positive simple samples, so as to alleviate the impact of the imbalance of positive and negative samples, thereby improving the robustness and accuracy of the algorithm in multi-intent recognition scenarios. In order to achieve joint optimization of the two tasks, the loss functions of the two tasks are weighted and summed to obtain the final loss function. For the slot filling task, it is regarded as a sequence labeling problem. In order to more accurately capture the dependencies in the labeled sequence and provide structured prediction output, the conditional random field CRF is used as the decoder. In the CRF framework, the negative log-likelihood loss function is used to ensure that the probability deviation between the predicted sequence and the true sequence is minimized during the model training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Figure 1: Model architecture diagram of RCGT-MT.

[0017] Figure 2For: DBPedia knowledge graph visualization example.

[0018] Figure 3 For: Flowchart of external knowledge attention mechanism.

[0019] Figure 4 : Global-local attention layer structure diagram. DETAILED DESCRIPTION

[0020] The present invention will be further described below in conjunction with the accompanying drawings.

[0021] First, the present invention designs a multi-round natural language understanding algorithm structure diagram based on historical dialogue and external knowledge. Figure 1 As shown. In order to solve the problems of accumulated errors in the dialogue management module and underutilization of historical dialogue information, historical dialogue context encoding and historical dialogue context attention mechanisms are introduced. These mechanisms help encode and aggregate historical dialogue information, provide more information for natural language understanding tasks, and thus reduce error accumulation and ambiguity problems in multi-round dialogues. In order to deal with contextual ambiguity and lack of background knowledge in user input, an external knowledge base is used to introduce professional domain knowledge. Especially in cases where specific background information is required to understand user intent, this knowledge can help the model better understand and resolve ambiguity. In order to make more full use of the historical context information accumulated as the number of dialogue turns increases, a global attention mechanism is introduced. This mechanism can effectively calculate the weight of each round of historical dialogue information, thereby making more effective use of historical dialogue information.

[0022] Specific methods

[0023] The present invention designs a multi-round natural language understanding algorithm based on historical dialogues and external knowledge. On top of the RCGT infrastructure, it also includes a historical context encoding layer, an external knowledge attention layer, a global-local attention layer, an intent-slot bidirectional interaction layer, and a decoding layer.

[0024] Figure 2 This is an example of DBPedia knowledge graph visualization. In a complex dialogue environment, participants often rely on their own professional background knowledge to interpret and understand the content of the communication. This phenomenon is extremely common in real-person communication, and it is equally important for building task-based dialogue systems that can understand and participate in such communication. In order to simulate the ability to query professional background knowledge, the present invention proposes a method for enhancing the understanding ability of the dialogue system by using an external knowledge base. Specifically, DBPedia is selected as the external knowledge base, which brings together structured information from Wikipedia, covering a wide range of fields and topics, and can provide a rich and extensive knowledge base for the system.

[0025] In order to effectively integrate knowledge from external knowledge bases into the dialogue system, the present invention adopts a method based on the attention mechanism. This mechanism can dynamically retrieve and select the most relevant knowledge information from the external knowledge base according to the needs of the dialogue context. This process is similar to the ability of humans to selectively call information based on context and background knowledge in a dialogue to enhance understanding and expression.

[0026] By introducing the attention mechanism, the system can not only identify the knowledge points required for the current conversation, but also adjust the weight of this information to ensure that only the most relevant knowledge is used to assist in conversation understanding and generation. This design allows the system to more accurately process and respond to complex conversation needs, improving the accuracy and naturalness of the conversation.

[0027] Figure 3 This is the flow chart of the external knowledge attention mechanism. Each word in the input is searched in the external knowledge base to obtain the corresponding triple i ={h i ,r i ,t i}, where h i Represents the head entity, r i Indicates the relationship, t i Represents the tail entity. And embeds the current round of dialogue into After linear transformation, we get the query Q of the attention mechanism and the relationship r i and entity t i After linear transformation and summation, we get the key K of the attention mechanism. After concatenating the relationship and entity, we get the value V of the attention mechanism. After attention calculation, we get the attention coefficient α i , and then sum up all the attention results to finally get the context vector that integrates external knowledge It is worth mentioning that all external knowledge of a word cannot be introduced into the model, otherwise it will lead to the introduction of redundant information, which will reduce the effect of the model. k Control the amount of external knowledge. The calculation process of the external knowledge attention mechanism is shown in formulas (1) to (3).

[0028]

[0029]

[0030] Among them, Sigmoid and tanh are nonlinear activation functions; W X , W R , W T are learnable weight coefficients; [·] represents a matrix concatenation operation.

[0031] Figure 4 The figure is a diagram of the global-local attention layer structure. The present invention proposes an improved historical context attention mechanism. By introducing the global attention mechanism, the model can carefully score and screen each round of dialogue and its related dialogue actions. This mechanism enables the system to dynamically assign weights to different parts of the historical context, giving priority to information that is more relevant to the current task. In particular, the present invention implements a global-local attention layer, which, through a carefully designed algorithm structure, not only considers the importance of historical information, but also optimizes the processing and integration of this information.

[0032] In this global-local attention layer, the global attention mechanism is responsible for evaluating the relevance of each round of dialogue from a macro level, while the local attention mechanism makes micro adjustments and analyses on the specific content of each dialogue action. This dual-layer attention design aims to accurately grasp the flow and changes of information in the dialogue flow, ensuring that the model can effectively identify and utilize key historical context information when processing multiple rounds of dialogue, thereby significantly improving the performance of the dialogue system.

[0033] After encoding the current dialogue, historical dialogue, and dialogue action, they are linearly transformed and mapped into Q, K, and V required by the attention mechanism, where Q is the current dialogue obtained by linear transformation, and K and V are the historical context and dialogue action obtained by linear transformation. The process of calculating the historical context attention weight and the dialogue action attention weight is shown in formulas (4) to (11).

[0034]

[0035]

[0036] Among them, W QAS , W KAS , W VAS , W QH , W KH , W VH is a learnable weight matrix; b QAS 、b KAS 、b VAS 、b QH 、b KH 、b VH is the learnable bias.

[0037] After calculating the attention weights of the historical context and dialogue actions, each round of historical dialogue and dialogue actions is scored through the global attention mechanism. The scoring process is shown in formulas (12) to (14).

[0038]

[0039] Among them, WQG , W KG is a learnable weight matrix; b QG 、b KG is a learnable bias; global attention weight

[0040] α G Expanded to α AS and α H The same shape is used to perform the Hadamard product. The attention weights including the weights of each round can be obtained. The calculation process is shown in formulas (15) and (16).

[0041] β AS =α AS ⊙α G (15)

[0042] β H =α H ⊙α G (16)

[0043] Among them, ⊙ represents the hadamard product of matrices, that is, the product of corresponding elements of two matrices of the same order.

[0044] After obtaining two attention weights with a global attention mechanism, we perform matrix multiplication with the same value V to obtain a context vector that contains historical context and dialogue actions. The calculation process is shown in formulas (17) and (18).

[0045]

Claims

1. A multi-round natural language understanding algorithm based on historical dialogue and external knowledge, characterized by The following steps are involved: Step 1: The historical context encoding layer uses TOD-BERT to encode the historical dialogue. The input of the historical context encoding layer is the historical dialogue text U prev = {U 1 ,U 2 ,…,U t-1 }, for any round of historical dialogue U i ∈U prev , [CLS], [SYS], [USR] tags are introduced in the word segmentation stage. The purpose of this step is to introduce more semantic information. After encoding, the contextual information of the historical rounds will be integrated into the embedding of the [CLS] tag. In the example, the embeddings of the [CLS] tags are concatenated. To avoid over-large input, the number of historical dialogue turns is determined by the hyperparameter t H Controlled, each time only the latest t H Encoding of historical conversations; Step 2: Historical dialogue actions also contain information, which can be used to guide the natural language understanding task to improve the effect of natural language understanding in the current round. In order to encode dialogue actions and their corresponding slots, the present invention introduces two embedding matrices in the dialogue action encoding layer: and Where |label A | and |label S | respectively represent the number of labels of the dialogue action and the number of labels of the slot. Through the dialogue action encoding layer, the embedding of the dialogue action and the corresponding slot is obtained. The present invention uses GloVe 300d as the external knowledge base to initialize the embedding matrix. Finally, the embedding vectors of the dialogue action and the corresponding slot are summed to obtain the dialogue action embedding; Step 3: In order to simulate the ability to query professional background knowledge, the present invention proposes a method to enhance the understanding ability of the dialogue system by using an external knowledge base. Specifically, DBPedia is selected as the external knowledge base, which aggregates structured information from Wikipedia, covers a wide range of fields and topics, and can provide a rich and extensive knowledge base for the system; Step 4: In order to effectively integrate historical context information, the present invention designs an innovative attention mechanism framework that combines the historical context attention mechanism and the dialogue action attention mechanism. In the traditional historical context attention mechanism, each round of dialogue information is often treated equally, which leads to potential problems: as the number of dialogue rounds increases, historical information irrelevant to the current task is introduced, which may cause parsing ambiguity and reduce the accuracy and efficiency of the system; Step 5: After the input passes through the word embedding encoding layer, the external knowledge attention layer, and the global-local attention layer, this step concatenates the context vectors obtained. In order to obtain the temporal features in the sentence and capture the longer-distance dependencies in the sentence, the context information of the sentence is modeled, and the context vector is captured through Bi-LSTM to capture the temporal features. This step reuses the RCGT algorithm framework, which explicitly constructs the intent and slot label context vectors through the slot-intent interaction layer through the full attention mechanism, and fuses the slot information, historical context information, and external knowledge based on the dynamic context in the conversation and the bidirectional dependency between the slot and the intent, and then uses the decoder to infer the intent label. After obtaining the intent label, the algorithm further combines it with the original input containing historical context information and external knowledge, and sends it to the intent-slot interaction layer. While capturing the semantic details of the current input, it also integrates the intent context information provided by the intent label inferred by the intent recognition layer, and combines the historical context information, thereby optimizing the prediction of the slot label; Step 6: Connect the above modules. In view of the imbalance problem of positive and negative samples often encountered in actual dialogue scenarios, that is, the number of positive samples and negative samples is significantly different, the adaptive selective loss function ASL is selected as the optimization target. ASL introduces the learning weight γ + and γ - , while increasing the attention to samples that are difficult to classify, the attention to positive simple samples will not be reduced, so as to alleviate the impact of the imbalance between positive and negative samples, thereby improving the robustness and accuracy of the algorithm in multi-intent recognition scenarios. In order to achieve joint optimization of the two tasks, the loss functions of the two tasks are weighted and summed to obtain the final loss function. For the slot filling task, it is regarded as a sequence labeling problem. In order to more accurately capture the dependencies in the labeled sequence and provide structured prediction output, the conditional random field CRF is used as the decoder. In the CRF framework, the negative log-likelihood loss function is used to ensure that the probability deviation between the predicted sequence and the true sequence is minimized during the model training process.

2. The method according to claim 1, characterized in that: The historical context encoding layer in step 1 adopts TOD-BERT as a solution to encode historical dialogues.

3. The method according to claim 1, characterized in that: The dialogue action encoding layer in step 2 uses GloVe 300d as an external knowledge base to initialize the embedding matrix.

4. The method according to claim 1, characterized in that: In step 3, DBPedia is selected as the external knowledge base in the external knowledge attention layer.

5. The method according to claim 1, characterized in that: In step 4, an innovative attention mechanism framework is designed, which combines the historical context attention mechanism and the dialogue action attention mechanism.

6. The method according to claim 1, characterized in that: In step 5, the context vector is used to capture the temporal features through Bi-LSTM.

7. The method according to claim 1, characterized in that: In step 6, in order to more accurately capture the dependencies in the labeled sequence and provide structured prediction output, the conditional random field CRF is used as the decoder.

Citation Information

Patent Citations

  • Knowledge-introduced generative session system

    CN112084314A

  • Anaphora Resolution Using Linguisitic Cues, Dialogue Context, and General Knowledge

    US20140257792A1

  • Multi-round dialogue semantic analysis method and system based on long short-term memory network

    WO2021042543A1

Cited By

  • Intelligent scene dialogue analysis method and system based on model recognition

    CN119940345A

  • Multi-round dialogue information feature updating method based on optimal matching mechanism

    CN120407747A

  • Smart phone voice client system and method

    CN120877721A

  • Intelligent decision support system and method based on cognitive logic and scenarized semantics

    CN121526095A