Proactive system for increasing proactivity in knowledge-driven proactive dialogues

Abstract

The application discloses a knowledge-driven initiative promotion system in an initiative in a dialogue, which comprises a dialogue state mapping unit, a dialogue initiative scoring unit, a data enhancement unit and a generated result checking unit, wherein the initiative of the dialogue robot is detected, secondary screening of training set data is carried out through an evaluation index, the score of a model generated reply in a test set is improved through a data enhancement method, and the automation degree is significantly improved.

Description

TECHNICAL FIELD

[0001] The application relates to a technology in the field of natural language processing, in particular to an initiative improvement system of a language model in a knowledge-driven initiative dialogue system. BACKGROUND

[0002] Natural language processing (NLP) is an important direction in the field of computer science and the field of artificial intelligence, wherein: a dialogue system (Dialog System) comprises two aspects of natural language understanding and generation technology, and a knowledge-driven initiative dialogue system is a dialogue system in which a dialogue robot is actively led based on a knowledge graph. However, the current knowledge-driven initiative dialogue dataset is not unified in format, and the information granularity is different, and the state tracking in the dialogue and the knowledge graph mapping path extraction are lacking; the initiative of the dataset lacks quantitative marking, and the initiative embodied in the dialogue lacks a unified standard measurement standard; the text generation quality depends on the similarity with a reference reply, and due to the particularity of the initiative dialogue, it is unreasonable to evaluate the initiative by using the similarity, the initiative of the reply generated by the model cannot be measured, and specific adjustment for the problem is lacking. SUMMARY

[0003] The application proposes an initiative improvement system in a knowledge-driven initiative dialogue, which detects the initiative of the dialogue robot and performs secondary screening of the training set data through evaluation indexes, so as to improve the score of the reply generated by the model in the test set by using the data enhancement method, and the degree of automation is significantly improved.

[0004] The application is implemented by the following technical scheme:

[0005] The application relates to an initiative improvement system in a knowledge-driven initiative dialogue, which comprises a dialogue state mapping unit, a dialogue initiative scoring unit, a data enhancement unit and a generation result verification unit, wherein: the dialogue state mapping unit performs regular matching of each entity and each dialogue history to obtain a topic transition chain; the dialogue initiative scoring unit describes the initiative of the dialogue robot in the dialogue from four aspects of information richness, non-chatting proportion, promotion speed and dialogue fluency according to the knowledge graph and the topic transition chain; the data enhancement unit performs secondary screening according to the initiative score of each sample in the training set to obtain training samples with high initiative for retraining of the model; and the generation result verification unit compares the accuracy of the generated reply before and after data enhancement in the test set and the initiative score of the test set dialogue to measure the improvement effect of the model.

[0006] Technical effects

[0007] This invention emphasizes the mapping relationship between responses and knowledge graphs through topic transition chains, addressing the shortcomings of current knowledge-driven proactive dialogue datasets, such as inconsistent formats, lack of dialogue state tracking, and extraction of knowledge graph mapping paths. Furthermore, it designs new evaluation metrics from four dimensions to define the proactivity of the chatbot in a more scientific and comprehensive way. A method for improving model performance is employed by selecting data from the training set for augmented training. Compared to existing technologies, this invention can enhance the proactivity of language models on multiple proactive dialogue datasets, while also offering strong interpretability, low training costs, and broad application scenarios. Attached Figure Description

[0008] Figure 1 This is a schematic diagram illustrating the principle of the present invention;

[0009] Figure 2 This is a flowchart of the present invention. Detailed Implementation

[0010] like Figure 1 As shown in the figure, this embodiment relates to a method for enhancing initiative based on a knowledge-driven dialogue system, which includes the following steps:

[0011] Step 1) Obtain all knowledge nodes of the knowledge graph E = [E1, E2, ... E n ], linking each entity with each dialogue history U j (j≤M) Perform regular expression matching, if node E i If (i∈N) appears in the j-th dialogue, then the dialogue state of the j-th dialogue is incremented by E. i And record the position of appearance in the dialogue, where: the knowledge graph is a directed graph composed of N [entity, relation, entity] triples [E i R i E i′ ], i∈N, the dialogue history U consists of M dialogues between the user and the chatbot, and either party can initiate a dialogue.

[0012] Step 2) For each dialogue j, obtain a list of all knowledge entities discussed, and reorder them according to their appearance position to obtain a topic transition chain C of L (L≤N) knowledge entities. j The topic shift chain C j The dialogue contains repeated entities, j∈M. The topic transition chain of each sentence is mapped as a path on the knowledge graph, reflecting the steps and direction the speaker takes to advance the dialogue topic. The initial state of the dialogue topic is empty; the last knowledge entity E in the dialogue topic chain is then used. i =C j[-1], i ∈ N marks the latest state of the current dialogue, if the dialogue does not mention any knowledge entity, the topic shift chain is empty, the dialogue belongs to chitchat, i.e. non-active dialogue, and the current dialogue state does not change.

[0013] Step 3) For evaluating the activity of the dialogue bot, specifically comprising:

[0014] 3.1) When there are P (P ≤ M) sentences of the dialogue bot's reply in the current dialogue and Q (Q ≤ P) sentences contain the topic shift chain corresponding to the knowledge entity, the dialogue has a specified topic starting point E s (E S and the target E T (E T , i.e. the target knowledge entity that the bot needs to lead the topic to, first use the tokenizer tool of nltk to split, and split each sentence of the bot and the user into S tokens;

[0015] 3.2) As shown in Figure 2 , calculate for each dialogue respectively:

[0016] Informativeness: the proportion of knowledge entities contained in each sentence of the dialogue bot on average, each topic shift chain C j (j ≤ P), The informativeness embodied by the bot in the dialogue is Where: L j is the number of knowledge entities contained in the jth round of the bot's reply, S j is the number of tokens contained in the jth round of the bot's reply, and P is the total number of rounds of dialogue.

[0017] Non-chitchat ratio: the proportion of the number of replies containing knowledge entities of the dialogue bot in the dialogue to the total number of replies, Where: Q is the number of replies containing knowledge entities in the number of replies of the dialogue bot, and P is the same as above.

[0018] Gained distance: the average speed at which the dialogue bot pushes the dialogue on the path of the knowledge graph, the speed of each round is equal to the gained distance divided by the shortest distance from E s to E T . The gained distance of the jth sentence of the dialogue bot's reply in the dialogue is the shortest distance from the current state to E T on the knowledge graph minus the shortest distance from the dialogue state after the user finishes speaking before the reply to E T ,

[0019]

[0020] where: d(E i , E i′ ) denotes the shortest distance between E i and E i′ on the knowledge graph, C j [-1] denotes the last mentioned knowledge entity in the jth turn of the dialogue by the robot. When the robot's reply does not contain any knowledge entity from the knowledge graph, the gained distance j is 0; if the dialogue has not mentioned any knowledge entity before this turn or the knowledge entity mentioned by the user before this turn does not exist in the path from the knowledge entity in the dialogue to the Target, the gained distance j is 1.

[0021] Smoothness of the dialogue: the average of the smoothness j of each turn of the dialogue. For each turn j, the smoothness j measures whether the robot successfully connects the topic of the previous turn. where: C j [0] denotes the first mentioned knowledge entity in the jth turn of the dialogue by the robot.

[0022] 3.3) Sum the information richness, the non-small-talk ratio, the pushing speed and the smoothness of the dialogue calculated in step 3.2 to obtain the initiative score of the dialogue robot in this dialogue.

[0023] The reply of the dialogue robot in the dialogue contains knowledge entity recognition as an action to promote the shift of the dialogue topic. The higher the proportion of non-small talk in the dialogue, the more frequent the action to promote the dialogue, and the more active the robot. The promotion speed is first calculated by comparing the dialogue state of each reply with the previous sentence, and then normalized to determine how close the dialogue is to the target entity. The design idea is that an active dialogue robot will promote the shift of the dialogue topic to the target entity at a faster speed, and the dialogue state after each reply is significantly closer to the target on average. There are three special cases in the definition of the promotion speed: if the reply of the dialogue robot is small talk, there is no shift of the topic, and the promotion speed is 0; if the dialogue has not been talking about any knowledge entity before, the dialogue topic is completely initiated by the robot and the user is chatting, and the promotion speed of this round is 1; when the knowledge entity in the dialogue before this does not exist on the graph to the Target, it is considered that the user has brought the dialogue to an irrelevant topic, and if the reply of the robot shifts the dialogue topic to an entity on the knowledge graph that has a path to the Target, the robot brings the topic back to the path towards the target, and the promotion speed of this round is 1. In addition, a qualified active dialogue robot needs to ensure the coherence of the current reply and the context of the previous text, and cannot just talk to itself. The beginning of the reply needs to be connected with the dialogue state of the previous text, and the higher the fluency of the dialogue, the more fluent the reply of the robot, and the higher the quality of the generated text.

[0024] Step 4) After scoring each sample in the training set according to step 3), select the top 70% of the training samples and train the model again to enhance the learning of the language model on the initiative and improve the generalization ability.

[0025] Through specific actual experiments, the end-to-end dialogue reply generation task was trained on the active dialogue dataset OpenDialKG. The training set (35564), validation set (4402), and test set (4607) were divided according to the original dataset, with a batch size of 4, 12 epochs, and a learning rate of 1e-3. The results of the test set using Google T5-base, GPT2-small, ChatGPT, and GPT2-aug model with GPT2 as the back bone and using the evaluation index to select the top 70% of the training samples for data augmentation training are as follows:

[0026]

[0027] The test process of ChatGPT is to input the task description prompt, obtain the reply by calling the GPT4 version API, and compare and score with the reference reply.

[0028] The embodiment is further trained on the end-to-end dialogue reply generation task on the proactive dialogue dataset OpenDialKG, uses GPT2 as the back bone, uses the evaluation index to screen the top 70% training samples in the OpenDialKG training set according to the initiative score, performs data augmentation training to obtain the GPT2-aug model, and compares the performance of the GPT2-aug model on the test set. Compared with before data augmentation, the F1, BLEU1 and DISTINCT1 and DISTINCT2 which measure the richness of generated language are all improved.

[0029] For example, in the following example, the three language models all correctly reply to the user, and the reply of GPT2-aug is more fluent and connects the user's question, while the question provides more information. ChatGPT, as a representative of current large language models, generates long and redundant replies and can only be accessed through API calls. It contains a lot of additional knowledge but does not actively interact with the user.

[0030]

[0031]

[0032] Compared with the prior art, GPT2-aug trained by the data augmentation of the application has higher accuracy and language richness than GPT2. ChatGPT, a large language model, tends to generate longer replies, so the language richness is high. However, due to the lack of targeted training, the understanding of the task is still lacking, and the generated reply is not accurate. Under the premise of the same model size, the T5 model is more suitable for generating relatively fixed context after giving the previous context due to its encoder-decoder design, and is not as good as the GPT2 model with only a decoder in generating new content type dialogue tasks. In summary, the method of screening the training set with stronger initiative and higher quality and then training the language model again achieves better results on the test set.

[0033] The above specific embodiments can be adjusted in different ways by those skilled in the art without departing from the principles and purposes of the application. The protection scope of the application is subject to the claims and is not limited by the above specific embodiments. Each implementation within the scope is subject to the constraints of the application.

Claims

1. A system for enhancing initiative in knowledge-driven proactive dialogue, characterized in that, include: The system comprises a dialogue state mapping unit, a dialogue initiative scoring unit, a data augmentation unit, and a generation result verification unit. Specifically: the dialogue state mapping unit performs regular expression matching between each entity and each dialogue history to obtain the topic transition chain; the dialogue initiative scoring unit characterizes the chatbot's initiative in the dialogue based on the knowledge graph and topic transition chain, considering four aspects: information richness, non-idle talk ratio, progression speed, and dialogue fluency; the data augmentation unit performs secondary screening based on the initiative score of each sample in the training set, obtaining high-initiative training samples for model retraining; and the generation result verification unit compares the accuracy of generated responses on the test set before and after data augmentation, as well as the initiative score of the test set dialogue, to measure the model's improvement effect. Information richness refers to the proportion of knowledge entities contained in each sentence of a chatbot, and the proportion of each topic transition chain. , The information richness demonstrated by the robot in this dialogue is: ,in: Let be the number of knowledge entities contained in the robot's response in the j-th round. For the first The number of tokens included in the robot's response. This represents the total number of rounds of dialogue. The aforementioned propagation speed refers to the average speed at which the chatbot advances the dialogue along the path of the knowledge graph. The speed in each round is equal to the distance traveled divided by [the speed of the robot's movement]. arrive The shortest distance; the dialogue The distance the dialogue robot's response travels from the current state to... The shortest distance on the knowledge graph minus the dialogue state after the user finished speaking before replying. The shortest distance, , ;in: and These are the starting point and the goal of the dialogue, respectively. Representing knowledge graphs arrive The shortest distance, Refers to the first The knowledge entity mentioned at the end of the bot's response in a turn-based dialogue; when the bot's response does not contain any entities from a knowledge graph. The value is 0; no knowledge entities were discussed before the dialogue, or the knowledge entity in the dialogue at that point was not mentioned on the graph after the user finished speaking. When the road does not exist, The value is 1.

2. The initiative enhancement system in knowledge-driven proactive dialogue according to claim 1, characterized in that, The non-casual conversation ratio refers to the proportion of responses from the chatbot that contain knowledge entities in the total number of responses. ,in: This refers to the number of responses from knowledge entities included in the chatbot's response count. Same as above.

3. The initiative enhancement system in knowledge-driven proactive dialogue according to claim 1, characterized in that, The fluency of the dialogue refers to the smoothness of each response from the chatbot in the dialogue. The average value for each dialogue; , To gauge whether the robot can smoothly continue the topic from the previous sentence, , ;in: No. The first knowledge entity mentioned in the robot's response during a turn-based dialogue.

4. A method for enhancing initiative in knowledge-driven proactive dialogue based on any one of the systems described in claims 1-3, characterized in that, Includes the following steps: Step 1) Obtain all knowledge nodes of the knowledge graph Connect each entity with each dialogue history Perform regular expression matching, if node ) in the If it appears in the dialogue, then the first one Add the dialogue status to the dialogue. And record the location of its appearance in the dialogue, where: the knowledge graph is A directed graph consisting of triples of [entity, relation, entity]. Dialogue with History Depend on A dialogue consists of interactions between a user and a chatbot, and either party can initiate the dialogue. Step 2) For each dialogue Obtain a list of all knowledge entities discussed, reorder them according to their order of appearance, and obtain... A topic shifting chain in a dialogue composed of knowledge entities , It contains duplicate entities; the topic transition chain of each sentence is mapped as a path on the knowledge graph, reflecting the steps and direction the speaker takes to advance the conversation; the initial state of the conversation topic is empty, and the last knowledge entity in the conversation topic chain is used as the starting point. The current state of the conversation is marked as the latest state. If the conversation does not mention any knowledge entities, the topic transition chain is empty. The conversation is chatter, i.e., a non-initiated conversation, and the current state of the conversation does not change. Step 3) Regarding the initiative of the chatbot to be evaluated, specifically including: 3.1) When there are a total of The reply from the chatbot The sentence contains a topic transition chain corresponding to a knowledge entity that is not empty, and the dialogue has a specified topic starting point. and target In other words, when the bot needs to steer the conversation toward a target knowledge entity, it first uses NLTK's tokenizer tool to segment the conversation, dividing each line of dialogue between the bot and the user into segments. Each token; 3.2) Calculate the information richness, non-idle talk ratio, pacing speed, and conversation fluency for each dialogue. 3.3) Sum the information richness, non-idle talk ratio, progress speed and dialogue fluency calculated in step 3.2 to get the initiative score of the chatbot in this dialogue. Step 4) After scoring each sample in the training set as in Step 3), select the top 70% of the training samples to train the model again, strengthen the language model's learning of initiative, and improve its generalization ability.

Images