A knowledge-driven dialogue method based on a unified generative model

Abstract

The present invention discloses a knowledge-driven dialogue method based on a unified generative model, which belongs to the technical field of natural language processing. The method comprises the following steps: unifying the format of input data fields through a data formatting module to obtain an entity prediction mode, an attribute prediction mode and a dialogue generation mode of the data formatting module; obtaining a mixed training set; training and obtaining a unified generative model based on the mixed training set and a negative log-likelihood loss function; utilizing the data formatting module, the unified generative model and a knowledge base to select external knowledge for historical dialogues to be tested, and predicting and replying to the historical dialogues to be tested; simplifying and merging numerous complex processes in a knowledge-driven dialogue system into three key tasks: entity prediction, attribute prediction and dialogue generation, and uniformly transforming them into a text generation problem; and using a unified generative model for multi-task modeling, the present invention solves the problems of numerous existing knowledge-driven dialogue models, complex implementation, large video memory usage and long inference time.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a knowledge-driven dialogue method based on a unified generation model. Background Art

[0002] Dialogue systems are at the core of human-computer interaction and are widely used in areas such as mobile assistants, chat apps, smart home appliances, and smart cockpits. Generative dialogue systems, owing to their flexible and diverse responses, are becoming a mainstream approach. Traditional generative dialogue systems primarily consist of a dialogue generation module, which takes historical conversations as input and directly generates responses. These responses and newly entered user sentences are then appended to the end of the historical conversations, fed back into the model as new historical conversations to predict responses, and this cycle repeats. However, traditional generative dialogue systems only utilize historical conversation information as input. Due to their lack of external knowledge, they are prone to generating uncontrollable, information-lacking, and distorted responses. Introducing external knowledge into the dialogue generation process can effectively mitigate these issues. Knowledge graphs are a commonly used form of external knowledge, typically in the form of (entity, attribute name, attribute value) triples. Compared to traditional dialogue systems, knowledge-driven dialogue systems require not only dialogue generation capabilities but also the ability to select knowledge from external knowledge bases.

[0003] Existing knowledge-driven dialogue systems typically involve numerous complex subtasks, with the two major subtasks at the first level being knowledge selection and knowledge-driven dialogue generation. The typical process for knowledge selection involves first detecting whether the current dialogue turn requires knowledge. If so, all entities are identified from the previous dialogue, followed by a selection of entity sets that may be used in the response. For each entity to be used, entity linking is used to locate its standard name and all attributes in the knowledge base, and then the specific attribute set to be used is selected. Once the entity set and the attribute set to be used for each entity are determined, the final set of knowledge triples is determined and fed into the dialogue generation model along with the dialogue for response generation. This shows that the knowledge selection subtask can be further broken down into five subtasks: knowledge turn detection, entity recognition, entity selection, entity linking, and attribute selection. Existing technical solutions typically use multiple independent deep models to model key subtasks in knowledge-driven dialogue. For example, classification models are used to model knowledge turn detection, sequence labeling models to model entity recognition, text matching models to model entity selection and attribute selection, and text generation models to model knowledge-driven dialogue generation. These existing solutions ignore the relationship between subtasks, use different modeling methods for different tasks, have numerous models, are complex to implement, occupy large amounts of video memory, and take a long time to infer. Summary of the Invention

[0004] In response to the above-mentioned deficiencies in the prior art, the present invention provides a knowledge-driven dialogue method based on a unified generative model. By simplifying and merging the numerous complex processes in the knowledge-driven dialogue system into three key tasks: entity prediction, attribute prediction, and dialogue generation, the method is uniformly transformed into a text generation problem. A unified generative model is used for multi-task modeling, thereby solving the problems of numerous models, complex implementation, large graphics memory usage, and long inference time in existing knowledge-driven dialogue technologies.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0006] The present invention provides a knowledge-driven dialogue method based on a unified generative model, comprising the following steps:

[0007] S1. Using the data formatting module to unify the input data fields into a unified format, and obtain the entity prediction model, attribute prediction model, and dialogue generation model of the data formatting module;

[0008] S2. Based on the knowledge-driven dialogue dataset Kdconv and the entity prediction model, attribute prediction model, and dialogue generation model of the data formatting module, a hybrid training set consisting of an entity prediction training set, an attribute prediction training set, and a dialogue generation training set is obtained;

[0009] S3, train and obtain a unified generative model based on the mixed training set and negative log-likelihood loss function;

[0010] S4. Use the data formatting module, unified generation model and knowledge base to select external knowledge for the historical dialogue to be tested, and predict responses to the historical dialogue to be tested.

[0011] The beneficial effects of the present invention are:

[0012] (1) The present invention simplifies the original complex process of the knowledge-driven dialogue system, simplifies and merges its many subtasks into three key tasks: entity prediction, attribute prediction, and dialogue generation, and uniformly models them into a flexible text generation task. Only one generation model is used to achieve the three key tasks through different inputs and different control codes during decoding, which simplifies the problem and reduces the implementation costs such as code writing, operation, maintenance, and model space occupation;

[0013] (2) The entity prediction method of the present invention defines the word "none" as an empty entity set, omitting the separate knowledge round detection step. Since it is modeled as a text generation task, the entity set used for end-to-end response generation can be directly obtained without the need for an entity selection step.

[0014] (3) The attribute prediction function of the present invention provides historical conversations and entity attributes, as well as comparison and interaction between attributes. It can quickly predict the attribute set used by the current entity at one time, without the need to predict the matching score of each attribute in isolation and then compare and sort.

[0015] (4) When generating dialogues, the present invention uses sentence patterns and rules to textualize candidate knowledge triples instead of directly splicing them together, which effectively shortens the final text length and makes the semantics more compact.

[0016] Furthermore, the step S1 includes the following steps:

[0017] S11. Based on the historical conversation History and entity set Entity_set, build the entity prediction model of the data formatting module;

[0018] S12. Based on the historical conversation History, a single entity Entity, all attributes All_attr of the entity, and the attribute set Attr_set actually used by the entity, construct an attribute prediction model for the data formatting module;

[0019] S13. Based on the historical conversation History, the knowledge triple list Kg_ls, and the response Response, a dialogue generation model of the data formatting module is constructed.

[0020] The beneficial effect of adopting the above further solution is that the incoming data of the three key tasks of entity prediction, attribute prediction and dialogue generation are uniformly processed into the format of (encoding end text, decoding end text), which meets the data format requirements of the text generation model and can be directly used for model training and prediction.

[0021] Furthermore, the step S11 includes the following steps:

[0022] S111. When the input data field is a historical conversation History and an entity set Entity_set, the input data field is formatted to obtain entity encoding text:

[0023] enc_text en =[conv]History[SEP]

[0024] Among them, enc_text en Indicates entity encoding text, [conv] indicates the conversation field token, and [SEP] indicates the conversation end token. Each conversation sentence in the historical conversation Hisitoty is separated by the first separator character ' / ';

[0025] S112: When the input data field does not include the entity set Entity_set, the entity decoding text dec_text is obtained. en is the empty string ε;

[0026] S113: When the input data field includes the entity set Entity_set, the entity decoding text dec_text is formatted. en :

[0027] dec_text en =[gen_entity]Entity_set[SEP]

[0028] [gen_entity] represents the control code token for generating an entity set, where each entity in the entity set Entity_set is separated by the first separator character ' / ';

[0029] S114. Based on the entity encoding text and the entity decoding text, complete the construction of the entity prediction model:

[0030] (enc_text en ,dec_text en ).

[0031] The beneficial effect of adopting the above further scheme is: providing a specific method for formatting unified entity predictions, and providing a basis for model training and prediction of unified generation models.

[0032] Furthermore, the step S12 includes the following steps:

[0033] S121. When the input data fields are a historical conversation History, a single entity Entity, all attributes All_attr of the entity, and the attribute set Attr_set actually used by the entity, the input data fields are formatted to obtain attribute encoding text:

[0034] enc_text attr =[conv]History[entity]Entity[attr]All_attr[SEP]

[0035] Among them, enc_text attr Indicates attribute-encoded text, [entity] represents a single entity token in the historical conversation, and [attr] represents the attribute field token of the entity. Each conversation sentence in the historical conversation History and each attribute in all attributes All_attr are separated by the first separator character ' / ';

[0036] S122: When the input data field does not include the attribute set Attr_set, the attribute decoding text dec_text is obtained. attr is the empty string ε;

[0037] S123: When the input data field includes the attribute set Attr_set, the attribute decoding text dec_text is formatted. attr :

[0038] dec_text attr =[gen_attr]Attr_set[SEP]

[0039] [gen_attr] represents the control code token for generating an attribute set, wherein each attribute in the attribute set Attr_set is separated by the first separator character ' / ';

[0040] S124. Based on the attribute encoding text and the attribute decoding text, the attribute prediction model is constructed:

[0041] (enc_text attr ,dec_text attr ).

[0042] The beneficial effects of adopting the above further scheme are: providing a specific method for formatting unified attribute prediction, and providing a basis for model training and prediction of a unified generation model.

[0043] Furthermore, step S13 includes the following steps:

[0044] S131. When the input data fields are the historical conversation History, the knowledge triple list Kg_ls, and the response Response, the triple list Kg_ls is converted into the knowledge text Kg_text, and the input data fields are formatted to obtain the conversation encoded text:

[0045] enc_text conv =[conv]History[kg]Kg_text[SEP]

[0046] Among them, enc_text conv Represents the dialogue encoding text, [kg] represents the external knowledge token;

[0047] S132: When the input data field does not include a response, the dialogue decoded text dec_text is obtained. conv is the empty string ε;

[0048] S133: When the input data field includes a response Response, the dialogue decoding text dec_text is formatted. conv :

[0049] dec_text conv =[gen_response]Response[SEP]

[0050] Among them, [gen_response] represents the control code token for generating a response;

[0051] S134. Based on the dialogue encoding text and the dialogue decoding text, a dialogue generation model is constructed:

[0052] (enc_text conv ,dec_text conv ).

[0053] The beneficial effect of adopting the above further solution is: providing a specific method for formatting unified dialogue generation, and providing a basis for model training and prediction of a unified generation model.

[0054] Furthermore, step S2 includes the following steps:

[0055] S21. Get the conversation dataset KdConv conv and external knowledge base KdConv kg The knowledge-driven dialogue dataset Kdconv is constructed, and the initial entity prediction training set, initial attribute prediction training set, and initial dialogue generation training set are all set to empty lists;

[0056] S22. Loop through the conversation dataset KdConv one by one conv The ith sentence in any dialogue sample in , and take the sentence as the ith response i , and take the sentence preceding it in the corresponding dialogue sample as the i-th historical dialogue History i , where i∈[2,conv_len], conv_len represents the total number of sentences in the conversation sample;

[0057] S23. Get the i-th response i The corresponding i-th triple list Kg_ls i For all entities in , construct the corresponding i-th entity set Entity_set i For the i-th triple list Kg_ls i If it is not an empty list, the i-th entity set Entity_set i From the i-th triple list Kg_ls iOtherwise, the i-th entity set Entity_set i Set it to an empty entity set, where the empty entity set only contains the empty character 'none';

[0058] S24, the i-th historical dialogue History i and the i-th entity set Entity_set i As the input data field, and using the entity prediction mode of the data formatting module, the i-th entity prediction result (enc_text eni ,dec_text eni ), and add the i-th entity prediction result to the initial entity prediction training set, where enc_text eni Indicates the i-th entity encoding text, dec_text eni Represents the decoded text of the i-th entity;

[0059] S25, for the i-th entity set Entity_set i When it is a non-empty set, the triple list Kg_ls is selected from the i-th triple list i Get the i-th entity set Entity_set i Any entity in j The actual attribute set Attr_set used j , from the external knowledge base KdConv kg Get the entity Entity j All attributes of All_attr j , where j∈[1,ent_set_len], ent_set_len represents the i-th entity set Entity_set i The total number of entities in ;

[0060] S26, the i-th historical dialogue History i , the i-th entity set Entity_set i Any entity in j Entity j All attributes of All_attr j And the actual attribute set Attr_set j As the input data field, and using the attribute prediction mode of the data formatting module, the j-th attribute prediction result (enc_text attrj ,dec_text attrj ), and add the j-th attribute prediction result to the initial attribute prediction training set, where enc_text attrjIndicates the jth attribute encoding text, dec_text attrj Indicates the j-th attribute decoded text;

[0061] S27, the i-th historical dialogue History i , the i-th triple list Kg_ls i and the i-th response i As the input data field, and using the dialogue generation mode of the data formatting module, the i-th dialogue generation result (enc_text convi ,dec_text convi ), and add the i-th dialogue generation result to the initial dialogue generation training set, where enc_text convi Indicates the coded text of the i-th conversation, dec_text convi Represents the decoded text of the i-th dialogue;

[0062] S28, repeat steps S22 to S27 until the conversation dataset KdConv conv All the dialogue samples in are looped through to obtain the complete entity prediction training set D entity , attribute prediction training set D attr And the dialogue generation training set D conv ;

[0063] S29, the entity prediction training set D entity , attribute prediction training set D attr And the dialogue generation training set D conv Mix to obtain a mixed training set, wherein each training sample in the mixed training set is in the format of (encoding end text, decoding end text).

[0064] The beneficial effect of adopting the above further solution is: using a mixed training set with consistent format to train a unified generative model, the model can simultaneously obtain three capabilities: entity prediction, attribute prediction, and dialogue generation.

[0065] Furthermore, the modeling probability of the unified generation model in step S3 is:

[0066]

[0067] Among them, enc_text represents the encoding end text, | represents the conditional probability calculation separator, dec_text represents the decoding end text, len represents the number of tokens of the decoding end text, dec_text j′ Indicates the j′th token of the decoded text, dec_text <j′ Represents a sequence of the first j' tokens of the decoded text, where j' is a non-zero natural number.

[0068] The beneficial effect of adopting the above further solution is: providing a specific calculation method for modeling probability, providing a basis for training to obtain a unified generative model.

[0069] Furthermore, the calculation expression of the negative log-likelihood loss function in step S3 is as follows:

[0070]

[0071] Among them, loss represents the loss function of the unified generation model, bs represents the number of batch samples for a single training, and dec_text i′ Indicates the i′th decoder text in a batch of samples during a single training, enc_text i′ Represents the i′th encoded end text in a batch of samples during a single training session, where i′ is a non-zero natural number.

[0072] The beneficial effect of adopting the above further scheme is: providing a specific calculation method for the negative log-likelihood loss function, which provides a basis for training a unified generative model.

[0073] Furthermore, step S4 includes the following steps:

[0074] S41. Using the historical conversation to be tested as an input data field, and using the entity prediction mode of the data formatting module, obtain the entity encoding end text corresponding to the historical conversation to be tested;

[0075] S42. Input the entity encoding end text into the unified generation model, and use [gen_entity] as the control code token of the initial input, and perform autoregressive decoding to obtain the predicted entity set for the reply;

[0076] S43. When the predicted entity set is an empty set, the reply does not require the use of knowledge, and the candidate knowledge triple list is also empty;

[0077] S44. When the predicted entity set is a non-empty set, an entity linking operation is performed on all entities in the predicted entity set, and the position of each entity in the knowledge base is located and replaced with a standard entity name;

[0078] S45. Obtain all attributes of each entity after replacing the standard entity name from the knowledge base, use the historical conversation to be tested, each entity, and all attributes corresponding to each entity as input data fields, and use the attribute prediction mode of the data formatting module to obtain the attribute encoding end text corresponding to each predicted entity in the historical conversation to be tested;

[0079] S46. Input the attribute encoding end text into the unified generation model, and use [gen_attr] as the control code token of the initial input, and perform autoregressive decoding to obtain the predicted attribute set corresponding to each entity;

[0080] S47. Based on the predicted entity set and the predicted attribute set of each entity, all triples are extracted from the knowledge base using the (entity name, attribute name) tuple as a keyword to form a list of candidate knowledge triples. The list of candidate knowledge triples is used as external knowledge for predicting the response.

[0081] S48. Using the historical conversation to be tested and external knowledge as input data fields, and using the conversation generation mode of the data formatting module, obtain the conversation encoding end text corresponding to the historical conversation to be tested;

[0082] S49. Input the dialogue encoding end text corresponding to the historical dialogue to be tested into the unified generation model, and use [gen_response] as the control code token of the initial input. Autoregressive decoding is performed to obtain the predicted response content of the historical dialogue to be tested, thereby completing the predicted response to the historical dialogue to be tested.

[0083] The beneficial effect of adopting the above further solution is that the unified generative model can fully understand the semantics of the historical conversation and pay attention to all entities appearing in it when predicting the entity set, and efficiently predict the entity set to be used at one time without the need for entity selection operation.

[0084] When predicting attribute sets, the unified generative model can perceive all attributes of an entity at once, fully understand the semantics of historical conversations, and efficiently predict the attribute set that the entity will use at one time.

[0085] The unified generative model can fully understand the semantics of historical conversations and incorporate external knowledge when predicting replies, making replies more controllable, more informative, and more logically sound.

[0086] Furthermore, the method for replacing the standard entity name in step S44 includes the following steps:

[0087] B1. When an entity in the predicted entity set exists in the knowledge base, the entity name is the standard entity name and is not replaced;

[0088] B2. When an entity in the predicted entity set does not exist in the knowledge base, the 1-gram matching score between the entity and each entity in the knowledge base is calculated, and the entity in the knowledge base with the highest 1-gram matching score is used to replace the corresponding entity in the predicted entity set, completing the replacement of the standard entity name;

[0089] The calculation expression of the 1-gram matching score is as follows:

[0090]

[0091] Among them, score(A,B) represents the 1-gram matching score, A represents the first entity that exists in the predicted entity set but does not exist in the knowledge base, B represents the second entity in the knowledge base, |A∩B| represents the number of overlaps between entities at the token level, |A| represents the number of tokens in the first entity, and |B| represents the number of tokens in the second entity.

[0092] The beneficial effect of adopting the above further solution is: it enhances the robustness of the system. Even if the entities appearing in the conversation and the entities predicted by the model are not standard expressions, the system is able to locate the standard entities in its corresponding knowledge base. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 This is a flowchart of the steps of a knowledge-driven dialogue method based on a unified generation model in an embodiment of the present invention.

[0094] Figure 2 Schematic diagram of a unified generation model in an embodiment of the present invention.

[0095] Figure 3 Schematic diagram of entity prediction in an embodiment of the present invention.

[0096] Figure 4 Schematic diagram of attribute prediction in an embodiment of the present invention.

[0097] Figure 5 This is a schematic diagram of dialogue generation in an embodiment of the present invention. DETAILED DESCRIPTION

[0098] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0099] like Figure 1 As shown, in one embodiment of the present invention, the present invention provides a knowledge-driven dialogue method based on a unified generative model, comprising the following steps:

[0100] S1. Using the data formatting module to unify the input data fields into a unified format, and obtain the entity prediction model, attribute prediction model, and dialogue generation model of the data formatting module;

[0101] The step S1 includes the following steps:

[0102] S11. Based on the historical conversation History and entity set Entity_set, build the entity prediction model of the data formatting module;

[0103] The step S11 includes the following steps:

[0104] S111. When the input data field is a historical conversation History and an entity set Entity_set, the input data field is formatted to obtain entity encoding text:

[0105] enc_text en =[conv]History[SEP]

[0106] Among them, enc_text en Indicates entity encoding text, [conv] indicates the conversation field token, and [SEP] indicates the conversation end token. Each conversation sentence in the historical conversation Hisitoty is separated by the first separator character ' / ';

[0107] S112: When the input data field does not include the entity set Entity_set, the entity decoding text dec_text is obtained. en is the empty string ε;

[0108] S113: When the input data field includes the entity set Entity_set, the entity decoding text dec_text is formatted. en :

[0109] dec_text en =[gen_entity]Entity_set[SEP]

[0110] [gen_entity] represents the control code token for generating an entity set, where each entity in the entity set Entity_set is separated by the first separator character ' / ';

[0111] S114. Based on the entity encoding text and the entity decoding text, complete the construction of the entity prediction model:

[0112] (enc_text en ,dec_text en );

[0113] S12. Based on the historical conversation History, a single entity Entity, all attributes All_attr of the entity, and the attribute set Attr_set actually used by the entity, construct an attribute prediction model for the data formatting module;

[0114] The step S12 includes the following steps:

[0115] S121. When the input data fields are a historical conversation History, a single entity Entity, all attributes All_attr of the entity, and the attribute set Attr_set actually used by the entity, the input data fields are formatted to obtain attribute encoding text:

[0116] enc_text attr =[conv]History[entity]Entity[attr]All_attr[SEP]

[0117] Among them, enc_text attr Indicates attribute-encoded text, [entity] represents a single entity token in the historical conversation, and [attr] represents the attribute field token of the entity. Each conversation sentence in the historical conversation History and each attribute in all attributes All_attr are separated by the first separator character ' / ';

[0118] S122: When the input data field does not include the attribute set Attr_set, the attribute decoding text dec_text is obtained. attr is the empty string ε;

[0119] S123: When the input data field includes the attribute set Attr_set, the attribute decoding text dec_text is formatted. attr :

[0120] dec_text attr =[gen_attr]Attr_set[SEP]

[0121] [gen_attr] represents the control code token for generating an attribute set, wherein each attribute in the attribute set Attr_set is separated by the first separator character ' / ';

[0122] S124. Based on the attribute encoding text and the attribute decoding text, the attribute prediction model is constructed:

[0123] (enc_text attr ,dec_text attr );

[0124] S13. Based on the historical conversation History, the knowledge triple list Kg_ls, and the response Response, a dialogue generation model of the data formatting module is constructed;

[0125] The step S13 includes the following steps:

[0126] S131. When the input data fields are the historical conversation History, the knowledge triple list Kg_ls, and the response Response, the triple list Kg_ls is converted into the knowledge text Kg_text, and the input data fields are formatted to obtain the conversation encoded text:

[0127] enc_text conv =[conv]History[kg]Kg_text[SEP]

[0128] Among them, enc_text conv Represents the dialogue encoding text, [kg] represents the external knowledge token;

[0129] The method for converting the triple list Kg_ls into the knowledge text Kg_text in step S131 includes the following steps:

[0130] A1. When the triple list Kg_ls is empty, the triple list Kg_ls is converted into an empty string ε;

[0131] A2. When the triple list Kg_ls is not empty, the triple list Kg_ls is converted into the knowledge text Kg_text using the sentence "{entity}'s {attribute name} is {attribute value}", where {} indicates replacement with the corresponding field in the triple list;

[0132] In order to reduce the length of the formatted text and make the semantics more compact, the triples with the same "entity-attribute name" are aggregated, and multiple attribute values are separated by the second separator character ','; then for the triples with the same entity, the texts constructed with different attributes are separated by the third separator character ',', and finally the texts constructed with different entities are separated by the fourth separator character '.'; for example, for the triple list [(Andy Lau, major achievements, top ten outstanding young people in the world), (Andy Lau, representative works, Infernal Affairs), (Andy Lau, representative works, A Simple Life), (A Simple Life, director, Ann Hui)], after textualization, it becomes "Andy Lau's major achievements are top ten outstanding young people in the world, and his representative works are Infernal Affairs and A Simple Life. A Simple Life is directed by Ann Hui";

[0133] S132: When the input data field does not include a response, the dialogue decoded text dec_text is obtained. conv is the empty string ε;

[0134] S133: When the input data field includes a response Response, the dialogue decoding text dec_text is formatted.conv :

[0135] dec_text conv =[gen_response]Response[SEP]

[0136] Among them, [gen_response] represents the control code token for generating a response;

[0137] S134. Based on the dialogue encoding text and the dialogue decoding text, a dialogue generation model is constructed:

[0138] (enc_text conv ,dec_text conv );

[0139] S2. Based on the knowledge-driven dialogue dataset Kdconv and the entity prediction model, attribute prediction model, and dialogue generation model of the data formatting module, a hybrid training set consisting of an entity prediction training set, an attribute prediction training set, and a dialogue generation training set is obtained;

[0140] The step S2 comprises the following steps:

[0141] S21. Obtain the open source knowledge-driven dialogue dataset KdConv conv and external knowledge base KdConv kg The knowledge-driven dialogue dataset Kdconv is constructed, and the initial entity prediction training set, initial attribute prediction training set, and initial dialogue generation training set are all set to empty lists;

[0142] KdConv conv Each sample in the KdConv consists of a complete historical dialogue and a list of knowledge triples on each dialogue sentence. kg It is in the form of a key-value dictionary, where the key is the entity and the value is all the knowledge triples under the entity. The format of the knowledge triple is (entity, attribute name, attribute value);

[0143] S22. Loop through the conversation dataset KdConv one by one conv The ith sentence in any dialogue sample in , and take the sentence as the ith response i , and take the sentence preceding it in the corresponding dialogue sample as the i-th historical dialogue History i , where i∈[2,conv_len], conv_len represents the total number of sentences in the conversation sample;

[0144] S23. Get the i-th response iThe corresponding i-th triple list Kg_ls i For all entities in , construct the corresponding i-th entity set Entity_set i For the i-th triple list Kg_ls i If it is not an empty list, the i-th entity set Entity_set i From the i-th triple list Kg_ls i Otherwise, the i-th entity set Entity_set i Set it to an empty entity set, where the empty entity set only contains the empty character 'none';

[0145] S24, the i-th historical dialogue History i and the i-th entity set Entity_set i As the input data field, and using the entity prediction mode of the data formatting module, the i-th entity prediction result (enc_text eni ,dec_text eni ), and add the i-th entity prediction result to the initial entity prediction training set, where enc_text eni Indicates the i-th entity encoding text, dec_text eni Represents the decoded text of the i-th entity;

[0146] S25, for the i-th entity set Entity_set i When it is a non-empty set, the triple list Kg_ls is selected from the i-th triple list i Get the i-th entity set Entity_set i Any entity in j The actual attribute set Attr_set used j , from the external knowledge base KdConv kg Get the entity Entity j All attributes of All_attr j , where j∈[1,ent_set_len], ent_set_len represents the i-th entity set Entity_set i The total number of entities in ;

[0147] S26, the i-th historical dialogue History i , the i-th entity set Entity_set i Any entity in j Entity j All attributes of All_attr jAnd the actual attribute set Attr_set j As the input data field, and using the attribute prediction mode of the data formatting module, the j-th attribute prediction result (enc_text attrj ,dec_text attrj ), and add the j-th attribute prediction result to the initial attribute prediction training set, where enc_text attrj Indicates the jth attribute encoding text, dec_text attrj Indicates the j-th attribute decoded text;

[0148] S27, the i-th historical dialogue History i , the i-th triple list Kg_ls i and the i-th response i As the input data field, and using the dialogue generation mode of the data formatting module, the i-th dialogue generation result (enc_text convi ,dec_text convi ), and add the i-th dialogue generation result to the initial dialogue generation training set, where enc_text convi Indicates the coded text of the i-th conversation, dec_text convi Represents the decoded text of the i-th dialogue;

[0149] S28, repeat steps S22 to S27 until the conversation dataset KdConv conv All the dialogue samples in are looped through to obtain the complete entity prediction training set D entity , attribute prediction training set D attr And the dialogue generation training set D conv ;

[0150] S29, the entity prediction training set D entity , attribute prediction training set D attr And the dialogue generation training set D conv Mixing to obtain a mixed training set, wherein each training sample in the mixed training set is in the format of (encoding end text, decoding end text);

[0151] like Figure 2 As shown, S3 is trained based on the mixed training set and negative log-likelihood loss function to obtain a unified generative model;

[0152] The backbone of the unified generative model can be any deep text generation model. In this embodiment, the T5 model based on the Transformers architecture is selected. A unified generative model is used to perform multi-task modeling for entity prediction, attribute prediction, and dialogue generation. Different functions are achieved through different inputs and different control codes during decoding. After training is completed, the obtained unified generative model can simultaneously obtain the three capabilities of entity prediction, attribute prediction, and dialogue generation.

[0153] The modeling probability of the unified generation model in step S3 is:

[0154]

[0155] Among them, enc_text represents the encoding end text, | represents the conditional probability calculation separator, dec_text represents the decoding end text, len represents the number of tokens of the decoding end text, dec_text j′ Indicates the j′th token of the decoded text, dec_text <j′ Represents a sequence of the first j' tokens of the decoded text, where j' is a non-zero natural number;

[0156] The calculation expression of the negative log-likelihood loss function in step S3 is as follows:

[0157]

[0158] Among them, loss represents the loss function of the unified generation model, bs represents the number of batch samples for a single training, and dec_text i′ Indicates the i′th decoder text in a batch of samples during a single training, enc_text i′ represents the i′th encoding end text in a batch of samples during a single training session, where i′ is a non-zero natural number;

[0159] S4. Use the data formatting module, unified generation model and knowledge base to select external knowledge for the historical dialogue to be tested, and predict responses to the historical dialogue to be tested.

[0160] The step S4 comprises the following steps:

[0161] like Figure 3 As shown, S41, the historical conversation to be tested is used as the input data field, and the entity prediction mode of the data formatting module is used to obtain the entity encoding end text corresponding to the historical conversation to be tested;

[0162] S42. Input the entity encoding end text into the unified generation model, and use [gen_entity] as the control code token of the initial input, and perform autoregressive decoding to obtain the predicted entity set for the reply;

[0163] S43. When the predicted entity set is an empty set, the reply does not require the use of knowledge, and the candidate knowledge triple list is also empty;

[0164] S44. When the predicted entity set is a non-empty set, an entity linking operation is performed on all entities in the predicted entity set, and the position of each entity in the knowledge base is located and replaced with a standard entity name;

[0165] If the predicted entity set is not an empty set (not "none"), since the entities that appear in the historical conversations may not be standard expressions, each entity in the predicted entity set may not be a standard name. Therefore, an entity linking operation needs to be performed on each entity to locate its position in the knowledge base and replace it with a standard entity name.

[0166] The method for replacing the standard entity name in step S44 includes the following steps:

[0167] B1. When an entity in the predicted entity set exists in the knowledge base, the entity name is the standard entity name and is not replaced;

[0168] B2. When an entity in the predicted entity set does not exist in the knowledge base, the 1-gram matching score between the entity and each entity in the knowledge base is calculated, and the entity in the knowledge base with the highest 1-gram matching score is used to replace the corresponding entity in the predicted entity set, completing the replacement of the standard entity name;

[0169] The calculation expression of the 1-gram matching score is as follows:

[0170]

[0171] Where score(A,B) represents the 1-gram matching score, A represents the first entity that exists in the predicted entity set but does not exist in the knowledge base, B represents the second entity in the knowledge base, |A∩B| represents the number of overlaps between entities at the token level, |A| represents the number of tokens in the first entity, and |B| represents the number of tokens in the second entity.

[0172] like Figure 4As shown, S45, all attributes of each entity after replacing the standard entity name are obtained from the knowledge base, the historical conversation to be tested, each entity, and all attributes corresponding to each entity are used as input data fields, and the attribute prediction mode of the data formatting module is used to obtain the attribute encoding end text corresponding to each predicted entity in the historical conversation to be tested;

[0173] S46. Input the attribute encoding end text into the unified generation model, and use [gen_attr] as the control code token of the initial input, and perform autoregressive decoding to obtain the predicted attribute set corresponding to each entity;

[0174] S47. Based on the predicted entity set and the predicted attribute set of each entity, all triples are extracted from the knowledge base using the (entity name, attribute name) tuple as a keyword to form a list of candidate knowledge triples. The list of candidate knowledge triples is used as external knowledge for predicting the response.

[0175] like Figure 5 As shown, S48, the historical dialogue to be tested and external knowledge are used as input data fields, and the dialogue generation mode of the data formatting module is used to obtain the dialogue encoding end text corresponding to the historical dialogue to be tested;

[0176] S49. Input the dialogue encoding end text corresponding to the historical dialogue to be tested into the unified generation model, and use [gen_response] as the control code token of the initial input. Autoregressive decoding is performed to obtain the predicted response content of the historical dialogue to be tested, thereby completing the predicted response to the historical dialogue to be tested.

[0177] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to specific details.

Claims

1. A knowledge-driven dialogue method based on a unified generative model, characterized in that: The steps include: S1. Using the data formatting module to unify the input data fields into a unified format, and obtain the entity prediction model, attribute prediction model, and dialogue generation model of the data formatting module; S2, based on knowledge-driven dialogue dataset Kdconv As well as the entity prediction mode, attribute prediction mode and dialogue generation mode of the data formatting module, a mixed training set consisting of entity prediction training set, attribute prediction training set and dialogue generation training set is obtained, specifically: S21. Get the dialogue dataset KdConv conv and external knowledge bases KdConv kg Knowledge-driven dialogue dataset Kdconv , and set the initial entity prediction training set, initial attribute prediction training set, and initial dialogue generation training set to empty lists; S22. Loop through the conversation dataset one by one KdConv conv In any of the dialogue samples i sentence and use it as the i Replies , and take the sentence preceding it in the corresponding dialogue sample as the first i A historical dialogue ,in, , Indicates the total number of sentences in the conversation sample; S23, get the i Replies The corresponding i List of triples Kg_ls i For all entities in , construct the corresponding i Entity Set Entity_set i , for the i List of triples Kg_ls i If it is not an empty list, i Entity Set Entity_set i By i List of triples Kg_ls i Otherwise, the i Entity Set Entity_ set i Set it to an empty entity set, where the empty entity set only contains the empty character 'none'; S24, the i A historical dialogue Hedi i Entity Set Entity_set i As the input data field, and using the entity prediction mode of the data formatting module, we get the i Entity prediction results , and the first i The entity prediction results are added to the initial entity prediction training set, where Indicates the i Entity encoding text, Indicates the i Entity decoded text; S25, for the i Entity Set Entity_set i If it is a non-empty set, then i List of triples Kg_ls i Get the first i Entity Set Entity_set i Any entity The actual set of properties used , from external knowledge base KdConv kg Get the entity All properties of All_attr j ,in, , Indicates the i Entity Set Entity_set i The total number of entities in ; S26, the i A historical dialogue , No. i Entity Set Entity_set i Any entity , the entity All properties of All_attr j and the actual set of properties used As the input data field, and using the attribute prediction mode of the data formatting module, we get the j Attribute prediction results , and the first j The attribute prediction results are added to the initial attribute prediction training set, where Indicates the j attribute encoding text, Indicates the j Attribute decoded text; S27, the i A historical dialogue , No. i List of triples Kg_ls i Hedi i Replies As the input data field, and using the dialogue generation mode of the data formatting module, the first i Dialogue generation results , and the first i The dialogue generation results are added to the initial dialogue generation training set, where Indicates the i Conversational coded text, Indicates the i A dialogue decoded text; S28, repeat steps S22 to S27 until the dialogue data set KdConv conv All the dialogue samples in are looped through to obtain a complete entity prediction training set , attribute prediction training set and dialogue generation training set ; S29, entity prediction training set , attribute prediction training set and dialogue generation training set Mixing to obtain a mixed training set, where each training sample in the mixed training set is in the format of (encoding end text, decoding end text); S3, train and obtain a unified generative model based on the mixed training set and negative log-likelihood loss function; S4. Use the data formatting module, unified generation model and knowledge base to select external knowledge for the historical dialogue to be tested, and predict responses to the historical dialogue to be tested.

2. The knowledge-driven dialogue method based on a unified generative model according to claim 1 is characterized in that: The step S1 includes the following steps: S11. Based on historical dialogue History and entity sets Entity_set , construct the entity prediction model of the data formatting module; S12. Historical dialogue History , single entity Entity , all attributes of the entity All_attr and the set of attributes actually used by the entity , construct the attribute prediction model of the data formatting module; S13, based on historical dialogue History , knowledge triple list Kg_ls and reply Response , build the dialogue generation mode of the data formatting module.

3. The knowledge-driven dialogue method based on a unified generative model according to claim 2, characterized in that: The step S11 includes the following steps: S111, for the input data field is the history dialogue History and entity sets Entity_set , the input data field is formatted to obtain entity-encoded text: in, represents entity-encoded text, Indicates a dialog field tag token , Indicates the end of the conversation token , among which, historical dialogue Hisitoty Each internal dialogue sentence is separated by the first separator character ' / '; S112, for input data fields that do not include entity sets Entity_set , then the entity decoded text is obtained Empty string ; S113, for the input data field including the entity set Entity_set Format entity decoded text : in, Control code tag indicating the generation of entity set token , where each entity in the entity set Entity_set is separated by the first separator character ' / '; S114. Based on the entity encoding text and the entity decoding text, complete the construction of the entity prediction model: 。 4. The knowledge-driven dialogue method based on a unified generative model according to claim 3 is characterized in that: The step S12 includes the following steps: S121, for the input data field is the history dialogue History , single entity Entity , all attributes of the entity All_attr and the set of attributes actually used by the entity , the input data field is formatted to obtain attribute encoding text: in, Represents attribute-encoded text, Represents a single entity token in the historical conversation token , Indicates the attribute field tag of the entity token , among which, historical dialogue History Every dialogue sentence and all attributes inside All_attr Each attribute in the first separator character is ' / 'To separate; S122: Input data fields do not include attribute sets When the attribute decoding text is obtained Empty string ; S123, including attribute set for input data field When the formatting attribute decodes the text : in, Control code tag indicating the generation of attribute sets token , where the attribute set Each attribute inside is separated by the first character ' / 'To separate; S124. Based on the attribute encoding text and the attribute decoding text, the attribute prediction model is constructed: 。 5. The knowledge-driven dialogue method based on a unified generative model according to claim 4 is characterized in that: The step S13 includes the following steps: S131, for the input data field is the history dialogue History , knowledge triple list Kg_ls and reply Response , then the triple list Kg_ls Converted into knowledge text Kg_text , and format the input data field to obtain the conversation encoding text: in, represents the conversational encoded text, External knowledge token token ; S132, for input data fields that do not include a response Response When , the dialogue decoded text is obtained Empty string ; S133, including responses for input data fields Response Format the dialogue decoded text : in, Control code flag indicating the generation of a reply token ; S134. Based on the dialogue encoding text and the dialogue decoding text, a dialogue generation model is constructed: 。 6. The knowledge-driven dialogue method based on a unified generative model according to claim 5 is characterized in that: The modeling probability of the unified generation model in step S3 is: in, Represents the encoding side text, Indicates the conditional probability calculation separator, Represents the decoded text. len Markup representing decoded text token Number, Indicates the decoded text indivual token , Indicates the front of the decoded text indivual token A sequence consisting of A non-zero natural number.

7. The knowledge-driven dialogue method based on a unified generative model according to claim 6, characterized in that: The calculation expression of the negative log-likelihood loss function in step S3 is as follows: in, represents the loss function of the unified generative model, bs Indicates the number of batch samples for a single training session, Indicates the first sample in a batch during a single training Decoder text, Indicates the first sample in a batch during a single training Coded end text, where is a non-zero natural number.

8. The knowledge-driven dialogue method based on a unified generative model according to claim 7, characterized in that: The step S4 comprises the following steps: S41. Using the historical conversation to be tested as an input data field, and using the entity prediction mode of the data formatting module, obtain the entity encoding end text corresponding to the historical conversation to be tested; S42, input the entity encoding end text into the unified generation model, and [ gen_entity ] as the control code mark for initial input token , autoregressive decoding obtains the predicted entity set for reply; S43. When the predicted entity set is an empty set, the reply does not require the use of knowledge, and the candidate knowledge triple list is also empty; S44. When the predicted entity set is a non-empty set, an entity linking operation is performed on all entities in the predicted entity set, and the position of each entity in the knowledge base is located and replaced with a standard entity name; S45. Obtain all attributes of each entity after replacing the standard entity name from the knowledge base, use the historical conversation to be tested, each entity, and all attributes corresponding to each entity as input data fields, and use the attribute prediction mode of the data formatting module to obtain the attribute encoding end text corresponding to each predicted entity in the historical conversation to be tested; S46, input the attribute encoding end text into the unified generation model, and [ gen_attr ] as the control code mark for initial input token , autoregressive decoding obtains the predicted attribute set corresponding to each entity; S47. Based on the predicted entity set and the predicted attribute set of each entity, all triples are extracted from the knowledge base using the (entity name, attribute name) tuple as a keyword to form a list of candidate knowledge triples. The list of candidate knowledge triples is used as external knowledge for predicting the response. S48. Using the historical conversation to be tested and external knowledge as input data fields, and using the conversation generation mode of the data formatting module, obtain the conversation encoding end text corresponding to the historical conversation to be tested; S49, input the dialogue encoding end text corresponding to the historical dialogue to be tested into the unified generation model, and [ gen_ response ] as the control code mark for initial input token , autoregressive decoding obtains the predicted response content of the historical dialogue to be tested, and completes the predicted response to the historical dialogue to be tested.

9. The knowledge-driven dialogue method based on a unified generative model according to claim 8, characterized in that: The method for replacing the standard entity name in step S44 includes the following steps: B1. When an entity in the predicted entity set exists in the knowledge base, the entity name is the standard entity name and is not replaced; B2. When an entity in the predicted entity set does not exist in the knowledge base, the 1-gram matching score between the entity and each entity in the knowledge base is calculated, and the entity in the knowledge base with the highest 1-gram matching score is used to replace the corresponding entity in the predicted entity set, completing the replacement of the standard entity name; The calculation expression of the 1-gram matching score is as follows: in, represents the 1-gram matching score, A Indicates the first entity that exists in the predicted entity set and does not exist in the knowledge base, B represents the second entity in the knowledge base, Indicates that entities are marked token The number of overlapping layers, Represents the token in the first entity token Number, Represents the token in the second entity token Number.

Images