Open-domain dialogue generation method based on dynamic multi-modal knowledge fusion

Abstract

The present invention provides a method for generating open-domain dialogues based on dynamic multi-modal knowledge fusion. The method comprises a dialogue module and a dynamic multi-modal knowledge fusion module. The dialogue module comprises a first Transformer encoder module, a second Transformer encoder module, a third Transformer encoder module, a Transformer decoder module, and a controller module. The dynamic multi-modal knowledge fusion module comprises a virtual knowledge reasoning module, a virtual knowledge dynamic memory module, a virtual knowledge dynamic selection module, and a common sense knowledge expansion module. In summary, the method efficiently searches for dialogue-related information and generates rich dialogue content.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an open-domain dialogue generation method based on dynamic multi-form knowledge fusion. Background Art

[0002] Open-domain dialogue generation, or casual conversation, is a key research direction in natural language processing. Achieving open-domain dialogue generation is also a key sign of AI maturity. Open-domain dialogue aims to ensure that machine-generated responses maintain continuity and meet human needs for conversational understanding and emotional resonance. Open-domain dialogue generation technology holds great potential for application. For example, it can be applied to systems like intelligent voice assistants, enabling more intelligent and natural human-computer interaction. Open-domain dialogue generation systems can also serve as companions for casual conversation. The technology can even be used to assist with topic exploration and stimulate user creativity.

[0003] In open-domain multi-turn dialogues, dialogue systems often face the challenge of enriching the content of the conversation. Traditional approaches to enriching dialogue content typically rely on a single form of knowledge, such as structured text or unstructured knowledge graphs, to generate rich semantics and information. Traditional dialogue generation methods based on information-rich structured text suffer from complex screening processes and difficulty finding appropriate knowledge to incorporate into the conversation. Traditional dialogue generation methods based on easily reasoned unstructured knowledge graphs also suffer from incomplete knowledge graphs and information loss during the knowledge graph construction process. Summary of the Invention

[0004] The present invention is made to solve the above problems, and its purpose is to provide an open domain dialogue generation method based on dynamic multi-form knowledge fusion.

[0005] The present invention provides an open domain dialogue generation method based on dynamic multi-form knowledge fusion, which has the following characteristics: a dialogue module and a dynamic multi-form knowledge fusion module, wherein the dialogue module includes a first Transformer encoder module, a second Transformer encoder module, a third Transformer encoder module, a Transformer decoder module and a controller module, and the dynamic multi-form knowledge fusion module includes a virtual knowledge reasoning module, a virtual knowledge dynamic memory module, a virtual knowledge dynamic selection module and a common sense knowledge expansion module, and the virtual knowledge reasoning module converts the user input X into n Input virtual knowledge reasoning model to obtain candidate document set According to the candidate document set The filtering scores of each candidate document in are arranged from high to low, and the top T candidate documents are selected as the candidate document set The first Transformer encoder module sets the candidate document set Encode to obtain the semantic feature matrix The second Transformer encoder module, for user input X n Encode semantic information The virtual knowledge dynamic memory module collects all historical semantic feature matrices obtained before the current conversation Dynamically stored as matrix H, according to semantic information Filter and screen the matrix H based on the attention mechanism to obtain historical virtual knowledge The third Transformer encoder module is used to analyze the historical dialogue C n and user input X n Encode semantic information The virtual knowledge dynamic selection module is based on the multi-head attention mechanism to analyze semantic information. Historical virtual knowledge and semantic feature matrix Integrate to obtain a vector that integrates semantic features and virtual knowledge features The common sense knowledge expansion module traverses the common sense knowledge graph ConceptNet to expand the historical dialogue C n 、User input X n and candidate document collection The information in the concept expansion, encoding and feature extraction are used to obtain extended knowledge Transformer decoder module, based on the generated word at the previous moment Pair Vector Decode and get the hidden layer state after decoding The controller module decodes the hidden layer state The probability distribution A of the generated words in the dialogue standard vocabulary is calculated by the Softmax function, and the knowledge is expanded Get the probability distribution B of the generated word in the extended knowledge entity set, and get the generated word at the current moment based on the probability distribution A and probability distribution B based on the gating mechanism By calling the Transformer decoder module and controller module multiple times, the generated word set at all times in the current dialogue round is obtained Output generated word set Y as a response to user input X n 's reply.

[0006] The open domain dialogue generation method based on dynamic multi-form knowledge fusion provided by the present invention may also have the following characteristics: wherein, the virtual knowledge reasoning model is obtained by training a model based on the virtual knowledge base reasoning algorithm DrKIT using a dialogue dataset and a virtual knowledge base.

[0007] The open-domain dialogue generation method based on dynamic multi-form knowledge fusion provided by the present invention may also have the following feature: wherein the dialogue datasets are open source dialogue datasets DailyDialog and PersonaChat.

[0008] The open-domain dialogue generation method based on dynamic multi-form knowledge fusion provided by the present invention may also have the following feature: wherein the virtual knowledge base is obtained by indexing the common sense story dataset ROCStories.

[0009] The open domain dialogue generation method based on dynamic multi-form knowledge fusion provided by the present invention may also have the following features: wherein the filtering score is the number of times the candidate document co-occurs with words in the word set, and the word set is input by the user X n and user input X n It is composed of related words obtained based on the common sense knowledge graph ConceptNet, and there are no repeated words in the word set.

[0010] The open-domain dialogue generation method based on dynamic multi-form knowledge fusion provided by the present invention may also have the following feature: wherein the value of T is 5.

[0011] The open domain dialogue generation method based on dynamic multi-form knowledge fusion provided by the present invention may also have the following features: wherein the concept expansion, encoding and feature extraction method of the common sense knowledge expansion module is: from the historical dialogue C n 、User input X n and candidate document collection Extract word sets as original entities By traversing the common sense knowledge graph ConceptNet, we get the original entity The associated word set as an extended entity Obtaining original entities based on the common sense knowledge graph ConceptNet and extended entities The relationship set of each corresponding word in is used as the relationship Based on the neural network CompGCN, the original entities Extended Entity and relationships Encode and obtain the encoding vectors Encoded vector and encoding vector According to the encoding vector Encoded vector and encoding vector Extended knowledge through attention mechanism

[0012] Functions and effects of the invention

[0013] According to the open-domain dialogue generation method based on dynamic multi-form knowledge fusion involved in the present invention, because it introduces the structured common sense knowledge graph ConceptNet, it can generate richer content and construct an unstructured virtual knowledge base, which can more efficiently search for text information related to the current dialogue topic. Therefore, the open-domain dialogue generation method based on dynamic multi-form knowledge fusion of the present invention can efficiently search for dialogue-related information and form rich dialogue content. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 1 is a flow chart of an open domain dialogue generation method based on dynamic multi-form knowledge fusion in an embodiment of the present invention;

[0015] Figure 2 3 is a schematic diagram of the effect of the open domain dialogue generation method based on dynamic multi-form knowledge fusion in an embodiment of the present invention. DETAILED DESCRIPTION

[0016] In order to make the technical means, creative features, objectives and effects of the present invention easier to understand, the following embodiments and accompanying drawings specifically illustrate the open-domain dialogue generation method based on dynamic multi-form knowledge fusion of the present invention.

[0017] Figure 1 4 is a flow chart of an open domain dialogue generation method based on dynamic multi-form knowledge fusion in an embodiment of the present invention.

[0018] like Figure 1 As shown in Figure 2, the open-domain dialogue generation method based on dynamic multi-modal knowledge fusion has the following process:

[0019] Virtual knowledge reasoning module, the user input X n That is, the user input content of the current dialogue round is input into the virtual knowledge reasoning model, namely DrKIT, to obtain the candidate document set According to the candidate document set The filtering scores of each candidate document in are arranged from high to low, and the top T candidate documents are selected as the candidate document set In the formula is the Tth candidate document.

[0020] Among them, the virtual knowledge reasoning model is obtained by training a model based on the virtual knowledge base reasoning algorithm DrKIT using a dialogue dataset and a virtual knowledge base. The dialogue dataset is the open source dialogue dataset DailyDialog and PersonaChat, and the virtual knowledge base is obtained by indexing the common sense story dataset ROCStories.

[0021] The filter score is the number of times the candidate document co-occurs with a word in the word set, where the word set is input by the user X n and user input X n It is composed of related words obtained based on the common sense knowledge graph ConceptNet, and there are no repeated words in the word set.

[0022] In this embodiment, the value of T is 5.

[0023] The first Transformer encoder module sets the candidate document set Encode each candidate document in to obtain the encoding set of each candidate document In the formula The output after encoding the T-th candidate document is then merged through the Concat operation to obtain the semantic feature matrix

[0024] The second Transformer encoder module, for user input X n Encode semantic information

[0025] The virtual knowledge dynamic memory module collects all historical semantic feature matrices obtained before the current conversation Dynamic storage is Update as matrix H, according to semantic information Filter and screen the matrix H based on the attention mechanism to obtain historical virtual knowledge As a supplement to knowledge.

[0026] The third Transformer encoder module is used to analyze the historical dialogue C n That is, all the dialogue content before the current dialogue round and the user input X n Encode semantic information

[0027] The virtual knowledge dynamic selection module is based on the multi-head attention mechanism to analyze semantic information. Historical virtual knowledge and semantic feature matrix Integrate to obtain a vector that integrates semantic features and virtual knowledge features

[0028] Among them, the multi-head attention mechanism specifically includes the following processes: Key values ​​and semantic information The query value is multiplied by the Product operation, and then goes through the Scaling operation and the Softmax operation, and the output of the weighting operation is combined with the semantic feature matrix The values ​​are sequentially subjected to Concat (merge operation) and Linear (linear operation) to obtain the output result Virtual knowledge of history Key values ​​and semantic information The query value is multiplied by the Product operation, and then goes through the Scaling operation and the Softmax operation, and the output of the weight operation is combined with the historical virtual knowledge. The values ​​are sequentially subjected to Concat (merge operation) and Linear (linear operation) to obtain the output result The virtual knowledge dynamic selection module will output the results Output and semantic information Merge to get the vector of fusion semantic features and virtual knowledge features

[0029] The common sense knowledge expansion module traverses the common sense knowledge graph ConceptNet to expand the historical dialogue C n 、User input X n and candidate document collection The information in the concept expansion, encoding and feature extraction are used to obtain extended knowledge

[0030] Among them, the concept expansion, encoding and feature extraction methods of the common sense knowledge expansion module are as follows: n 、User input X n and candidate document collection Extract word sets as original entities By traversing the common sense knowledge graph ConceptNet, we get the original entity The associated word set as an extended entity Obtaining original entities based on the common sense knowledge graph ConceptNet and extended entities The relationship set of each corresponding word in is used as the relationship Based on the neural network CompGCN, the original entities Extended Entity and relationships Encode and obtain the encoding vectors Encoded vector and encoding vector According to the encoding vector Encoded vector and encoding vector Extended knowledge through attention mechanism

[0031] Transformer decoder module, based on the generated word at the previous moment Pair Vector Decode and get the hidden layer state after decoding

[0032] Among them, when t=1, the word is generated is the starting character.

[0033] The controller module decodes the hidden layer state The probability distribution A of the generated words in the dialogue standard vocabulary is calculated by the Softmax function, and the knowledge is expanded Get the probability distribution B of the generated word in the extended knowledge entity set, and get the generated word at the current moment based on the probability distribution A and probability distribution B based on the gating mechanism That is, the word produced at the tth moment in the nth dialogue round.

[0034] The Transformer decoder module and controller module generate words one by one, so by calling the Transformer decoder module and controller module multiple times, we can get the generated word set at all times in the current dialogue round. Output generated word set Y as a response to user input X n 's reply.

[0035] Figure 2 3 is a schematic diagram of the effect of the open domain dialogue generation method based on dynamic multi-form knowledge fusion in an embodiment of the present invention.

[0036] like Figure 2 As shown in the figure, the open domain dialogue generation method based on dynamic multi-form knowledge fusion of the present invention obtains core concepts such as "people" and "lose weight" from user input, expands these core concepts through the virtual knowledge reasoning module, obtains adjacent concepts such as "overweight", and converts the candidate document set into The number of core concepts and adjacent concepts contained in is used as the filtering score to filter and obtain the first hop knowledge of DMKCM, namely the candidate document set. In order to better capture the conversion of concepts in multi-round dialogues to help generate more coherent dialogues, the common sense knowledge expansion module takes historical dialogues, user inputs, and first-hop knowledge as inputs, and traverses the common sense knowledge graph along the common sense relationship, thereby finding the related entities of words such as "diet" in historical dialogues, "cat" in first-hop knowledge, and "weight" in user input, namely the words "physical", "exercise", "healthy", and "food", as second-hop knowledge. The result of encoding and feature extraction of the second-hop knowledge is the extended knowledge. When the topic shifts during a multi-round conversation, it's difficult to find the right knowledge to generate a response from only the first hop of knowledge. To avoid this problem, this patent incorporates a virtual knowledge dynamic memory module to provide knowledge support for the current round of conversation. For example, the "fatening foods" in the machine's response comes from the "fatty foods" in the virtual knowledge dynamic memory module's historical virtual knowledge base. The machine's response to user input is generated through the interaction between the Transformer decoder module and the controller module.

[0037] In this embodiment, based on the content of the existing datasets PersonaChat and DailyDialog, machine replies are performed on the model constructed by the open domain dialogue generation method based on dynamic multi-modal knowledge fusion of the present invention and other existing models, namely the Aten-S2S model, the Transformer model, the Dir-VHRED model, the GLKS model, the CCM model, and the MSTK model. The generated dialogue content is as follows:

[0038]

[0039]

[0040] The first row shows two existing datasets, PersonaChat and DailyDialog. The second row shows historical conversation content and user input from the current conversation round, both selected from these datasets. Rows three through nine show the machine responses generated by each model to the user input. For example, the third column of row nine shows the machine responses to user input from the DailyDialog dataset, generated by a model constructed using the present invention's open-domain conversation generation method based on dynamic multi-modal knowledge fusion. As can be seen from the table, the machine responses generated by the model constructed using the present invention's open-domain conversation generation method based on dynamic multi-modal knowledge fusion are more diverse and coherent.

[0041] Functions and Effects of the Embodiments

[0042] According to the open-domain dialogue generation method based on dynamic multi-modal knowledge fusion involved in this embodiment, the virtual knowledge reasoning module obtains candidate documents related to the current dialogue content as first-hop knowledge, which can enrich the content of the current dialogue. The virtual knowledge dynamic memory module stores historical first-hop knowledge data, providing knowledge support for the current dialogue and further enriching the dialogue content. The common sense knowledge expansion module traverses the common sense knowledge graph of the dialogue content, effectively coping with topic shifts. In short, this method can generate rich dialogue content in open-domain multi-round dialogues and has good adaptability to topic shifts.

[0043] The above embodiments are preferred examples of the present invention and are not intended to limit the scope of protection of the present invention.

Claims

1. A method for generating open-domain dialogues based on dynamic multi-form knowledge fusion, characterized in that: include: Dialogue module and dynamic multi-modal knowledge fusion module, The dialogue module includes a first Transformer encoder module, a second Transformer encoder module, a third Transformer encoder module, a Transformer decoder module and a controller module. The dynamic multi-form knowledge fusion module includes a virtual knowledge reasoning module, a virtual knowledge dynamic memory module, a virtual knowledge dynamic selection module and a common sense knowledge expansion module. The virtual knowledge reasoning module takes the user input X n Input virtual knowledge reasoning model to obtain candidate document set According to the candidate document set The filtering scores of each candidate document in are arranged from high to low, and the top T candidate documents are selected as the candidate document set The first Transformer encoder module sets the candidate document set Encode to obtain the semantic feature matrix The second Transformer encoder module processes the user input X n Encode semantic information The virtual knowledge dynamic memory module collects all historical semantic feature matrices obtained before the current dialogue Dynamically stored as matrix H, according to the semantic information The matrix H is filtered and screened based on the attention mechanism to obtain historical virtual knowledge The third Transformer encoder module is used to process the historical dialogue C n and the user inputs X n Encode semantic information The virtual knowledge dynamic selection module, based on the multi-head attention mechanism, selects the semantic information The historical virtual knowledge and the semantic feature matrix Integrate to obtain a vector that integrates semantic features and virtual knowledge features The common sense knowledge expansion module traverses the common sense knowledge graph ConceptNet to expand the historical dialogue C n , the user input X n and the candidate document set The information in the concept expansion, encoding and feature extraction are used to obtain extended knowledge The Transformer decoder module generates words based on the previous moment , t≥1, for the vector Decode and get the hidden layer state after decoding The controller module converts the decoded hidden layer state The probability distribution A of the generated words in the dialogue standard vocabulary is calculated by the Softmax function, and the extended knowledge Obtain the probability distribution B of the generated word in the extended knowledge entity set, and obtain the generated word at the current moment based on the probability distribution A and the probability distribution B based on the gating mechanism By repeatedly calling the Transformer decoder module and the controller module, the generated word set at all times in the current dialogue round is obtained. , N≤50, output the generated word set Y as the response to the user input X n 's reply.

2. The method for generating open-domain dialogue based on dynamic multi-modal knowledge fusion according to claim 1, characterized in that: in, The virtual knowledge reasoning model is obtained by training a model based on a virtual knowledge base reasoning algorithm DrKIT using a dialogue dataset and a virtual knowledge base.

3. The method for generating open-domain dialogue based on dynamic multi-modal knowledge fusion according to claim 2, characterized in that: in, The conversation datasets are open source conversation datasets DailyDialog and PersonaChat.

4. The method for generating open-domain dialogue based on dynamic multi-modal knowledge fusion according to claim 2, characterized in that: in, The virtual knowledge base is obtained by indexing the common sense story dataset ROCStories.

5. The method for generating open-domain dialogue based on dynamic multi-modal knowledge fusion according to claim 1, characterized in that: in, The filtering score is the number of times the candidate document co-occurs with words in a word set, where the word set is input by the user X. n and the user inputs X n The word set is composed of related words obtained based on the common sense knowledge graph ConceptNet, and there are no repeated words in the word set.

6. The method for generating open-domain dialogue based on dynamic multi-modal knowledge fusion according to claim 1, characterized in that: in, The value of T is 5.

7. The method for generating open-domain dialogue based on dynamic multi-modal knowledge fusion according to claim 1, characterized in that: in, The concept expansion, encoding and feature extraction methods of the common sense knowledge expansion module are as follows: From the historical dialogue C n , the user input X n and the candidate document set Extract word sets as original entities By traversing the common sense knowledge graph ConceptNet, the original entity is obtained The associated word set as an extended entity The original entity is obtained according to the common sense knowledge graph ConceptNet and the extension entity The relationship set of each corresponding word in is used as the relationship Based on the neural network CompGCN, the original entities The extended entity and the relationship Encode and obtain the encoding vectors Encoded vector and encoding vector According to the encoding vector The encoding vector and the encoding vector The extended knowledge is obtained through the attention mechanism

Images