Abstract generation method, model training method and electronic equipment

Abstract

The invention discloses an abstract generation method, a model training method and electronic equipment, which are used for accurately capturing the semantic coherence between the subject content of a long dialogue text and front and back statements and generating an accurate and coherent abstract. The abstract generation method comprises the steps of determining keywords of a first dialogue text based on the similarity between sentences in the first dialogue text and the similarity between words in the sentences; dialogue state tracking is carried out based on the context of the first dialogue text, and dialogue state information of the first dialogue text is determined; obtaining first entity information matched with the dialogue state information from the first knowledge graph; nodes in the first knowledge graph represent entities related to a service to which the first dialogue text belongs, and edges between the nodes represent relationships between the entities; and generating a first abstract of the first dialogue text based on the keyword, the first entity information and the image corresponding to the first entity information.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a summary generation method, a model training method, and an electronic device. Background Art

[0002] Text summarization is a key technology in the field of natural language processing. It aims to extract key information from text and generate concise summaries. It is widely used in various business scenarios such as intelligent customer service, news reporting, academic research, and business analysis.

[0003] In conversational scenarios, long conversational texts often contain a large amount of redundant information, and the semantic relevance between preceding and following sentences is often weakened. Accurately capturing the main content of long conversational texts and the semantic coherence between preceding and following sentences, and generating accurate and coherent summaries, has become a hot research topic. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a summary generation method, a model training method, and an electronic device for accurately capturing the main content of a long conversation text and the semantic coherence between the preceding and following sentences, and generating an accurate and coherent summary.

[0005] In order to achieve the above objectives, the embodiments of the present application adopt the following technical solutions: In a first aspect, an embodiment of the present application provides a method for generating a summary, comprising: determining keywords of the first dialogue text based on similarities between sentences in the first dialogue text and similarities between words in the sentences; Tracking the conversation state based on the context of the first conversation text to determine conversation state information of the first conversation text; Obtaining first entity information matching the conversation state information from a first knowledge graph; nodes in the first knowledge graph represent entities involved in the business to which the first conversation text belongs, and edges between nodes represent relationships between the entities; A first summary of the first conversation text is generated based on the keyword, the first entity information, and an image corresponding to the first entity information.

[0006] The summary generation method provided in the embodiments of the present application accurately identifies key sentences in the first dialogue text based on similarities, since the similarity between sentences in the first dialogue text can reflect the correlation between sentences, and the similarity between words in sentences can reflect the correlation between words. Therefore, based on these similarities, key sentences in the first dialogue text can be accurately identified, providing key clues for understanding the main content of the first dialogue text. Furthermore, dialogue state information of the first dialogue text is obtained by tracking the dialogue state based on the context of the first dialogue text, so as to adapt to the changing dialogue environment during the summary generation process for the first dialogue text. Furthermore, graph reasoning is performed on a knowledge graph based on the dialogue state information to obtain entity information that matches the dialogue state information. Because the entity information and the image corresponding to the entity information can provide the background knowledge required to summarize the first dialogue text, they help prevent the presence of untrue content in the generated summary. Therefore, by integrating the keywords of the first dialogue text, the entity information that matches the dialogue state information of the first dialogue text, and the image corresponding to the entity information, multiple modal information complement each other, providing a richer and more comprehensive basis for accurately understanding the semantics of the first dialogue text, capturing the correlation between previous and subsequent sentences in the dialogue text, and changes in the dialogue environment, thereby generating an accurate, truthful, and coherent summary for the first dialogue text.

[0007] In a second aspect, an embodiment of the present application provides a model training method, comprising: determining keywords of the second dialogue text based on similarities between sentences in the second dialogue text and similarities between words in the sentences; performing dialogue state tracking based on the context of the second dialogue text to determine dialogue state information of the second dialogue text; Obtaining second entity information matching the conversation state information from a second knowledge graph; nodes in the second knowledge graph represent entities involved in the business to which the second conversation text belongs, and edges between nodes represent relationships between the entities; generating a first summary of the second conversation text based on the keyword, the second entity information, and an image corresponding to the second entity information; Adjust parameters of the model based on the first summary and the reference summary of the second conversation text.

[0008] The model training method provided in an embodiment of the present application determines key sentences of the second dialogue text based on the similarity between sentences and the similarity between words in sentences. In addition, the method also tracks the dialogue state based on the context of the second dialogue text to obtain dialogue state information of the second dialogue text, and performs graph reasoning on a knowledge graph based on the dialogue state information to obtain entity information matching the dialogue state information. Furthermore, the method integrates keywords of the second dialogue text, entity information matching the dialogue state information of the second dialogue text, and images corresponding to the entity information, so that multiple modal information complement each other, providing a richer and more comprehensive basis for accurately understanding the semantics of the second dialogue text, capturing the correlation between previous and subsequent sentences in the second dialogue text, and changes in the dialogue environment, thereby generating a summary of the second dialogue text. Finally, based on the summary of the second dialogue text and a reference summary, the model parameters are adjusted so that the model can generate an accurate, true, and coherent summary for any dialogue text, especially long dialogue texts.

[0009] In a third aspect, an embodiment of the present application provides a summary generation device, comprising: a determination module, configured to determine keywords of the first dialogue text based on similarities between sentences in the first dialogue text and similarities between words in the sentences; a state tracking module, configured to determine dialogue state information of the first dialogue text based on the context of the first dialogue text; A query module, configured to obtain first entity information matching the conversation state information from a first knowledge graph, wherein the first knowledge graph is configured to describe entities involved in the business to which the first conversation text belongs and relationships between the entities; A generation module is used to generate a first summary of the first conversation text based on the keyword, the first entity information and the image corresponding to the first entity information.

[0010] In a fourth aspect, an embodiment of the present application provides a model training device, comprising: a determination module, configured to determine keywords of the second dialogue text based on similarities between sentences in the second dialogue text and similarities between words in the sentences; a state tracking module, configured to determine dialogue state information of the second dialogue text based on the context of the second dialogue text; A query module, configured to obtain second entity information matching the conversation state information from a second knowledge graph; the second knowledge graph is configured to describe entities involved in the business to which the second conversation text belongs and the relationships between the entities; a generating module, configured to generate a first summary of the second conversation text based on the keyword, the second entity information, and an image corresponding to the second entity information; An adjustment module is used to adjust parameters of the model based on the first summary and the reference summary of the second dialogue text.

[0011] In a fifth aspect, an embodiment of the present application provides an electronic device, including: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the summary generation method provided in the first aspect or the model training method provided in the second aspect.

[0012] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the summary generation method provided in the first aspect or the model training method provided in the second aspect.

[0013] In a seventh aspect, an embodiment of the present application provides a computer program product, comprising a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to execute part or all of the steps in the summary generation method provided in the first aspect or the model training method provided in the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A schematic diagram of a process for generating a summary according to an embodiment of the present application; Figure 2 A flowchart of a method for generating a summary according to another embodiment of the present application is provided; Figure 3 A flowchart of a model training method provided for one embodiment of the present application; Figure 4 A schematic structural diagram of a summary generation device provided in one embodiment of the present application; Figure 5 A schematic structural diagram of a model training device provided in one embodiment of the present application; Figure 6 A schematic structural diagram of an electronic device provided in accordance with an embodiment of the present application. DETAILED DESCRIPTION

[0015] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0016] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0017] Key terms explained: Dialogue State Tracking (DST): This technology tracks the state of a conversation in real time, linking the current conversation text with previous ones. It continuously updates key information such as the intent, slot information, and sentiment of the conversation text, ensuring the model understands user intent and maintains semantic consistency. This is particularly important in multi-turn conversations, as it prevents the generation of content that is inconsistent with the current context.

[0018] Instruction Tuning: By designing prompts, the Large Language Model (LLM) can better understand the conversation state and user intent, achieving more accurate state tracking and information extraction in complex, multi-round conversations.

[0019] Anti-hallucination mechanism: Large language models can generate "hallucinations" when summarizing long texts, where the generated content deviates from the input. To address this issue, anti-hallucination mechanisms have been developed. By comparing generated content with real content and applying Kullback-Leibler Divergence (KL-D) constraints, they suppress the generation of false content and improve the authenticity of summaries.

[0020] Graph Neural Network (GNN) and Variational Graph Autoencoder: Graph-structured models such as GNN and Variational Graph Autoencoder can model relationships during reasoning and are suitable for information fusion and hierarchical reasoning in multi-round interactions in long conversations.

[0021] Memory enhancement and reflection mechanism: In multi-round dialogues, the memory enhancement and reflection mechanism can improve the model's reasoning consistency and decision-making accuracy by preserving important information and self-correcting the reasoning path.

[0022] The inventors conducted extensive research and found that: On the one hand, relevant technologies typically use extractive summarization and generative summarization techniques to generate summaries of text. Extractive summarization involves directly extracting important sentences or phrases from text to form a summary. This technique is simple to implement and can preserve the authenticity of the original sentences, but it is effective for long conversations and can easily miss cross-sentence correlation information. Generative summarization, based on models such as recurrent neural networks (RNNs) or long short-term memory networks (LSTMs), captures sequential information in text and generates new sentences or paragraphs to summarize the main content of the text. This technique can flexibly capture semantic relationships in conversational text, but it faces problems such as vanishing gradients when processing long conversations, resulting in limited contextual memory capabilities. Consequently, it still lacks accuracy and coherence in summary generation.

[0023] On the other hand, keywords in a text often cover the core points of the text and are the key clues to understanding the main content of the text. Keywords are often closely related to other words in the text, and keywords often appear in key sentences that are closely related to other sentences.

[0024] On the other hand, the dialogue state information of a conversation often contains key information such as the intent, sentiment, and context of the conversation. This helps the summary generation process adapt to the changing conversation environment and ensure the coherence and accuracy of the summary. Furthermore, entity information and images related to the dialogue state information can provide the background knowledge needed to summarize the conversation, helping to prevent untrue content in the generated summary.

[0025] Taking the above considerations into consideration, the embodiment of the present application proposes a summary generation method that abandons extractive summarization technology and instead adopts summary generation technology to generate a summary of the dialogue text, so as to avoid missing the correlation information between the previous and next sentences in the dialogue text, accurately capture the semantic relationship in the dialogue text, and thus improve the accuracy of the summary.

[0026] In addition, in response to the problems of insufficient summary accuracy and coherence in summary generation technology in related technologies, improvements have been made to the summary generation technology to ensure the accuracy and coherence of summary generation. Specifically, since the similarity between sentences in a conversation text can reflect the correlation between sentences, and the similarity between words in a sentence can reflect the correlation between words, these similarities can be used to accurately identify key sentences in the conversation text, providing key clues for understanding the main content of the text. In addition, by tracking the conversation state of the context based on the conversation text, the conversation state information of the conversation text is obtained, so as to adapt to the changing conversation environment during the process of generating a summary for the conversation text. Furthermore, graph reasoning is performed on the knowledge graph based on the conversation state information to obtain entity information that matches the conversation state information. Since the entity information and the image corresponding to the entity information can provide the background knowledge required to summarize the conversation text, it helps to suppress the appearance of untrue content in the generated summary. Therefore, the keywords of the conversation text, the entity information matching the conversation state information of the conversation text, and the image corresponding to the entity information are integrated to enable multiple modal information to complement each other, providing a richer and more comprehensive basis for accurately understanding the semantics of the conversation text, capturing the correlation between previous and subsequent sentences in the conversation text, and changes in the conversation environment, thereby generating an accurate, true, and coherent summary of the conversation text.

[0027] The embodiment of the present application also proposes a model training method, which determines the key sentences of a dialogue text based on the similarity between sentences in the dialogue text and the similarity between words in the sentences; in addition, it also obtains the dialogue state information of the dialogue text by tracking the dialogue state based on the context of the dialogue text, and obtains entity information matching the dialogue state information by performing graph reasoning on the knowledge graph based on the dialogue state information; further, the keywords of the dialogue text, the entity information matching the dialogue state information of the dialogue text, and the images corresponding to the entity information are integrated to make the multiple modal information complement each other, providing a richer and more comprehensive basis for accurately understanding the semantics of the dialogue text, capturing the correlation between previous and next sentences in the dialogue text and changes in the dialogue environment, and generating a summary of the dialogue text; finally, based on the summary of the dialogue text and the reference summary, the parameters of the model are adjusted so that the model can generate an accurate, true and coherent summary for any dialogue text, especially long dialogue text.

[0028] It should be understood that the summary generation method and model training method proposed in the embodiments of the present application can be executed by an electronic device. As an example, they can be executed by software in the electronic device. The electronic devices mentioned here can include terminal devices, such as smartphones, tablets, laptops, desktop computers, intelligent voice interaction devices, smart home appliances, smart watches, vehicle terminals, aircraft, etc.; or the electronic device can also include a server, such as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0029] Next, the summary generation method and model training method provided in the embodiments of the present application are described in detail with reference to the accompanying drawings.

[0030] Please refer to Figure 1 , is a flow chart of a method for generating a summary according to an embodiment of the present application, the method comprising the following steps: S102: Determine keywords of the first dialogue text based on similarities between sentences in the first dialogue text and similarities between words in the sentences.

[0031] The first conversation text is the text to be processed as the target for summary generation. The first conversation text includes multiple sentences. For any two sentences, the similarity between the two sentences can be determined based on the distance between the sentence vectors of each sentence. Similarly, for any two words in a sentence, the similarity between the two words can be determined based on the distance between the word vectors of each word.

[0032] In one implementation, the above S102 includes the following steps: S1021: Determine key sentences of the first dialogue text based on similarities between sentences in the first dialogue text.

[0033] Key sentences are sentences that contain the main content of the first conversation text. Exemplarily, the TextRank graph ranking algorithm can be used to determine the key sentences of the first conversation text based on the similarity between sentences in the first conversation text. Specifically, each sentence in the first conversation text is treated as a node, and edges are established between the nodes based on the similarity between the sentences. For example, if the similarity between two sentences exceeds a preset threshold, an edge is established between the nodes corresponding to the two sentences, thereby generating a sentence graph. Then, based on the similarity between each node and its neighboring nodes, an importance score is determined for each node. Finally, sentences corresponding to nodes with importance scores greater than a preset score are determined as key sentences.

[0034] S1022: Determine a first weight of the words in the key sentence based on the similarity between the words in the key sentence.

[0035] For example, after obtaining the similarity between words in a key sentence, an attention mechanism can be used to determine the attention weight of the words in the key sentence. This attention weight is the first weight of the word. The first weight of a word represents the degree of influence of the word on the semantics of the key sentence. That is, words with a large first weight have a greater influence on the semantics of the key sentence, while words with a small first weight have a smaller influence on the semantics of the key sentence.

[0036] S1023: Determine keywords of the first dialogue text from the words included in the key sentence based on the first weights of the words in the key sentence.

[0037] As an example, words in the key sentence whose first weight is greater than the first threshold are determined as keywords of the first dialogue text.

[0038] As another example, considering that the words in the key sentence whose first weight is greater than the first threshold may only be important words to the key sentence, but in the global context of the first dialogue text they are redundant information that cannot reflect the main content, which will have an adverse effect on understanding the semantics of the first dialogue text. For this reason, a context comparison strategy is adopted to compare the semantics of the important words in the key sentence with the semantics of the first dialogue text, and further identify words from these words that are consistent with the global semantics of the first dialogue text as keywords, thereby enhancing the accuracy of understanding the semantics of the first dialogue text and providing reliable data support for the subsequent generation of a summary of the first dialogue text.

[0039] Specifically, a word in the key sentence whose first weight is greater than a first threshold is determined as the first word; the first word and the first conversation text are projected into the same semantic space to obtain a semantic vector for the first word and a semantic vector for the first conversation text; the relevance between the first word and the first conversation text is determined based on the distance between the semantic vector of the first word and the semantic vector of the first conversation text; and the first word whose relevance to the first conversation text is greater than a second threshold is determined as a keyword of the first conversation text. In an application, the relevance between the first word and the first conversation text can be obtained by performing a predetermined operation on the distance between their semantic vectors, such as determining the reciprocal of the distance as the relevance between the two.

[0040] For example, the words in the key sentence whose first weight is greater than the first threshold include word 1 and word 2. Assuming that the correlation between word 1 and the first dialogue text is less than the second threshold, and the correlation between word 2 and the first dialogue text is greater than the second threshold, this shows that the semantics of word 1 is inconsistent with the global semantics of the first dialogue text, and the semantics of word 2 is consistent with the global semantics of the first dialogue text, and thus word 2 is determined as the keyword of the first dialogue text.

[0041] Since keywords are often closely related to other words in the text, and keywords often appear in key sentences that are closely related to other sentences, through the above implementation method, the key sentences that are closely related to other sentences in the first dialogue text are first determined, and then the keywords that are closely related to other words are determined from the key sentences. This can ensure that the keywords cover the core points of the first dialogue text and provide key clues for understanding the main content of the first dialogue text.

[0042] In another implementation, for each sentence in the first dialogue text, the frequency of each word in the sentence appearing in the first dialogue text is counted, and words with a frequency greater than a frequency threshold are determined as keywords of the first dialogue text.

[0043] The above describes some implementations of the above S102. Of course, it should be understood that the above S102 can also be implemented in other ways, which are not limited in the present embodiment.

[0044] S104: Tracking the dialogue state based on the context of the first dialogue text to determine dialogue state information of the first dialogue text.

[0045] The context of the first conversation text may include conversation text generated before the first conversation text. The conversation state information of the first conversation text may include, but is not limited to, at least one of the following information: the intention of the first conversation text, slot information (such as the type of associated product, date, location, etc.), emotional information, etc.

[0046] Exemplarily, a first instruction can be generated based on the first dialogue text, its context, and the first prompt template, and the first instruction can be input into a model (such as a large language model) to guide the model to understand the semantics of each sentence in the first dialogue text, and track the dialogue state of the first dialogue text in combination with the context to obtain dialogue state information of the first dialogue text.

[0047] The first prompt template can be set according to actual needs, and the embodiments of this application do not limit this. For example, the first prompt template can be "Please generate dialogue state information based on the following dialogue. Dialogue intent, slot information, emotion, etc. need to be generated together. Historical dialogue content: {context}, current dialogue content: {first dialogue text}". This prompt template can help the model clarify the specific content of its dialogue state tracking task.

[0048] In another embodiment, prior to S104, the model may be further fine-tuned for instructions, enabling the model to determine specific dialogue state information based on a given input text. Specifically, historical dialogue texts are pre-labeled with corresponding reference dialogue state information. Then, based on the historical dialogue texts, their context, and prompt templates, corresponding instructions are generated and input into a model (e.g., a large language model) to guide the model in understanding the semantics of each sentence in the historical dialogue texts. The model then tracks the dialogue state of the historical dialogue texts in conjunction with the context, obtaining dialogue state information for the historical dialogue texts. Finally, based on the differences between this dialogue state information and the reference dialogue state information, the model parameters are adjusted, enabling the model to accurately track the dialogue state of the input texts.

[0049] In another embodiment, considering that the model may produce hallucinations during the process of dialogue state tracking, causing the output dialogue state information to deviate from the actual dialogue state information, an anti-hallucination mechanism is introduced in the dialogue state tracking process. By controlling the difference between the output dialogue state information and the actual dialogue state information, the model maintains context consistency in multiple rounds of interaction, ensures semantic coherence in multiple rounds of dialogue, and avoids generating content that deviates from actual user needs.

[0050] Specifically, the words in the first dialogue text are compared with preset semantically ambiguous words (such as "very good" and "some risks") to obtain the semantically ambiguous words in the first dialogue text, and the semantically ambiguous words in the first dialogue text are deleted to obtain a new first dialogue text; then, based on the context of the first dialogue text, the dialogue state of the new first dialogue text is tracked to obtain dialogue state information.

[0051] S106: Obtain first entity information that matches the dialogue state information from the first knowledge graph.

[0052] The first knowledge graph is used to describe the entities involved in the business to which the first conversation text belongs, as well as the relationships between these entities. The first knowledge graph includes multiple nodes and edges between these nodes. A node represents an entity involved in the business to which the first conversation text belongs, such as a person, place, time, organization, event, product, etc. An edge between any two nodes represents the relationship between the entities corresponding to these two nodes. Each node has corresponding node information, which includes the attributes of the entity corresponding to the node.

[0053] The first knowledge graph can be constructed in various appropriate ways, which are not limited in the embodiments of the present application. In one implementation, entity recognition is performed on documents such as news articles, social media posts, and scientific research documents related to the business scenario to which the first conversation text belongs, to obtain entities in these documents and the relationships between entities, and each entity is treated as a node. Connection edges are established between nodes based on the relationships between entities to obtain a knowledge graph; then, entity recognition is performed on the descriptive text (such as question-answer pairs) of images related to these entities (such as promotional images of a product, etc.), and the knowledge graph is updated based on the identified entities and the relationships between entities to obtain the first knowledge graph.

[0054] The first entity information matching the dialog state information may be entity information that provides background knowledge for the dialog state information.

[0055] In one implementation, the first knowledge graph and the dialogue state information are semantically parsed using a model, such as a graph neural network or a large language model, to obtain first entity information that matches the dialogue state information.

[0056] In another implementation, the above S106 includes the following steps: S1061: Generate a first reasoning task based on the dialogue state information.

[0057] As an example, the intent of the first conversation text and the first entity corresponding to the first intent are obtained from the conversation state information. This information is then concatenated to obtain the first reasoning task. For example, if the intent of the first conversation text is "low risk + high return" and the first entity includes product A and product B, concatenating the two results in the first reasoning task: determining whether product A and product B have the characteristics of low risk and high return.

[0058] For example, the intention of the first dialogue text is "determine the topic of discussion", and the first entities corresponding to the intention are "Zhang San" and "Li Si". By splicing the two together, we get the first reasoning task: determine what topics Zhang San and Li Si discussed together.

[0059] As another example, based on the dialogue state information and the second prompt template, a third instruction is generated, and the third instruction is input into the model to guide the model to generate the corresponding first reasoning task according to the dialogue state information.

[0060] S1062: Determine a first reasoning path on the first knowledge graph that matches the first reasoning task based on the nodes and the edges between the nodes on the first knowledge graph.

[0061] As an example, based on the nodes and edges between the nodes on the first knowledge graph, a first reasoning task is performed on the first knowledge graph, and the reasoning path in the reasoning process is recorded as the first reasoning path.

[0062] In the application, the execution of the first reasoning task on the first knowledge graph can be completed by a model, such as a large language model, a graph neural network, etc., which is not limited in the embodiments of the present application. In the process of executing the first reasoning task, the model is also integrated with the variational graph autoencoder, and the first knowledge graph is encoded into a corresponding representation vector through the variational graph autoencoder; further, the model executes the first reasoning task based on the representation vector of the first knowledge graph, and reflects on and self-corrects the wrong reasoning path during the reasoning process. Exemplarily, the model records the current reasoning path in the memory bank based on the memory update mechanism, and dynamically updates the memory bank during the reasoning process, and self-corrects the current reasoning path based on the reflection mechanism to ensure the accuracy and robustness of the reasoning, so that it can be continuously improved in complex reasoning scenarios and ultimately obtain high-quality reasoning results.

[0063] As another example, the conversation state information includes the intent of the first conversation text and the first entity corresponding to the intent. In this case, the first reasoning task is decomposed into multiple subtasks. For each subtask, the node corresponding to the first entity is used as the starting point and the subtask is executed on the first knowledge graph to obtain the second reasoning path corresponding to the subtask. The second reasoning paths corresponding to the subtasks are then merged to obtain the first reasoning path that matches the first reasoning task.

[0064] For example, the intent of the first conversation text is "low risk + high return", the first entity includes product A and product B, and the first reasoning task is to determine whether product A and product B have the characteristics of low risk and high return. The first reasoning task can be decomposed into the following two subtasks: Subtask 1: Determine the risk level of product A and the risk level of product B Subtask 2: Determine the revenue guarantee for Product A and Product B For subtask 1, take node A corresponding to product A as the starting point, find the node information of node A and node C representing the risk level connected to node A, and record the second reasoning path: node A corresponding to product A → node information of node A → edge connected to node A → node C representing the risk level connected by the edge.

[0065] Similarly, node B corresponding to product B is taken as the starting point, the node information of node B and node D representing the risk level connected to node B are found, and the second reasoning path is recorded: node B corresponding to product B → node information of node B → edge connected to node B → node D representing the risk level connected by the edge.

[0066] For subtask 2, take node A corresponding to product A as the starting point, find the node information of node A and node E representing the revenue characteristics connected to node A, and record the second reasoning path: node A corresponding to product A → node information of node A → edge connected to node A → node E representing the revenue characteristics connected by the edge.

[0067] Similarly, node B corresponding to product B is taken as the starting point to find the node information of node B and node F representing the revenue characteristics connected to node B, and record the second reasoning path: node B corresponding to product B → node information of node B → edge connected to node B → node F representing the revenue characteristics connected by the edge.

[0068] By combining the above second reasoning paths, we obtain the following first reasoning path: The first reasoning path 1: Node A → Node information of Node A → Edge connected to Node A → Node C representing the risk level connected by the edge → Node E representing the return characteristics connected by the edge.

[0069] First reasoning path 2: Node B → Node information of Node B → Edge connected to Node B → Node D representing the risk level connected to the table → Node F representing the return feature connected to the edge.

[0070] For example, the first reasoning task is to determine what topics Zhang San and Li Si discussed together. The first entity includes "Zhang San" and "Li Si". The first reasoning task can be decomposed into the following two subtasks: Subtask 1: Determine the relationship between Zhang San and Li Si (e.g., whether they know each other, whether they have discussed topics before) Subtask 2: Find all the topics discussed by Zhang San and Li Si For subtask 1, take the nodes corresponding to "Zhang San" and "Li Si" as the starting point, and find out whether there is an edge between the two nodes. If so, determine the relationship represented by the edge as the relationship between Zhang San and Li Si, and record the second reasoning path: Zhang San → Li Si → connected edge.

[0071] For sub-goal 2, take the nodes corresponding to "Zhang San" and "Li Si" as the starting point, find other nodes representing topics connected to these nodes, and record the second reasoning path: Zhang San → Li Si → node representing the topic.

[0072] The second reasoning path corresponding to each subtask is merged to obtain the first reasoning path: Zhang San → Li Si → connecting edge → node representing the topic.

[0073] S1063: Determine first entity information that matches the dialogue state information based on the nodes and edges between the nodes on the first reasoning path.

[0074] The first entity information may include entities that match the dialog state information, attribute information of the entities, and relationships between these entities. To this end, the entities that match the dialog state information may be determined from the entities represented by the nodes on the first reasoning path, the attribute information of these entities may be determined from the node information of the nodes on the first reasoning path, and the relationships between these entities may be determined from the relationships represented by the edges between the nodes on the first reasoning path.

[0075] For example, taking the above-mentioned first reasoning task "determine whether product A and product B have the characteristics of low risk and high return" and its corresponding first reasoning path 1 and first reasoning path 2 as an example, for the first reasoning path 1, it can be determined from the node information of node A and the node information of node C on the reasoning path whether product A has the characteristics of low risk, and from the node information of node A and the node information of node E, it can be determined whether product A has the characteristics of high return.

[0076] For the second reasoning path 2, it can be determined from the node information of node B and the node information of node D on the reasoning path whether product B has low-risk characteristics, and from the node information of node B and the node information of node F, it can be determined whether product B has high-yield characteristics.

[0077] For example, taking the first reasoning task "Determine what topics Zhang San and Li Si discussed together, the first entity" and its corresponding first reasoning path "Zhang San → Li Si → connecting edge → node representing the topic" as an example, the topics discussed by Zhang San and Li Si can be determined from the nodes representing the topic on the reasoning path.

[0078] In the above implementation, the first reasoning task is first generated based on the dialogue state information, and then the first reasoning task is performed on the first knowledge graph according to the entities and the relationships between entities described by the first knowledge graph. This can ensure the comprehensiveness and accuracy of the retrieval and reasoning of information on the first knowledge graph, thereby ensuring that the obtained first entity information is highly matched with the dialogue state information, providing reliable data support for the subsequent generation of accurate summaries.

[0079] The above describes some implementations of the above S106. Of course, it should be understood that the above S106 can also be implemented in other ways, which are not limited in the present embodiment.

[0080] S108 : Generate a first summary of the first conversation text based on the keyword, the first entity information, and the image corresponding to the first entity information.

[0081] The image corresponding to the first entity information refers to an image that can describe, explain, or display the first entity information. Specifically, the first entity information may include entities that match the conversation state information, attribute information of the entities, and relationships between these entities. The image corresponding to the first entity information may include, but is not limited to, images that describe, explain, or display these entities, their attribute information, and the relationships between these entities.

[0082] For example, the first entity information includes a product, the risk characteristics and the return characteristics of the product. The image corresponding to the first entity information may include but is not limited to a promotional picture of the product, a return trend chart and a risk trend chart of the product, etc.

[0083] The image corresponding to the first entity information can be retrieved from a gallery based on the first entity information. In one implementation, the gallery includes a large number of images and description text for each image. These images can be pre-collected or crawled from the Internet, and the description text of the image can be a title used to briefly describe the content of the image. Exemplarily, for each image in the gallery, the image features of the image can be extracted, and the image features of the image and the instructions for summarizing the content of the image can be input into a large language model. The semantic understanding ability and language generation ability of the large language model are used to capture the semantics of the image from the image features, and accordingly generate a description text for the image.

[0084] In this case, for each image in the gallery, the image features and the first entity information of the image can be mapped to the same semantic space to obtain the similarity between the two in the semantic space; then, the image described by the descriptive text whose similarity with the first entity information is greater than the similarity threshold is determined to be the image corresponding to the first entity information.

[0085] In one implementation, the above S108 includes the following steps: S1081: Encode the keyword to obtain a representation vector of the keyword.

[0086] S1082: Encode the first dialogue text to obtain a representation vector of the first dialogue text.

[0087] As an example, for each sentence in the first dialogue text, each word in the sentence is embedded to obtain a word vector for each word, and the word vectors for each word are concatenated to obtain a sentence vector for the sentence; further, the sentence vectors for each sentence in the first dialogue text are concatenated to obtain a representation vector for the first dialogue text.

[0088] As another example, considering that each sentence in the first conversation text has a different degree of influence on the semantics of the first conversation text, the representation vector obtained using the first example above cannot highlight the key information of the first conversation text, which will affect the capture of the main content of the first conversation text. Therefore, a hierarchical attention mechanism is used to encode the first conversation text. Specifically, the above S1082 includes the following steps: Step A1: Perform embedding processing on each word contained in the sentence in the first conversation text to obtain a word vector for each word.

[0089] In step A2, for each word in the sentence, the context of the word is determined from the sentence, and the probability of the word is predicted based on the context of the word.

[0090] For each word in a sentence, the words preceding the word in the sentence can be determined as the context of the word, and then the pre-trained language model can be used to predict the conditional probability distribution corresponding to the position of the word. The conditional probability distribution includes the probability that each word in the preset vocabulary is suitable for the position.

[0091] Step A3: Determine the second weight of each word and the perplexity of the sentence based on the probability of each word in the sentence, and determine the sentence vector of the sentence based on the word vector of each word and the second weight of each word.

[0092] Perplexity is a metric used in natural language processing to evaluate language model performance. It can be understood as the degree of confusion a language model has when predicting the next word. Lower perplexity indicates greater confidence in the language model's prediction of the next word, meaning it can more accurately predict the next word.

[0093] For each word in a sentence, its perplexity can be determined based on its probability. The inverse of the perplexity is then used as the second weight for that word. This means that the lower the perplexity of a word, the more informative the language model's prediction of it is, and the greater its influence on the semantics of the sentence, resulting in a higher second weight assigned to the word.

[0094] Optionally, after obtaining the probability of each word, the loss caused by the sentence is determined based on the probability of each word and the cross entropy loss function, that is, ,in, Indicates the loss caused by the sentence, represents the i-th word, represents the context of the i-th word, Represents the i-th word, represents the second weight of the i-th word, Represents the number of words in a sentence. Then, with the goal of reducing this loss, the second weight of each word is adjusted. In this way, words with lower losses can be assigned higher second weights. This is because words with lower losses contribute more to the language model's prediction and typically contain important content for the sentence.

[0095] After obtaining the word vector of each word in the sentence, the word vector of each word is weighted and summed based on the second weight of each word to obtain the sentence vector of the sentence.

[0096] In addition, each sentence also has a corresponding perplexity. The perplexity of a sentence can be determined by the following formula (1): (1) in, represents the perplexity of the sentence, Indicates the number of words in the sentence. represents the i-th word in the sentence, represents the probability of the i-th word in the sentence.

[0097] Step A4: determining a third weight of the sentence based on the perplexity of the sentence, and determining a representation vector of the first dialogue text based on the sentence vector and the third weight of the sentence.

[0098] The inverse of the perplexity of a sentence can be determined as the third weight of the sentence. This means that the lower the perplexity of a sentence, the more informative the language model's prediction of the sentence is, and the greater the influence of the sentence on the semantics of the first dialogue text, and thus the sentence is assigned a higher third weight.

[0099] After obtaining the sentence vector of each sentence, the sentence vector of each sentence is weighted and summed based on the third weight of each sentence to obtain the representation vector of the first dialogue text.

[0100] Through the above method, the key sentences in the first dialogue text and the keywords in the key sentences are given higher weights to imitate the way humans focus on key information in layers when reading text, which helps to focus on the key sentences in the first dialogue text and the keywords in the key sentences, thereby accurately capturing the main content of the first dialogue text and generating an accurate summary for the first dialogue text.

[0101] S1083: Encode the image to obtain a representation vector of the image.

[0102] As an example, image features of an image are extracted and encoded to obtain a representation vector of the image features, and the representation vector is determined as the representation vector of the image.

[0103] As another example, image features of an image are extracted and encoded to obtain a representation vector of the image features; description text of the image is generated based on the image features, and the description text is encoded to obtain a representation vector of the description text; the representation vector of the image features and the representation vector of the description text are fused to obtain a representation vector of the image.

[0104] The image description text can be a title that briefly describes the image content. Specifically, the image features and instructions for summarizing the image content can be input into the large language model. The large language model's semantic understanding and language generation capabilities are used to capture the image semantics from the image features and generate the image description text accordingly.

[0105] Furthermore, the image and its description text are encoded separately to obtain the representation vector of the image and the representation vector of the description text; then, the two representation vectors are semantically verified and aligned using a large language model, and the two aligned representation vectors are fused to obtain the representation vector of the image.

[0106] In this application, the BLIP-2 visual language model is used to extract image features and generate description text. BLIP-2 effectively combines image information with natural language to enhance semantic understanding of image content. This improves image understanding by generating description text. This reduces the complexity of image understanding, improves the semantic alignment between image and description text, and ensures that the final output summary is more accurate and relevant.

[0107] The representation vector of the image obtained in the above manner can more accurately and completely describe the content of the image, providing reliable data support for accurately generating a summary of the first conversation text.

[0108] S1084 , decoding the representation vector of the keyword, the representation vector of the first dialogue text, and the representation vector of the image to obtain a first summary of the first dialogue text.

[0109] As an example, the representation vector of the keyword, the representation vector of the first conversation text, and the representation vector of the image are fused and decoded to obtain a first summary of the first conversation text.

[0110] As another example, the representation vector of the keyword and the representation vector of the first dialogue text are fused and decoded to obtain a second summary of the first dialogue text; the second summary is encoded to obtain a representation vector of the second summary; based on the distance between the representation vector of the second summary and the representation vector of the image, a consistency score between the second summary and the image is determined; and based on the consistency score, the second summary is modified to obtain a first summary of the first dialogue text.

[0111] Specifically, the consistency score between the second summary and the image represents the semantic consistency between the second summary and the image content. In an application, the reciprocal of the distance between the representation vector of the second summary and the representation vector of the image can be determined as the consistency score between the second summary and the image.

[0112] If the consistency score is less than the score threshold, the second summary is modified based on the image description text to obtain the first summary of the first conversation text. If the consistency score is greater than or equal to the score threshold, the second summary is determined as the first summary of the first conversation text.

[0113] For example, the second summary is "The customer prefers low-risk, high-return products. Product A and Product B are recommended. Product A has guaranteed returns, but the risk is slightly higher." However, the image of Product A indicates that Product A has moderate risk, which is inconsistent with the risk description of Product A in the second summary. Therefore, the second summary is corrected to obtain the first summary "The customer prefers low-risk, high-return products. Product A (guaranteed returns, moderate risk) and Product B are recommended."

[0114] Through the above implementation, it can be ensured that the first summary accurately summarizes the content of the first dialogue text, and the relevant knowledge covered in the first summary is true and reliable.

[0115] In another implementation, the above S108 includes the following steps: generating a third instruction based on keywords and a third prompt template, and inputting the third instruction into the model to guide the model to capture the main content of the first dialogue text through keywords to obtain a second summary of the first dialogue text; encoding the first entity information to obtain a representation vector of the first entity information; encoding the image to obtain a representation vector of the image; determining a consistency score between the first entity information and the image based on the distance between the representation vector of the first entity information and the representation vector of the image; correcting the first entity information based on the consistency score, and combining the corrected first entity information and the second summary to obtain a first summary of the first dialogue text.

[0116] The above describes a specific implementation of S108. Of course, it should be understood that S108 can also be implemented in other ways, which are not limited in this embodiment of the present application.

[0117] To facilitate understanding of the abstract generation method provided in the embodiment of the present application, the following is combined with Figure 2 and one A specific dialogue text is given as an example.

[0118] like Figure 2 As shown, first, based on the similarity between sentences and words in the first dialogue text, combined with the long and short context comparison strategy, the keywords of the first dialogue text are determined. For example, the first dialogue text is as follows: Customer: I would like to learn about financial products with high returns, but not too high risks.

[0119] Sales: There are several financial products with higher returns and moderate risks, such as Product A and Product B.

[0120] Customer: What is the approximate profit of Product A? Is it guaranteed?

[0121] Through the above-mentioned long-short context comparison strategy, the key words are identified from the first dialogue text: high return, moderate risk, and guaranteed return.

[0122] At the same time, the dialogue state is tracked based on the context of the first dialogue text to obtain dialogue state information of the first dialogue text.

[0123] For example, the dialogue state of the first dialogue text is tracked to obtain the following dialogue state information: Intention: Customers prefer low-risk, high-yield financial products First entity: Product A, Product B Furthermore, through the graph planning reasoning strategy, based on the dialogue state information of the first dialogue text, reasoning is performed on the first knowledge graph to obtain first entity information that matches the dialogue state information.

[0124] For example, by reasoning on the knowledge graph in the financial field, we can obtain the return and risk characteristics of product A and the return and risk characteristics of product B.

[0125] Furthermore, based on the keywords, the first entity information, and the corresponding instructions, a second summary of the first conversation text is generated.

[0126] For example, based on the above keywords, the benefits and risk characteristics of Product A and Product B, and the instruction "Generate a product overview that meets the customer's needs," the second summary is as follows: The customer prefers financial products with low risk and high returns. We recommend Product A and Product B. Product A has guaranteed returns but slightly higher risks.

[0127] Furthermore, a multimodal comparison strategy is adopted to determine the consistency score between the second summary and the image based on the distance between the representation vector of the second summary and the representation vector of the image corresponding to the first entity information; if the consistency score is less than a score threshold, the second summary is modified based on the image to obtain a first summary of the first dialogue text; if the consistency score is less than the score threshold, the second summary is determined as the first summary.

[0128] For example, if the distance between the representation vector of the second summary and the representation vector of the image of product A is determined to be less than a threshold value, the summary of product A in the second summary is modified based on the image of product A to obtain the following first summary: The customer prefers financial products with low risk and high returns. We recommend Product A (guaranteed returns, moderate risk) and Product B.

[0129] The summary generation method provided in the embodiments of the present application accurately identifies key sentences in the first dialogue text based on similarities, since the similarity between sentences in the first dialogue text can reflect the correlation between sentences, and the similarity between words in sentences can reflect the correlation between words. Therefore, based on these similarities, key sentences in the first dialogue text can be accurately identified, providing key clues for understanding the main content of the first dialogue text. Furthermore, dialogue state information of the first dialogue text is obtained by tracking the dialogue state based on the context of the first dialogue text, so as to adapt to the changing dialogue environment during the summary generation process for the first dialogue text. Furthermore, graph reasoning is performed on a knowledge graph based on the dialogue state information to obtain entity information that matches the dialogue state information. Because the entity information and the image corresponding to the entity information can provide the background knowledge required to summarize the first dialogue text, they help prevent the presence of untrue content in the generated summary. Therefore, by integrating the keywords of the first dialogue text, the entity information that matches the dialogue state information of the first dialogue text, and the image corresponding to the entity information, multiple modal information complement each other, providing a richer and more comprehensive basis for accurately understanding the semantics of the first dialogue text, capturing the correlation between previous and subsequent sentences in the dialogue text, and changes in the dialogue environment, thereby generating an accurate, truthful, and coherent summary for the first dialogue text.

[0130] In applications, S102 and S108 can be implemented by different models. For example, S104 can be implemented by a Longformer model suitable for processing long texts, or by a model combining BERT and a Gated Recurrent Unit (GRU). BERT extracts pre-trained representations of text, while the GRU helps capture long-range contextual dependencies within the text. Its recurrent structure allows the GRU to process long-range dependencies and effectively reduces information loss in long text processing. S106 can be implemented by a graph convolutional network, and S108 by a multimodal model. In this case, a corresponding loss function can be designed for each model and pre-trained accordingly to optimize the corresponding steps. Specifically, for the model implementing S104, a KL divergence loss function can be designed to represent the loss caused by the difference between the dialogue state information obtained by the model for a sample conversation and the reference dialogue state information. Based on this loss, the model parameters can be adjusted to ensure that the model outputs dialogue state information that is more consistent with the true dialogue state, thereby alleviating the problem of hallucination.

[0131] For the model that implements the above S106, a multi-objective loss function can be designed to represent the loss caused by the difference between the entity information obtained by the model for the sample dialogue text and the reference entity information, and then the parameters of the model are adjusted based on the loss, so that when the model performs reasoning on the knowledge graph, it can weigh the completion of the reasoning task and the reflective adjustment of the reasoning path to ensure the accuracy and breadth of the reasoning.

[0132] For the model implementing S108 above, a cross-entropy loss function can be designed to represent the loss caused by the difference between the summary generated by the model for the sample conversation text and the reference summary. Based on this loss, the model parameters can be adjusted to enable the model to more accurately capture the main content of the conversation text and generate a more accurate summary. Furthermore, a multimodal contrastive loss can be designed to represent the loss caused by the semantic difference between the summary generated for the sample conversation text and the images associated with the sample conversation text. Based on this loss, the model parameters can be adjusted to improve the model's performance on multimodal information processing tasks and ensure that the summary generated for the conversation text is semantically consistent with the associated images.

[0133] Of course, the above S102 and S108 can also be implemented by the same model, such as a multimodal model. In this regard, the embodiment of the present application also provides a model training method for training the model so that the model can generate accurate, true and coherent summaries for any dialogue text, especially long dialogue texts.

[0134] Please refer to Figure 3, is a flow chart of a model training method provided in one embodiment of the present application, the method comprising the following steps: S302: Determine keywords of the second dialogue text based on similarities between sentences and similarities between words in sentences in the second dialogue text.

[0135] The second dialogue text is a dialogue text used as a training sample. The second dialogue text has corresponding annotation data, which may include: a reference summary of the second dialogue text, reference dialogue state information, reference entity information matching the reference dialogue state information, and an image corresponding to the reference entity information.

[0136] The specific implementation of the above S302 is the same as Figure 1 The specific implementation of S102 in the illustrated embodiment is similar and will not be described in detail.

[0137] S304: Track the dialogue state based on the context of the second dialogue text to determine dialogue state information of the second dialogue text.

[0138] The specific implementation of the above S304 is the same as Figure 1 The specific implementation of S104 in the illustrated embodiment is similar and will not be described in detail.

[0139] S306: Obtain second entity information that matches the dialogue state information from the second knowledge graph.

[0140] The nodes in the second knowledge graph represent entities involved in the business to which the second conversation text belongs, and the edges between the nodes represent the relationships between the entities.

[0141] The second knowledge graph can be constructed in various appropriate ways, which are not limited in the embodiments of the present application. In one implementation, entity recognition is performed on documents such as news articles, social media posts, and scientific research documents related to the business scenario to which the second conversation text belongs, to obtain entities in these documents and the relationships between entities, and each entity is treated as a node. Connection edges are established between nodes based on the relationships between entities to obtain a knowledge graph; then, entity recognition is performed on the descriptive text (such as question-answer pairs) of images related to these entities (such as promotional images of a certain product, etc.), and the knowledge graph is updated based on the identified entities and the relationships between entities to obtain a second knowledge graph.

[0142] The specific implementation of the above S306 is the same as Figure 1 The specific implementation of S106 in the illustrated embodiment is similar and will not be described in detail.

[0143] S308 : Generate a first summary of the second conversation text based on the keyword, the second entity information, and the image corresponding to the second entity information.

[0144] In one implementation, S308 includes the following steps: fusing the representation vector of the keyword and the representation vector of the second dialogue text and decoding them to obtain a second summary of the second dialogue text; encoding the second summary to obtain a representation vector of the second summary; determining a consistency score between the second summary and the image based on the distance between the representation vector of the second summary and the representation vector of the image; and modifying the second summary based on the consistency score to obtain a first summary of the second dialogue text.

[0145] The specific implementation of the above S308 is the same as Figure 1 The specific implementation of S108 in the illustrated embodiment is similar and will not be described in detail.

[0146] S310 , adjusting parameters of the model based on the first summary and the reference summary of the second conversation text.

[0147] In one implementation, a first loss is determined based on a first summary of the second conversation text and a reference summary; and parameters of the model are adjusted based on the first loss.

[0148] For example, minimizing the first loss is set as the goal, and the gradient descent algorithm, back propagation algorithm, etc. are used to adjust the parameters of the model.

[0149] In another implementation, considering that the model may deviate from the actual results in various links such as dialogue state tracking, graph reasoning, and final summary generation, these deviations may affect the summary generation effect of the model. Therefore, the losses caused by the deviations in each environment are also considered separately, and the model parameters are adjusted based on these losses to ensure that the model can obtain accurate results in each link, thereby improving the training effect of the model.

[0150] Specifically, the first summary of the second conversation text is obtained by decoding the representation vector of the keyword and the representation vector of the second conversation text to obtain the second summary, and then correcting the second summary based on the distance between the representation vector of the second summary and the representation vector of the image corresponding to the second entity information. In this case, the above S310 includes the following steps: S3101: Determine a first loss based on a first summary of the second conversation text and the reference summary.

[0151] Exemplarily, the first loss may be determined based on a difference between the first summary and a reference summary and a cross-entropy loss function.

[0152] S3102 : Determine a second loss based on the similarity between the representation vector of the second summary and the representation vector of the image.

[0153] Exemplarily, the second loss is determined by the following formula (2): (2) in, represents the second loss, represents the representation vector of the second summary, The representation vector representing the image, Indicates similarity, Represents the temperature parameter of the model.

[0154] S3103: Determine a third loss based on the dialogue state information and the reference dialogue state information of the second dialogue text.

[0155] The third loss can be determined by KL divergence, as shown in the following formula (3): (3) in, represents the third loss, Indicates the conversation status information. Indicates reference to conversation state information, represents the KL divergence, Represents the distribution of dialogue state information output by the model, Represents the reference conversation state information distribution.

[0156] S3104: Determine a fourth loss based on the second entity information and the reference entity information involved in the second conversation text.

[0157] Exemplarily, the first loss may be determined based on a difference between the second entity information and the reference entity information and a cross entropy loss function.

[0158] S3105 , adjusting parameters of the model based on the first loss, the second loss, the third loss, and the fourth loss.

[0159] For example, the weighted sum of the first loss, the second loss, the third loss, and the fourth loss is determined as the total loss of the model. Based on the total loss, the model parameters are adjusted. For example, with minimizing the total loss as the goal, the model parameters are adjusted using a gradient descent algorithm, a backpropagation algorithm, or the like.

[0160] The above describes some implementations of the above S310. Of course, it should be understood that the above S310 can also be implemented in other ways, which are not limited in the present embodiment.

[0161] It is worth noting that the above steps S302 to S310 represent only one round of model training. In practical applications, the model can be trained for multiple rounds until a training stop condition is met. The training stop condition can be set based on actual needs, such as when the number of training rounds reaches a round threshold or when the total loss of the model converges, etc., and this embodiment of the application does not limit this.

[0162] The model training method provided in an embodiment of the present application determines key sentences of the second dialogue text based on the similarity between sentences and the similarity between words in sentences. In addition, the method also tracks the dialogue state based on the context of the second dialogue text to obtain dialogue state information of the second dialogue text, and performs graph reasoning on a knowledge graph based on the dialogue state information to obtain entity information matching the dialogue state information. Furthermore, the method integrates keywords of the second dialogue text, entity information matching the dialogue state information of the second dialogue text, and images corresponding to the entity information, so that multiple modal information complement each other, providing a richer and more comprehensive basis for accurately understanding the semantics of the second dialogue text, capturing the correlation between previous and subsequent sentences in the second dialogue text, and changes in the dialogue environment, thereby generating a summary of the second dialogue text. Finally, based on the summary of the second dialogue text and a reference summary, the model parameters are adjusted so that the model can generate an accurate, true, and coherent summary for any dialogue text, especially long dialogue texts.

[0163] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0164] Based on the same inventive concept, this embodiment of the application also provides a summary generation device. Figure 4 , is a structural diagram of a summary generation device 400 provided in an embodiment of the present application. The device 400 includes: a determination module 410, a state tracking module 420, a query module 430 and a generation module 440.

[0165] The determination module 410 is configured to determine keywords of the first dialogue text based on similarities between sentences in the first dialogue text and similarities between words in the sentences.

[0166] The state tracking module 420 is configured to track the conversation state based on the context of the first conversation text, and determine the conversation state information of the first conversation text.

[0167] The query module 430 is used to obtain first entity information matching the conversation state information from the first knowledge graph; the nodes in the first knowledge graph represent entities involved in the business to which the first conversation text belongs, and the edges between the nodes represent the relationships between the entities.

[0168] The generating module 440 is configured to generate a first summary of the first conversation text based on the keyword, the first entity information, and the image corresponding to the first entity information.

[0169] In another embodiment, the determining module is configured to: determining key sentences of the first dialogue text based on similarities between sentences in the first dialogue text; Determining a first weight of the words in the key sentence based on similarities between the words in the key sentence; Based on the first weights of the words in the key sentence, keywords of the first dialogue text are determined from the words included in the key sentence.

[0170] In another embodiment, when determining the keywords of the first dialogue text from the words included in the key sentence based on the first weights of the words in the key sentence, the determination module performs the following steps: Determining a word in the key sentence whose first weight is greater than a first threshold as a first word; Projecting the first word and the first conversation text into the same semantic space to obtain a semantic vector of the first word and a semantic vector of the first conversation text; determining a relevance between the first word and the first conversation text based on a distance between a semantic vector of the first word and a semantic vector of the first conversation text; A first word whose relevance to the first dialogue text is greater than a second threshold is determined as a keyword of the first dialogue text.

[0171] In another embodiment, the search module is used to: generating a first reasoning task based on the dialogue state information; Determining a first reasoning path on the first knowledge graph that matches the first reasoning task based on nodes and edges between nodes on the first knowledge graph; Based on the nodes on the first reasoning path and the edges between the nodes, first entity information matching the dialog state information is determined.

[0172] In another embodiment, the dialogue state information includes the intent of the first dialogue text and a first entity corresponding to the intent; When the search module determines a first reasoning path on the first knowledge graph that matches the first reasoning task based on the nodes and the edges between the nodes on the first knowledge graph, the search module performs the following steps: Decomposing the first reasoning task to obtain multiple subtasks; For each subtask, taking the node corresponding to the first entity as the starting point, executing the subtask on the first knowledge graph to obtain a second reasoning path corresponding to the subtask; The second reasoning paths corresponding to the subtasks are merged to obtain a first reasoning path that matches the first reasoning task.

[0173] In another embodiment, the generating module is configured to: Encoding the keyword to obtain a representation vector of the keyword; encoding the first conversation text to obtain a representation vector of the first conversation text; Encoding the image to obtain a representation vector of the image; The representation vector of the keyword, the representation vector of the first dialogue text, and the representation vector of the image are decoded to obtain a first summary of the first dialogue text.

[0174] In another embodiment, when the generation module decodes the representation vector of the keyword, the representation vector of the first conversation text, and the representation vector of the image to obtain the first summary of the first conversation text, the generation module performs the following steps: fusing the representation vector of the keyword and the representation vector of the first conversation text and decoding them to obtain a second summary of the first conversation text; Encoding the second digest to obtain a representation vector of the second digest; determining a consistency score between the second summary and the image based on a distance between a representation vector of the second summary and a representation vector of the image; The second summary is modified based on the consistency score to obtain a first summary of the first conversation text.

[0175] Obviously, the summary generation device provided in the embodiment of the present application can be used as Figure 1 The execution body of the summary generation method shown is, for example Figure 1 In the summary generation method shown in FIG. 1 , step S102 can be performed by Figure 4 The summary generating apparatus shown in FIG. 1 is executed by the determining module 410, and step S104 can be performed by Figure 4 The state tracking module 420 in the summary generation device shown in FIG. 1 is executed, and step S106 can be performed by Figure 4 The query module 430 in the summary generation device shown in FIG. 1 is executed, and step S108 can be performed by Figure 4 The generation module 440 in the summary generation device shown is executed.

[0176] According to another embodiment of the present application, Figure 4The various modules in the summary generation device shown can be individually or entirely combined into one or more other modules to form a structure, or one (or more) of the modules can be further divided into multiple functionally smaller modules to form a structure, which can achieve the same operation without affecting the realization of the technical effects of the embodiment of the present application. The above-mentioned units are divided based on logical functions. In actual applications, the functions of one module can also be implemented by multiple modules, or the functions of multiple modules can be implemented by one module. In the embodiment of the present application, the summary generation device can also include other modules. In actual applications, these modules can also be assisted by other modules and can be implemented by the collaboration of multiple modules.

[0177] According to another embodiment of the present application, a general computing device such as a computer including a central processing unit (CPU), a random access memory (RAM), a read-only memory (ROM) and other processing elements and storage elements can be run to execute the following operations: Figure 1 A computer program (including program code) for each step involved in the corresponding method shown in FIG. Figure 4 The computer program can be recorded on a computer-readable storage medium, for example, and transferred to an electronic device through the computer-readable storage medium and run therein.

[0178] Based on the same inventive concept, this embodiment of the application also provides a model training device. Figure 5 , is a structural diagram of a model training device 500 provided in an embodiment of the present application, wherein the device 500 includes: a determination module 510, a state tracking module 520, a query module 530, a generation module 540 and an adjustment module 550.

[0179] The determination module 510 is configured to determine keywords of the second dialogue text based on similarities between sentences in the second dialogue text and similarities between words in the sentences.

[0180] The state tracking module 520 is configured to track the dialogue state based on the context of the second dialogue text, and determine the dialogue state information of the second dialogue text.

[0181] The query module 530 is used to obtain second entity information matching the conversation state information from the second knowledge graph; the nodes in the second knowledge graph represent entities involved in the business to which the second conversation text belongs, and the edges between the nodes represent the relationships between the entities.

[0182] The generating module 540 is configured to generate a first summary of the second conversation text based on the keyword, the second entity information, and the image corresponding to the second entity information.

[0183] The adjustment module 550 is configured to adjust parameters of the model based on the first summary and the reference summary of the second conversation text.

[0184] In another embodiment, the first summary of the second conversation text is obtained by decoding the representation vector of the keyword and the representation vector of the second conversation text to obtain a second summary, and then correcting the second summary based on the distance between the representation vector of the second summary and the representation vector of the image corresponding to the second entity information; The adjustment module is used for: determining a first loss based on the first summary of the second conversation text and the reference summary; determining a second loss based on a similarity between a representation vector of the second summary and a representation vector of the image; determining a third loss based on the dialogue state information and reference dialogue state information of the second dialogue text; determining a fourth loss based on the second entity information and reference entity information involved in the second conversation text; Parameters of a model are adjusted based on the first loss, the second loss, the third loss, and the fourth loss.

[0185] Obviously, the model training device provided in the embodiment of the present application can be used as Figure 3 The execution body of the model training method shown, for example Figure 3 In the model training method shown, step S302 can be performed by Figure 5 The determination module 510 in the model training device shown in FIG. 1 is executed, and step S304 can be performed by Figure 5 The state tracking module 520 in the model training device shown in FIG. 5 is executed, and step S306 can be performed by Figure 5 The query module 530 in the model training device shown in FIG. 1 is executed, and step S308 can be performed by Figure 5 The generation module 540 in the model training device shown in FIG. 1 is executed, and step S310 can be performed by Figure 5 The adjustment module 550 in the model training device is shown to be executed.

[0186] According to another embodiment of the present application, Figure 5The various modules in the model training device shown can be individually or completely combined into one or several other modules to form a whole, or one (or more) of the modules can be further divided into multiple smaller modules to form a whole, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In actual applications, the functions of a module can also be implemented by multiple modules, or the functions of multiple modules can be implemented by one module. In an embodiment of the present application, the model training device may also include other modules. In actual applications, these modules can also be implemented with the assistance of other modules, and can be implemented by the collaboration of multiple modules.

[0187] According to another embodiment of the present application, a general computing device such as a computer including a central processing unit (CPU), a random access memory (RAM), a read-only memory (ROM) and other processing elements and storage elements can be run to execute the following operations: Figure 3 A computer program (including program code) for each step involved in the corresponding method shown in FIG. Figure 5 The model training device shown in the figure and the model training method of the embodiment of the present application are implemented. The computer program can be recorded on a computer-readable storage medium, for example, and transferred to an electronic device through the computer-readable storage medium and run therein.

[0188] Figure 6 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 6 At the hardware level, the electronic device includes a processor and, optionally, an internal bus, a network interface, and memory. The memory may include internal memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for its services.

[0189] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0190] The memory is used to store programs. Specifically, the program may include program code, which includes computer operating instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0191] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming a summary generation device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations: determining keywords of the first dialogue text based on similarities between sentences in the first dialogue text and similarities between words in the sentences; Tracking the conversation state based on the context of the first conversation text to determine conversation state information of the first conversation text; Obtaining first entity information matching the conversation state information from a first knowledge graph; nodes in the first knowledge graph represent entities involved in the business to which the first conversation text belongs, and edges between nodes represent relationships between the entities; A first summary of the first conversation text is generated based on the keyword, the first entity information, and an image corresponding to the first entity information.

[0192] Alternatively, the processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming a model training device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations: determining keywords of the second dialogue text based on similarities between sentences in the second dialogue text and similarities between words in the sentences; performing dialogue state tracking based on the context of the second dialogue text to determine dialogue state information of the second dialogue text; Obtaining second entity information matching the conversation state information from a second knowledge graph; nodes in the second knowledge graph represent entities involved in the business to which the second conversation text belongs, and edges between nodes represent relationships between the entities; generating a first summary of the second conversation text based on the keyword, the second entity information, and an image corresponding to the second entity information; Adjust parameters of the model based on the first summary and the reference summary of the second conversation text.

[0193] The above application Figure 1 The summary generation method disclosed in the embodiment shown or the above-mentioned method of the present application Figure 3 The model training methods disclosed in the illustrated embodiments can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be performed by hardware integrated logic circuits in the processor or by software instructions. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0194] The electronic device may also perform Figure 1 Method, and realize the summary generation device in Figure 1 、 Figure 2 Alternatively, the electronic device may also perform the functions of the embodiment shown. Figure 3 Method, and realize the model training device in Figure 3The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0195] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0196] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, can enable the electronic device to execute Figure 1 The method of the embodiment shown is specifically used to perform the following operations: determining keywords of the first dialogue text based on similarities between sentences in the first dialogue text and similarities between words in the sentences; Tracking the conversation state based on the context of the first conversation text to determine conversation state information of the first conversation text; Obtaining first entity information matching the conversation state information from a first knowledge graph; nodes in the first knowledge graph represent entities involved in the business to which the first conversation text belongs, and edges between nodes represent relationships between the entities; A first summary of the first conversation text is generated based on the keyword, the first entity information, and an image corresponding to the first entity information.

[0197] Alternatively, when the instruction is executed by an electronic device including multiple applications, the electronic device can execute Figure 3 The method of the embodiment shown is specifically used to perform the following operations: determining keywords of the second dialogue text based on similarities between sentences in the second dialogue text and similarities between words in the sentences; performing dialogue state tracking based on the context of the second dialogue text to determine dialogue state information of the second dialogue text; Obtaining second entity information matching the conversation state information from a second knowledge graph; nodes in the second knowledge graph represent entities involved in the business to which the second conversation text belongs, and edges between nodes represent relationships between the entities; generating a first summary of the second conversation text based on the keyword, the second entity information, and an image corresponding to the second entity information; Adjust parameters of the model based on the first summary and the reference summary of the second conversation text.

[0198] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps in the summary generation method or model training method provided in the embodiment of the present application.

[0199] In short, the above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

[0200] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0201] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0202] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0203] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

Claims

1. A method for generating a summary, characterized in that: include: determining keywords of the first dialogue text based on similarities between sentences in the first dialogue text and similarities between words in the sentences; Tracking the conversation state based on the context of the first conversation text to determine conversation state information of the first conversation text; Obtaining first entity information matching the conversation state information from a first knowledge graph; nodes in the first knowledge graph represent entities involved in the business to which the first conversation text belongs, and edges between nodes represent relationships between the entities; A first summary of the first conversation text is generated based on the keyword, the first entity information, and the image corresponding to the first entity information.

2. The method according to claim 1, characterized in that The determining of keywords of the first dialogue text based on similarities between sentences in the first dialogue text and similarities between words in the sentences includes: determining key sentences of the first dialogue text based on similarities between sentences in the first dialogue text; Determining a first weight of the words in the key sentence based on similarities between the words in the key sentence; Based on the first weights of the words in the key sentence, keywords of the first dialogue text are determined from the words included in the key sentence.

3. The method according to claim 2, characterized in that The determining of the keywords of the first dialogue text from the words contained in the key sentence based on the first weights of the words in the key sentence includes: Determining a word in the key sentence whose first weight is greater than a first threshold as a first word; Projecting the first word and the first conversation text into the same semantic space to obtain a semantic vector of the first word and a semantic vector of the first conversation text; determining a relevance between the first word and the first conversation text based on a distance between a semantic vector of the first word and a semantic vector of the first conversation text; A first word whose relevance to the first dialogue text is greater than a second threshold is determined as a keyword of the first dialogue text.

4. The method according to claim 1, wherein The acquiring, from the first knowledge graph, first entity information matching the conversation state information includes: generating a first reasoning task based on the dialogue state information; Determining a first reasoning path on the first knowledge graph that matches the first reasoning task based on nodes and edges between nodes on the first knowledge graph; Based on the nodes on the first reasoning path and the edges between the nodes, first entity information matching the dialog state information is determined.

5. The method according to claim 4, characterized in that The dialogue state information includes the intention of the first dialogue text and the first entity corresponding to the intention; The determining, based on the nodes and edges between the nodes on the first knowledge graph, a first reasoning path on the first knowledge graph that matches the first reasoning task includes: Decomposing the first reasoning task to obtain multiple subtasks; For each subtask, taking the node corresponding to the first entity as the starting point, executing the subtask on the first knowledge graph to obtain a second reasoning path corresponding to the subtask; The second reasoning paths corresponding to the subtasks are merged to obtain a first reasoning path that matches the first reasoning task.

6. The method according to claim 1, characterized in that Generating a first summary of the first conversation text based on the keyword, the first entity information, and the image corresponding to the first entity information includes: Encoding the keyword to obtain a representation vector of the keyword; encoding the first conversation text to obtain a representation vector of the first conversation text; Encoding the image to obtain a representation vector of the image; The representation vector of the keyword, the representation vector of the first dialogue text, and the representation vector of the image are decoded to obtain a first summary of the first dialogue text.

7. The method according to claim 6, characterized in that The decoding of the representation vector of the keyword, the representation vector of the first conversation text, and the representation vector of the image to obtain a first summary of the first conversation text includes: fusing the representation vector of the keyword and the representation vector of the first conversation text and decoding them to obtain a second summary of the first conversation text; Encoding the second digest to obtain a representation vector of the second digest; determining a consistency score between the second summary and the image based on a distance between a representation vector of the second summary and a representation vector of the image; The second summary is modified based on the consistency score to obtain a first summary of the first conversation text.

8. A model training method, characterized in that: include: determining keywords of the second dialogue text based on similarities between sentences in the second dialogue text and similarities between words in the sentences; performing dialogue state tracking based on the context of the second dialogue text to determine dialogue state information of the second dialogue text; Obtaining second entity information matching the conversation state information from a second knowledge graph; nodes in the second knowledge graph represent entities involved in the business to which the second conversation text belongs, and edges between nodes represent relationships between the entities; generating a first summary of the second conversation text based on the keyword, the second entity information, and an image corresponding to the second entity information; Adjust parameters of the model based on the first summary and the reference summary of the second conversation text.

9. The method according to claim 8, characterized in that The first summary of the second conversation text is obtained by decoding the representation vector of the keyword and the representation vector of the second conversation text to obtain a second summary, and then correcting the second summary based on the distance between the representation vector of the second summary and the representation vector of the image corresponding to the second entity information; The adjusting the parameters of the model based on the first summary and the reference summary of the second conversation text includes: determining a first loss based on the first summary of the second conversation text and the reference summary; determining a second loss based on a similarity between a representation vector of the second summary and a representation vector of the image; determining a third loss based on the dialogue state information and reference dialogue state information of the second dialogue text; determining a fourth loss based on the second entity information and reference entity information involved in the second conversation text; Parameters of a model are adjusted based on the first loss, the second loss, the third loss, and the fourth loss.

10. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the summary generation method according to any one of claims 1 to 7 or the model training method according to any one of claims 8 to 9.

Images