Entity-oriented multi-document abstract generation system and method

Abstract

The invention provides a multi-document abstract generation system and method based on entity orientation, and relates to the field of natural language processing. A first encoder is used for effectively analyzing cross-document entity information of a model and analyzing significance of entities, a second encoder is constructed in combination with a hierarchical self-attention mechanism to analyze a cross-document relationship, and a hierarchical cross-attention mechanism is adopted to integrate information of entities, text segments, documents and a Token level into abstract generation. The technical effects that the significant entity in the long text formed by multiple documents can be analyzed, the fact loyalty of the abstract is improved, and the illusion problem is reduced are achieved.

Description

Technical Field

[0001] The present application relates to the field of natural language processing, and in particular to an entity-oriented multi-document summary generation system and method. Background Art

[0002] Text summarization technology refers to the natural language understanding and generation technology that generates a relatively short summary that can summarize the main content of the source text after reading the source text. The generated summary can compress the text content but retain the core information in the source text. Text summarization technology mainly includes extractive summary generation and generative summary generation. The extractive technology is mainly based on statistical ideas. It extracts keywords and sentences recognized by statistical models from existing news texts to form a relatively short summary, while the generative technology generates summaries based on the model's learning and understanding of the source text. Multi-document summarization (MDS) is an effective information aggregation tool that can generate information-rich and concise summaries from a group of documents related to the topic. It has a wide range of applications, such as creating news summaries, product review summaries, and scientific literature summaries. Compared with single-document summaries, it has higher technical difficulty, but also has higher application value.

[0003] Entity-guided summarization refers to the process of generating a contextualized summary that conforms to the entity relationships in the original document by combining the entity information in the document given a document. This is a technology that combines traditional summary generation and entity analysis in documents. Its goal is to improve the quality of generated summaries, especially the factual fidelity to the original text.

[0004] In the existing technology, long texts of multiple documents are a challenge in themselves. Coupled with the sparsity of entity information, this not only increases the difficulty of entity analysis, but may also lead to confusion of entity information in the generated summary.

[0005] Therefore, how to effectively integrate salient entity information and generate more accurate, clear summaries that contain rich entity relationships has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] In order to achieve low-error, low-bias exposure entity relationship extraction in overlapping entity relationship tasks, the present application provides an entity-guided multi-document summary generation system and method.

[0007] In the first aspect, the present application provides an entity-oriented multi-document summary generation method using the following technical solution: A multi-document summary generation system based on entity orientation, comprising: a first encoder, a second encoder and a decoder; The first encoder is used to generate target cross-document information according to the multi-document information when receiving the multi-document information; The second encoder is used to output a set of time steps according to the multi-document information in combination with a hierarchical self-attention mechanism; The decoder is used to receive the target cross-document information and the time step set and combine them with the target summary to generate reasoning summary information using a hierarchical cross-attention mechanism.

[0008] Optionally, the first encoder is further configured to generate text information according to the multiple document information when receiving the multiple document information, and simultaneously obtain entity information from the multiple document information by entity clustering; Performing graph construction on the text segment information and the entity information to generate a graph construction result; A graph attention network is introduced to update the node information in the graph construction result to generate target cross-document information.

[0009] Optionally, the first encoder is further configured to use To represent any node of the graph in the graph construction result, To represent the node representation on any two nodes, the dimension of any node is ,use To represent the node The set of nearby nodes; Perform feature transformation to generate feature representation , ,in is the weight matrix; Calculate the attention coefficient of the feature representation: in is the trained attention mechanism vector, || represents the connection operation, for matrix The derived edge weights; Normalize the attention coefficient: Calculate the updated node representation using the normalized attention coefficient: in express function, is the transformed feature of node b, is the attention coefficient between node a and node b; Use residual links: Each iteration includes a paragraph-to-entity and an entity-to-paragraph update process. After that, the updated text segment is represented as Connect to the corresponding Token representation and get ,Right now .

[0010] Optionally, the second encoder is further used to introduce a pre-trained language model and use Longformer to initialize the attention weight; Building a hierarchical attention mechanism based on the multi-head self-attention in the pre-trained language model; Optionally, the second encoder is also used to decode the document sequence , define the corresponding document as sequence, ,in Represents the document A token in, the quantity is ; The self-attention of the mth document in the document sequence is calculated as: , , in , and Represents the weight parameter matrix, which is optimized during training; Each self-attention layer in the hierarchical attention mechanism is followed by a residual link and then layer normalization: in is the input before self-attention, is the output after self-attention, ; For a feed-forward network: After the residual connection, the layer is normalized: The output of the second encoder is represented by is the total number of time steps A collection of: .

[0011] Optionally, the decoder comprises a first layer cross attention mechanism and a second layer cross attention mechanism; The first layer crisscross attention mechanism is used to generate document-level information crisscross attention according to the time step set and the target summary; In a given documents, the cross-attention score for each individual document in the decoder is expressed as: in Indicates in the document ; Calculate the number of The cross attention weights are: Get the layer normalization scaling factor for each document as: in Indicates Attention scores for documents; Calculate the value of each document The standardized attention weights are: Cross-attention for document-level information.

[0012] Optionally, the second layer cross attention mechanism is also used to set the state of the decoder to , computing entity cluster nodes Attention scores on: in is the weight vector of the attention mechanism, Used to convert decoder status, To convert the embedding of entity cluster nodes, use Activation function, Represents the concatenation of vector matrices; Combined with the attention score of the entity node and the edge weights , to calculate the weighted attention score of the entity to the paragraph node ; use Normalize to get the attention weight : Based on the attention score Select the first K paragraph nodes from high to low, and then apply the attention mechanism to the selected paragraph nodes indivual : in is another attention weight vector, Used to convert the current decoder state , To convert text Embed ; use Normalize to get the attention weight : .

[0013] Optionally, the decoder is further used to convert the attention score at the paragraph level and The attention scores of each level are multiplied to obtain : Compute the context vector for the decoding step : The context vector is a weighted sum, where each Contribution by Adjusted attention scores were determined; Make predictions about vocabulary distribution: Get a target summary, where each Defined as and time step Prediction ,use ; in Indicates that at time step When the model predicts correctly The probability of Indicates the length of the target sequence; For the entire training batch, the loss function is the average of the individual sample losses: in, is the number of samples in the batch, It is The target sequence length is samples, It is The model predicts the probability distribution of samples, It is samples at time step The actual target word; x and y represent the target summary, Are the parameters that need to be learned and trained: For each candidate sequence, calculate , expressed as The cumulative score for the entire sequence is calculated as follows: Reasoning summary information is determined based on the cumulative score.

[0014] In a second aspect, the present application provides an entity-oriented multi-document summary generation method, the entity-oriented multi-document summary generation method comprising: When receiving the multi-document information, the first encoder generates target cross-document information according to the multi-document information; The second encoder outputs a set of time steps based on the multi-document information combined with a hierarchical self-attention mechanism; The decoder receives the target cross-document information and the time step set and combines them with the target summary to generate reasoning summary information using a hierarchical cross-attention mechanism.

[0015] Optionally, when the first encoder receives the multi-document information, the step of generating target cross-document information according to the multi-document information includes: When receiving multiple document information, generating text segment information according to the multiple document information, and acquiring entity information from the multiple document information by entity clustering; Performing graph construction on the text segment information and the entity information to generate a graph construction result; A graph attention network is introduced to update the node information in the graph construction result to generate target cross-document information.

[0016] In summary, the present application includes the following beneficial technical effects: This application uses a first encoder to effectively analyze the model's cross-document entity information and analyze the significance of entities, combines a hierarchical self-attention mechanism to build a second encoder to analyze cross-document relationships, and uses a hierarchical cross-attention mechanism to integrate entity, paragraph, document, and token level information into summary generation, thereby achieving the technical effect of being able to analyze significant entities in long texts consisting of multiple documents, and improving the factual fidelity of summaries and reducing hallucination problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1It is a structural block diagram of the first embodiment of the entity-oriented multi-document summary generation system of the present application.

[0018] Figure 2 This is a diagram of an entity-oriented multi-document summary generation framework based on salient entity analysis in the first embodiment of the entity-oriented multi-document summary generation system of the present application.

[0019] Figure 3 This is a structural diagram of a first encoder of the first embodiment of the entity-oriented multi-document summary generation system of the present application; Figure 4 This is a structural diagram of the second encoder of the first embodiment of the entity-oriented multi-document summary generation system of the present application; Figure 5 This is a decoder structure diagram of the first embodiment of the entity-oriented multi-document summary generation system of the present application; Figure 6 This is a flowchart of the first embodiment of the entity-oriented multi-document summary generation method of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below through the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0021] Reference Figure 1 , Figure 1 This is a structural block diagram of the first embodiment of the entity-oriented multi-document summary generation system of this application.

[0022] like Figure 1 As shown, the entity-oriented multi-document summary generation system proposed in the embodiment of the present application includes: a first encoder 10, a second encoder 20 and a decoder 30; A first encoder 10 is used to generate target cross-document information according to the multi-document information when receiving the multi-document information; A second encoder 20, configured to output a set of time steps according to multi-document information combined with a hierarchical self-attention mechanism; The decoder 30 is used to receive the target cross-document information and the time step set and combine them with the target summary to generate reasoning summary information using a hierarchical cross-attention mechanism.

[0023] It should be noted that multi-document summarization (MDS) is an effective information aggregation tool that can generate information-rich and concise summaries from a set of topic-related documents. It has a wide range of applications, such as creating news summaries, product review summaries, and scientific literature summaries. Compared with single-document summaries, it has higher technical difficulty, but also has higher application value. Different documents cover different perspectives, and multi-document summarization technology attempts to address potential diversity and redundant information, so it is more complicated to implement. It is more difficult to retain the key content in complex input sequences while generating coherent, non-redundant, factually consistent, and grammatically readable summaries. For the research on multi-document text summarization technology, the extractive methods mainly include word frequency statistics method, LDA model, Lexrank and BERT-based extractive summarization technology; the generative technology mainly includes graph model-based methods, Pointer-generator, ABS, CopyNet, BERT-PGN and GPT-2, T5, BART, Longformer and other generative summarization technologies, as well as summary generation or summary extraction technology based on large model technologies such as Llama, GPT-3.5, GPT-4 and Gemma. The existing large models have achieved relatively good results in general summary generation and can generate more fluent and easy-to-accept summaries.

[0024] Entity-guided summarization refers to the process of generating a contextualized summary that conforms to the entity relationships in the original document by combining the entity information in the document given a document. This is a technology that combines traditional summary generation and entity analysis in documents. Its goal is to improve the quality of generated summaries, especially the factual fidelity to the original text. Most existing technologies divide entity-oriented methods into two stages. The first is to obtain entity information in the original text. This stage currently mostly uses pre-trained language models, such as BERT, Roberta and other fine-tuned models that have been recognized by entities, and also uses existing APIs, such as OpenIE and other professional information extraction tools to obtain entities. The second stage is to integrate entity information into the summary generation task. The current methods mainly use entities as keywords and combine the Copy mechanism to increase the proportion of entities in the summary, thereby improving the quality of the summary, or encode the acquired entities and combine them with the summary generation model. The encoding here mainly includes information encoding and feature processing based on graphs and Transformer encoders. There are also methods that model the two stages as a unified knowledge graph construction process, build a knowledge graph from the original text, and combine the knowledge graph to include query, reasoning, encoding and other methods to integrate the information in the knowledge graph into summary generation, thereby guiding the model to generate entity-oriented summaries. The development of large models has also promoted the progress of this task. Methods such as entity chaining and prompt engineering combined with large models such as Mistral and Gemma can also achieve good results. Multi-document summarization faces more problems than single-document summarization. In entity-oriented multi-document summary generation, it is necessary to perform entity analysis on multiple documents and then combine entity information with summary generation. However, as the original text, multiple documents may have decentralized centers, a large number of entities described, and sparse information. Using directly acquired entity information will affect the central summary of multiple documents, and even have a negative impact. It is easy to lose significant entities or even produce hallucinatory entities. Similar to Baidu Encyclopedia and Wikipedia, multi-document summaries are developed around the central significant entity. That is, although the contents described in multiple documents are quite different, the multi-document summary should be developed towards the overall center when no target entity is provided. The center here can be regarded as a significant entity, that is, the main entity described after multiple documents are summarized, and it is also the main element of summary writing. That is, in the case of multiple reference documents, a brief description of an entity similar to that in Baidu Encyclopedia and Wikipedia can be generated.

[0025] The entity-oriented multi-document summarization technology based on salient entity analysis combines the multi-document summarization technology and the entity analysis technology to realize the description of an entity summarized from multiple references in Wikipedia and Baidu Encyclopedia. The entity may be a celebrity or a thing, and the reference documents are mostly documents related to the entity. In the multi-document summarization task, the generated summary can achieve a concise description of the overall content. In practical applications, more attention is paid to the central entity described in multiple documents and the description of the entity, showing a clear description subject to users, similar to Wikipedia, which gives a concise and general entity description based on the reference document. However, the existing entity-oriented multi-document summarization technology mainly adopts the knowledge enhancement method, combines entity information for knowledge enhancement, or directly limits the entities that appear in the summary. It lacks attention to the salient entities in multiple documents and the guiding role they bring. Moreover, the method in a single document does not perform well in multiple documents. The salient entity analysis in multi-document summarization needs to analyze its saliency based on the cross-document relationship between multiple documents. In addition to overcoming the sparsity and redundancy problems brought by long texts, it is also necessary to consider the relationship between multiple documents. How to effectively model cross-document entity information, how to effectively analyze the relationship between the saliency of the entity and the target summary, how to effectively integrate the information obtained from the salient entity analysis into summary generation, and how to use the entity information obtained from the analysis to control the factual fidelity and illusion level in summary generation are key issues.

[0026] It should be noted that the entity-oriented multi-document summary generation framework based on salient entity analysis proposed in this application is as follows Figure 2 As shown, it follows the Transformer-based encoder-decoder architecture. In this embodiment, the first encoder is Figure 2 The graph encoder in the figure, the second encoder is Figure 2 The encoder in .

[0027] Using graph encoder and cross-attention mechanism, entity information and graph representation can be integrated to analyze salient entities. Cross-attention mechanism can explicitly deal with saliency and redundancy issues. The generative model of the pre-trained encoder-decoder architecture combines the graph encoder for salient entity analysis and the improved Transformer encoder and decoder to integrate cross-document information analysis into multi-document summary generation. This method adopts a unified framework to analyze and integrate entities in multiple documents and entity information across documents, and pays special attention to cross-document entity information and relationships to analyze salient entities. The entire framework includes two encoders and one decoder. Most of the parameters of the encoder and decoder come from the pre-trained BART model. The text encoding in the graph encoder also comes from the encoder of the pre-trained language model and the parameters are shared. These encoders and decoders are initialized using Longformer. However, in order to adapt to the analysis and integration of cross-document and entity information in multiple documents, the decoder innovatively adopts hierarchical attention mechanism and hierarchical cross-attention mechanism to improve the original multi-head attention mechanism, thereby better analyzing the key cross-document information in multiple documents, especially the entity information.This application constructs a graph encoder module, an improved Transformer encoder module and an improved Transformer decoder module for the analysis and integration of cross-document information in multiple documents and entity information therein. The graph encoder is used to analyze cross-document information in multiple documents, including text and entity information, and the hierarchical attention mechanism is used to realize the analysis of document information in cross-documents. Most of the parameters of the encoder and decoder modules based on Transformer are initialized from BART-large to Longformer, which can obtain information at the Token level, and BART-large, as a pre-trained generation model, has good results in summary generation tasks. Using Longformer for initialization can break through the original BART input length limit, which can be set to 4096 Tokens or even up to 8000 Tokens. Fine-tuning after parameter initialization can optimize the parameters of the newly added multi-head hierarchical attention mechanism, and most parameters are still retained. Maintaining the parameters of the original pre-trained language model can greatly reduce the training cost of new tasks. The Transformer-based encoder and decoder incorporate a hierarchical attention mechanism. In the multi-document summary task, more tokens need to be analyzed, especially in the analysis of cross-document information and entity information. The cost will be relatively high and the attention will be relatively scattered. Based on the original attention, it is improved to a hierarchical attention mechanism. The hierarchical attention in the encoder can analyze document-level information, and the hierarchical cross-attention mechanism in the decoder can integrate the updated text and entity node information in the graph encoder, and then use cross-attention to analyze key text and significant entities, and integrate them into summary generation. The two-level decoding process of the hierarchical cross-attention mechanism in the decoder first focuses on several central entity cluster nodes. These nodes can be regarded as indicators of significance, to combine text information to realize significant entity analysis, and then select text in combination with significant entity information analysis. On the basis of significance analysis, certain redundant information is removed, reducing the negative impact of redundant information on summary generation.

[0028] In one embodiment, the first encoder 10 is also used to generate text information based on the multi-document information when receiving multi-document information, and at the same time obtain entity information in the multi-document information by entity clustering; perform graph construction on the text information and entity information to generate a graph construction result; introduce a graph attention network to update the node information in the graph construction result to generate target cross-document information.

[0029] In one embodiment, the first encoder 10 is further configured to use To represent any node in the graph construction result, To represent the node representation on any two nodes, the dimension of any node is ,use To represent the node A collection of nearby nodes; feature transformation to generate feature representation , ,in is a weight matrix; calculate the attention coefficient of the feature representation: in is the trained attention mechanism vector, || represents the connection operation, for matrix Derived edge weights; normalized attention coefficients: Calculate the updated node representation using the normalized attention coefficient: in express function, is the transformed feature of node b, is the attention coefficient between node a and node b; using residual link: Each iteration includes a paragraph-to-entity and an entity-to-paragraph update process. After that, the updated text segment is represented as Connect to the corresponding Token representation and get ,Right now .

[0030] In the specific implementation, Figure 3 The graph encoder structure diagram shown in the figure, the multi-document summarization task can be modeled as: for multiple documents , which requires obtaining a summary based on multiple document analyses ,in Indicates summary No. Tokens, because this embodiment needs to analyze paragraph content, that is, text information, which is more related to entity information. The number of entities will also be controlled within a certain number for easy analysis. , using line breaks in the document to divide the source document into multiple segments, where the segments are represented by For the text, the entities in it are used as the semantic units of analysis. This is because most previous studies use words as semantic units to enrich the relationship between sentences or texts, but this will bring huge computational costs. For multi-document summarization tasks, the model is usually required to process a large number of documents, and the number of words will be large, which will bring huge costs and obstacles to the construction and update process of the graph. Entity clustering can be used as the semantic unit of analysis. This application uses Spacy to obtain entities and coreference resolution. In order to analyze the cross-document information of entities, the acquisition of entities requires global analysis, that is, all documents The complete long text is concatenated and then analyzed using Spacy. The extracted entity clusters are represented as ,in ,in Represents a cluster The number of times the entity is mentioned in the document, and then the text and entity information are constructed into a graph. In the text information construction, all the texts in the document are constructed into , and then construct a heterogeneous graph , Include paragraph nodes as nodes on the graph and entity cluster nodes , Represents an undirected edge between nodes. There are no edges within paragraph nodes and entity cluster nodes. Only edges exist between these nodes. Connecting two nodes is connecting paragraph nodes. and entity cluster nodes The edge represents the paragraph and There is an inclusion relationship between them, that is, the text will contain the entities mentioned in this entity cluster. Here we also need to build a matrix Indicates that the values ​​in the matrix Refers to the text Include Entities in and mentioned times, among which , where the number of mentions is used to indicate the importance of the entity in the passage, which can reflect the prominence of the entity to a certain extent. Based on this matrix Continue calculation matrix , modeling the importance of the relationship between entity clusters and paragraphs, and the dimensions of both matrices are , that is, the number of whole paragraphs and the number of entity clusters. Based on the above analysis, the graph can be constructed.

[0031] For the context encoding of the text segment, the encoder is constructed with the encoder parameters of the BART-large pre-trained language model initialized with Longformer , the parameters are initialized with the pre-trained BART-large model, and the structure is consistent with it. It is used as a context encoder to obtain information at the paragraph level. The input paragraph is , that is, the document The passage in , the text segment calculated by the encoder is represented as , which is the text representation at the Token level, with a size of , then through The text is weighted pooled. After this step of calculation, a fixed-length and information-rich text representation can be obtained. The multi-head pooling layer calculates the weight distribution on the tag, allowing the model to flexibly encode paragraphs in different representation subspaces according to different heads. After calculation, After the above processing, we can get , which is used to represent the feature matrix at the paragraph level, and the dimension is expressed as ,in Expressed as the number of paragraphs, Represents the dimension of the hidden layer, that is Parameters of the model.

[0032] Similarly, the same structure is used to encode entities. In order to encode the text and entities in a unified space, the encoder parameters here implement the parameter sharing mechanism in this embodiment, but the pooling layer does not implement parameter sharing in order to distinguish different attentions. Two independent layers are used for encoding, and then similar to the previous one, we can get , the dimension of any node is expressed as ,in It indicates the number of entity clusters obtained. Indicates the dimension of the hidden layer.

[0033] Used in graph encoders That is, the graph attention network updates the representation on the node, using To represent any node in the graph, To represent the node representation on any two nodes in the graph, the dimension of any node is ,use To represent the node The set of nearby nodes, The calculation of is as follows: The first is feature transformation, applying a nonlinear function before calculating the attention: , ,in is the weight matrix, which needs to be adjusted during training.

[0034] Then calculate the attention coefficient of the connected feature representation: in is the trained attention mechanism vector, || represents the connection operation, for matrix The derived edge weights are because The matrix value is more important for analyzing the cross-segment information and cross-segment information of the entity, so the value of the matrix is ​​included in the update mechanism in the iterative update of the node representation, and then the attention coefficient is normalized: Finally, these normalized attention coefficients are used to calculate the updated node representation: in express function, is the transformed feature of node b, is the attention coefficient between node a and node b.

[0035] Then use the residual module for processing, that is, use the residual connection: GATv2 layers and normalization and feed-forward layers are used to iteratively update node representations.

[0036] Each iteration consists of a paragraph-to-entity and an entity-to-paragraph update. After that, the updated text segment is represented as Connect to the corresponding Token representation and get , that is .

[0037] In one embodiment, the second encoder 20 is also used to introduce a pre-trained language model and use Longformer to initialize attention weights; and to construct a hierarchical attention mechanism based on the multi-head self-attention in the pre-trained language model.

[0038] In the specific implementation, Figure 4As shown in the encoder structure diagram, the main part of this module refers to the pre-trained language model BART, and Longformer is used to initialize the attention weight. However, on the original BART model, the model pre-trained by the generative task shows good results in single-text summaries, but it is difficult to deal with the information redundancy and sparsity problems brought by multiple documents in the multi-document summary generation task, and it is also difficult to model the relationship between cross-documents, which will have a greater impact on the quality of the summary. Since the self-attention of the pre-trained language model is only applied to tokens from the same consistent source during pre-training, it cannot handle cross-document representations well. Therefore, in order to comply with the pre-training process, this application constructs a hierarchical attention mechanism based on multi-head self-attention to limit the tokens referenced in the self-attention mechanism. After the analysis of this attention, the hierarchical attention mechanism can only see the words in its own document, thereby avoiding information leakage or being affected by out-of-context or contradictory information in other documents. In previous methods, tokens can be affected by tokens from other documents, which will be negatively affected by redundant information, thereby affecting the quality of the generated summary.

[0039] The encoding here uses <s> To start the document, use< / s> To mark the end of the document, use <s>As the beginning of each document, of course, the beginning position of the following document means the end position of the previous document. This application uses this symbol <s>To obtain document information, use it as a document representation to promote cross-document relations in the encoder and decoder. The input is as follows: Here, a hierarchical self-attention mechanism is used to limit As a representation of the entire document, and most importantly, the restrictions Only focusing on other document start symbols, rather than on the tokens of other documents, this mechanism is an improvement over the original self-attention mechanism. It can interact with information at the document level and analyze cross-segment information and relationships. The position embedding in front of the encoder is combined with different documents to adjust the start symbol and token in a single document for position embedding encoding. That is, for each document, the position encoding is from <s>The positional encoding is re-encoded at the beginning of a new document so that the information can be passed to the encoder, that is, the next Token comes from the new document, not the continuous paragraph of the previous document.

[0040] First, at the Tokenizer position, the long text consisting of multiple documents needs to be encoded, that is, the Embedding layer includes The difference between the encoder and the original one is that the position encoding is different. The tokens in the same document can use the document number, and then the two are summed to get the final The difference between the encoder and the original encoder is the use of hierarchical self-attention. Different levels are used to identify different documents, and only the tokens in the same document are calculated for self-attention. The calculation of hierarchical attention is different in that the token embedding of a long text composed of multiple documents is represented as , the position encoding is expressed as , the position code of each document is incremented starting from 0, that is, in each document <s>It is reset to 0 at the beginning, incremented for each subsequent Token in each document, and then the hierarchical self-attention is calculated: For a sequence of documents , the document can be defined as sequence, ,in Representation Document A token in the number is , with the Taking a document as an example, this embodiment uses document-level self-attention calculation: , , Among them , and Represents the weight parameter matrix, which is optimized during training.

[0041] After residual connection, layer normalization and fully connected feedforward network processing, the encoder output is finally obtained. Each self-attention layer is followed by a residual link followed by layer normalization in is the input before self-attention, is the output after self-attention, For the subsequent feed-forward network: The encoder also needs residual connections and layer normalization processing: The final encoder output representation is the total number of time steps A collection of: In one embodiment, the second encoder 20 is further configured to decode the document sequence , define the corresponding document as sequence, ,in Representation Document A token in, the quantity is ; The self-attention calculation for the mth document in the document sequence is: , , in , and Represents the weight parameter matrix, which is optimized during training; Each self-attention layer in the hierarchical attention mechanism is followed by a residual link and then layer normalization is performed: in is the input before self-attention, is the output after self-attention, ; For a feed-forward network: After the residual connection, the layer is normalized: The output of the second encoder is represented as is the total number of time steps A collection of: .

[0042] In one embodiment, the decoder 30 includes a first layer cross attention mechanism and a second layer cross attention mechanism; the first layer cross attention mechanism is used to generate document-level information cross attention according to the time step set and the target summary; documents, the cross-attention score for each individual document in the decoder is expressed as: in Indicates in the document ; Calculate the number of The cross attention weights are: Get the layer normalization scaling factor for each document as: in Indicates The attention scores of each document are calculated. The standardized attention weights are: Cross-attention for document-level information.

[0043] In the specific implementation, Figure 5 As shown in the decoder structure diagram, in multi-document summarization, the input source document will involve a large number of tokens. If the decoder calculates the attention weights of all tokens, the computational cost will be relatively high, and the attention will be relatively dispersed during the calculation process. Therefore, the decoder constructs a hierarchical cross-attention mechanism.

[0044] The decoder contains two layers of hierarchical cross-attention mechanism. During the training phase, the mechanism processes the encoder output through the decoder self-attention and normalization layers based on the encoder output and the target summary. The first layer of the hierarchical cross-attention mechanism, given documents, the cross-attention score for each individual document in the decoder is expressed as: in Indicates in the document , here we calculate the The key here is to distinguish the levels of each document. Calculation of criss-cross attention weights.

[0045] Next, with the help of <s>The symbol is used to determine the relative importance of each document, and then the cross attention weights are scaled accordingly to obtain the layer normalization scaling factor for each document: in Indicates in the document <s>The attention score of the symbol, that is, the attention score of the beginning symbol of the document, and then calculate the The standardized attention weights are: In this way, the decoder can better grasp the relative importance of each document, while retaining the relative cross-attention weights within the same document during the decoding process, thereby capturing document-level informative cross-attention.

[0046] After that, it passes through the normalization layer, and then the second layer of hierarchical cross attention, which processes the information from the graph encoder and processes the information at the paragraph and entity levels. This layer uses a two-layer decoding mechanism to first focus on the entity cluster nodes, which are regarded as saliency indicators in our model. Attention also includes cross-attention mechanism analysis at the paragraph level. At each decoding step, the state of the decoder is , computing entity cluster nodes Attention score on: in is the weight vector of the attention mechanism, and is a trainable weight matrix used to transform the decoder state and the embedding of the entity cluster nodes, respectively, using Activation function, Represents the concatenation of vector matrices. The calculation is to combine attention to evaluate the importance of each entity cluster node to the current decoder state.

[0047] This formula is to combine the attention scores of entity nodes and the edge weights , to calculate the weighted attention score from entity to paragraph node . Then use Normalize to get the attention weight , that is, describing the relative importance of each text node for the decoding process: Then select the attention score The top K paragraph nodes with the highest score, and then apply the attention mechanism to the selected paragraph nodes indivual : in is another attention weight vector, and is the weight matrix used to transform the current decoder state and paragraph Embed , and then use Normalize to get the attention weight : Next, we use the paragraph-level attention score and The attention scores of each level are multiplied to get The importance of paragraph level and individual The importance of Attention: Next, the context vector for the decoding step is calculated : The context vector is a weighted sum, where each The contribution of is determined by its adjusted attention score.

[0048] Finally, we can predict the vocabulary distribution: For each That is , and the model at the corresponding time step Prediction , using the negative log-likelihood loss function for training and learning. The training phase uses Teacher-Forcing, directly using the real previous Token as the input for the next time step. in Indicates that at time step When the model predicts correctly The probability of Indicates the length of the target sequence.

[0049] For the entire training batch, the loss function is the average of the individual sample losses: in, is the number of samples in the batch, It is The target sequence length is samples, It is The model predicts the probability distribution of samples, It is samples at time step The actual target word. The training phase is to minimize this loss function. The training process follows the traditional sequence-to-sequence modeling. and Indicates that it comes from the data set Target summary in - multiple document pairs, Are the parameters that need to be learned and trained: The reasoning phase uses the beam search algorithm, also known as beam search, for each time step , generating an extension from each candidate sequence at the previous time step, starting with <s>, construct candidate sequences, and for each candidate sequence use the decoder to generate the next possible The probability distribution of , for each candidate sequence, the calculation includes the newly generated , expressed as The cumulative score of the entire sequence is calculated as follows: Among all generated candidate sequences, select the top sequence with the highest score. sequences as candidate sequences for the next time step, if the latest Is the end symbol< / s> Or if the maximum length is reached, it is removed from the candidate sequence and saved as the completed sequence. If the condition is met, the decoding process is stopped, and the sequence with the highest score is selected from the completed sequence according to the final score as the final generated summary.

[0050] In one embodiment, the decoder 30 is further configured to set the state of the decoder to , computing entity cluster nodes Attention scores on: in is the weight vector of the attention mechanism, Used to convert decoder status, To convert the embedding of entity cluster nodes, use Activation function, Represents the concatenation of vector matrices; Combined with the attention scores of entity nodes and the edge weights , to calculate the weighted attention score from entity to paragraph node ;use Normalize to get the attention weight : Based on the attention score Select the first K paragraph nodes from high to low, and then apply the attention mechanism to the selected paragraph nodes indivual : in is another attention weight vector, Used to convert the current decoder state , To convert text Embed ;use Normalize to get the attention weight : .

[0051] In one embodiment, the decoder 30 is further configured to convert the attention score at the paragraph level into and The attention scores of each level are multiplied to obtain : Compute the context vector for the decoding step : The context vector is a weighted sum, where each Contribution by The adjusted attention score is determined; the vocabulary distribution is predicted: Get the target summary. For each Defined as and time step Prediction ,use ; in Indicates that at time step When the model predicts correctly The probability of represents the length of the target sequence; for the entire training batch, the loss function is the average of the single sample loss: in, is the number of samples in the batch, It is The target sequence length is samples, It is The model predicts the probability distribution of samples, It is samples at time step The actual target word; x and y represent the target summary, Are the parameters that need to be learned and trained: For each candidate sequence, calculate , expressed as The cumulative score for the entire sequence is calculated as follows: The inference summary information is determined based on the cumulative scores.

[0052] It should be noted that this embodiment covers a graph encoder for salient entity analysis, a Transformer-based encoder and decoder that integrates information at different levels of entities, paragraphs and documents. The main advantages and innovations of this technology include: this embodiment combines the hierarchical self-attention mechanism and the hierarchical cross-attention mechanism on the basis of the pre-trained language model to improve the encoder and decoder, so that it can analyze information at different levels, especially cross-document information, reduce the information redundancy caused by the long text composed of multiple documents, combine the entity information and paragraph information in the data to analyze the cross-paragraph and document information of the entity, construct a graph encoder adapted to the task and use the graph attention mechanism to update the node information, so as to analyze the saliency of the entity as a whole framework during the training process, and merge the output information of the graph encoder and the encoder into the decoder. The decoder innovatively uses two hierarchical cross-attention mechanism modules to complete the integration of information at the entity, paragraph, document and token levels into summary generation, that is, the integration in the decoder, realizing a comprehensive information capture and integration framework, especially the understanding and utilization of salient entities. By treating entities as central nodes of information flow, this embodiment can not only identify and weight the importance of each entity, but also effectively track the development trajectory of entities in a multi-document environment. This is critical for building coherent and informative summaries, especially in scenarios that require cross-document consistency and accurate delivery of entity information. In addition, this technology uses a graph encoder to analyze not only the internal relationships of entities, but also the association of entities with other textual content, such as paragraph and document level connections. This multi-level information processing method can enhance the model's grasp of context, so that when generating summaries, it can more accurately reflect the core issues and events in the document set. At the same time, the application of hierarchical cross-attention mechanism modules in the decoding process further optimizes the selection and integration of information, making the summary not only compact but also highly relevant in content. More importantly, this technology strengthens the focus of the summary on salient entities through refined entity-level analysis. This means that when faced with large amounts of data, the model can more effectively filter out key information, thereby providing users with more direct and accurate information excerpts.

[0053] In summary, the innovation of this embodiment lies in the unique application of hierarchical self-attention and hierarchical cross-attention mechanisms, as well as the innovative integration of graph encoders in entity importance analysis. These features not only make the summary generation process more intelligent and adaptable, but also significantly improve the quality and usability of the generated summaries. These innovations give this technology significant technical advantages in the field of entity-oriented summary generation and text generation, and have good application value in scenarios such as Wikipedia generation.

[0054] This embodiment determines the target entity information by overlapping multiple entity markers when receiving the target to be extracted; obtains entity category information through the classification layer; injects the entity category information into the preset rule module to determine the candidate triple set; determines the target original output information by combining the candidate triple set through a complex labeling method; and inputs the target original input information into the preset relationship extraction model to obtain the entity relationship extraction result. It is achieved that all entities can be extracted at the same time and entity category information is given, solving the nested entity problem. It is further achieved that the entity extraction effect can reduce errors and mitigate bias exposure.

[0055] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present application. In specific applications, technicians in this field can make settings as needed, and the present application does not impose any limitation on this.

[0056] The present application embodiment provides a method for generating multi-document summaries based on entity orientation, referring to Figure 6 , Figure 6 This is a flowchart of the first embodiment of the entity-oriented multi-document summary generation method of the present application.

[0057] In this embodiment, the entity-oriented multi-document summary generation method includes the following steps: Step S10: When receiving the multi-document information, the first encoder generates target cross-document information according to the multi-document information.

[0058] Furthermore, in order to reasonably generate target cross-document information, the first encoder generates target cross-document information based on the multi-document information when receiving multi-document information, including: generating text information based on the multi-document information when receiving multi-document information, and obtaining entity information in the multi-document information by entity clustering; performing graph construction on the text information and entity information to generate a graph construction result; introducing a graph attention network to update the node information in the graph construction result to generate target cross-document information.

[0059] Step S20: The second encoder outputs a set of time steps based on multi-document information combined with a hierarchical self-attention mechanism.

[0060] Step S30: The decoder receives the target cross-document information and the time step set and combines them with the target summary to generate reasoning summary information using a hierarchical cross-attention mechanism.

[0061] This embodiment uses the first encoder to effectively analyze the model's cross-document entity information and the significance of the entity, and combines the hierarchical self-attention mechanism to build a second encoder to analyze cross-document relationships. The hierarchical cross-attention mechanism is used to integrate the information at the entity, paragraph, document, and token levels into summary generation, thereby achieving the technical effect of being able to analyze the significant entities in a long text consisting of multiple documents, and improving the factual fidelity of the summary and reducing the hallucination problem.

[0062] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present application. In practical applications, technicians in this field can select part or all of it according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0063] In addition, for technical details not fully described in this embodiment, please refer to the entity-oriented multi-document summary generation method provided in any embodiment of the present application, which will not be repeated here.

[0064] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0065] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0066] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0067] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.< / s> < / s> < / s> < / s> < / s> < / s>

Claims

1. A multi-document summary generation system based on entity orientation, characterized in that: include: a first encoder, a second encoder, and a decoder; The first encoder is used to generate target cross-document information according to the multi-document information when receiving the multi-document information; The second encoder is used to output a set of time steps according to the multi-document information in combination with a hierarchical self-attention mechanism; The decoder is used to receive the target cross-document information and the time step set and combine them with the target summary to generate reasoning summary information using a hierarchical cross-attention mechanism.

2. The entity-oriented multi-document summary generation system according to claim 1 is characterized in that: The first encoder is further configured to generate text information according to the multiple document information when receiving the multiple document information, and simultaneously obtain entity information from the multiple document information by entity clustering; Performing graph construction on the text segment information and the entity information to generate a graph construction result; A graph attention network is introduced to update the node information in the graph construction result to generate target cross-document information.

3. The entity-oriented multi-document summary generation system according to claim 2 is characterized in that: The first encoder is also used to use To represent any node of the graph in the graph construction result, To represent the node representation on any two nodes, the dimension of any node is ,use To represent the node The set of nearby nodes; Perform feature transformation to generate feature representation , ,in is the weight matrix; Calculate the attention coefficient of the feature representation: in is the trained attention mechanism vector, || represents the connection operation, for matrix The derived edge weights; Normalize the attention coefficient: Calculate the updated node representation using the normalized attention coefficient: in express function, is the transformed feature of node b, is the attention coefficient between node a and node b; Use residual links: Each iteration includes a paragraph-to-entity and an entity-to-paragraph update process. After that, the updated text segment is represented as Connect to the corresponding Token representation and get ,Right now .

4. The entity-oriented multi-document summary generation system according to claim 1, characterized in that: The second encoder is also used to introduce a pre-trained language model and use Longformer to initialize the attention weight; A hierarchical attention mechanism is constructed based on the multi-head self-attention in the pre-trained language model.

5. The entity-oriented multi-document summary generation system according to claim 4 is characterized in that: The second encoder is also used to , define the corresponding document as sequence, ,in Represents the document A token in, the quantity is ; The self-attention of the mth document in the document sequence is calculated as: ， ， in , and represents the weight parameter matrix; Each self-attention layer in the hierarchical attention mechanism is followed by a residual link and then layer normalization: in is the input before self-attention, is the output after self-attention, ; For a feed-forward network: After the residual connection, the layer is normalized: The output of the second encoder is represented by is the total number of time steps A collection of: 。 6. The entity-oriented multi-document summary generation system according to claim 1, characterized in that: The decoder comprises a first layer crisscross attention mechanism and a second layer crisscross attention mechanism; The first layer crisscross attention mechanism is used to generate document-level information crisscross attention according to the time step set and the target summary; In a given documents, the cross-attention score for each individual document in the decoder is expressed as: in Indicates in the document ; Calculate the number of The cross attention weights are: Get the layer normalization scaling factor for each document as: in Indicates Attention scores for documents; Calculate the value of each document The standardized attention weights are: Cross-attention for document-level information.

7. The entity-oriented multi-document summary generation system according to claim 6, characterized in that: The second layer of cross attention mechanism is also used to set the state of the decoder to , computing entity cluster nodes Attention scores on: in is the weight vector of the attention mechanism, Used to convert decoder status, To convert the embedding of entity cluster nodes, use Activation function, Represents the concatenation of vector matrices; Combined with the attention score of the entity node and the edge weights , to calculate the weighted attention score of the entity to the paragraph node ; use Normalize to get the attention weight : According to the attention weight Select the first K paragraph nodes from high to low, and then apply the attention mechanism to the selected paragraph nodes indivual : in is another attention weight vector, Used to convert the current decoder state , To convert text Embed ; use Normalize to get the attention weight : 。 8. The entity-oriented multi-document summary generation system according to claim 7, characterized in that: The decoder is also used to convert the attention score at the paragraph level and The attention scores of each level are multiplied to obtain : Compute the context vector for the decoding step : The context vector is a weighted sum, where each Contribution by Adjusted attention scores were determined; Make predictions about vocabulary distribution: Get a target summary, where each Defined as and time step Prediction ,use ; in Indicates that at time step When the model predicts correctly The probability of Indicates the length of the target sequence; For the entire training batch, the loss function is the average of the individual sample losses: in, is the number of samples in the batch, It is The target sequence length is samples, It is The model predicts the probability distribution of samples, It is samples at time step The actual target word; x and y represent the target summary, Are the parameters that need to be learned and trained: For each candidate sequence, calculate , expressed as The cumulative score for the entire sequence is calculated as follows: Reasoning summary information is determined based on the cumulative score.

9. A method for generating multi-document summaries based on entity orientation, characterized in that: The entity-oriented multi-document summary generation method comprises: When receiving the multi-document information, the first encoder generates target cross-document information according to the multi-document information; The second encoder outputs a set of time steps based on the multi-document information combined with a hierarchical self-attention mechanism; The decoder receives the target cross-document information and the time step set and combines them with the target summary to generate reasoning summary information using a hierarchical cross-attention mechanism.

10. The entity-oriented multi-document summary generation method according to claim 9, characterized in that: When the first encoder receives the multi-document information, the step of generating target cross-document information according to the multi-document information includes: When receiving multiple document information, generating text segment information according to the multiple document information, and acquiring entity information from the multiple document information by entity clustering; Performing graph construction on the text segment information and the entity information to generate a graph construction result; A graph attention network is introduced to update the node information in the graph construction result to generate target cross-document information.

Images