A generative multi-document summarization method using entity explicit graph

Abstract

The present application provides a generative multi-document summarization method using Entity explicit graph to solve the problems of the prior art. The Entity graph based on SimBert is used instead of the traditional document similarity graph to express the connection between documents. Entity extraction is performed on each paragraph, and the Entity of each paragraph is spliced into a sentence. Then, the cosine similarity between all paragraphs is calculated using SimBert. After threshold filtering, an explicit graph representing the connection between paragraphs is obtained. The graphing method based on neural networks and aimed at the entities within the document is more effective than the inflexible rules defined by humans. In addition, the present application improves the attention mechanism and hierarchical graph attention mechanism of the graph perception, introduces a gating mechanism and a residual connection, so that the explicit graph and implicit graph information can be better integrated, and the implicit relationship learned by the attention mechanism is reserved with a residual path to ensure that the information learned by the network occupies a more important position, thereby guiding the generative multi-document summarization.

Description

Technical Field

[0001] This invention relates to the field of computer natural language processing technology, and in particular to a generative multi-document summarization method using an explicit entity graph. Background Technology

[0002] With the rapid development of society, the amount of data on the internet is growing exponentially. The internet provides many convenient services, and the forms of information are becoming increasingly diverse, including images, audio, and video in addition to text. However, due to the relatively low cost of text transmission, text remains the most widely disseminated form. The value of text lies not in its quantity but in the absorption and refinement of textual data. Due to the rapid increase in data volume, analyzing and understanding text has become an extremely tedious task. From the reader's perspective, even with initial filtering through search engines, the result is still an excessive amount of information. Extracting significant information from massive amounts of textual data is a labor-intensive task. Therefore, the need for automated processing and refinement of textual data has arisen. This necessitates that machines be trained with the help of large amounts of data to learn human abilities to understand and process textual information—the automatic text summarization technology studied in this paper, also known as automatic text extraction.

[0003] Automatic text summarization is an important branch of natural language processing. In this age of information overload, people urgently need data reduction, but manual summarization is clearly an impossible task. Automatic summarization solves this problem by simplifying information. Automatic summarization technology can automatically convert texts or collections of texts containing the same topic into concise summaries that include core semantic information, filtering out redundant information and reflecting the key points of the content. The summaries are significantly shorter than the original text. The significance of automatic text summarization lies not only in helping ordinary users quickly extract information, but also in assisting many downstream artificial intelligence applications, such as creating news summaries and generating reports. Furthermore, this technology also plays a supporting role in information retrieval (IR) and data mining tasks.

[0004] Automatic text summarization has been developing for over 70 years. Currently, it has a wider range of applications, extending beyond just text summarization in the narrow sense to include summarizing "text" from multi-source data structures, such as extracting summaries from video and audio formats. While significant progress has been made in this field, its quality still lags behind that of human summarization.

[0005] Based on the number of input documents, text summarization can be categorized into single-document summarization and multi-document summarization. Multi-document summarization is more complex and challenging than single-document summarization because the information between documents is more diverse and conflicting. The number of documents is typically large, and the relationships between them are complex. With such a large number of documents, there are bound to be complementary, overlapping, and conflicting elements. Furthermore, excessively long input documents often lead to model degradation. Generating coherent, non-redundant, factually accurate, and grammatically readable summaries while preserving the most important content from a complex input sequence is a significant challenge. Therefore, multi-document summarization requires models with stronger capabilities to analyze corpora and identify and merge consistent information.

[0006] Existing multi-document summarization methods suffer from poor generation results. Summary of the Invention

[0007] This invention addresses the shortcomings of existing technologies by providing a generative multi-document summarization method using explicit entity graphs. It proposes a SimBert-based entity graph to replace traditional document similarity graphs for expressing document relationships. Entities are extracted from each paragraph, and the entities from each paragraph are concatenated into a sentence. SimBert is then used to calculate the cosine similarity between all paragraphs. After thresholding, an explicit graph representing paragraph relationships is obtained. This neural network-based graph construction method, which focuses on entities within the document, outperforms inflexible, manually defined rules. Furthermore, this invention improves graph-aware attention mechanisms and hierarchical graph attention mechanisms by introducing gating mechanisms and residual connections. This allows for better integration of explicit and implicit graph information and preserves residual paths for implicit relationships learned by the attention mechanism, ensuring that the information learned by the network has a more significant role in guiding generative multi-document summarization.

[0008] The technical solution adopted in this invention is as follows:

[0009] The first aspect provides a generative multi-document summarization method using an explicit entity graph. A generative multi-document summarization model is pre-built, comprising a BERT pre-trained model input module, a character-level encoding Transformer module, a global graph encoding module, and a graph decoding layer module. The method includes:

[0010] After dividing the multiple documents corresponding to the human summary into multiple paragraphs and preprocessing them, the input paragraphs are obtained.

[0011] The Bert pre-trained model input module obtains preliminary character embeddings based on the characters in the input paragraph.

[0012] Input the initial character embedding into the character-level encoding Transformer module in units of paragraphs to obtain the initial embedding of each paragraph and the embedding of all characters after information exchange within the paragraph;

[0013] The initial embedding of each paragraph is input into the global graph encoding module. The global graph encoding module replaces the multi-head attention mechanism in the Transformer encoder structure with a graph-aware attention mechanism and introduces an explicit graph structure. By combining the explicit graph information with the learned implicit relationships, information interaction between paragraphs is carried out to obtain the paragraph embedding after paragraph information interaction.

[0014] The initial character embedding, the embeddings of all characters after information exchange within the paragraph, and the paragraph embedding after paragraph information exchange are input into the graph decoding layer module. The graph decoding layer module replaces the multi-head attention module of the Transformer decoder with a hierarchical graph attention module. The graph decoding layer module is divided into upper and lower layers. The upper layer is a global graph attention mechanism, which uses a graph structure aligned with the predicted characters to improve the paragraph-level context information obtained by the encoder. The lower layer is a character-level local attention, which improves the character-level context information within each paragraph under the correction of the global graph attention. Then, the paragraph-level context information and the character-level context information are concatenated and linearly transformed. After that, the vector obtained by linear transformation is passed through a fully connected layer and Softmax to find the character with the highest probability in the character space as the next predicted character.

[0015] In one implementation, the multiple documents corresponding to the human-generated summaries are divided into multiple paragraphs and then preprocessed, including:

[0016] Entities are extracted from each paragraph using an entity extraction tool. The entities from each paragraph are concatenated into a sentence. SimBert is then used to represent these sentences. Cosine similarity is calculated between all paragraphs in the resulting sentence embeddings. An explicit graph representing the paragraph relationships is obtained based on the cosine similarity.

[0017] In one implementation, the graph-aware attention mechanism employed by the global graph encoding module incorporates residual connections and a gating mechanism. The specific processing steps include:

[0018] Introduce a bias to represent the pairwise relationships between paragraphs. Its calculation is as follows:

[0019]

[0020]

[0021] Among them, let This indicates that the (l-1)th graph coding layer corresponds to paragraph P. i The output, This indicates that the (l-1)th graph coding layer corresponds to paragraph P. j The output σ is the standard deviation affecting the strength of the graph structure, and d is the output. head It is the dimension of the vector, e ij It is an implicit paragraph relationship learned by the network, for paragraph P. i For paragraph P j Attention score Used to import and display graph information;

[0022] A gating mechanism is used to process the displayed image information:

[0023] Y = σ(F(W) y E+b)*G

[0024] Where σ is the sigmoid activation function, F is the feature mapping layer, and W... y σ(F(W)) and b are the learnable parameters of the convolutional layer. y The E+b) output is a score between [0, 1], which is applied to each dimension of the input graph G, where E represents all e. ij That is, the implicit relation weights of all paragraphs, and Y represents the display graph information activated by the gating mechanism;

[0025] The final contextual representation of the paragraph is obtained by using an attention mechanism:

[0026] new Attention=Y+alpha*conv(E)

[0027] α ij =Softmax(new Attention)

[0028]

[0029] This part is a variant implementation of the attention mechanism, W Q W K W V It is the weight matrix, α ij These are the attention weights. `conv` is a 1x1 convolutional layer. `newAttention` is the fusion attention score between the explicit graph and the implicit relationship. `alpha` is the balancing weight, and `u`... i For paragraph P i The final context representation.

[0030] In one implementation, the processing procedure of the hierarchical graph attention module in the graph decoding layer module includes:

[0031] A feedforward network is used to convert the character into a positional hidden state vector, and then a linear projection is used to convert it into a scalar s. t s t This represents the ID of the corresponding central paragraph, calculated as follows:

[0032]

[0033] in W represents the t-th decoded character of the (l-1)-th decoding layer. p U p This is the weight matrix. Some are feedforward networks. The transformation is linear, and L is the total number of paragraphs;

[0034] The attention weights are adjusted using a graph structure of the central paragraph and related paragraphs, calculated as follows:

[0035]

[0036] Where, β tj For the corrected attention weights, e tj Let t be the attention weight of the t-th decoded character for the j-th paragraph, calculated using the following formula:

[0037]

[0038] Among them, W′ Q 、W′ K Let d′ be the weight matrix. head Let x be the vector dimension of the projection space. j The paragraph code for the j-th paragraph, i.e., paragraph P. i The final context representation, u j =x j ;

[0039] The global context vector representation g is obtained according to the following formula. t :

[0040]

[0041] The character-level local attention mechanism is computed. With the help of the global graph attention mechanism, it captures character-level context vectors and is applied independently to each paragraph. First, the character-level attention score within each paragraph is calculated:

[0042]

[0043] γ t，jk Let represent the attention score of the t-th decoded character to the k-th character of the j-th paragraph. It is a character-level attention score improved by the upper-level global graph attention;

[0044] Compute the local context vector representation l t :

[0045]

[0046] Where, x jk This represents the vector encoding of the k-th character in the j-th paragraph, where n represents the total number of characters in each paragraph.

[0047] The global context vector representation and the local context vector are concatenated and then subjected to a linear transformation. It is a linear transformation that yields the output of the hierarchical graph attention layer:

[0048]

[0049] in, Denotes a linear transformation, d l This is the output of the attention layer in the hierarchical graph.

[0050] Based on the same inventive concept, a second aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0051] Based on the same inventive concept, a third aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0052] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0053] This invention provides a generative multi-document summarization method using explicit entity graphs. Specifically, it proposes a SimBert-based entity graph to replace traditional document similarity graphs for expressing document relationships. Entities are extracted from each paragraph, and the entities from each paragraph are concatenated into a sentence. SimBert is then used to calculate the cosine similarity between all paragraphs. After thresholding, an explicit graph representing paragraph relationships is obtained. This neural network-based graph construction method, which focuses on entities within the document, outperforms inflexible, manually defined rules. Furthermore, the model improves graph-aware attention mechanisms and hierarchical graph attention mechanisms by introducing gating mechanisms and residual connections. This allows for better fusion of explicit and implicit graph information and preserves residual paths for implicit relationships learned by the attention mechanism, ensuring that the information learned by the network has a more significant role in guiding generative multi-document summarization, thus improving the multi-document summarization generation performance. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a structural diagram of the generative multi-document summarization model in an embodiment of the present invention;

[0056] Figure 2 This is the explicit relationship graph extraction process in the method of this embodiment of the invention.

[0057] Figure 3 This is a schematic diagram of the perceptual attention method in an embodiment of the present invention.

[0058] Figure 4 This is a schematic diagram of the hierarchical graph attention mechanism in the method of this embodiment of the invention.

[0059] Figure 5 This is a schematic diagram of the global graph attention mechanism in the method of this embodiment of the invention.

[0060] Figure 6 This is a schematic diagram of the residual connection and gating mechanism in the method of this invention embodiment. Detailed Implementation

[0061] This invention is primarily used for automatic summarization of multiple documents with the same topic. Inheriting the idea of ​​using explicit graph structures to guide the model in learning the relationships between multiple documents, it proposes a generative multi-document summarization method using an explicit entity graph. Addressing the shortcomings of existing technologies, this invention provides a generative multi-document summarization method using an explicit entity graph. It proposes an entity graph based on SimBert to replace the traditional document similarity graph to express the connections between documents. Furthermore, this invention improves the graph-aware attention mechanism and hierarchical graph attention mechanism, introducing gating mechanisms and residual connections to guide generative multi-document summarization.

[0062] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] Example 1

[0064] This invention provides a generative multi-document summarization method using Entity explicit graphs. A generative multi-document summarization model is pre-built, comprising a BERT pre-trained model input module, a character-level encoding Transformer module, a global graph encoding module, and a graph decoding layer module. The method includes:

[0065] After dividing the multiple documents corresponding to the human summary into multiple paragraphs and preprocessing them, the input paragraphs are obtained.

[0066] The Bert pre-trained model input module obtains preliminary character embeddings based on the characters in the input paragraph.

[0067] Input the initial character embedding into the character-level encoding Transformer module in units of paragraphs to obtain the initial embedding of each paragraph and the embedding of all characters after information exchange within the paragraph;

[0068] The initial embedding of each paragraph is input into the global graph encoding module. The global graph encoding module replaces the multi-head attention mechanism in the Transformer encoder structure with a graph-aware attention mechanism and introduces an explicit graph structure. By combining the explicit graph information with the learned implicit relationships, information interaction between paragraphs is carried out to obtain the paragraph embedding after paragraph information interaction.

[0069] The initial character embedding, the embeddings of all characters after information exchange within the paragraph, and the paragraph embedding after paragraph information exchange are input into the graph decoding layer module. The graph decoding layer module replaces the multi-head attention module of the Transformer decoder with a hierarchical graph attention module. The graph decoding layer module is divided into upper and lower layers. The upper layer is a global graph attention mechanism, which uses a graph structure aligned with the predicted characters to improve the paragraph-level context information obtained by the encoder. The lower layer is a character-level local attention, which improves the character-level context information within each paragraph under the correction of the global graph attention. Then, the paragraph-level context information and the character-level context information are concatenated and linearly transformed. After that, the vector obtained by linear transformation is passed through a fully connected layer and Softmax to find the character with the highest probability in the character space as the next predicted character.

[0070] like Figure 1 The diagram shown is a structural diagram of the generative multi-document summarization model in an embodiment of the present invention.

[0071] In one implementation, the multiple documents corresponding to the human-generated summaries are divided into multiple paragraphs and then preprocessed, including:

[0072] Entities are extracted from each paragraph using an entity extraction tool. The entities from each paragraph are concatenated into a sentence. SimBert is then used to represent these sentences. Cosine similarity is calculated between all paragraphs in the resulting sentence embeddings. An explicit graph representing the paragraph relationships is obtained based on the cosine similarity.

[0073] In the specific implementation, a threshold filter is applied; in this example, it is set to 0.5, meaning paragraph relationships with a correlation score less than 0.5 are ignored. This results in an explicit graph representing paragraph connections, displayed as a matrix. The corresponding numbers are then added to the dataset as part of the input. Please refer to [link to relevant documentation]. Figure 2 This is the explicit relationship graph extraction process in the method of this embodiment of the invention.

[0074] To better illustrate the use of this invention, specific details are provided in conjunction with embodiments. For ease of understanding, this embodiment assumes that for a given abstract, there are L input paragraphs P1, ..., P2. L And the correlation graph G of each paragraph, the input form of graph G is an L*L matrix, G[i][j] represents paragraph P i and P j The relational weights are used to generate a summary S of multiple documents.

[0075] The specific implementation process is explained as follows:

[0076] 1) This embodiment uses the WikiSum dataset to train the model. Other datasets can be processed similarly. Multiple documents corresponding to the human summaries are divided into multiple paragraphs. Note that the paragraph length should ideally not exceed 60 characters; otherwise, truncation will occur. Entities are extracted from each paragraph using an entity extraction tool. The entities from each paragraph are concatenated into a sentence. SimBert is then used to represent these sentences. Cosine similarity is calculated between all paragraphs among the resulting sentence embeddings. After thresholding (set to 0.5 in this example), paragraph relationships with a similarity less than 0.5 are ignored, resulting in an explicit graph representing paragraph connections, represented as a matrix. The corresponding numbers are added to the dataset as part of the input.

[0077] 2) Obtain the embedding of all paragraph characters using BERT.

[0078] 3) Input the Emdding from the previous step into the Transformer block of the character-level encoding layer by paragraph to obtain the initial Embedding of each paragraph and the character Embedding after information interaction within the paragraph.

[0079] 4) Input the paragraph embedding into the graph encoding layer, which includes three parts: graph-aware attention mechanism, residual pathway and layer normalization, and fully connected layer, such as... Figure 3 As shown, in the attention mechanism of graph perception, it is assumed that... This indicates that the first-to-first graph coding layer corresponds to paragraph P. i The output, This indicates that the character-level encoding Transformer is applied to paragraph P. i The output of the graph is as follows. In the GraphSum model, to improve the implicit relationships between paragraphs obtained by the attention mechanism, a bias representing the pairwise relationships between paragraphs is introduced to incorporate the graph G into the attention mechanism. Its calculation is as follows:

[0080]

[0081]

[0082] σ is the standard deviation affecting the strength of the graph structure. It is set based on experience in optimizing different datasets. In this example, σ is set to 2, and d head It is the dimension of the vector, e ij It represents the implicit paragraph relationships learned by the network, specifically the attention score of paragraph i to paragraph j. This introduces information about the display diagram.

[0083] Y = σ(F(W) y E+b)*G

[0084] Where σ is the sigmoid activation function. F is the feature mapping layer, which is two stacked 3x3 convolutional layers. W y σ(F(W)) and b are the learnable parameters of the convolutional layer. y The output of E+b) is a score between [0, 1], which is applied to each dimension of the input G. Here, E represents all e. ij , which represents the implicit relation weights of all paragraphs, and Y represents the displayed graph information activated by the gating mechanism.

[0085] new Attention=Y+alpha*conv(E)

[0086] α ij =Softmax(new Attention)

[0087]

[0088] This part is a variant implementation of the attention mechanism, W Q W K W V It is the weight matrix, obtained through training, e ij This is a traditional attention score implementation, representing paragraph P. i and P j The implicit relationship weights between them, α ij These are the attention weights. `conv` is a 1x1 convolutional layer that organizes the dimensions. `newAttention` is the fusion attention score between the explicit graph and the implicit relationship. `alpha` is the balanced weight; theoretically, the optimal value can be found through testing, but 1 was used in the experiment. Then, paragraph P can be calculated. i The final context representation u i It serves as the output of the graph-aware attention module and the input of the hierarchical graph attention module.

[0089] 5) The characters are then predicted at the decoder, and the core computation is performed in the hierarchical graph attention module. Assume... Let represent the t-th decoded character in the (l-1)-th decoding layer. Using this character, we find the aligned center paragraph. First, a feedforward network is used to convert this character into a positional hidden state vector, and then a linear projection is used to convert it into a scalar s. t This can be understood as the ID of the corresponding central paragraph, and its calculation process is as follows:

[0090]

[0091] Where Wp and Up are weight matrices. Part of it is a feedforward network, multiplied by Partially, the transformation is linear, where L is the total number of paragraphs. After obtaining the central paragraph, the explicit graph structure of the input determines the remaining paragraphs. Then, the attention mechanism is modified using the graph structure of the central paragraph and related paragraphs, calculated as follows:

[0092]

[0093] e tj Let be the attention weight for the t-th decoded character to the j-th paragraph. Its calculation is similar to that in the encoder:

[0094]

[0095] Among them, W′ Q 、W′ K Let d′ be the weight matrix. head Let x be the vector dimension of the projection space. j The paragraph code for the j-th paragraph obtained in the previous text, which is paragraph P. i The final context representation, u j =x j ;

[0096] The final result is a global context vector representation, such as Figure 5 As shown:

[0097]

[0098] Next, a character-level local attention mechanism is calculated. This mechanism, aided by the global graph attention mechanism, captures character-level context vectors and is applied independently to each paragraph. First, the character-level attention score within each paragraph is calculated:

[0099]

[0100] γ t，jk This represents the attention score of the t-th decoded character to the k-th character of the j-th paragraph. It is a character-level attention score improved by the upper-level global graph attention.

[0101]

[0102] x jk This represents the vector encoding of the k-th character in the j-th paragraph, where n represents the total number of characters in each paragraph.

[0103] Then the global and local context vectors are concatenated and subjected to a linear transformation. It is a linear transformation that yields the output of the hierarchical graph attention layer:

[0104]

[0105] Please see Figure 4 and Figure 6 ,in, Figure 4 This is a schematic diagram of the hierarchical graph attention mechanism in the method of this embodiment of the invention. Figure 6 This is a schematic diagram of the residual connection and gating mechanism in the method of this invention embodiment.

[0106] 6) Then the output of step 5) is input into the next module, and prediction and decoding are performed according to the structure of the Transformer decoder, combining global and local context vectors. The decoded character is found in the character space through operations such as softmax.

[0107] 7) In addition, some general module calculations are involved, including layer normalization, full connection, backpropagation, etc., which will not be elaborated here.

[0108] The parameter settings for this example are shown in the table below:

[0109] Table 1 Example Parameter Settings

[0110]

[0111] The beneficial effects of this invention are:

[0112] This paper proposes an Entity Graph based on SimBert to replace the traditional document similarity graph to express the connections between documents. At the same time, this invention improves the graph-aware attention mechanism and the hierarchical graph attention mechanism by introducing a gating mechanism and residual connections to guide generative multi-document summarization and improve the generation effect.

[0113] The final experimental results of this invention demonstrate the effectiveness of the improvement and enhance the quality of automatic summarization.

[0114] Example 2

[0115] Based on the same inventive concept, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the method described in Embodiment 1.

[0116] Since the computer-readable storage medium described in Embodiment 3 of this invention is the same computer-readable storage medium used in implementing the generative multi-document summarization method using Entity explicit graphs in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this computer-readable storage medium based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All computer-readable storage media used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0117] Example 3

[0118] Based on the same inventive concept, this application also provides a computer device, including storage, a processor, and a computer program stored in the storage and executable on the processor, wherein the processor executes the program to implement the method in Embodiment 1.

[0119] Since the computer device described in Embodiment 4 of this invention is the same computer device used to implement the generative multi-document summarization method using Entity explicit graphs in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this computer device based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All computer devices used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0120] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0121] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0122] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0123] Obviously, those skilled in the art can make various modifications and variations to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations to the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A generative multi-document summarization method using Entity Explicit Graph, characterized in that, A generative multi-document summarization model is pre-built, comprising a BERT pre-trained model input module, a character-level encoding Transformer module, a global graph encoding module, and a graph decoding layer module. The method includes: After dividing the multiple documents corresponding to the human summary into multiple paragraphs, preprocessing is performed to obtain the input paragraphs. Specifically, the entity extraction tool is used to extract the entities in each paragraph. The entities in each paragraph are concatenated into a sentence. Then, SimBert is used to represent the concatenated sentence. The cosine similarity between all paragraphs is calculated between the obtained sentence embeddings. An explicit graph representing the paragraph relationship is obtained based on the cosine similarity. The Bert pre-trained model input module obtains preliminary character embeddings based on the characters in the input paragraph. Input the initial character embedding into the character-level encoding Transformer module in units of paragraphs to obtain the initial embedding of each paragraph and the embedding of all characters after information exchange within the paragraph; The initial embedding of each paragraph is input into the global graph encoding module. The global graph encoding module replaces the multi-head attention mechanism in the Transformer encoder structure with a graph-aware attention mechanism and introduces an explicit graph structure. By combining the explicit graph information with the learned implicit relationships, information interaction between paragraphs is carried out to obtain the paragraph embedding after paragraph information interaction. The initial character embedding, the embeddings of all characters after information exchange within the paragraph, and the paragraph embedding after paragraph information exchange are input into the graph decoding layer module. The graph decoding layer module replaces the multi-head attention module of the Transformer decoder with a hierarchical graph attention module. The graph decoding layer module is divided into upper and lower layers. The upper layer is a global graph attention mechanism, which uses a graph structure aligned with the predicted characters to improve the paragraph-level context information obtained by the encoder. The lower layer is a character-level local attention, which improves the character-level context information within each paragraph under the correction of the global graph attention. Then, the paragraph-level context information and the character-level context information are concatenated and linearly transformed. After that, the vector obtained by linear transformation is passed through a fully connected layer and Softmax to find the character with the highest probability in the character space as the next predicted character.

2. The generative multi-document summarization method using Entity Explicit Graph of claim 1, wherein, The graph-aware attention mechanism used in the global graph encoding module incorporates residual connections and gating mechanisms. The specific processing steps include: A bias representing pairwise relationships between paragraphs is introduced which is computed as follows: Among them, let Indicates the first Each graph encoding layer for paragraphs The output, Indicates the first Each graph encoding layer for paragraphs The output, It is the standard deviation that affects the structural strength of the diagram. It is the dimension of the vector. It is the implicit paragraph relationship learned by the network, which is a paragraph For paragraphs Attention score Used to import and display graph information; A gating mechanism is used to process the displayed image information: Where σ is the sigmoid activation function. It is a feature mapping layer. and These are the learnable parameters of the convolutional layer. The output is a score between [0,1], which is applied to each dimension of the input graph G, where E represents all... That is, the implicit relation weights of all paragraphs, and Y represents the display graph information activated by the gating mechanism; The final contextual representation of the paragraph is obtained by using an attention mechanism: This part is a variant implementation of the attention mechanism. , , It is a weight matrix. It is attention weight. It is a 1x1 convolutional layer. It is a fusion attention score of explicit graphs and implicit relationships, with alpha as the balancing weight. For paragraph The final context representation.

3. The generative multi-document summarization method using an explicit Entity graph as described in claim 1, characterized in that, In the graph decoding layer module, the processing steps of the hierarchical graph attention module include: A feedforward network is used to convert the character into a positional hidden state vector, and then a linear projection is used to convert it into a scalar. , This represents the ID of the corresponding central paragraph, calculated as follows: in Indicates the first The first decoding layer One decoded character, W p U p This is the weight matrix. Some are feedforward networks. Part of it is a linear transformation. Total number of paragraphs; The attention weights are adjusted using a graph structure of the central paragraph and related paragraphs, calculated as follows: in, For the corrected attention weights, For the first The decoded character pair The attention weight of each paragraph is calculated using the following formula: in, , This is the weight matrix. Let be the vector dimension of the projection space. The paragraph code for the j-th paragraph, i.e., the paragraph The final context representation, ; The global context vector representation is obtained according to the following formula. : The character-level local attention mechanism is computed. With the help of the global graph attention mechanism, it captures character-level context vectors and is applied independently to each paragraph. First, the character-level attention score within each paragraph is calculated: Indicates the first The decoded character pair The first paragraph Attention score per character It is a character-level attention score improved by the upper-level global graph attention; Compute local context vector representation : in, This represents the vector encoding of the k-th character in the j-th paragraph. This represents the total number of words in each paragraph; The global context vector representation and the local context vector are concatenated and then subjected to a linear transformation. It is a linear transformation that yields the output of the hierarchical graph attention layer: in, Represents a linear transformation. This is the output of the attention layer in the hierarchical graph.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 3.

5. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 3.

Images