Multi-document abstract generation method for crisis help information

Abstract

The invention discloses a multi-document abstract generation method oriented to crisis help information. The method is divided into an extraction type abstract stage and a generation type abstract stage. In the extraction type abstract stage, context information is fused in a deep neural network by utilizing a sentence graph structure, and key information is effectively extracted from numerous documents by optimizing a sub-graph structure during information extraction; in the generative abstract stage, by means of the strong sequence generation capability of the BART model and the characteristics of the pointer generation network in the aspects of accurately copying key information and avoiding repeated content generation, the extracted information is further simplified and summarized, and an accurate and compact crisis help information abstract is generated. The method is suitable for processing and summarizing mass help seeking information in a crisis scene, the two-stage method can quickly and accurately provide clear information abstracts for rescue and emergency management, and the crisis response efficiency and effect are greatly improved.

Description

Technical Field

[0001] The invention belongs to the technical field of text processing, and in particular relates to a multi-document summary generation method oriented to crisis help-seeking information. Background Art

[0002] (1) Crisis information processing technology

[0003] During crisis and disaster events, social media has become an important channel for information dissemination, carrying all kinds of information that is critical to rescue work. Effectively parsing and managing this information is crucial for the successful implementation of rescue operations. To meet this challenge, a series of studies have been devoted to identifying, classifying and processing crisis-related information from social media content.

[0004] Among them, some studies aim to identify and classify crisis-related content from social media tweets, and propose an algorithm that combines traditional support vector machines, Gini index decision trees, and naive Bayes to improve overall recognition performance. The algorithm was successfully applied to social media tweet data during the Indonesian earthquake, showing high feasibility and accuracy in actual scenarios. There are also studies that extract grammatical features from online public opinion event datasets to construct a grammatical feature library of events. By representing event text in the form of syntactic vectors and using similarity recognition, this method shows that it can maintain a high recognition accuracy even under conditions of few samples when facing natural disasters. In addition, there are also studies on the recognition and extraction of help information in social media when floods occur. The research framework combines BERT-CNN technology and disaster knowledge system, and successfully realizes the automatic extraction of help information by fine-tuning the pre-trained language model. Taking a rainstorm event in a certain place as an example, the model can automatically identify key help information, such as trapped conditions and scarce materials, under conditions of few samples.

[0005] Although many studies have made progress in the micro-extraction and identification of crisis information, how to better integrate information during a crisis to support decision-making and response is still in the exploratory stage. This is an important direction for future research, namely, developing more efficient methods to comprehensively process and analyze the large amount of data generated in crisis situations in order to provide more accurate and timely information for rescue operations and disaster response.

[0006] (2) Pointer Network Technology

[0007] Pointer Networks (PN) are designed to address the limitation that the elements of the output sequence depend on the position of the elements in the input sequence. In traditional models, the dimension of the output sequence is usually limited by the size of the dictionary, which makes them unable to effectively deal with the problem of unknown words. In contrast, pointer networks integrate attention mechanisms to enable the model to learn the relative position information of elements in the input sequence, and then directly point to the specific position in the input sequence when outputting.

[0008] The core components of the pointer network include encoders and decoders, which are usually constructed using recurrent neural networks. In this architecture, the encoder is responsible for encoding the input sequence into a vector representation, and the decoder is responsible for generating the output at the current moment based on the encoder's output and the previous state. The decoder uses a pointer mechanism in this process to calculate and select the probability of an element in the input sequence as the output. In this way, the model can eventually directly output the elements in the sequence based on the calculated probability, thereby effectively solving the problem of unregistered words. Summary of the invention

[0009] In view of the deficiencies of the above-mentioned prior art, the purpose of the present invention is to provide a multi-document summary generation method for crisis help information, which is suitable for processing and summarizing massive help information in crisis scenarios; through the two-stage method, the present invention can quickly and accurately provide clear information summaries for rescue and emergency management, thereby greatly improving the efficiency and effectiveness of crisis response.

[0010] The technical solution of the present invention is specifically described as follows.

[0011] The present invention provides a multi-document summary generation method for crisis help information, which includes two stages: an extractive summary stage and a generative summary stage;

[0012] (i) Extractive summarization stage: extract key information from multiple documents through a crisis information extractive summarization model based on subgraph selection;

[0013] The subgraph selection-based crisis information extractive summarization model includes a graph-based multi-document encoder and a subgraph-based sentence selection module. The graph-based multi-document encoder uses the sentence graph structure to fuse context information in a deep neural network, which includes a text encoder, a graph encoder, and a subgraph encoder; the subgraph-based sentence selection module is optimized during information extraction through the subgraph structure, which includes a graph pooling layer and a subgraph selection layer;

[0014] First, the document collection is taken as input, and after passing through the text encoder, the vector representation corresponding to each sentence is obtained; then, the graph encoder converts the document collection into a relationship graph through two graph construction methods: similarity graph and topic graph; the vector representation of each sentence constitutes a node in the relationship graph, and the relationship between sentences forms an edge in the relationship graph; then, the graph encoder captures the relationship between sentences through the explicit sentence relationship graph, and integrates the relationship between sentences into the neural network through the graph structure; then, the subgraph encoder further models the subgraph structure based on the output of the graph encoder, and uses the ROUGE score as the extraction criterion, and uses the greedy algorithm to construct a candidate summary set. The graph pooling layer calculates the global pooling representation of the whole graph and subgraphs, and the subgraph selection layer selects the subgraph with the highest ROUGE score as the final summary;

[0015] (ii) Generative summary stage: The generative summary model based on BART-PGN further summarizes and simplifies the extracted information to generate an accurate and compact summary of the crisis help information;

[0016] The generative summary model is based on BART. By introducing a pointer generation network, the probability distribution mechanism of the words generated by its autoregressive decoder is improved. The BART model adopts an encoder-decoder architecture.

[0017] First, a pointer generation mechanism is used so that the model can consider not only the possibility of generating new words through the vocabulary, but also the possibility of directly copying vocabulary from the input text; second, a coverage mechanism is used to accumulate the attention distribution in each time step and reduce the generation of repeated content by paying attention to the coverage vector.

[0018] In the present invention, the text encoder adopts the BERT pre-trained language model, and the graph encoder and the subgraph encoder have the same structure, including a graph self-attention layer, a residual layer, a normalization layer and a feedforward layer.

[0019] In the present invention, the graph pooling layer uses a multi-head pooling mechanism, takes a set of sentence representations as input, and outputs a global pooled representation P.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] In the extractive summarization stage, the present invention designs an extraction method based on subgraph selection, which can effectively extract key information from numerous documents. It not only uses the sentence graph structure to fuse contextual information in the deep neural network, but also optimizes the information extraction through the subgraph structure, which significantly improves the coherence and information richness of the summary.

[0022] In the generative summary stage, the present invention designs a model that combines the advantages of the BART model and the pointer generation network. It makes use of the powerful sequence generation ability of the BART model and the characteristics of the pointer generation network in accurately copying key information and avoiding duplicate content generation, so as to further streamline and summarize the extracted information and generate an accurate and compact crisis help information summary. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a diagram of the extractive model structure of the present invention.

[0024] Figure 2 It is a diagrammatic representation of the structure of the encoder of the present invention.

[0025] Figure 3 It is a diagram of the generative model structure of the present invention. DETAILED DESCRIPTION

[0026] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and embodiments.

[0027] The present invention proposes a multi-document summary generation method for crisis help information, which is divided into two stages. The specific steps are as follows:

[0028] (1) In response to the massive amount of crisis help texts, a crisis information extractive summarization model based on subgraph selection is used in the first stage of the processing flow to extract key sentences from each document;

[0029] (2) Using the generative summary model based on BART-PGN, the extracted key sentences are further simplified and summarized to generate a highly generalized summary of the crisis help information. Ultimately, a comprehensive summary of massive crisis help information can be generated.

[0030] 1. Crisis Information Extractive Summarization Model Based on Subgraph Selection

[0031] The overall structure of the model is as follows Figure 1 As shown in the figure, the model is divided into two parts: graph-based multi-document encoder and subgraph-based sentence selection.

[0032] First, the model takes a document collection as input, and after passing through the BERT model, it obtains the vector representation corresponding to each sentence. Subsequently, the document collection is converted into a relationship graph through two graph construction methods: similarity graph and topic graph. Among them, the vector representation of each sentence constitutes a node in the relationship graph, and the relationship between sentences forms the edge in the relationship graph. Then, the graph encoder captures the relationship between sentences through the explicit sentence relationship graph, and integrates the relationship between sentences into the neural network through the graph structure. Subsequently, based on the output of the graph encoder, the model uses the subgraph encoder to further model the subgraph structure, and calculates the global pooling representation of the full graph and subgraph through the graph pooling layer. This process can distinguish the quality of different subgraph candidate summaries. Finally, the subgraph selection layer will select the subgraph with the highest score as the final summary.

[0033] 1. Graph-based Multi-Document Encoder

[0034] The graph-based multi-document encoder consists of three parts: text encoder, graph encoder and subgraph encoder.

[0035] (1) Text Encoder

[0036] The text encoder is responsible for processing the document text and outputting the corresponding sentence representation. In this method, the BERT pre-trained language model is used to encode the document text. Specifically, [CLS] and [SEP] tags are first inserted at the beginning and end of each sentence, and then input into the BERT model to obtain the sentence representation.

[0037] (2) Image Encoder

[0038] The graph encoder updates the sentence representation by modeling the relationship graph between sentences. The graph encoder structure is as follows Figure 2 As shown in the figure, the input first passes through the graph self-attention layer, then the residual is added to the input and the layer is normalized, and the output is passed to the feedforward layer for similar residual and layer normalization processing. The output of the graph encoder is a sentence vector that incorporates the graph relationship between sentences.

[0039] In the graph encoder, we first need to obtain the sentence relationship graph. In the sentence relationship graph, sentences are regarded as nodes of the graph, and the relationship between sentences is regarded as edges. Therefore, the graph representation matrix G is used to represent the graph structure information. Specifically, the element G[i][j] in the matrix represents the sentence s i With sentence s j The relationship weight E ij ,Relation weights use different standard methods in different graph types (similarity graphs and topic graphs).

[0040] The graph-aware self-attention mechanism is then used to enable the model to more effectively capture and understand the association information between sentences and other related sentences. The graph-aware self-attention mechanism extends the self-attention mechanism to consider the relationship between sentences in the explicit sentence relationship graph. Assume For sentence s i At the output of the l-1th graph encoding layer, u i For sentence s i The context representation, e ij Indicates sentence s i and j The original self-attention weight between ij represents the adjusted weight. Then its calculation method is as follows:

[0041]

[0042]

[0043] α ij =softmax(e ij +R ij )

[0044] The key to the graph-aware self-attention mechanism lies in the bias R of the relationship between sentences. ij , that is, this step, integrates the sentence relationship graph into the neural network. ij The calculation method of is as follows:

[0045]

[0046] Next, the result of the operation will be passed to a feedforward neural network and then residual connection and layer normalization will be performed. This feedforward neural network contains two linear layers and ReLU activation function. Through this process, the vector representation x of each sentence will be obtained. l .

[0047] (3) Sub-image encoder

[0048] The subgraph encoder is used to model each subgraph, and its architecture is exactly the same as that of the graph encoder. The input of the subgraph encoder is the output of the graph encoder; the output of the subgraph encoder is a subgraph vector consisting of a set of sentences.

[0049] There are two challenges in the subgraph encoder training process: (1) The golden summaries of most datasets are written by humans and need to be converted into sentence tags for extractive summaries. (2) There are hundreds of sentences in the document collection, and there are thousands of subgraphs to choose from. The efficiency and performance of subgraph construction need to be guaranteed. To solve these two problems, the present invention uses ROUGE score as the extraction criterion and a greedy algorithm to construct a candidate summary set.

[0050] Specifically, we first select several sentences with the highest ROUGE scores as initial candidate nodes, which are then used as initial nodes of multiple candidate subgraphs. Then, we gradually add sentences to the summary set, so that the ROUGE score of the currently selected sentence set is maximized relative to the golden summary, until the ROUGE score cannot be improved by adding any remaining sentences. Finally, these sentence sets correspond to multiple constructed subgraphs. The process algorithm is shown in Table 1:

[0051] Table 1

[0052]

[0053]

[0054] 2. Subgraph-based statement selection

[0055] Subgraph-based sentence selection consists of two parts: graph pooling layer and subgraph selection layer.

[0056] (1) Graph Pooling Layer

[0057] The main goal of the graph pooling layer is to capture the overall semantic information of the entire sentence relationship graph and each subgraph. The graph pooling layer uses a multi-head pooling mechanism, takes the sentence representation set as input, and outputs the global pooled representation P.

[0058] For each independent attention head z∈{1,…,n head}, first calculate the attention score and value vector for the vector representation of the sentence, and calculate the attention distribution of each sentence based on the attention score. Then perform weighted summation on each attention head, and obtain the vector representation head corresponding to each attention head through linear transformation and layer normalization. z Finally, concatenate the vectors of all attention heads and perform linear transformation to obtain the final global pooling representation P. Therefore, the calculation method of the global pooling representation is as follows:

[0059]

[0060] P=W d [head1||…||head z ]

[0061] (2) Sub-image selection layer

[0062] The subgraph selection layer scores each subgraph to select the best subgraph as the final summary. Two losses are used as the loss function of the entire model. Among them, cosine similarity is used to calculate the similarity between the golden summary C* and the global pooling representation P as the loss function of the summary task, and binary cross entropy loss is used to learn more accurate sentence and summary representations. Assume Indicates whether the sentence should be considered as a summary sentence, then the overall loss function is calculated as follows:

[0063]

[0064] 2. Generative Summarization Model Based on BART-PGN

[0065] The BART model uses an encoder-decoder architecture. Based on BART, the generative summary model improves the probability distribution mechanism of the autoregressive decoder to generate words by introducing a pointer generation network. First, the pointer generation mechanism is used so that the model can consider not only the possibility of generating new words through the vocabulary, but also the possibility of directly copying words from the input text. Secondly, the coverage mechanism is used to accumulate the attention distribution in each time step, and the generation of repeated content is reduced by paying attention to the coverage vector. The overall structure of the model is as follows Figure 3 shown.

[0066] a) Pointer generation mechanism

[0067] First, at each moment, the model is divided into generation mode and replication mode. In the generation mode, the model needs to generate words from the vocabulary, which allows the model to introduce new words when generating text. The hidden state and context vector output by the decoder are concatenated, and after two linear transformations and a softmax layer, the probability distribution P of the vocabulary at that moment is obtained. vocab In copy mode, the model can directly select and copy words from the input sequence. The model will select and copy words based on the current attention distribution. To determine the probability P of copying word w in the input sequence copy (w), calculated as follows:

[0068]

[0069] In order to choose between the generation mode and the replication mode, a generation probability P is introduced gen As a soft switch, it is used to determine which method is currently used for decoding, whether to generate new words or directly copy words from the input sequence. The soft switch is calculated by the context vector, the decoder hidden state, and the decoder input after passing through the Sigmoid function.

[0070] Finally, at each moment, the model performs a weighted calculation on the probability distribution obtained in the generation mode and the probability distribution obtained in the replication mode to obtain the probability distribution P(w) of the decoded word. The calculation method is as follows:

[0071] P(w)=P gen P vocab (w)+(1-P gen )copy (w)

[0072] b) Coverage Mechanism

[0073] The core of the coverage mechanism is to adjust the attention mechanism to prevent the probability of attention distribution of a certain word in the original text from being too high, resulting in too large a weight. It can effectively reduce or avoid repeated concentration of attention at the same position, thereby significantly reducing the repeated content in the generated summary.

[0074] Specifically, through the coverage vector c t Attention distribution for generated words Penalize, the coverage vector value is the sum of the attention distribution at time t-1 and before. Assume that W h , W o , W c represents the linear transformation matrix, v T represents the weight vector, represents the coverage vector, b attn Represents the bias matrix, the original attention distribution is output by the encoder h i and decoder output o t Calculated, the adjusted coverage vector and attention distribution are calculated as follows:

[0075]

[0076]

[0077] Implementation examples:

[0078] This example involves the development of a crisis help platform system, which aims to provide users with a platform to publish and obtain crisis-related help information and support the need to process large-scale data. The system is programmed in JavaScript, Golang and Python languages, and the system and the model proposed in this article are deployed on a Linux server. Among them, the system can generate summaries of help information related to a specific crisis event at a regular interval. The specific steps are: first, the crisis help information posted by the user is preprocessed to improve the data quality and prepare for summary generation. Then the preprocessed text data is input into the summary model proposed in this article to generate a coherent and information-rich summary. Finally, the obtained crisis information summary is displayed to the public through the user interface; the inventor conducted relevant experiments, and the results showed that compared with multiple traditional summary models, the summary results of the present invention showed obvious superiority in ROUGE-1, ROUGE-2 and ROUGE-L.

Claims

1. A method for generating multi-document summaries for crisis assistance information, characterized by: The method consists of two stages: extractive summarization stage and generative summarization stage; (i) Extractive summarization stage: extract key information from multiple documents through a crisis information extractive summarization model based on subgraph selection; The subgraph selection-based crisis information extractive summarization model includes a graph-based multi-document encoder and a subgraph-based sentence selection module. The graph-based multi-document encoder utilizes the sentence graph structure to fuse context information in a deep neural network, which includes a text encoder, a graph encoder, and a subgraph encoder. The subgraph-based sentence selection module optimizes information extraction through the subgraph structure, which includes a graph pooling layer and a subgraph selection layer; First, the document collection is taken as input, and after passing through the text encoder, the vector representation corresponding to each sentence is obtained; then, the graph encoder converts the document collection into a relationship graph through two graph construction methods: similarity graph and topic graph; the vector representation of each sentence constitutes a node in the relationship graph, and the relationship between sentences forms an edge in the relationship graph; then, the graph encoder captures the relationship between sentences through the explicit sentence relationship graph, and integrates the relationship between sentences into the neural network through the graph structure; then, the subgraph encoder further models the subgraph structure based on the output of the graph encoder, and uses the ROUGE score as the extraction criterion, and uses the greedy algorithm to construct a candidate summary set. The graph pooling layer calculates the global pooling representation of the whole graph and subgraphs, and the subgraph selection layer selects the subgraph with the highest ROUGE score as the final summary; (ii) Generative summary stage: The generative summary model based on BART-PGN further summarizes and simplifies the extracted information to generate an accurate and compact summary of the crisis help information; The generative summary model is based on BART. By introducing a pointer generation network, the probability distribution mechanism of the words generated by its autoregressive decoder is improved. The BART model adopts an encoder-decoder architecture. First, a pointer generation mechanism is used so that the model can consider not only the possibility of generating new words through the vocabulary, but also the possibility of directly copying vocabulary from the input text; second, a coverage mechanism is used to accumulate the attention distribution in each time step and reduce the generation of repeated content by paying attention to the coverage vector.

2. The multi-document summary generation method according to claim 1, characterized in that: The text encoder uses the BERT pre-trained language model. The graph encoder and subgraph encoder have the same structure, including graph self-attention layer, residual layer, normalization layer and feedforward layer.

3. The multi-document summary generation method according to claim 1, characterized in that: In the graph encoder, we first construct a sentence relationship graph; In the sentence relationship graph, sentences are regarded as nodes of the graph, and the relationships between sentences are regarded as edges, so the graph representation matrix G is used to represent the graph structure information; The element G[i][j] in the matrix represents the sentence s i With sentence s j The relationship weight E ij ; Then, a graph-aware self-attention mechanism is used to consider the relationships between sentences in the explicit sentence relationship graph; assuming For sentence s i At the output of the l-1th graph encoding layer, u i For sentence s i The context representation, e ij Indicates sentence s i and j The original self-attention weight between Q , W K , W V are the Query, Key, and Value weight matrices in the self-attention mechanism, respectively. head is the attention head, α ij represents the adjusted weight, α ij The calculation method of is as follows: a ij =softmax(e ij +R ij ) Where R ij The bias that represents the relationship between sentences, R ij The calculation method of is as follows: The graph-aware self-attention mechanism integrates the sentence relationship graph into the deep neural network by extending the self-attention mechanism; Next, the adjusted weight α of the sentence ij Passed into the feedforward neural network and then subjected to residual connection and layer normalization, the vector representation x of each sentence is obtained l .

4. The multi-document summary generation method according to claim 1, characterized in that: In the subgraph encoder, the method of using the greedy algorithm to construct the candidate summary set is as follows: First, several sentences with the highest ROUGE scores are selected as initial candidate nodes, which are then used as initial nodes of multiple candidate subgraphs. Then, gradually add sentences to the summary set so that the ROUGE score of the currently selected sentence set is maximized relative to the golden summary C*, until adding any remaining sentences does not improve the ROUGE score; Finally, these sentence sets correspond to multiple constructed subgraphs.

5. The multi-document summary generation method according to claim 1, characterized in that: The graph pooling layer uses a multi-head pooling mechanism, takes a set of sentence representations as input, and outputs a global pooled representation P.

6. The method for generating multi-document summaries according to any one of claims 1 to 5, characterized in that: Two losses are used as the overall loss function of the crisis information extractive summary model based on subgraph selection; among them, cosine similarity is used to calculate the similarity between the golden summary C* and the global pooling representation P as the loss function of the summary task, and binary cross entropy loss is used to learn more accurate sentence and summary representations; assuming y i represents the true label of the sentence, Indicates whether the sentence should be considered as a summary sentence, then the overall loss function is calculated as follows:

7. The multi-document summary generation method according to claim 1, characterized in that: The generative summary model based on BART-PGN is divided into generation mode and copy mode. When generating text, new words are generated through the vocabulary in the generation mode, and words are selected and copied from the input sequence in the copy mode. Improvements are made to the probability distribution mechanism of words generated by its autoregressive decoder, including: Using the pointer generation mechanism, the probability of generating new words through the vocabulary and the probability of directly copying words from the input text at each moment are considered to obtain the probability distribution of decoded words at each moment. Using the coverage mechanism, we accumulate the attention distribution at each time step and reduce the generation of duplicate content by focusing on the coverage vector.

8. The method for generating multi-document summaries according to claim 7, characterized in that: In the generative summary stage, at the current moment, the hidden state and context vector output by the decoder are concatenated, and after two linear transformations and a layer of softmax, the probability distribution P of generating new words through the vocabulary at this moment is obtained. vocab ; According to the current moment of attention distribution Determine the probability P of copying word w in the input sequence copy (w), calculated as follows: In order to choose between the generation mode and the replication mode, a generation probability P is introduced gen As a soft switch, it is used to determine which method is currently used for decoding; The soft switch is calculated by the context vector, the decoder hidden state, and the decoder input after passing through the Sigmoid function; Finally, at each moment, the model performs a weighted calculation on the probability distribution obtained in the generation mode and the probability distribution obtained in the replication mode to obtain the probability distribution P(w) of the decoded word; the calculation method is as follows: P(w)=P gen P vocab (w)+(1-P gen )P copy (w) 9. The method for generating multi-document summaries according to claim 7, characterized in that: In the generative summary stage, the coverage vector c t Attention distribution for generated words Penalization is performed, and the value of the coverage vector is the sum of the attention distribution at the t-1th moment and before.

Images