Generative summarization method and system based on bert vectorization and dependency syntax

Abstract

The application provides a BERT vectorization and dependency syntax generative abstract method and system, and the generative abstract method comprises the following steps: inputting original text and abstract text into a BERT pre-training model to obtain a sentence feature vector; performing syntax dependency analysis on the original text and the abstract text by using an LTP model to construct a dependency type matrix and an adjacency matrix; performing attention weight calculation on the dependency type matrix and the adjacency matrix, and fusing the obtained sentence feature vector to obtain a multi-granularity feature semantic intermediate vector; taking the sentence feature vector as an input sequence, and encoding the input sequence by using an encoder with a network structure of a BiLSTM model to obtain an output sequence; splicing the multi-granularity feature semantic intermediate vector and the output sequence to input a decoder of the BiLSTM model to generate a result sequence, and generating a final text abstract by using a beam search. The generative abstract method is simple, and can better capture key information.

Description

Technical Field

[0001] This invention relates to the field of text summarization technology, and more specifically, to a generative summarization method and system based on BERT vectorization and dependency syntax. Background Technology

[0002] Text summarization technology uses machine learning and deep learning algorithms to automatically extract concise, clear, and easy-to-understand summaries from long texts. Generative summarization, in particular, achieves a deeper understanding of the text content and learns the complex mapping relationship from the original text to the summary. Therefore, the generated summaries more accurately summarize the main idea of ​​the original text while retaining key details, resulting in higher quality summaries.

[0003] However, existing generative summarization technologies also face problems such as high model complexity and difficulty in capturing key information.

[0004] In view of this, the present invention is hereby proposed. Summary of the Invention

[0005] In view of this, the present invention discloses a generative summarization method and system based on BERT vectorization and dependency syntax to improve the summarization quality of text summarization tasks.

[0006] Specifically, the present invention is achieved through the following technical solutions:

[0007] This invention provides a generative summarization method based on BERT vectorization and dependency syntax, comprising the following steps:

[0008] The original text and the summary text are fed into the BERT pre-trained model to obtain sentence feature vectors;

[0009] Syntactic dependency analysis was performed on the original text and the summary text using the LTP model to construct a dependency type matrix and an adjacency matrix;

[0010] Attention weights are calculated on the dependency type matrix and the adjacency matrix, and then fused with the obtained sentence feature vector to obtain a multi-granularity feature semantic intermediate vector.

[0011] The sentence feature vector is used as the input sequence, and the encoder with a BiLSTM network structure is used to encode the output sequence.

[0012] The multi-granularity feature semantic intermediate vector is concatenated with the output sequence and input into the decoder of the BiLSTM model to generate a result sequence. The final text summary is generated by beam search.

[0013] Specifically, the technical solution of the present invention mainly includes the following steps:

[0014] (1) The original text and the summary text are fed into the Chinese BERT pre-trained model. The model inserts a [CLS] marker at the beginning of each sentence and adds a [SEP] marker at the end of each sentence. For a training batch of original text, Q1 = (Q11, Q12, ... Q1...). N ) and the abstract text Q2=(Q21,Q22,...Q2 N After passing through the BERT model, the corresponding vectorized result is E1 = (E11, E12, ..., E1). N E2 = (E21, E22, ..., E2) N );

[0015] (2) Construct syntactic features from the original text and the summary text. Specifically, firstly, LTP is used for syntactic dependency parsing to obtain a dependency syntactic tree based on the dependency relationships between each word in the sentence. Then, the syntactic dependency tree is parsed to generate an association matrix M and a dependency type matrix M_Type. Subsequently, M and M_Type are input into the GCN for attention weight calculation. The formula for the attention weight between nodes i and j in the l-th layer GCN is as follows: Where, m i,j These are the values ​​of i and j in the correlation matrix, and "·" represents the vector inner product. It is calculated from the dependency type matrices of nodes i and j and the GCN vector, and the formula is as follows: in This represents the vector representation of the corresponding dependency type matrix. This represents the vector representation of node i in the (l-1)th layer of the GCN. Next, the attention weights are combined using an l-layer graph convolutional neural network. The formula for processing graph-structured data is as follows: Where σ represents the ReLU activation function, W (l) and b (l) Let L represent the trainable matrix and the bias term of the l-th layer, respectively. Output from the GCN of layer (l-1) Vector representation of dependency type matrix It is calculated. In order to further integrate word features and syntactic information, the output of the graph convolutional neural network is fused with the sentence vector feature E in (1) by vector concatenation. Finally, a semantic vector h that integrates multi-granularity features is obtained;

[0016] (3) The encoder network structure is a BiLSTM model. The forward propagation LSTM layer and the backward propagation LSTM layer accept the vectorized result E as the input sequence to obtain the forward and backward hidden states h of BiLSTM at time t. t and h t Then perform vector fusion on them, with the formula s.t =g(w1h) t +w2h t ′), where w represents the weight parameter of each layer. Finally, the encoder output sequence C is obtained;

[0017] (4) The decoder of the model adopts a BiLSTM network structure. Furthermore, to enable the decoder to make reasonable use of the intermediate semantic information generated by the encoder when generating prediction results, an Attention mechanism is introduced. The multi-granularity feature semantic vector h obtained in (2) and the encoder output sequence C in (3) are concatenated and then input into the decoder. The decoder first calculates the hidden layer state h based on the model input. j The formula for calculating attention is as follows: Among them, s t-1 h represents the hidden layer state of the decoder at time t-1. j Let α(·) represent the hidden layer states of the encoder at all time points, and let α(·) be an alignment model based on a nonlinear function. Based on the calculated attention weights, the hidden layer vectors (h1, h2, ..., h...) of the encoder are then aligned. n ) and attention weight α ij The weighted summation yields the semantic vector C. i Finally, based on semantic vector C i Information interaction was achieved through an attention mechanism, and the result sequence of the nth character was predicted accordingly. The formula is as follows: Where g(·) represents the LSTM prediction function, H f and H b These represent the result sequences generated by the L2R decoder and the R2L decoder, respectively. This represents the prediction result of the nth word in the decoder. When using an attention mechanism as features to combine H during decoding... f and H b When predicting the nth word, the formula is: In the decoder, Q = H is satisfied. f K = V = H b ;

[0018] (5) Finally, beam search is used to generate the final text summary. Beam search takes the output prediction results of the bidirectional decoder as input. The algorithm can record the top-k results with the highest prediction probabilities in both directions at each time step, and finally selects the result with the highest probability as the summary.

[0019] In addition, this invention also provides a generative summarization system based on BERT vectorization and dependency syntax, specifically including:

[0020] The acquisition module is used to input the original text and the summary text into the BERT pre-trained model to obtain sentence feature vectors;

[0021] Builder module: Used to perform syntactic dependency analysis on the original text and the summary text using the LTP model, and to build the dependency type matrix and adjacency matrix;

[0022] The fusion module is used to calculate attention weights for the dependency type matrix and the adjacency matrix, and fuse them with the obtained sentence feature vector to obtain a multi-granularity feature semantic intermediate vector.

[0023] Output module: Used to encode the sentence feature vector as the input sequence and use an encoder with a BiLSTM network structure to obtain the output sequence;

[0024] The generation module is used to concatenate the multi-granularity feature semantic intermediate vector with the output sequence and input it into the decoder of the BiLSTM model to generate a result sequence, and then use beam search to generate the final text summary.

[0025] The present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed, implements the steps of the generative summarization method.

[0026] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the generative summarization method.

[0027] In summary, the generative summarization method and system provided by this invention first utilize the BERT model to capture global semantic features of the text during the feature construction stage, and then extracts syntactic structure information through dependency parsing. Furthermore, the model integrates an attention mechanism and a graph convolutional neural network to effectively fuse syntactic features with BERT features, thereby enhancing the model's ability to represent deep semantics and its in-depth understanding of global text information. In addition, a bidirectional decoding structure is used in the decoding stage for forward and backward semantic analysis, effectively mitigating the bias accumulation problem of unidirectional decoding structures when understanding the full text, thus improving the coherence and comprehensiveness of the summary, ultimately achieving the goal of improving the summarization quality of text summarization tasks. Attached Figure Description

[0028] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0029] Figure 1 This is a diagram illustrating the algorithm structure involved in the embodiments of the present invention;

[0030] Figure 2 This is a flowchart of the algorithm involved in the embodiments of the present invention;

[0031] Figure 3 This is a flowchart illustrating a computer device provided in an embodiment of the present invention. Detailed Implementation

[0032] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure; rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0033] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms. Unless the context clearly indicates otherwise, it should be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0034] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, depending on the context. For example, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0035] Example

[0036] (1) Overall structure of the model in this invention:

[0037] The overall structure of the model in this embodiment of the invention is as follows: Figure 1 As shown, the overall structure is divided into three modules. The algorithm flow in this embodiment of the invention is as follows: Figure 2As shown, in the feature construction module, the original text and summary are used as input. First, a global vectorized representation is obtained through the BERT model. Then, dependency parsing is used to parse the text sentences into a set of relation triples (i.e., if there are two nodes with label types, it is parsed as (head, type, tail)). These relation triples are then transformed into the form of an association matrix and a dependency type matrix, and syntactic features in the dependency parsing tree are obtained using a graph convolutional neural network and attention mechanism. In the Seq2Seq module based on bidirectional decoding, the encoder and decoder modules take the fusion result of syntactic features and BERT vectorization as input. First, the encoder extracts key information. Then, the decoder fuses the decoding results from both directions during prediction through bidirectional decoding. Through bidirectional decoding, the model can more accurately capture key information in the text and generate more accurate, coherent, and comprehensive summaries. In the beam search algorithm module, the model first takes the output prediction result of the bidirectional decoder as input. Then, the model adopts the beam search algorithm, which can record the top-k results with the highest prediction probabilities in both directions at each time step. Finally, the result with the highest probability is selected as the summary.

[0038] (2) BERT vectorization:

[0039] BERT, through conditionalization of all layers in a bidirectional context and pre-training of deep bidirectional representations, enables it to fully consider word information, positional information, and contextual information when understanding text. This allows the model to more accurately understand the meaning of each word in the text, effectively solving the problem of polysemy. In this invention, BERT is used as the baseline model for initially extracting Chinese text features. The text is input into the BERT preprocessing model, which inserts a [CLS] marker at the beginning of each sentence and adds a [SEP] marker at the end. The vectorized result obtained from the Pooler_out output layer vector of the BERT model is E1 = (E11, E12, ..., E1...). N E2 = (E21, E22, ..., E2) N The formulas are described as shown in (1) and (2).

[0040] E1 = [BERT([S1])] Pooler_out (1);

[0041] E2 = [BERT([S2])] Pooler_out (2);

[0042] (3) Syntactic feature construction:

[0043] The syntactic feature construction employs the LTP method for syntactic dependency analysis. Specifically, firstly, a dependency syntactic tree is obtained based on the dependency relationships between each word in the sentence. Then, the dependency tree is parsed to generate an association matrix and a dependency type matrix. In the association matrix M, if two words have a dependency relationship, the corresponding position in the matrix is ​​marked as 1; otherwise, it is marked as 0. The dependency type matrix M_Type is also constructed based on the dependency relationships between words. For words with dependency relationships, the corresponding position in the M_Type matrix is ​​set to the specific type between those words.

[0044] Next, the character feature vectors, association matrix, and dependency type matrix of the text are integrated into a unified graph structure. In this graph structure, the feature vector corresponding to each character is represented as the value of the node. At the same time, the weight information of the connection edges between nodes is composed of the attribute values ​​of the corresponding positions in the association matrix and dependency type matrix. Subsequently, this graph structure data is input into the attention mechanism and the graph convolutional neural network (GCN) for attention weight calculation. Specifically, the calculation of the attention weight between nodes i and j in the l-th layer of the graph neural network is shown in (3).

[0045]

[0046] Where, m i,j These are the values ​​of i and j in the correlation matrix, and "·" represents the vector inner product. It is calculated from the dependency type matrix of nodes i and j and the GCN vector, and its specific calculation is shown in formulas (4) and (5).

[0047]

[0048] in This represents the vector representation of the corresponding dependency type matrix. This represents the vector representation of node i in the (l-1)th layer of the GCN.

[0049] Next, the attention weights obtained are combined using an l-layer graph convolutional neural network. The graph structure data is processed. The calculation is shown in formula (6).

[0050]

[0051] Where σ represents the ReLU activation function, W (l) and b (l) Let L represent the trainable matrix and the bias term of the l-th layer, respectively. Output from the GCN of layer (l-1) Vector representation of dependency type matrix It is calculated, as shown in formula (7).

[0052]

[0053] Among them, matrix Its function is to Mapping to AND They share the same dimension to ensure compatibility and matching during subsequent calculations.

[0054] The intermediate vector representation was obtained through graph convolutional neural network operations. To further integrate word features and syntactic information, the intermediate vector was fused with the sentence features E by vector concatenation. In this way, a semantic vector h that integrates multi-granularity features was obtained. This vector not only contains word-level information but also incorporates deeper syntactic structure information, thus more comprehensively representing the intrinsic meaning of the text.

[0055] (4) Encoder part of the Seq2Seq module based on bidirectional decoding:

[0056] The encoder is a BiLSTM network consisting of a forward LSTM layer and a backward LSTM layer. It accepts the vectorized result E as the input sequence. First, the forward LSTM layer processes the input data in the order of the sequence from beginning to end. At each time step, it considers the current input and the hidden state of the previous time step, then calculates and outputs the hidden state of the current time step. Similarly, the backward LSTM layer processes the same input data in the order of the sequence from end to beginning. Then, the hidden states output by the forward and backward LSTMs at the same time step are concatenated to obtain the bidirectional hidden state for that time step, as shown in formulas (8), (9), and (10).

[0057] h t =f(w1e t +w2h t-1 (8);

[0058] h t ′=f(w3e t +w5h t ′ -1 (9);

[0059] s t =g(w4h) t +w6h t ′)(10);

[0060] Among them, h t and h t ′ represent the forward and reverse hidden states of the BiLSTM at time t, respectively, e t s represents the input at time t. t This indicates that at time t, h t and ht The result after vector fusion. This represents the weight parameters for each layer.

[0061] (5) Decoder part of the Seq2Seq module based on bidirectional decoding:

[0062] The decoder is a BiLSTM network incorporating an attention mechanism. The decoder's main task is to generate the target sequence based on the output of the BiLSTM encoder. Specifically, during the decoding process, the BiLSTM decoder combines the encoder's output C = (c1, c2, ..., c...) n The decoder generates each element of the target sequence step by step, using its internal state, the output of the previous time step, and the multi-granularity feature semantic vector h. At each time step t, the decoder updates its internal state and predicts the output of the next time step based on the current state, the input, and other possible information.

[0063] In addition, to further improve performance, the BiLSTM decoder introduces an attention mechanism. This mechanism enables the decoder to give different attention to different parts of the input sequence according to the attention weight of different words when generating each output, thereby capturing the correspondence between input and output more accurately, as shown in (11) and (12).

[0064]

[0065] e ij =α(s i-1 ,h j (12);

[0066] Among them, s t-1 h represents the hidden layer state of the decoder at time t-1. j α(·) represents the hidden layer state of the encoder at all time points, and is an alignment model based on a nonlinear function.

[0067] Based on the calculated attention weights, the hidden layer vectors (h1, h2, ..., h) of the encoder are then used to... n ) and attention weight α ij The weighted summation yields the semantic vector C. i As shown in (13):

[0068]

[0069] Based on semantic vector C iThe bidirectional decoding process begins, which is executed in parallel by a forward (L2R) decoder and a backward (R2L) decoder. Each decoder considers not only previously generated results but also those generated by other decoders when generating a new result. Ultimately, the two decoders interact through an attention mechanism to predict the result sequence for the nth character, as shown in (14), (15), and (16).

[0070]

[0071]

[0072] Where g(·) represents the LSTM prediction function, H f and H b These represent the result sequences generated by the L2R decoder and the R2L decoder, respectively. and Let H and R represent the prediction results of the nth word in the L2R decoder and R2L decoder, respectively. When using an attention mechanism as features to combine H during decoding... f and H b When predicting the nth word, the formula is shown in 4-14, which satisfies Q = H in the L2R decoder. f K = V = H b In the R2L decoder, Q = H is satisfied. b K = V = H f .

[0073] In summary, the technical effects of this invention are as follows: The invention designs a generative summarization model that integrates BERT vectorization and dependency parsing information, solving the problems of insufficient accuracy in sentence-level vocabulary understanding and inadequate utilization of syntactic information in generative summarization techniques. It utilizes the BERT model to capture the global semantic features of the text and extracts syntactic structure information through dependency parsing. Furthermore, the model integrates an attention mechanism and a graph convolutional neural network to achieve effective fusion of syntactic features and BERT features, thereby enhancing the model's ability to represent deep semantic meanings of the text. Ultimately, this achieves the goal of improving the quality of text summarization.

[0074] The present invention also provides a generative summarization system, specifically comprising:

[0075] The acquisition module is used to input the original text and the summary text into the BERT pre-trained model to obtain sentence feature vectors;

[0076] Builder module: Used to perform syntactic dependency analysis on the original text and the summary text using the LTP model, and to build the dependency type matrix and adjacency matrix;

[0077] The fusion module is used to calculate attention weights for the dependency type matrix and the adjacency matrix, and fuse them with the obtained sentence feature vector to obtain a multi-granularity feature semantic intermediate vector.

[0078] Output module: Used to encode the sentence feature vector as the input sequence and use an encoder with a BiLSTM network structure to obtain the output sequence;

[0079] The generation module is used to concatenate the multi-granularity feature semantic intermediate vector with the output sequence and input it into the decoder of the BiLSTM model to generate a result sequence, and then use beam search to generate the final text summary.

[0080] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of each unit, please refer to the previous method implementation examples, which will not be repeated here.

[0081] Figure 3 This is a schematic diagram of the structure of a computer device disclosed in this invention. (Reference) Figure 3 As shown, the computer device 400 includes at least a memory 402 and a processor 401; the memory 402 is connected to the processor via a communication bus 403 and is used to store computer instructions executable by the processor 401. The processor 401 is used to read computer instructions from the memory 402 to implement the steps of the generative summarization method described in any of the above embodiments.

[0082] For the above-described apparatus embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The apparatus embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0083] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal disks or removable disks), magneto-optical disks, and CD-ROMs and DVD-ROMs. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0084] Finally, it should be noted that although this specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily used to describe the features of specific embodiments of a particular invention. Certain features described in the various embodiments of this specification may also be implemented in combination in a single embodiment. On the other hand, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation of a sub-combination.

[0085] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0086] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0087] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A generative summarization method based on BERT vectorization and dependency syntax, characterized in that, The steps include the following: The original text and the summary text are fed into the BERT pre-trained model to obtain sentence feature vectors; Syntactic dependency analysis was performed on the original text and the summary text using the LTP model to construct a dependency type matrix and an adjacency matrix; Attention weights are calculated on the dependency type matrix and the adjacency matrix, and then fused with the obtained sentence feature vector to obtain a multi-granularity feature semantic intermediate vector. The sentence feature vector is used as the input sequence, and the encoder with a BiLSTM network structure is used to encode the output sequence. The multi-granularity feature semantic intermediate vector is concatenated with the output sequence and input into the decoder of the BiLSTM model to generate a result sequence. The final text summary is generated by beam search. The method for calculating attention weights includes the following steps: Adjacency matrix Dependency type matrix Attention weights are calculated from the input GCN. Nodes in layer GCN and The formula for attention weights is: ; in, It is the corresponding one in the correlation matrix , The value, " represents the dot product of vectors. , By node , The dependency type matrix and GCN vector are calculated using the following formula: ,in This represents the vector representation of the corresponding dependency type matrix. Represents a node In the Vector representation of a layer GCN; use Attention weights obtained by combining layered graph convolutional neural networks The formula for processing graph-structured data is as follows: ; in, Represents the ReLU activation function. and They represent the first The trainable matrix and bias terms of the layer, By the GCN output of the layer Vector representation of dependency type matrix It is calculated.

2. The generative summarization method according to claim 1, characterized in that, The method for obtaining sentence feature vectors includes the following steps: Insert a [CLS] tag at the beginning of each sentence in both the original text and the summary text, and add a [SEP] tag at the end of each sentence; For a training batch of original text and abstract text The corresponding vectorized results obtained after passing through the BERT model are as follows: and .

3. The generative summarization method according to claim 1, characterized in that, Syntactic dependency parsing involves obtaining a dependency syntactic tree based on the dependency relationships between each word in a sentence, and then parsing the syntactic dependency tree to generate a dependency type matrix and an adjacency matrix.

4. The generative summarization method according to claim 1, characterized in that, The method for obtaining the output sequence includes the following steps: The encoder's forward propagation LSTM layer and backward propagation LSTM layer accept sentence feature vectors. Given the input sequence, obtain the forward and reverse hidden states of the BiLSTM at time t. and Then perform vector fusion on it, the formula is as follows: , This represents the weight parameters of each layer, resulting in the encoder output sequence. .

5. The generative summarization method according to claim 4, characterized in that, The method for concatenating the multi-granularity feature semantic intermediate vector with the output sequence and inputting it into the decoder of the BiLSTM model to generate the result sequence includes the following steps: The Attention mechanism is introduced, and the decoder determines the hidden layer state based on the model input. The formula for calculating attention is as follows: , ; in, Indicate decoder The hidden layer state at any given time. This represents the hidden layer states of the encoder at all time points. It is an alignment model based on a nonlinear function; By the encoder's hidden layer vector and attention weight The weighted summation yielded the semantic vector. ; Based on semantic vectors Information interaction was achieved through an attention mechanism, and the result sequence of the nth character was predicted accordingly. The formula is as follows: ; in, This represents the LSTM prediction function. and These represent the result sequences generated by the L2R decoder and the R2L decoder, respectively. This represents the prediction result of the nth word in the decoder; When using an attention mechanism as features for combination during decoding and When predicting the nth word, the formula is: Satisfying in the decoder .

6. The generative summarization method according to claim 1, characterized in that, The beam search method includes the following steps: taking the result sequence generated by the decoder as input, recording the top-k results with the highest prediction probabilities in two directions at each time step, and finally selecting the result with the highest probability as the summary.

7. The system corresponding to the generative summarization method according to any one of claims 1-6, comprising: The acquisition module is used to input the original text and the summary text into the BERT pre-trained model to obtain sentence feature vectors; Builder module: Used to perform syntactic dependency analysis on the original text and the summary text using the LTP model, and to build the dependency type matrix and adjacency matrix; The fusion module is used to calculate attention weights for the dependency type matrix and the adjacency matrix, and fuse them with the obtained sentence feature vector to obtain a multi-granularity feature semantic intermediate vector. Output module: Used to encode the sentence feature vector as the input sequence and use an encoder with a BiLSTM network structure to obtain the output sequence; The generation module is used to concatenate the multi-granularity feature semantic intermediate vector with the output sequence and input it into the decoder of the BiLSTM model to generate a result sequence, and then use beam search to generate the final text summary.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed, it implements the steps of the generative summarization method according to any one of claims 1-6.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the generative summarization method as described in any one of claims 1-6.

Images