Chinese abstract generation method based on component syntactic analysis

Abstract

The invention discloses a Chinese abstract generation method based on component syntactic analysis, which includes: preprocessing a document to obtain a text sentence set; based on the text sentence set, using a semantic extraction model to obtain text semantic information coding; based on the text sentence set, generating a component syntactic analysis structure tree of each sentence, and converting the component syntactic analysis structure tree of each sentence into a component syntactic structure serialization coding based on a span method; inputting the text semantic information coding and the component syntactic structure serialization coding into an encoder for integrated coding; decoding the integrated coding transmitted by the encoder through a decoder to generate a text abstract. The original grammatical structure of the text can be brought out for supervising the text abstract generation process, thereby solving the problems of text abstract accuracy and readability.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a Chinese abstract generation method based on component syntactic analysis. Background Art

[0002] The National Natural Science Foundation of China includes applications for basic theory and applied basic theory research, which is theoretical work that reveals the universal laws, basic principles, and movement nature of natural phenomena in nature. In the application process for the fund, review experts need to efficiently and accurately obtain effective information from the massive application text and make a review. Text summarization technology aims to automatically extract key information from a large amount of application text data, which can play an auxiliary role in the expert review process to a certain extent. However, the fund application contains a large number of scientific research professional terms. The existing text summary model is difficult to fully mine and understand the semantic information and grammatical structure information contained in the professional terms. The generated summaries often have defects such as missing key information, incomplete coverage, and incoherent grammar.

[0003] Most of the application documents are long, and the traditional sequence-based text summary generation model cannot be calculated in parallel and does not pay attention to the text grammatical information, resulting in the generated summary being unclear and not in line with grammatical rules when processing long texts. In text summary generation, the main problems faced in the generation process are that the summary is inadequate, inaccurate, and does not conform to human language habits. If the deep semantic information of the text can be obtained and the syntactic structure information can be introduced, the integrated encoding will contain text semantic information and grammatical information, which can make the summary more grammatical and the main theme more prominent. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a Chinese abstract generation method, device and storage medium based on component syntactic analysis to solve the problem that the abstracts obtained by the existing text abstract generation methods are unclear in theme and low in readability.

[0005] In order to achieve the above object, the present invention adopts the following technical solution.

[0006] A Chinese summary generation method based on component syntactic analysis, comprising:

[0007] Preprocess the document to obtain a text sentence set;

[0008] Based on the text sentence set, the semantic information encoding of the text is obtained using the semantic extraction model;

[0009] Based on the text sentence set, generate a component syntactic analysis structure tree for each sentence, and convert the component syntactic analysis structure tree of each sentence into a component syntactic structure serialization encoding based on a span method;

[0010] The text semantic information encoding and the component syntactic structure serialization encoding are input into the encoder for integrated encoding;

[0011] The decoder decodes the integrated code sent by the encoder to generate a text summary.

[0012] Furthermore, the semantic extraction model adopts a PEGASUS model.

[0013] Furthermore, Stanford CoreNLP is used to generate a component syntactic analysis structure tree for each sentence.

[0014] Furthermore, the method of converting the component syntactic analysis structure tree of each sentence into a component syntactic structure serialization encoding based on the span includes:

[0015] For each sentence's component syntactic analysis structure tree, recursively combine the two rightmost child nodes to convert it into a right binary tree;

[0016] Represent the resulting right binary tree as a span table;

[0017] The span table is divided into n parts according to its right boundary, where n is the sentence length. All left children including the root node in the binary tree are distributed in the n parts, and the right boundaries of all left children correspond one by one to the values ​​in [1, n]. The right boundaries of all left children are used as serialized subscripts, and the corresponding left boundaries are used as serialized values, thus obtaining the serialized encoding of the component syntactic structure after the linearization of the span table.

[0018] Furthermore, the encoder adopts a semantic structure encoder based on an attention mechanism, which first fuses the text semantic information encoding and the component syntactic structure serialization encoding, as shown in the following formula:

[0019]

[0020] In the formula, represents the final hidden state of the encoder, d represents the serialization encoding of the component syntactic structure, h represents the text semantic information encoding, is the GLU activation function, b represents the offset, and W represents the learnable parameter;

[0021] The encoder's attention medium will reallocate the attention size of other words according to the decoder input at the current time t, and generate a contextual semantic vector C that changes with the current word t , the attention mechanism formula is as follows:

[0022]

[0023]

[0024]

[0025] In the formula, a t，i Represents the attention weight, through e t，i Score to calculate; represents the i-th hidden state of the encoder, S t-1 represents the hidden state of the decoder at the previous time t-1, n represents the length of the sentence, W h 、V h Both represent weight matrices.

[0026] Furthermore, the decoder uses a unidirectional GRU network, whose input is the output y of the decoder at the previous time t-1 t-1 , the hidden state S of the decoder at the previous time t-1 t-1 , the context semantic vector C at the current time t t Together, the final hidden state of the encoder As the first input of the decoder, the unidirectional GRU network structure formula is:

[0027] z t =σ(W z S t-1 +W z C t +W z y t-1 )

[0028] r t =σ(W r +S t-1 +W r C t +W r y t-1 )

[0029]

[0030]

[0031] In the formula, z t and r t Represent the update gate and reset gate respectively; Yes C t ,y t-1 and S at the previous moment t-1 The summary of the new vector is calculated Contains the above information and y t-1 ; σ and tanh represent activation functions; ⊙ represents the product of matrices; W z is the weight parameter of the update gate, W r To reset the gate weight parameters, for The weight parameter S t represents the hidden state of the decoder at the current time t;

[0032] The position of the word in the vocabulary is obtained through the softmax layer, as shown in the formula:

[0033] P(y t |y1,y2,...,y n , C t )=softmax(S t )

[0034] The decoder’s hidden state S t As shown in the following formula, GRU stands for Gated Recurrent Unit Decoder;

[0035] S t =GRU(S t-1 , C t ,y t-1 )

[0036] Finally, a search algorithm is used to generate the optimal solution.

[0037] Furthermore, the search algorithm adopts a beam search algorithm.

[0038] Beneficial Effects

[0039] The invention proposes a Chinese summary generation method based on component syntactic analysis. First, a semantic extraction model is used to extract the text semantic information encoding of the text, which contains the deep semantic information of the text; at the same time, the component syntactic analysis structure tree of the text is obtained, and a component syntactic structure serialization encoding is obtained through a span-based method. The encoding contains the syntactic structure information of the text, which can be used to supervise the summary generation process, so that the generated summary is more in line with human language habits; then, the text semantic information encoding and the component syntactic structure serialization encoding are jointly input into the attention-based encoder structure, and the encoder is mainly used to fuse the text semantic information encoding and the component syntactic structure serialization encoding, and the keyword semantic vector is calculated using the attention mechanism, and it is combined with the component syntactic structure serialization encoding and then derived to the next word item, so that the integrated encoding contains the grammatical structure information while retaining the semantic information. Finally, a decoder is used to decode the integrated encoding transmitted by the encoder, and finally a summary result that is more in line with grammatical rules and has a more significant theme is generated.

[0040] The present invention can realize the task of generating Chinese abstracts of long texts, solve the problem of unclear abstract theme and abstract not conforming to human language habits caused by too long texts, and play an important auxiliary role in the quantification of application documents. The core significance of this solution is that it can bring out the original grammatical structure of the text, which is used to supervise the text abstract generation process, and solve the problems of text abstract accuracy and readability. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 is a flow chart of a Chinese abstract generation method based on component syntactic analysis provided by an embodiment of the present invention;

[0043] Figure 2 It is a schematic diagram of the component syntactic structure serialization encoding generation process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] To make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described in detail below. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other implementation methods obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0045] like Figure 1 As shown, the embodiment of the present invention discloses a Chinese abstract generation method based on component syntactic analysis, comprising:

[0046] S1: Preprocess the document for which summary is to be generated to obtain a set of text sentences.

[0047] Specifically, stop words are filtered out from each sentence in the document in turn, and only words with specified parts of speech are retained, thereby obtaining a new set of text sentences.

[0048] S2: Based on the text sentence set, the semantic extraction model is used to obtain the text semantic information encoding.

[0049] During implementation, the semantic extraction model adopts the PEGASUS model. The PEGASUS model uses a new self-supervised target to pre-train on a large text corpus. The model adds a generator to the pre-training structure of Bert, so that it can pre-train for generation tasks. As pre-training targets, the model uses GSG (Gap Sentences Generation) and MLM (Masked Language Model) at the same time. Specifically, assuming that the original text has a total of m sentences in the sentence set after text preprocessing, 2 / 3 of the sentence set is regarded as the original text, and 1 / 3 is regarded as the summary to form a training set, forming a pseudo data set, and the PEGASUS model is pre-trained using the pseudo data set.

[0050] GSG (Gap Sentences Generation): Cover a key sentence from the document and generate it based on other sentences in the document. This helps the model understand the entire document and generate similar summaries.

[0051] MLM (Masked Language Model): According to BERT, 15% of the characters in the input text are selected, and (1) 80% of them are replaced with the symbol [MASK]; (2) 10% are randomly replaced with other characters; and (3) 10% remain unchanged.

[0052] S3: Based on the text sentence set, generate a component syntactic analysis structure tree for each sentence, and convert the component syntactic analysis structure tree of each sentence into a component syntactic structure serialization encoding based on the span method.

[0053] The component syntactic analysis structure tree of a sentence is to divide the sentence into components, and the larger components are obtained by combining smaller components. In this embodiment, Stanford CoreNLP is used to generate the component syntactic analysis structure tree of each sentence. StanfordCoreNLP is a natural language processing toolkit, which currently supports Arabic, Chinese, English, French, German, Spanish, etc. Stanford CoreNLP integrates functions such as word segmentation, part-of-speech tagging, and syntactic analysis. Therefore, by using StanfordCoreNLP to perform grammatical analysis on the sentence, the component syntactic analysis structure tree corresponding to each sentence is obtained.

[0054] Directly constructing a model with grammatical information can not only capture and record the structure of the entire component syntactic analysis structure tree, but also distinguish the content of any two different words. However, there is no effective method to directly construct such a model with grammatical structure information. This embodiment uses an alternative method - converting the component syntactic analysis structure tree into a linearized structure label sequence through a span-based method. Figure 2Shown is the linearization process of the sentence "I love writing code."

[0055] First, we use StanfordCoreNLP to obtain the original component parsing structure tree, and then recursively combine the two rightmost child nodes to convert it into a right binary tree. Next, we represent the tree as a span table and divide it into five parts according to the right boundary of the span table.

[0056] Definition: Let W = (w1, w2, ..., w n ) is a sentence; define (i, j): is a sentence from w i+1 to w i span, 0≤i<j≤n; given a sentence W and its component syntactic analysis structure tree T, we call d=(d1,d2,...,d n ) is the linearization of T (i.e., the serialization encoding of the component syntactic structure), where d i ∈{0, 1, ..., i-1} and (d i , i) is the longest span in T that ends with i.

[0057] In the span table (c) of the right binary tree (b), the gray part is all the left children (including the root node), and the black part is all the right children. The right boundaries of all left children must not be repeated, so they must correspond one-to-one to the values ​​in [1, n]. Then they can be used as serialized subscripts, and the corresponding left boundaries are used as serialized values. For example, Figure 2 The span (1, 4) in the span table represents the phrase "I love writing code", so the value at index 4 of the serialized array d is d4=1. i The sequence is the serialized encoding of the component syntactic structure we need.

[0058] S4: The text semantic information encoding and the component syntactic structure serialization encoding are input into the encoder for integrated encoding.

[0059] In this embodiment, the encoder adopts a semantic structure encoder based on an attention mechanism. The semantic structure encoder based on an attention mechanism is a dual encoder model. Encoder encoder1 uses the generative pre-training model PEGASUS to obtain the corresponding hidden state information h in the text semantic information encoding. i(i=1, 2, ..., p); encoder encoder2 uses CNN (convolutional neural networks, CNN) to extract text grammatical information, and serializes and vectorizes the component syntactic structure tree; after fusing the feature information extracted by CNN with the PEGASUS hidden layer state, an attention mechanism is constructed to fuse the text semantic information encoding with the span-based component syntactic structure serialization encoding, so that the integrated encoding contains the overall information of the text.

[0060] First, the text semantic information encoding and the component syntactic structure serialization encoding are integrated as shown in the following formula:

[0061]

[0062] In the formula, represents the final hidden state of the encoder, d represents the serialized encoding of the component syntactic structure, h is the hidden state value of the PEGASUS model, is the GLU activation function, b represents the offset, and W represents the learnable parameter, which can be obtained through training;

[0063] After fusion, the hidden state value containing the syntactic structure features of the text is obtained. In the traditional Seq2Seq model, the encoder encodes the full text and generates a fixed context semantic vector C, and the decoder decodes C to output the final result. The model in this embodiment uses the attention mechanism to weight the target data. The attention medium of the encoder will reallocate the attention size of other words according to the decoder input at the current time t, and generate a context semantic vector C that changes with the current word. t , the attention mechanism formula is as follows:

[0064]

[0065]

[0066]

[0067] In the formula, a t，i Represents the attention weight, through e t，i Score to calculate; represents the i-th hidden state of the encoder, S t-1 represents the hidden state of the decoder at the previous time t-1, W h 、V h Both represent weight matrices, which can be obtained through training. According to the hidden state of the encoder and the decoder hidden state S t-1 To calculate e t，i Score, then according to et，i Score calculation attention weight a t，i , each hidden state of the encoder With attention weight a t，i Multiply and then perform weighted summation to obtain the context semantic vector C at the current time t t .

[0068] S5: The decoder decodes the integrated code sent by the encoder to generate a text summary.

[0069] In this embodiment, the decoder adopts a unidirectional GRU network, whose input is the output y of the decoder at the previous time t-1 t-1 , the hidden state S of the decoder at the previous time t-1 t-1 , the context semantic vector C at the current time t t Together, the final hidden state of the encoder As the first input of the decoder, the unidirectional GRU network structure formula is:

[0070] z t =σ(W z S t-1 +W z C t +W z y t-1 )

[0071] r t =σ(W r +S t-1 +W r C t +W r y t-1 )

[0072]

[0073]

[0074] In the formula, z t and r t They represent the update gate and reset gate respectively. The update gate can decide which information to discard and which new information to add, and the reset gate is used to decide the extent to which the previous information is discarded. Yes C t ,y t-1 and S at the previous moment t-1 The summary of the new vector is calculated Contains the above information and y t-1 ; σ and tanh represent activation functions; ⊙ represents the product of matrices; W z , W r , Represents the weight matrix, which can be obtained through training, W z is the weight parameter of the update gate, W r To reset the gate weight parameters, for The weight parameter S t represents the hidden state of the decoder at the current time t, Indicates S t intermediate state.

[0075] The decoder obtains the output state at the current moment, and then the softmax layer obtains the position of the word in the vocabulary, as shown in the formula:

[0076] P(y t |y1,y2,...,y n , C t )=softmax(S t )

[0077] The decoder’s hidden state S t As shown in the following formula, GRU stands for Gated Recurrent Unit Decoder;

[0078] S t =GRU(S t-1 , C t ,y t-1 )

[0079] Finally, a search algorithm is used to generate the optimal solution. In this embodiment, a beam search algorithm is used to reduce computational complexity and improve accuracy. The traditional decoder uses a greedy search algorithm, that is, the maximum probability is selected in the probability matrix to generate the target word. The beam search algorithm is a heuristic graph search algorithm. The beam search selects the first q maximum probabilities and considers more candidate spaces, so that better generation results can be obtained. In this embodiment, the width q of the beam search is set to 10.

[0080] The present invention can realize the task of generating Chinese abstracts of long texts, solve the problem of unclear abstract theme and abstract not conforming to human language habits caused by too long texts, and play an important auxiliary role in the quantification of application documents. The core significance of this solution is that it can bring out the original grammatical structure of the text, which is used to supervise the text abstract generation process, and solve the problems of text abstract accuracy and readability. Firstly, a generative pre-training model, PEGASUS, is used. This model combines the GSG (Gap Sentences Generation) and MLM (Masked Language Model) methods to extract the text semantic information encoding of the text, which contains the deep semantic information of the text. At the same time, the component syntactic analysis structure tree of the text is obtained through Stanford coreNLP, and the component syntactic structure serialization encoding is obtained through a span-based method. The encoding contains the syntactic structure information of the text and can be used to supervise the summary generation process to make the generated summary more in line with human language habits. Then, the text semantic information encoding and the component syntactic structure serialization encoding are jointly input into the attention-based encoder structure. The encoder is mainly used to fuse the text semantic information encoding and the component syntactic structure serialization encoding. The attention mechanism is used to calculate the keyword semantic vector, which is combined with the component syntactic structure serialization encoding and then derived to the next word item, so that the integrated encoding contains grammatical structure information while retaining the semantic information. Finally, a unidirectional GRU network based on beam search is used to decode the integrated code transmitted by the decoder, thereby reducing the computational complexity while improving the accuracy, and finally generating a summary result that is more grammatical and has a more prominent theme.

[0081] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0082] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A Chinese abstract generation method based on component syntactic analysis, characterized in that: include: Preprocess the document to obtain a text sentence set; Based on the text sentence set, the semantic information encoding of the text is obtained using the semantic extraction model; Based on the text sentence set, generate a component syntactic analysis structure tree for each sentence, and convert the component syntactic analysis structure tree of each sentence into a component syntactic structure serialization encoding based on the span method; The text semantic information encoding and the component syntactic structure serialization encoding are input into the encoder for integrated encoding; The decoder decodes the integrated code sent by the encoder to generate a text summary.

2. The Chinese abstract generation method based on component syntactic analysis according to claim 1 is characterized in that: The semantic extraction model adopts the PEGASUS model.

3. The Chinese abstract generation method based on component syntactic analysis according to claim 1 is characterized in that: Stanford CoreNLP is used to generate a component syntactic analysis structure tree for each sentence.

4. The Chinese abstract generation method based on component syntactic analysis according to claim 1 or 3 is characterized in that: The method of converting the component syntactic analysis structure tree of each sentence into a component syntactic structure serialization encoding based on the span method includes: For each sentence's component syntactic analysis structure tree, recursively combine the two rightmost child nodes to convert it into a right binary tree; Represent the resulting right binary tree as a span table; The span table is divided into n parts according to its right boundary, where n is the sentence length. All left children including the root node in the binary tree are distributed in n parts, and the right boundaries of all left children correspond one by one to the values ​​in [1,n]. The right boundaries of all left children are used as serialized subscripts, and the corresponding left boundaries are used as serialized values, thus obtaining the serialized encoding of the component syntactic structure after the linearization of the span table.

5. The Chinese abstract generation method based on component syntactic analysis according to claim 1 is characterized in that: The encoder adopts a semantic structure encoder based on the attention mechanism, which first integrates the text semantic information encoding and the component syntactic structure serialization encoding, as shown in the following formula: In the formula, represents the final hidden state of the encoder, d represents the serialization encoding of the component syntactic structure, h represents the text semantic information encoding, is the GLU activation function, b represents the offset, and W represents the learnable parameter; The encoder's attention medium will reallocate the attention size of other words according to the decoder input at the current time t, and generate a contextual semantic vector C that changes with the current word t , the attention mechanism formula is as follows: In the formula, a t,i Represents the attention weight, through e t,i Score to calculate; represents the i-th hidden state of the encoder, S t-1 represents the hidden state of the decoder at the previous time t-1, n represents the length of the sentence, W h 、V h Both represent weight matrices.

6. The Chinese abstract generation method based on component syntactic analysis according to claim 5 is characterized in that: The decoder uses a unidirectional GRU network, whose input is the output y of the decoder at the previous time t-1 t-1 , the hidden state S of the decoder at the previous time t-1 t-1 , the context semantic vector C at the current time t t Together, the final hidden state of the encoder As the first input of the decoder, the unidirectional GRU network structure formula is: With t =σ(W z S t-1 +W z C t +W z and t-1 ) r t =σ(W r +S t-1 +W r C t +W r y t-1 ) In the formula, z t and r t represent the update gate and reset gate respectively; Yes C t ,y t-1 and S at the previous moment t-1 The summary of the new vector is calculated Contains the above information and y t-1 ; σ and tanh represent activation functions; ⊙ represents the product of matrices; W z is the weight parameter of the update gate, W r To reset the gate weight parameters, for The weight parameter S t represents the hidden state of the decoder at the current time t; The position of the word in the vocabulary is obtained through the softmax layer, as shown in the formula: P(and t |y1,y2,…,y n ,C t )=softmax(S t ) The decoder’s hidden state S t As shown in the following formula, GRU stands for Gated Recurrent Unit Decoder; S t =GRU(S t-1 ,C t ,y t-1 ) Finally, a search algorithm is used to generate the optimal solution.

7. The Chinese abstract generation method based on component syntactic analysis according to claim 6 is characterized in that: The search algorithm adopts a beam search algorithm.

Images