Method and system for automatic generation of text summaries in the field of earth sciences

By using the combination of BERT network model and BiLSTM components, Chinese text summary is extracted and generated, and the ambiguity and inaccuracy of the generation of Chinese text summary in the prior art is solved, and more efficient Chinese text summary generation is achieved, covering the core information of the original text effect.

CN113935308BActive Publication Date: 2024-12-06CHINA UNIV OF GEOSCIENCES (WUHAN)
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202111051884.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-08
Publication Date
2024-12-06
Estimated Expiration
2041-09-08

AI Technical Summary

Technical Problem

The existing text abstract generation methods are mainly aimed at English text and are difficult to apply to Chinese texts. Especially in the field of geoscience, there are problems of ambiguity and inaccuracy in the generation of Chinese abstracts, and traditional methods are difficult to effectively represent and capture the complex grammar of Chinese text. and semantics.

Method used

Using the text summary generation method based on the BERT network model, convert unit words into word vector matrix through word embedding operation, local and global features are extracted using convolutional neural network and self-attention mechanism, and encoding and decoding are combined with gating units and BiLSTM components. , generate accurate Chinese abstracts.

Benefits of technology

It improves the accuracy and reliability of Chinese text summary, can effectively cover the core information of the original text, and the generated summary is smoother and simpler, with better performance than traditional models, such as word2vec and GloVe.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113935308B_ABST
    Figure CN113935308B_ABST
Patent Text Reader

Abstract

The present invention provides a method and system for automatically generating text summaries in the field of earth science. The method comprises the following steps: using a fine-tuned BERT embedding component to initialize the value of an input sequence based on word embedding, a BiLSTM component in an encoder to encode the input received from a previous layer; a convolutional gating unit to retrain core information according to the previous output of each time step; calling a self-attention mechanism to calculate the weight of the vocabulary itself, fully considering the semantic and grammatical connections between different words in a sentence, thereby further strengthening the global information; a decoder part to decode a context vector output by the encoder, a BiLSTM component in the decoder to decode the sentence output by the encoder to obtain an output sequence; a fine-tuned BERT network model to capture the contextual relationship between sentences, so that a complete context is obtained at every moment, and ultimately the generated summary is more accurate.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method and system for automatically generating text summaries in the field of earth science. Background Art

[0002] Automatic summarization is an important task in the field of natural language processing. Its purpose is to retain the core elements of a set of documents or a set of documents related to a topic while generating a condensed text expression, which can effectively alleviate the contradiction between the huge amount of data, low information density and the user's efficient and accurate acquisition of information. In recent years, automatic text summarization technology has been used in many fields such as automatic report generation and news headline generation.

[0003] Existing automatic summary generation methods are mainly divided into three categories: extractive summary generation, generative summary generation and hybrid summary generation.

[0004] Extractive summarization evaluates the importance of sentences by considering their position, word frequency, keywords, etc. in the original text, and selects sentences with high importance from the original text to form a summary; generative summarization compresses and refines sentences at the word level based on an understanding of the semantics of the article, and ultimately generates a summary with fewer words and clearer language; and hybrid summarization combines the advantages of extractive text summarization and generative text summarization, improving the language fluency and conciseness of the final generated summary.

[0005] At present, research on automatic summary generation has only provided partial solutions. First, many existing text summary generation methods are mainly for English texts, and are not suitable for Chinese texts, especially lacking Chinese summary generation scenarios for specific professional fields. The grammar and semantics of Chinese texts are much more complex than those of English texts, especially in the field of earth sciences, where the presence of a large number of professional vocabulary further increases the difficulty. Chinese is character-based, while English is word-based, and there are no separators between words in Chinese, so it is difficult to determine the composition of a word, and there are a lot of ambiguous situations in Chinese texts. Second, most existing text summarization methods rely on traditional bag-of-words representations involving high-dimensional sparse data, which makes it difficult to represent and capture relevant information. Third, recent research on Chinese text summarization is still in its infancy, and Chinese text summarization cannot reach the same level of maturity and reliability as English text summarization. Therefore, there is an urgent need to build an automatic text summarization algorithm for the field of Chinese earth sciences. Summary of the invention

[0006] A major problem solved by the present invention is that the traditional abstract generation method is mainly for English texts, which is not applicable to the field of Chinese geology, and the generated Chinese abstract texts are highly ambiguous and inaccurate.

[0007] According to one aspect of the present invention, a method for automatically generating text summaries in the field of earth science is provided, the method comprising:

[0008] S1. Obtain the original data of titles and abstracts of geological literature;

[0009] S2, converting the unit words in the original data into a word vector matrix based on a word embedding operation;

[0010] S3. Encode using the pre-trained BERT network model:

[0011] S301, input the word vector matrix as an input sequence into the BERT network model, use a convolutional neural network to extract local connection features between unit words, and use a ReLU function to generate an output result, the output result is:

[0012]

[0013] Among them, m i is the i-th feature map, f represents the ReLU function, w k represents the first local vector, q represents the second local vector, k∈{3,5,7}, w j k is the jth first local vector, q j is the jth second local vector, q [i:i+k-1] represents the [i:i+k-1] vector;

[0014] S302, using the output result, calling the self-attention mechanism to obtain global connection features;

[0015] S303, using a gating unit to globally encode the local connection features and the global connection features to generate an encoder output, where the encoder output is:

[0016]

[0017] in, is the encoder output, σ(g) is the convolutional neural network function of the gating unit g, R T×dim It is a T×dim matrix, where T represents the sequence length and dim represents the vector dimension, i.e. the number of layers;

[0018] S4. Decode the encoder output:

[0019] S401, using the global attention mechanism, using the encoder hidden layer state to define an attention mechanism-based context vector for the decoder step;

[0020] S402, predicting the next word according to the context vector using the BiLSTM component embedded in the decoder, and obtaining an output sequence;

[0021] S403: Use the BERT network model to capture the contextual relationship in the output sequence and generate a complete summary.

[0022] Furthermore, the word vector matrix is ​​expressed as:

[0023]

[0024] Among them, X 1:T represents the input sequence, x1, x2, x3 and x T Respectively represent the 1st, 2nd, 3rd and Tth basic unit words in the sequence, R 1×T Represents a matrix of length 1×T.

[0025] Furthermore, step S301 also includes: using the "BERT-Base Chinese" model to fine-tune the input sequence.

[0026] Furthermore, step S301 further includes: encoding the output result as input of a BiLS TM component embedded in the encoder, so that the encoded content is transmitted bidirectionally.

[0027] Furthermore, step S401 further includes: considering the hidden layer states of all encoders, and comparing the current hidden layer state of the decoder with the hidden layer state of each encoder.

[0028] According to another aspect of the present invention, a system for automatically generating text summaries in the field of earth science is also disclosed. The system includes: an encoder module, a decoder module and a training module. The training module is used to pre-train the encoder module, the encoder module is used to encode an input sequence, and the decoder module is used to decode the output result of the encoder module.

[0029] Further, the encoder module includes: a first BERT network model, a first BiLSTM component, a convolutional gating unit, and a self-attention mechanism;

[0030] The first BERT network model is fine-tuned to initialize the value of the input sequence through a word embedding operation;

[0031] The first BiLSTM component is used to encode the received input sequence;

[0032] The convolutional gating unit is used to retrain the core information based on the previous output at each time step;

[0033] The self-attention mechanism is used to calculate the weight of the word itself, fully consider the semantic and grammatical connections between different words in the sentence, and strengthen the global information.

[0034] Furthermore, the decoder module comprises:

[0035] Global attention mechanism, second BiLSTM component and second BERT network model;

[0036] The global attention mechanism is used to define an attention-based context vector for the decoder step using the encoder hidden layer state;

[0037] The second BiLSTM component is used to decode the sentence output by the encoder to obtain an output sequence;

[0038] The second BERT network model is fine-tuned to capture the contextual relationship between sentences in the output sequence, thereby obtaining a complete summary context.

[0039] The dataset used in the method of the present invention is mainly retrieved from published literature data, including geological reviews, geological journals, mineral geology and Chinese geology. We collect abstracts and titles from published literature, and then use data preprocessing to clean the collected data. A series of rules are developed to clean and format the corpus, and since the length of the abstract and title may affect the performance of the model, we have statistically analyzed the length distribution of the abstract and title, and developed an efficient and effective automatic summarization system using BERT, and proposed a new Chinese text summarization method based on a bidirectional long short-term memory network model. In the evaluation of the automatic summary generation task, ROUGE-1, ROUGE-2, and ROUGE-L are used as evaluation indicators, and a series of tests are conducted on real data sets based on the trained model, and the generated titles are directly compared with the original text summary titles. The results show that the titles generated by this method are better and can cover the core information of the original text, further confirming the effectiveness of the method proposed in this method. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0041] Figure 1 Schematic diagram of the process flow of automatic text summary generation in an embodiment of the present invention.

[0042] Figure 2 Schematic diagram of the overall structure of the text summarization model in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] Various exemplary embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0044] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0045] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0046] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0047] Technologies, methods, and apparatus known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, such technologies, methods, and apparatus should be considered part of the authorization specification.

[0048] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0049] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0050] Embodiment 1, as Figure 1 As shown, it is a schematic diagram of the steps of automatic generation of text summaries, and the method steps specifically include:

[0051] S1. Obtain the raw data of titles and abstracts of geological literature; collect abstracts and titles from published literature by retrieving published literature data, including Geological Reviews, Acta Geologica Sinica, Mineral Geology, and Geology of China, and then clean the collected raw data through preprocessing.

[0052] S2. Convert the unit words in the original data into a word vector matrix based on word embedding operation; the vector matrix is ​​expressed as:

[0053]

[0054] Among them, X 1:T represents the input sequence, x1, x2, x3 and x T Respectively represent the 1st, 2nd, 3rd and Tth basic unit words in the sequence, R 1×T Represents a matrix of length 1×T.

[0055] Specifically, taking the collected raw data as input, the basic word units in the collected literature data are converted into vectors based on the word embedding operation. This process is initialized by using the fine-tuned BERT network model. The BERT network model uses the "BERT-Base Chinese" model to fine-tune the input sequence during training.

[0056] S3. Encoding using the pre-trained BERT network model: The encoder is mainly divided into four parts: a fine-tuned BERT model, a BiLSTM component, a set of convolutional gated units, and a self-attention mechanism. The purpose of fine-tuning the BERT embedding component is to initialize the value of the input sequence based on word embedding. For the input sequence, the BiLSTM component is used to encode the input received from the previous layer; the convolutional gated unit focuses on retraining the core information based on the previous output of each time step; the self-attention mechanism is to calculate the weight of the vocabulary itself, fully considering the semantic and grammatical connections between different words in the sentence, so as to further strengthen the subsequent global information.

[0057] S301, input the above-mentioned word vector matrix as an input sequence into the BERT network model, use the convolutional neural network to extract the local connection features between unit words, especially obtain n-gram features. And based on the average length of the input sequence X, use filters and receptive fields to grasp richer local connection features. Then, the two local vectors q∈R |q| and w k ∈R k∈{3,5,7} Perform convolution to form a feature map m∈R |q|-k+1 . And use the ReLU function to generate the output result, the output result is:

[0058]

[0059] Among them, m i is the i-th feature map, f represents the ReLU function, w k represents the first local vector, q represents the second local vector, k∈{3,5,7}, w j k is the jth first local vector, q j is the j-th second local vector.

[0060] The output of the BERT network model is used as the input of the BiLSTM component and then encoded so that the encoded content can be output from left to right and from right to left.

[0061] Let the expression:

[0062] h=GLU(h1,h2,h3,...ht )∈R T×dim (3)

[0063] is the content output of the BiLSTM component from left to right and from right to left, where dim is the number of layers, GLU(·) represents the gated linear unit, and h i The hidden state of the encoder at time step t can be expressed as:

[0064]

[0065] Among them, h t is the hidden state at time t, represents the hidden state on the left at time t, Represents the hidden state on the right at time t, R 1×2dim is a matrix of length 1×2dim;

[0066] S302, using the output result, calling the self-attention mechanism to obtain global connection features;

[0067] Specifically, the outputs obtained by CNN convolution are weighted by the self-attention mechanism, which fully considers the semantic and grammatical connections between different words in the sentence, thereby further strengthening the subsequent global information. This enables the network model to learn long-term dependencies without consuming too much computing resources. The weight calculation formula is as follows:

[0068]

[0069] Among them, Q and V represent different matrices generated by CNN, K represents the learnable matrix, and d k is the dimension of vector Q.

[0070] S303, using a gating unit to globally encode the local connection features and the global connection features to generate an encoder output, where the encoder output is:

[0071]

[0072] in, is the encoder output, σ(g) is the convolutional neural network function of the gating unit g, R T×dim It is a T×dim matrix, where T represents the sequence length, dim represents the vector dimension, i.e. the number of layers, and h represents the hidden layer state of the encoder;

[0073] S4. Decode the encoder output: The decoder part decodes the context vector c output by the encoder. T×dimT represents the sequence length, dim represents the vector dimension, and the decoder consists of three parts: Globalattention, BiLSTM, and fine-tuned BERT model.

[0074] S401, using the global attention mechanism, using the encoder hidden layer state to define the context vector based on the attention mechanism for the decoder step; it can make the encoder encode a different context vector c for each time step of the input sequence. When decoding, combine each different context vector c for decoding output, so that the result is more accurate. Considering the hidden layer states of all encoders, the current hidden state h of the decoder is t With each encoder’s hidden state For comparison, the calculation formula is as follows:

[0075]

[0076] Among them, t is the current time, a t is a variable-length alignment vector whose length is the length of the encoder part in the time series. t (s) is a vector obtained by comparing the state of an encoder and the state of a decoder. s is the original sentence. t (s) vectors are integrated into a weight matrix W a , the calculation formula of the weight matrix is ​​as follows:

[0077] a t =softmax(W a h t ) (8)

[0078] Perform a weighted average operation on the weight matrix to obtain the vector c at time t t .

[0079] S402, predicting the next word according to the context vector using the BiLSTM component embedded in the decoder, and obtaining an output sequence;

[0080] S403: Utilize the BERT network model to capture the contextual relationship in the output sequence, so that the complete context can be obtained at every moment, and a complete and accurate summary can be generated.

[0081] Embodiment 2 discloses a system for automatically generating text summaries in the field of earth science, including: an encoder module, a decoder module and a training module, the training module is used to pre-train the encoder module, the encoder module is used to encode an input sequence, and the decoder module is used to decode the output result of the encoder module.

[0082] The encoder module includes a first BERT network model, a first BiLSTM component, a convolutional gating unit and a self-attention mechanism; the first BERT network model is fine-tuned to initialize the value of the input sequence based on the word embedding operation; the first BiLSTM component is used to encode the received input sequence; the convolutional gating unit is used to retrain the core information according to the previous output of each time step; the self-attention mechanism is used to calculate the weight of the vocabulary itself, fully consider the semantic and grammatical connections between different words in the sentence, and strengthen the global information.

[0083] The decoder module includes: a global attention mechanism, a second BiLSTM component and a second BERT network model; the global attention mechanism is used to define an attention-based context vector for the decoder step using the encoder hidden layer state; the second BiLSTM component is used to decode the sentence output by the encoder to obtain an output sequence; the second BERT network model is fine-tuned to capture the contextual relationship between sentences in the output sequence, thereby obtaining a complete summary context.

[0084] The experimental results are given below to prove the technical effect of the present invention:

[0085] According to Tables 1 and 2, on the one hand, the summary generation method of the present invention is practiced using the fine-tuned BERT model, and better and more accurate summary text results are achieved than other models such as word2vec and GloVe models. On the other hand, generating text summaries based on the neural network model trained on the development dataset has better performance than generating text summaries based on other models (i.e., RNN, RNN-context, Super-AE, CGU). On the other hand, the experimental results in Tables 1 and 2 show that deep learning represented by BERT significantly improves the results of the summary system and outperforms the most advanced generation methods. Since the BERT model learns most of the language information during unsupervised pre-training, it can be fine-tuned even with a small dataset, so the performance is better than that of CNN or RNN downstream-based models that must be trained from scratch.

[0086] Table 1 Quantitative evaluation results compared with other word embedding models

[0087] Model ROUGE-1 ROUGE-2 ROUGE-L BLEU Word2Vec+Classifier 0.705 0.611 0.625 0.513 GloVe+Classifier 0.713 0.623 0.641 0.531 GloVe+RL 0.744 0.649 0.669 0.599 GloVe+BiLSTM 0.775 0.691 0.701 0.609 GloVe+BiLSTM+Attention 0.789 0.711 0.725 0.631 The present invention 0.814 0.781 0.846 0.676

[0088] Table 2 Quantitative evaluation results compared with other models

[0089] Model ROUGE-1 ROUGE-2 ROUGE-L BLEU RNN 0.738 0.705 0.781 0.621 RNN-context 0.744 0.721 0.797 0.633 Super-AE 0.781 0.743 0.805 0.642 CGU 0.792 0.765 0.821 0.655 The present invention 0.814 0.781 0.846 0.676

[0090] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

[0091] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Claims

1. A method for automatically generating text summaries in the field of earth sciences, characterized in that: The method comprises: S1. Obtain the original data of titles and abstracts of geological literature; S2, converting the unit words in the original data into a word vector matrix based on a word embedding operation; S3. Encode using the pre-trained BERT network model: S301, input the word vector matrix as an input sequence into the BERT network model, use a convolutional neural network to extract local connection features between unit words, and use a ReLU function to generate an output result, the output result is: Among them, m i is the i-th feature map, f represents the ReLU function, w k represents the first local vector, q represents the second local vector, k∈{3,5,7}, w j k is the jth first local vector, q j is the jth second local vector, q [i:i+k-1] is a vector of [i:i+k-1]; S302, using the output result, calling the self-attention mechanism to obtain global connection features; S303, using a gating unit to globally encode the local connection features and the global connection features to generate an encoder output, where the encoder output is: in, is the encoder output, σ(g) is the convolutional neural network function of the gating unit g, R T×dim It is a T×dim matrix, where T represents the sequence length, dim represents the vector dimension, i.e. the number of layers, and h represents the hidden layer state of the encoder; S4. Decode the encoder output: S401, using the global attention mechanism, using the encoder hidden layer state to define an attention mechanism-based context vector for the decoder step; S402, predicting the next word according to the context vector using the BiLSTM component embedded in the decoder, and obtaining an output sequence; S403: Use the BERT network model to capture the contextual relationship in the output sequence and generate a complete summary.

2. The method according to claim 1, characterized in that The word vector matrix is ​​expressed as: Among them, X 1:T represents the input sequence, x1, x2, x3 and x T Respectively represent the 1st, 2nd, 3rd and Tth basic unit words in the sequence, R 1×T Represents a matrix of length 1×T.

3. The method according to claim 1, characterized in that Step S301 also includes: using the "BE RT-BaseChinese" model to fine-tune the input sequence.

4. The method according to claim 1, characterized in that Step S301 also includes: inputting the output result into a BiLSTM component embedded in the encoder for encoding, so that the encoded content is transmitted bidirectionally.

5. The method according to claim 1, characterized in that Step S401 also includes: considering the hidden layer states of all encoders, and comparing the hidden layer state of the decoder with the hidden layer state of each encoder.

6. A system for automatically generating text summaries in the field of earth sciences, used to implement the method for automatically generating text summaries in the field of earth sciences as claimed in any one of claims 1 to 5, characterized in that: The system comprises: an encoder module, a decoder module and a training module, wherein the training module is used to pre-train the encoder module, the encoder module is used to encode an input sequence, and the decoder module is used to decode an output result of the encoder module.

7. The system according to claim 6, characterized in that The encoder module includes: a first BERT network model, a first BiLSTM component, a convolutional gating unit, and a self-attention mechanism; The first BERT network model is fine-tuned to initialize the value of the input sequence based on the word embedding operation; The first BiLSTM component is used to encode the received input sequence; The convolutional gating unit is used to retrain the core information based on the previous output at each time step; The self-attention mechanism is used to calculate the weight of the word itself, fully consider the semantic and grammatical connections between different words in the sentence, and strengthen the global information.

8. The system according to claim 6, characterized in that The decoder module comprises: Global attention mechanism, second BiLSTM component and second BERT network model; The global attention mechanism is used to define an attention-based context vector for the decoder step using the encoder hidden layer state; The second BiLSTM component is used to decode the sentence output by the encoder to obtain an output sequence; The second BERT network model is fine-tuned to capture the contextual relationship between sentences in the output sequence, thereby obtaining a complete summary.

Citation Information

Patent Citations

  • Case-related news viewpoint sentence recognition method based on BERT and BiLSTM-Attention

    CN111401061A

  • Method for generating guided text abstract based on Transformer

    CN111897949A