A long text processing method based on hierarchical attention mechanism and BERT

Abstract

The application discloses a long text processing method based on a hierarchical attention mechanism and BERT, and particularly relates to obtaining a text vector by processing long text through a BERT model, and improving the information loss caused by processing long text through an existing splitting method, and comprises the following steps: splitting a text into multiple pieces of content according to sentences; inputting each sentence into a BERT pre-training language model in the form of [CLS] sentence [SEP], obtaining a last-layer hidden layer vector and a [CLS] corresponding vector; obtaining a sentence vector by using a word attention mechanism; processing all sentences according to step two, obtaining sentence vectors of all sentences of the original text, and splicing to input into a transformer model in the form of [SCLS] vector, obtaining a last-layer hidden layer vector and a [SCLS] corresponding vector; obtaining a text vector by using a sentence attention mechanism; training a retrieval network model and updating parameters, extracting text features on a test set and testing. The application can obtain better long text processing, can more effectively improve long text classification accuracy, and has lower complexity.

Description

TECHNICAL FIELD

[0001] The present application relates to a text processing method in the field of natural language processing, and particularly relates to obtaining a text vector by processing a long text based on a BERT model. BACKGROUND

[0002] The Transformer model has achieved excellent results in many natural language processing tasks including classification, text generation and the like, and has promoted the birth of many large-scale pre-training models based on the Transformer model, such as BERT. The main reason for the success of the Transformer is that the self-attention mechanism of the Transformer can capture the information of the entire sequence context. However, due to the complexity of the self-attention mechanism of the Transformer, the complexity of the Transformer model is, where is the sequence length and is the hidden layer dimension, which leads to a limited sequence length that can be processed by the model. Further, the sequence length that can be processed by the BERT model based on the Transformer is limited, and in general, it is limited by the performance of the computer hardware. The BERT model limits the maximum input sequence length to 512 words. Moreover, the BERT model divides an input word into multiple components and adds multiple special markers such as [CLS] and [SEP], so the length of the text that can be processed by the BERT in actual application is much shorter than 512 words. However, in actual application, most of the texts such as news scripts and patent texts are much longer than 512 words, and these long texts cannot be directly processed by the BERT model or cannot be well processed by the BERT model.

[0003] Therefore, the BERT model generally has four methods to process long text. The first method is truncation, that is, a certain length of text is cut at the beginning or end of the text as the original text for processing, but this method will lose a lot of text information. The second method is segmentation, that is, the long text is divided into multiple short texts, each text is processed respectively, and then spliced by using pooling and other means to obtain a text vector, which is suitable for long text, but part of the information between the segments is lost. The third method is compression, that is, the long text is divided into multiple short texts, and then a rule or a trained other screening or scoring model is used to select meaningless paragraphs and delete them, but the effect of this method is seriously limited by the effect of the screening method. If the screening method is good, it can effectively shorten the text length and retain the text information. If the effect is not good, a large number of meaningful paragraphs will be deleted, which will affect the overall effect. The fourth method is to change the model structure, such as Transformer-XL, longformer, BigBird and other models. Since the high complexity of the Transformer is mainly in the self-attention mechanism, some methods reduce the complexity of the self-attention mechanism by limiting the range of information captured by the self-attention mechanism and the way of capturing information, thereby improving the ability of the model to process long text.

[0004] In summary, the traditional BERT model has limited ability to process long text, and the existing methods have a lot of drawbacks, which are easy to lose text information and have poor effect. SUMMARY

[0005] To solve the above technical problems, the purpose of the present application is to solve the problem that the existing BERT model cannot process more than 512 words, and a long text processing method based on hierarchical attention mechanism and BERT is proposed.

[0006] The long text processing method based on hierarchical attention mechanism and BERT of the present application comprises the following steps.

[0007] 1. The original text is divided into multiple contents according to sentences.

[0008] 2. Each sentence is transmitted to the pre-trained language model BERT in the form of [CLS] sentence [SEP], and the last layer hidden layer vector and the [CLS] corresponding vector are obtained.

[0009] 3. The word attention mechanism is used to obtain the sentence vector according to the obtained last layer hidden layer vector and the [CLS] corresponding vector.

[0010] 4. All sentences are processed according to step 2 to obtain the sentence vectors of all sentences of the original text and are spliced into the form of [SCLS] vector, which is transmitted to the transformer model to obtain the last layer hidden layer vector and the [SCLS] corresponding vector.

[0011] 5. Using the sentence attention mechanism to obtain the text vector according to the obtained last layer hidden layer vector of the transformer model and the [SCLS] corresponding vector.

[0012] 6. Training the retrieval network model according to the training data and updating the parameters, then extracting the text features on the test set and testing.

[0013] The long text processing method based on the hierarchical attention mechanism and BERT, in the step 1, the widely used public data sets IMDb, Hyperpartisan, Reuters-21578 and WikiHop are used as training data to train the model, and in order to ensure that the length of each text sentence is the same, all sentences are supplemented to the same length as the longest sentence by using [PAD].

[0014] The long text processing method based on the hierarchical attention mechanism and BERT, in the step 2, the [CLS] corresponding vector can represent the overall information of the sentence, and the last layer hidden layer vector can represent the information of each word.

[0015] The long text processing method based on the hierarchical attention mechanism and BERT, in the step 3, the word attention mechanism is introduced to better extract the relationship between words and the word meaning information to extract the sentence vector.

[0016] The long text processing method based on the hierarchical attention mechanism and BERT, in the step 4, in order to ensure that the number of sentences of each text is the same, all sentences are supplemented to the same number by using [SPAD].

[0017] The long text processing method based on the hierarchical attention mechanism and BERT, in the step 5, the sentence attention mechanism is introduced to better extract the relationship between sentences and the word meaning information to further improve the text feature extraction effect.

[0018] Compared with the prior art, the long text processing method based on the hierarchical attention mechanism and BERT has the following beneficial effects: compared with the traditional large-scale pre-training model, the length of the text that can be processed is improved; the semantic features and structural features of the text are extracted as much as possible and comprehensively from the input text, laying a foundation for text vector extraction; the accuracy of long text classification and question answering is improved, and the longer the text length is, the more significant the effect is; the word attention mechanism and the sentence attention mechanism can make the model pay more attention to the words and sentences that can express the semantics more, so that the text vector is more accurate; the model can process longer text while reducing the complexity and improving the accuracy of classification and question answering. BRIEF DESCRIPTION OF DRAWINGS

[0019] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments with reference made to the accompanying drawings. The drawings are for purposes of illustration only and are not intended to limit the present application thereto, as the present application can have general applicability. Like reference numerals can be used to refer to like components throughout the drawings.

[0020] Figure 1 is a flow chart of a long text processing method based on a hierarchical attention mechanism and BERT according to the present application.

[0021] Figure 2 is a model structure diagram of a long text processing method based on a hierarchical attention mechanism and BERT according to the present application.

[0022] Figure 3 is a performance diagram of different models on long text classification and question answering tasks. DETAILED DESCRIPTION

[0023] Exemplary embodiments of the present application will be described herein below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thoroughly and completely understood, and so that the scope of the present application will be completely conveyed to those skilled in the art.

[0024] Figure 1 and Figure 2 are a flow chart of a long text processing method based on a hierarchical attention mechanism and BERT according to the present application and a model structure diagram of a long text processing method based on a hierarchical attention mechanism and BERT according to the present application, respectively, and include the following steps.

[0025] 1. The text is divided by sentence, because according to language habits, a sentence contains a complete semantic, structural information and part of the context association information. If it is divided by fixed length, the division position cannot be guaranteed, and the text meaning will be broken, so the long text is divided into multiple parts according to the sentence. Therefore, each long text can be represented as a set of multiple sentences, i.e. , D represents the text, represents the i-th sentence of the text, 1, 2…n represents that the text contains n sentences. Among them, the sentence can be represented as , represents the i-th word of the sentence, 1, 2…m represents that the sentence is composed of m words.

[0026] 2. The BERT model can obtain text information of two granularities. The vector representation of the sentence granularity can be obtained through the output vector corresponding to [CLS], and the vector representation of the word granularity can be obtained through the output vector corresponding to each word of the last hidden layer. The sentence vector is obtained by comparing all word granularity vectors with the sentence granularity vector and weighted summation using the word attention method. The calculation formula is shown in (1):

[0027] (1)

[0028] wherein, represents the [CLS] corresponding vector output by BERT, and θ represents the parameter of the Bert model, represents the first, second, …, and m vectors corresponding to the words of the sentence i.

[0029] 3. The sentence vector is obtained by comparing all word granularity vectors with the sentence granularity vector and weighted summation using the word attention method. The calculation formula is shown in (2):

[0030] (2)

[0031] wherein, represents the sentence vector of the i-th sentence, and W_Att represents the word attention calculation process.

[0032] The calculation formula of the word attention is shown in (3)-(5):

[0033] (3)

[0034] (4)

[0035] (5)

[0036] wherein, represents the similarity matrix of [CLS] and the word vector, which is obtained by multiplying the transpose of [CLS] and each word vector. If the word vector is more similar to the corresponding vector of [CLS], the result is larger. represents the weight of each word vector. The similarity matrix is obtained using the softmax function, which is the weight of the weighted summation of each word vector in a sentence, represents the sentence vector after word attention calculation

[0037] 4. All sentence vectors are spliced, and a randomly initialized [SCLS] vector is spliced before all sentence vector representations. The vectors are input into the Transformer encoder to obtain the output of the last hidden layer The Transformer encoder is composed of multiple layers, each of which is composed of two sub-layers of multi-head attention mechanism and feedforward neural network. The calculation formula is shown in (6):

[0038] (6)

[0039] wherein, denotes the [SCLS] vector output by the Transformer encoder, denotes the vectors of sentences 1, 2…n output by the Transformer encoder

[0040] 5. The text vector is obtained by weighted sum of all vectors using sentence attention. The calculation formula is shown in (7):

[0041] (7)

[0042] wherein, S_Att denotes the sentence attention calculation process, denotes the text vector.

[0043] The purpose of sentence attention is to make full use of the sentence granularity information output by the last layer while obtaining the text granularity information using the [SCLS] corresponding vector output by the Transformer encoder. The calculation formula of sentence attention is shown in (8)-(10):

[0044] (8)

[0045] (9)

[0046] (10)

[0047] wherein, denotes the similarity matrix of [SCLS] and sentence vector, denotes the weight vector of each sentence vector, denotes the text vector obtained by sentence attention calculation.

[0048] Sentence attention first multiplies the [SCLS] vector with other vectors to obtain a similarity matrix. The more similar the sentence vector is to the [SCLS] corresponding vector, the greater the result is. According to the similarity matrix, the weight of each sentence is calculated using the softmax function. The greater the weight, the higher the relevance of the sentence meaning to the text. The weighted average of all sentence vectors using the calculated weight can obtain the text vector.

[0049] 6. The model can obtain semantic information at the word granularity and semantic information at the sentence granularity by the hierarchical method. Meanwhile, compared with the limitation that the basic Bert can only process 512 words, this method can process longer texts without deleting sentences in the article, and all context information is retained.

[0050] Embodiment 1:

[0051] The experimental results in this embodiment are obtained by using the widely used public data sets IMDb, Hyperpartisan, Reuters-21578 and WikiHop as data sets and testing on the data sets. The technical effects embodied in the present application are as follows.

[0052] Figure 3 The effect figures of different models, long text classification and question answering, wherein: HAN represents the results of classification and question answering after using the hierarchical attention mechanism to obtain the text vector. Roberta-base represents the results of classification and question answering after using the Roberta-base model to delete the long part of the text to obtain the text vector. Longformer represents the results of classification and question answering after using the longformer model to obtain the text vector.

[0053] From Figure 3 It can be seen that the method proposed in this paper is on par with RoBERTa in the classification task on the IMDb data set, slightly lower than longformer, 0.9% higher than longformer on the Hyperpartisan data set, the same as longformer on the Reuters-21578 data set, and 0.2% higher than longformer on the WikiHop data set in the question answering task. The reason is that the method proposed in this paper can better extract the full text features of the text, but compared with longformer, it reduces the influence of the local features of the text, so it can perform better on longer texts.

[0054] The above only describes the preferred embodiments of the present application. It should be noted that for those skilled in the art, without departing from the technical principles of the present application, several improvements and modifications can be made, and these improvements and modifications should also be considered as the protection scope of the present application.

Claims

1. A long text processing method based on hierarchical attention mechanism and BERT, characterized in that, Includes the following steps: Step (1): Divide the original text into multiple segments based on sentences. Each long segment can be represented as a set of multiple sentences, i.e., D = (sen1, ..., sen2) i ,...,sen n ), D represents text, sen i Let 1 represent the i-th sentence of the text, and 1, 2, ..., n represent that the text contains n sentences; Step (II): Input each sentence into the pre-trained language model BERT in the form of [CLS] sentence [SEP], where [CLS] is the sentence beginning tag of the BERT model; obtain the last hidden layer vector and the corresponding vector of [CLS], specifically through formula (H CLS H Wi1 ,...,H Wim ) = Bert(sen i Calculate θ), where H CLS This represents the [CLS] vector output by BERT, where θ represents the parameters of the BERT model, and H... Wi1 ,...,H Wim This represents the vector corresponding to the 1st, 2nd...mth words of sentence i; Step (3): Using the obtained last hidden layer vector and the corresponding [CLS] vector, a word attention mechanism is used to obtain the sentence vector, specifically through the formula w = H. CLS (H Wi1 ,...,H Wim ) T Calculate the similarity matrix between the [CLS] corresponding vector and the word vectors. Use weight = softmax(w) to calculate the weight of each word vector, and then use W_Att = weight(H) Wi1 ,...,H Wim The sentence vector obtained after word attention calculation is obtained; Step (iv): Process all sentences as in Step 2 to obtain sentence vectors for all sentences in the original text and concatenate them. Then, prepend a randomly initialized [SCLS] vector before all sentence vector representations, where [SCLS] is the introduced text-level tag. Input the [SCLS] vector into the transformer model to obtain the last hidden layer vector and the [SCLS] vector, specifically through formula (H). SCLS H S1 ,...,H Sn )=T(S SCLS S1,...,S n ) calculate, where H SCLS H represents the [SCLS] vector output by the Transformer encoder. S1 ,...,H Sn A vector representing sentences 1, 2, ..., n output by the Transformer encoder; Step (5): Using the obtained vectors from the last hidden layer of the Transformer model and the corresponding [SCLS] vectors, a sentence attention mechanism is used to obtain text vectors. By introducing [SCLS] tags and interacting with the Transformer model, global relationships between sentences are dynamically captured, specifically through the formula w_s = H SCLS (H S1 ,...,H Sn ) T Calculate the similarity matrix between [SCLS] and sentence vectors, and use weight_s = softmax(w_s) to calculate the weight vector for each sentence vector, and then use S_Att = weight_s(H S1 ,...,H Sn The text vector obtained after sentence attention calculation is obtained; Step (6): Train the retrieval network model based on the training data and update the parameters. Then, extract text features on the test set and perform testing.

2. The long text processing method based on hierarchical attention mechanism and BERT as described in claim 1, characterized in that: For the long text classification datasets IMDb and Hyperpartisan, and the question-answering dataset WikiHop, all texts are segmented into segments according to sentence granularity.

3. The long text processing method based on hierarchical attention mechanism and BERT as described in claim 2, characterized in that: For each sentence in the original text that has been segmented, the pre-trained language model BERT is used to obtain the context representation, and a hierarchical attention mechanism is designed, including word attention mechanism and sentence attention mechanism, to extract the local and global features of the text.

4. The long text processing method based on hierarchical attention mechanism and BERT as described in claim 3, characterized in that: For the input paragraph, sentence vectors are used to obtain text vectors through a transformer model to extract inter-sentence relationships and sentence-level features.

5. The long text processing method based on hierarchical attention mechanism and BERT as described in claim 4, characterized in that: Based on the large-scale pre-trained model BERT, hierarchical feature extraction is implemented to extract features and process long texts.

Images