Chinese abstract classification research method based on BERTBiLSTM

Through the BERT_BiLSTM-based method, the problem of unused word relationships in Chinese abstract classification is solved, the classification accuracy and efficiency are improved, and the extraction of deep semantic information and the transferability of the model are realized.

CN119003770APending Publication Date: 2024-11-22XI'AN UNIVERSITY OF ARCHITECTURE AND TECHNOLOGY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411033966.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing Chinese abstract classification algorithm fails to fully explore and utilize the inherent relationships and features between words, resulting in inaccurate classification, redundant model, inefficient classification, and machine challenges to natural language semantic understanding.

Method used

Using the Chinese abstract classification research method based on BERT_BiLSTM, through data preprocessing and label classification, a BERT pretrained model and a bidirectional long and short-term memory network were built, the word vector matrix was extracted and deep feature splicing was performed, and Dropout and L2 regularization were combined to improve the model's Simplicity and robustness.

Benefits of technology

It improves the disadvantages of traditional machine learning failing to fully explore word relationships, improves the accuracy and efficiency of text classification, effectively extracts deep semantic information, and enhances the transferability and robustness of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003770A_ABST
    Figure CN119003770A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of natural language processing, and particularly relates to a Chinese abstract classification research method based on BERTBiLSTM. The method comprises the following steps: preprocessing Chinese abstract data, and labeling according to abstract types; a Chinese abstract model based on BERTBiLSTM is built; and inputting the tagged abstract into a BERTBiLSTM model for training, and performing classification test on test data through rounds of training to obtain a classification result. According to the BERTBiLSTM model, the abstract statement pre-trained by BERT is converted into a word vector sequence, and the abstract statement is represented by a machine-readable digital matrix, so that a relatively good effect can be obtained without a large number of test data samples, and the BERT pre-training model learns large-scale text data, has relatively good mobility, and has relatively good application prospects. And abstract statement learning training is carried out in combination with own training data, so that the test data is more robust.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and specifically relates to a Chinese abstract classification research method based on BERT_BiLSTM. Background Art

[0002] With the massive growth in the number of papers and the extreme refinement of research directions, the previous title and keyword search can no longer meet the needs of current scholars to query and obtain information. New search methods are needed to meet the needs of scientific researchers. For example, scientific researchers can query relevant articles by keywords, but much of the information in the articles is useless to relevant personnel. Now let's use a specific example to illustrate. For example, we can use search engines to search for articles about "support vector machines" by title or keyword. Then the searched content will roughly include: introduction to the principle of support vector machines, application research on support vector machines, and performance improvement of support vector machines. These contents may not be the same for the specific research directions of scientific researchers, so refined search is needed. However, the title and keywords provide less information and cannot provide refined classification. Then it is necessary to start with the abstract, analyze and process the abstract, and obtain refined classification information to help readers make accurate queries. Of course, readers are limited by their background knowledge and are unlikely to find relevant research methods through known nouns. To this end, the present invention provides readers with abstract classification based on the general characteristics of scientific papers. On the basis of this classification, it provides readers with a powerful reference to help readers find the articles they need. According to the general characteristics of scientific papers, abstracts are divided into the following categories. 1) New methods are proposed completely independently, 2) Performance improvement of original methods, 3) Migration application of original methods, 4) Synthesis of new methods from original methods. Abstracts are classified in this way. Therefore, Chinese abstract classification can be used as a basic task in text processing related research, which is a research of great significance.

[0003] In the existing text classification algorithms, the performance of such algorithms needs to be improved and enhanced because the intrinsic relationship and features between words are not fully explored and utilized. One difficulty in classifying Chinese abstracts is: the diversity and flexibility of text expression. The text styles written by different authors are different. Some abstracts are unstructured abstracts, so they need to be changed to structured abstracts. Some abstracts do not belong to the four categories (new methods are proposed completely independently, the performance of the original methods is improved, the original methods are migrated and applied, and the original methods are synthesized into new methods). These abstracts need to be classified into other categories or eliminated, otherwise they will be difficult for machines to understand.

[0004] Another difficulty in classifying Chinese abstracts is that there is no perfect mathematical model to accurately describe the content expressed in natural language, and there are great challenges for machines to understand the semantics of natural language.

[0005] For the problem of Chinese abstract classification, researchers generally use methods based on traditional machine learning and deep learning. The methods based on traditional machine learning and deep learning for Chinese abstract classification are mostly based on structured abstracts; but in rule-based methods, rule formulation is crucial to the classification effect, and the rules cannot cover all classification situations, so the classification effect needs to be further improved; Chinese abstract classification methods based on traditional machine learning and deep learning rely on the vocabulary, morphology and other features of the text, but machine learning cannot deeply learn its semantics, so the effect is not optimal. Summary of the invention

[0006] The purpose of the present invention is to overcome the problem that traditional machine learning cannot fully explore and utilize the intrinsic relationship and features between words, resulting in inaccurate text classification, and the existing text classification model is relatively redundant and has low classification efficiency. A Chinese abstract classification research method based on BERT_BiLSTM is provided.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A Chinese abstract classification research method based on BERT_BiLSTM includes the following steps:

[0009] Collect Chinese abstracts, perform data preprocessing on the collected Chinese abstracts, label them according to classification categories, and divide the Chinese abstract data into training data and test data;

[0010] Build a BERT pre-training model, input training data and test data into the BERT pre-training model, extract features from the labeled Chinese summary sentences, and obtain a word vector matrix;

[0011] Build a bidirectional long short-term memory network, input the word vector matrix obtained by the BERT pre-training model into the bidirectional long short-term memory network, perform deep extraction of the input context features, and concatenate the extracted forward feature vector and reverse feature vector to obtain the concatenated feature vector matrix;

[0012] The BERT pre-trained model is combined with the bidirectional long short-term memory network to obtain the BERT_BiLSTM Chinese summary classification model. The concatenated feature vector matrix is ​​input into the classifier for judgment, and the classification result is obtained through function calculation.

[0013] The labeled summary training data and test data are input into the BERT_BilSTM model training. After rounds of training, the test data is classified and tested to obtain the classification results.

[0014] The BERT pre-training model includes an input layer, an intermediate layer, and an output layer. The input layer uses word embedding to directly obtain the position vector.

[0015] The middle layer is the Encoder layer, which includes two parts: residual connection and layer normalization. The residual connection divides the position vector information of the input layer into two parts. One part is transformed through the multi-head attention mechanism of this layer, and the other part is directly passed to the next layer. The results of these two parts are then added as the output result of the output layer.

[0016] Layer normalization is calculated as follows:

[0017]

[0018] Among them, n represents the number of words in a sentence, x i Represents each element in the matrix, μ i Represents each element X i The mean value, σ i Represents each element X i The variance of .

[0019] The result of the residual connection needs to be normalized again. The calculation formula is as follows:

[0020] LayerNorm(X+MultiHead(Q,K,V)) (11)

[0021] Among them, X represents the position vector matrix of the input layer, and MultiHead(Q,K,V) represents the attention output matrix calculated by the multi-head attention mechanism.

[0022] The Encoder layer includes a feed-forward neural network submodule FFN, which includes two fully connected layers. The two fully connected layers are activated by the Relu function. The mathematical formula is as follows:

[0023] FFN(x)=max(0,W1x+b1)W2+b2 (14)

[0024] Where x is the output feature vector of the residual connection and layer normalization submodule, W1 and W2 are the weight vectors of [embed_dim, FFN_dim] and [FFN_dim, embed_dim] respectively, b1 is the bias vector of the W1 layer, and b2 is the bias vector of the W2 layer.

[0025] The bidirectional long short-term memory network model consists of two independent LSTM models. The input of the LSTM model structure is the hidden state h of the previous moment. t-1 , old cell state C t-1 , the current input x t, the output is the next hidden state h t , the new cell state C t , which contains forget gate, input gate and output gate.

[0026] The forget gate is based on the hidden state h of the previous moment. t-1 and the current input x t Determine the old cell state C t-1 What information has been forgotten?

[0027] The input gate determines the information that can be input, and the old cell state C t-1 Together they determine the new output cell state C t ;

[0028] The output gate determines what contents of the output vector will be output.

[0029] The BERT_BiLSTM model adds a BiLSTM to the BERT pre-trained model.

[0030] The BERT_BiLSTM model uses the Dropout method to reduce the number of network layers and neurons to limit the model fitting ability; and L2 regularization to reduce the weight parameters.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] The present invention provides a research method for Chinese abstract classification based on BERT_BiLSTM. Chinese abstract data is preprocessed and labeled according to the abstract type; a Chinese abstract model based on BERT_BiLSTM is built; the labeled abstract is input into the BERT_BiLSTM model for training, and the test data is classified and tested after rounds of training to obtain a classification result. The disadvantage of traditional machine learning that the intrinsic relationship and features between words are not fully explored and utilized is improved, and the accuracy of text classification is improved. The BERT_BiLSTM model effectively extracts deep semantic information in the abstract that is conducive to classification, so the model is more streamlined and more efficient. In addition, the BERT_BiLSTM model converts the abstract sentence after BERT pre-training into a word vector sequence, uses a machine-readable digital matrix to represent the abstract sentence, and undergoes deep semantic learning of the BERT pre-training model, so a good effect can be obtained without a large number of test data samples, and the BERT pre-training model learns large-scale text data, has good transferability, and combines its own training data to perform abstract sentence learning and training, so it is more robust to test data.

[0033] Furthermore, the BERT pre-training model is used to convert the preprocessed summary sentences into word vector sequences. The BERT pre-training model can learn the contextual dependencies of the text, thereby providing very rich contextual features for the subsequent BiLSTM model.

[0034] Furthermore, BiLSTM has a unique advantage in processing text sequences. BiLSTM can deeply extract the contextual information of the input. Therefore, after BERT performs feature extraction, BiLSTM encodes the feature vectors with important meanings, which can make up for the contextual information that BERT easily forgets and better capture the bidirectional semantic dependency, thereby improving the performance of the BERT_BiLSTM model. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of the present invention.

[0036] Figure 2 This is a structural diagram of the BERT model of the present invention.

[0037] Figure 3 This is a specific structural diagram of the BERT model of the present invention.

[0038] Figure 4 This is the input layer structure diagram of the BERT model of the present invention.

[0039] Figure 5 This is the architecture diagram of the Encoder model in the BERT model.

[0040] Figure 6 This is the architecture diagram of the Multi-Head Attention model in the Encoder model architecture.

[0041] Figure 7 This is the self-attention operation diagram in the Encoder model architecture.

[0042] Figure 8 This is the structure diagram of the LSTM model.

[0043] Fig. 9 This is the calculation diagram of the forget gate in the LSTM model structure.

[0044] Fig.10 is the sigmoid function image.

[0045] Fig.11 Input gate computation graph in LSTM model structure.

[0046] Fig.12 is the image of the tanh function.

[0047] Fig.13This is the output gate calculation graph in the LSTM model structure.

[0048] Fig.14 This is the BiLSTM model structure.

[0049] Fig.15 BERT_BiLSTM model structure DETAILED DESCRIPTION

[0050] In order to further understand the content of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the embodiments are only for explaining the present invention and are not intended to limit it.

[0051] like Figure 1 As shown in the figure, a Chinese abstract classification research method based on BERT_BiLSTM is shown in the figure. The specific steps are as follows:

[0052] Collect Chinese abstracts, perform data preprocessing on the collected Chinese abstracts, label them according to classification categories, and divide the Chinese abstract data into training data and test data;

[0053] Build a BERT pre-training model, input training data and test data into the BERT pre-training model, extract features from the labeled Chinese summary sentences, and obtain a word vector matrix;

[0054] Build a bidirectional long short-term memory network, input the word vector matrix obtained by the BERT pre-training model into the bidirectional long short-term memory network, perform deep extraction of the input context features, and concatenate the extracted forward feature vector and reverse feature vector to obtain the concatenated feature vector matrix;

[0055] The BERT pre-trained model is combined with the bidirectional long short-term memory network to obtain the BERT_BiLSTM Chinese summary classification model. The concatenated feature vector matrix is ​​input into the classifier through the fully connected layer for judgment, and the classification result is obtained through function calculation.

[0056] The labeled summary training data and test data are input into the BERT_BilSTM model training. After rounds of training, the test data is classified and tested to obtain the classification results.

[0057] Specifically, the collected Chinese abstract data are selected from the abstracts of articles in the publicly available Journal of Electronics, Journal of Computers, and Acta Automatica Sinica. These Chinese abstract data are standardized and structured, and format symbols such as "\n", "\t", and spaces in the abstracts are cleaned to retain the complete text content of the original data.

[0058] Labeling: If the abstract type is a completely independent proposal of a new method, its label is marked as 0; if the abstract type is a performance improvement of an existing method, its label is marked as 1; if the abstract type is a migration application of an existing method, its label is marked as 2; if the abstract type is a synthesis of an existing method into a new method, its label is marked as 3.

[0059] These summary data are divided into training data and test data in a ratio of 8:2.

[0060] like Figure 2 As shown in the figure, the BERT pre-trained model is a bidirectional Transformer Encoder, which can be simply summarized as an input layer, an intermediate layer, and an output layer. The input layer of the BERT pre-trained model uses word embedding to directly obtain the position vector, and the intermediate layer uses a multi-head attention mechanism and a feedforward neural network. ⊕ represents residual connection and layer normalization, which includes residual connection and layer normalization. The residual connection divides the information of the input layer into two parts, one part is transformed through this layer, and the other part is directly passed to the next layer, and then the results of the two parts are added as the output result of the output layer. The layer normalization calculation method is as follows:

[0061]

[0062] Among them, x i Represents each element in the matrix, μ i Represents each element X i The mean value, σ i Represents each element X i The variance of the neurons is normalized by layer, the mean and variance of the neurons about the sample are normalized, the dependence between neurons is reduced, the model training process is accelerated, and the model converges faster.

[0063] like Figure 3As shown in the figure, the BERT pre-training model itself is an autoencoder language model, which mainly consists of two stages: pre-training task and fine-tuning. The BERT pre-training model has two pre-training tasks, namely Masked Language Model (MLM) and Next Sentence Prediction (NSP). The MLM task is similar to the cloze test in English, which is to cover several words in a given sentence and require the model to predict these covered words through the remaining words. The NSP task is to predict whether the second sentence is the direct sequel of the first sentence, enhance the model's understanding of the context, and allow the model to further understand the causal or logical relationship between the contexts during training. The WWM strategy (whole word mask) is that if the subword of a part of a complete word is masked, then the other parts of the word will also be masked. The Chinese BERT pre-training model segments sentences based on the granularity of characters, without considering the Chinese word segmentation in traditional natural language processing. The whole word mask method is applied to Chinese, and all Chinese characters that make up the same word are masked. In the process of continuous input of large amounts of data, multi-faceted learning is carried out to enrich semantic information, thereby obtaining a vector representation that incorporates global semantic information. After the word vector E is encoded by the BERT pre-trained model layer, the summary sentence features contained in each word in the summary sentence in different sentences are obtained, expressed as X = (X1, X2, X3, ..., X N ).

[0064] For example, Figure 4 As shown in the figure, take "proven the effectiveness of this method" as an example, where [CLS] represents the start position separator and [SEP] represents the sentence end identifier. For the text to be detected, firstly, after data preprocessing and cleaning, the BERT pre-training model is built. By querying the word vector table, the words in the text are converted into one-dimensional vectors as the input of the model. Assuming that the input sentence H contains at most h words, the sentence H after word segmentation is divided into h words, which are specifically expressed as H: {H1, H2, H3, ..., H h}, each word is converted into a corresponding word vector through position encoding, and the global semantic information of the text is captured through automatic learning methods to generate a text vector E: {E A ,E A , E A ,...,E A}, and then use the position vector E:{E1,E2,E3,...,E 11} Get the meaning of the words at different positions, and combine the position vector with the text vector to get the word vector E = (E [CLS] ,E 证 , E 明 ,...,E[CLS] ), and add the obtained word vector, text vector and position vector to get the input of the BERT pre-trained model.

[0065] Specifically, Figure 5 As shown in the figure, the middle layer adopts the Encoder model architecture. The Encoder first converts the input into a word embedding vector, and adds the position encoding vector to the word embedding to form the input vector of the model; then the input vector is put into the multi-head attention mechanism for calculation, and the result of the calculation is residually connected with the input vector and layer normalized, thereby obtaining the input vector of the feedforward neural network; this input vector is put into the feedforward neural network for calculation, and the result of the calculation is residually connected with the input vector and layer normalized, thus obtaining the output of the Encoder part.

[0066] like Figure 6 As shown, the input of the BERT pre-trained model is divided into three equal parts Q, K, V. Each attention operation is a head, and each head has a set of weight matrices, the calculation result of which is [sequence_length, d v ×h], h heads means there are h groups of different weight matrices, and h results are obtained. The Concat part first concatenates the h results, and then passes a shape of [d v ×h,embed_size] weight matrix is ​​linearly transformed, and the output result is a matrix with a shape of [sequence_length,embed_size]. MultiHead and head i The mathematical formula is as follows:

[0067]

[0068] Among them, MultiHead(Q,K,V) represents the multi-head attention mechanism function, w 0 Represents the weight matrix of the linear transformation layer, head i Represents the output of the i-th head, Concat(head1,head2,…,head h ) means concatenating the outputs of all heads along the depth dimension. Represents the calculation of attention weights, and enhances the model's expressiveness and ability to model the intrinsic structure of complex data by running multiple self-attention layers in parallel.

[0069] like Figure 7 As shown, in the attention operation, each head contains 3 weight matrices W Q , W K and WV , W Q and W K All of them are of shape [embed_size,d k ], and W V For specifications [embed_size,d v ], let d k =d v =(embed_size) / h.

[0070] Assume that a sentence has n words. For the i-th word, its query vector, key vector, and value vector are Q i , K i and V i , the calculation formula is as follows:

[0071] X i W K =K i (7)

[0072] x i w Q =Q i (6)

[0073] X i W V =V i (8)

[0074] Therefore, the mathematical formula of attention operation is as shown in the following formula (9):

[0075]

[0076] Among them, W Q , W K and W V Both represent weight matrices, Attention(Q i ,K i ,V i ) represents the attention output matrix, represents the scaling factor, represents the transposed matrix of Ki,

[0077] When paying attention to each word in a sentence, multiply the query vector (Q) of the current word by the transpose of the key vector (K) of all words and divide it by (mainly to control the scale, prevent it from being too large or too small) will get a set of data, the size of the data represents the similarity between the current word and other words, and then perform softmax processing to get a set of probability values, and the number of probability values ​​is the same as the number of value vectors, and then add this set of probability values ​​to the value vector V of all words iMultiplying and adding together gives the result of the current word and other words after attention. The process of attention for other words is the same. This process calculates the similarity between the query and the key and uses these similarities as weights to combine the value vectors, thereby generating a representation that can capture the dependencies between different positions in the input sequence.

[0078] Residual connection refers to adding the input and output of a module. For the Multi-Head Attention module, since its input X and output MultiHead (Q, K, V) have the same shape, the result of residual connection is to add the two matrices, as shown below.

[0079] X+MultiHead(Q,K,V) (10)

[0080] The residual connection mainly plays a corrective role, adding the extracted information to the original information to prevent the extracted information from being too biased. The result of the residual connection needs to be normalized again. The calculation formula is as follows:

[0081] LayerNorm(X+MultiHead(Q,K,V)) (11)

[0082] Layer normalization has two main functions: (1) making the input data distribution in the network relatively stable and speeding up model training; (2) making the model less sensitive to the parameters in the network, simplifying the parameter adjustment process, and making network learning more stable. The specific implementation process is as follows:

[0083] For each row i in the matrix (i = 1, 2, ..., sequence_length), find the element X i The mean μ i , the specific formula is as follows:

[0084]

[0085] Among them, embed_dim represents the embedding vector dimension of a word or part of speech.

[0086] By setting embed_dim=n, formula (12) can be simplified to obtain the above formula (1).

[0087] For each row i in the matrix (i = 1, 2, ..., sequence_length), find the variance σ of the row data i , the calculation formula is as follows:

[0088]

[0089] Setting embed_dim = n can simplify formula (13) to the above formula (2).

[0090] So the formula for layer normalization is the above formula (3), that is,

[0091] The Encoder includes a feedforward neural network submodule: The feedforward neural network (FFN) is essentially two fully connected layers. The Relu function is used to activate the two fully connected layers, which mainly plays the role of connection conversion. The mathematical formula is as follows:

[0092] FFN(x)=max(0,W1x+b1)W2+b2 (14)

[0093] Where x is the output feature vector of the residual connection and layer normalization submodule. W1 and W2 are weight vectors of [embed_dim, FFN_dim] and [FFN_dim, embed_dim] respectively, and FFN_dim is the number of neurons in the hidden layer. The feedforward neural network does not change the size of the matrix when processing data, and the shape of the result is still [sequence_length, embed_dim].

[0094] Specifically, the Bidirectional Long Short-Term Memory Network (BiLSTM) model is built as follows: The Bidirectional Long Short-Term Memory Network model consists of two independent LSTM models;

[0095] like Figure 8 As shown, the LSTM model structure input is h t-1 , C t-1 , x t , the output is h t , C t , which contains the forget gate, input gate and output gate, where h t-1 is the hidden state of the previous moment, C t-1 is the old cell state, x t is the input at the current moment, h t is the output of the next hidden state, C t The new cell state.

[0096] 1) Forget gate: The forget gate is based on the input information h t-1 and x t Determines cell state C t-1 What information is forgotten? The calculation process is as follows Fig. 9 The bold part shows the mathematical formula as follows:

[0097] f t =o(W t [h t-i ,xt l+h t ) (15)

[0098] First, h t-1 and x t splicing, where σ represents the sigmoid activation function. Use the forget weight matrix W f Do matrix multiplication with the concatenated result, add a bias bf, and use the result Fig.10 The sigmoid function shown is processed, and the mathematical formula is as follows:

[0099]

[0100] So the forgetting matrix f t All values ​​are between 0 and 1, 0 means no t-1 Any information in the t-1 All information is passed through.

[0101] 2) Input gate: The input gate determines the information that can be input and is related to the input cell unit C t-1 Together they determine the output cell unit C t The calculation process is as follows Fig.11 The bold part shows the mathematical formula as follows:

[0102] Ct1=tanh(W c [h t-1 ,x t ]+b c ) (17)

[0103] C t1 The calculation process and f t Same, but different weight matrices, biases, and activation functions, C t1 The activation function used is tanh, and the updated content C is obtained after the activation function tanh t1 , C t1 All values ​​in are between (-1,1), and the tanh function is Fig.12 As shown, the tanh function formula and input matrix i t The calculation formula is as follows:

[0104]

[0105] i t =o(W i [h t-i ,x t l+b i ) (19)

[0106] i t The calculation process and ft Same, but the weight matrix and bias are different, i t Determines which updates can be imported. t With C t1 Afterwards, combined with the previous forgetting matrix f t , we can get the output cell C t , the formula is as follows:

[0107] C t =f t ×C t-1 +i t ×C t1 (20)

[0108] 3) Output gate: The output gate determines which contents of the output vector will be output. The calculation process of the output gate is as follows Fig.13 The bold part shows:

[0109] In calculating the output vector h t When , first calculate the output matrix o t , the calculation formula is as follows:

[0110] o t =σ(W o [h t-1 ,x t ]+b o ) (twenty one)

[0111] Then calculate the output vector h t ,h t It is both the output at the current moment and the input at the next moment. t The calculation formula is as follows:

[0112] h t =o t ×tanh(C t ) (twenty two)

[0113] like Fig.14 As shown in the figure, the Bidirectional Long Short-Term Memory (BiLSTM) model adds a reverse LSTM to the forward LSTM. It can extract information from the Chinese summary from both the forward and reverse directions. Compared with the unidirectional LSTM, it has better information extraction capabilities. In addition, it also considers the relevance of the Chinese summary data and makes the most of the summary information. For the word vectors W1, W2, W3, …, W n First, we extract the relevant information of the summary through the forward LSTM to obtain a set of feature vectors {h L1 ,h L2 ,h L3 ,…,h Ln}, and then extract the relevant information of the summary through the reverse LSTM to obtain a set of feature vectors {h R1 ,h R2 ,h R3 ,…,h Rn}, concatenate the forward and reverse feature vectors to obtain {[h L1 ,h Rn ],[h L2 ,h Rn-1 ],…,[h Ln ,h R1 ]}, that is, {C1,C2,C3,…,C n}, this is the process of extracting information by the Bidirectional Long Short-Term Memory Network (BiLSTM).

[0114] Specifically, Fig.15 As shown in the figure, the BERT_BiLSTM model is based on the BERT pre-trained model with a BiLSTM. Before training the model, the summary sentence sequence is converted into a word vector [[CLS], Tok 1, Tok 2, …, TokN], and then processed into [E [CLS] , E1, E2,…,E n ], the BERT pre-trained model extracts information from the sentence into the output vector [C, T1, T2, …, T n ], and [C,T1,T2,…,T n ] is input into BiLSTM for feature extraction to obtain [P c ,P1,P2,…P n ], and finally take P n Classification is performed through a fully connected layer, and the corresponding probability is calculated through the softmax function to obtain the classification result. To prevent the model from overfitting, the model uses the Dropout method to reduce the number of network layers and neurons to limit the model fitting ability; and L2 regularization to reduce the weight parameters. Keeping a smaller weight can help the model smoother, thereby improving the generalization ability of the model.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A Chinese abstract classification research method based on BERT_BiLSTM, characterized in that: The following steps are involved: Collect Chinese abstracts, perform data preprocessing on the collected Chinese abstracts, label them according to classification categories, and divide the Chinese abstract data into training data and test data; Build a BERT pre-training model, input training data and test data into the BERT pre-training model, extract features from the labeled Chinese summary sentences, and obtain a word vector matrix; Build a bidirectional long short-term memory network, input the word vector matrix obtained by the BERT pre-training model into the bidirectional long short-term memory network, perform deep extraction of the input context features, and concatenate the extracted forward feature vector and reverse feature vector to obtain the concatenated feature vector matrix; The BERT pre-trained model is combined with the bidirectional long short-term memory network to obtain the BERT_BiLSTM Chinese summary classification model. The concatenated feature vector matrix is ​​input into the classifier for judgment, and the classification result is obtained through function calculation. The labeled summary training data and test data are input into the BERT_BilSTM model training. After rounds of training, the test data is classified and tested to obtain the classification results.

2. According to a BERT_BiLSTM-based Chinese abstract classification research method according to claim 1, it is characterized in that: The BERT pre-training model includes an input layer, an intermediate layer, and an output layer. The input layer uses word embedding to directly obtain the position vector.

3. According to a BERT_BiLSTM-based Chinese abstract classification research method according to claim 2, it is characterized in that: The middle layer is the Encoder layer, which includes two parts: residual connection and layer normalization. The residual connection divides the position vector information of the input layer into two parts. One part is transformed through the multi-head attention mechanism of this layer, and the other part is directly passed to the next layer. The results of these two parts are then added as the output result of the output layer.

4. According to a BERT_BiLSTM-based Chinese abstract classification research method according to claim 3, it is characterized in that: Layer normalization is calculated as follows: Among them, n represents the number of words in a sentence, x i Represents each element in the matrix, μ i Represents each element X i The mean value, σ i Represents each element X i The variance of .

5. According to a BERT_BiLSTM-based Chinese abstract classification research method according to claim 3, it is characterized in that: The result of the residual connection needs to be normalized again. The calculation formula is as follows: LayerNorm(X+MultiHead(Q,K,V)) (11) Among them, X represents the position vector matrix of the input layer, and MultiHead(Q,K,V) represents the attention output matrix calculated by the multi-head attention mechanism.

6. According to a BERT_BiLSTM-based Chinese abstract classification research method according to claim 3, it is characterized in that: The Encoder layer includes a feed-forward neural network submodule FFN, which includes two fully connected layers. The two fully connected layers are activated by the Relu function. The mathematical formula is as follows: FFN(x)=max(0,W1x+b1)W2+b2 (14) Where x is the output feature vector of the residual connection and layer normalization submodule, W1 and W2 are the weight vectors of [embed_dim, FFN_dim] and [FFN_dim, embed_dim] respectively, b1 is the bias vector of the W1 layer, and b2 is the bias vector of the W2 layer.

7. According to the BERT_BiLSTM-based Chinese abstract classification research method of claim 1, it is characterized in that: The bidirectional long short-term memory network model consists of two independent LSTM models. The input of the LSTM model structure is the hidden state h of the previous moment. t-1 , old cell state C t-1 , the current input x t , the output is the next hidden state h t , the new cell state C t , which contains forget gate, input gate and output gate.

8. According to the BERT_BiLSTM-based Chinese abstract classification research method of claim 7, it is characterized in that: The forget gate is based on the hidden state h of the previous moment. t-1 and the current input x t Determine the old cell state C t-1 What information has been forgotten? The input gate determines the information that can be input, and the old cell state C t-1 Together they determine the new output cell state C t ; The output gate determines what contents of the output vector will be output.

9. According to a BERT_BiLSTM-based Chinese abstract classification research method according to claim 1, it is characterized in that: The BERT_BiLSTM model adds a BiLSTM to the BERT pre-trained model.

10. According to claim 9, a Chinese abstract classification research method based on BERT_BiLSTM is characterized in that: The BERT_BiLSTM model uses the Dropout method to reduce the number of network layers and neurons to limit the model fitting ability; and L2 regularization to reduce the weight parameters.