A method for classifying a Chinese super-long text based on a large language model

Abstract

The application discloses a Chinese super-long text classification method based on a large language model, relates to the technical field of super-long text processing, and comprises the following steps: (1) obtaining Chinese super-long text classification data and corresponding classification labels; (2) pre-processing Chinese super-long text data; (3) defining a downstream task model of the large language model; (4) fine-tuning the large language model BERT for Chinese long text; and (5) predicting the category of Chinese super-long text by using the trained model weight. The Chinese super-long text classification method based on the large language model can significantly improve the accuracy of Chinese super-long text classification in engineering application.

Description

Technical Field

[0001] This invention relates to the field of ultra-long text processing technology, specifically to a classification method for ultra-long Chinese texts based on a large language model. Background Technology

[0002] With the development of the internet, the amount of text data generated in people's daily lives is increasing dramatically, and the amount of text data on the internet is growing rapidly. Therefore, it is particularly important to use text classification technology to scientifically organize and manage this massive amount of data. Text classification is commonly used in fields such as digital libraries and email filtering, providing strong support for the querying and retrieval of text resources, and is currently one of the main research hotspots. How to efficiently classify these texts has become an important issue. Currently, some researchers have proposed text classification techniques based on traditional machine learning and deep learning methods, but these methods have some problems, such as insufficient ability to process extremely long texts and insufficient ability to understand text semantics.

[0003] Many text classification methods based on traditional machine learning and deep learning only perform well on specific, relatively clean datasets, but their classification accuracy is low in practical engineering applications.

[0004] The existing technology has the following two drawbacks:

[0005] (1) The length of Chinese text is too long. If the length of each line of text is several thousand, but the maximum length of the large language model BERT is relatively short, the model will directly delete most of the data when loading data, discard most of the data features, and leave less data features. The deleted data may be the key content of the text, which greatly reduces the accuracy of text classification.

[0006] (2) Some models are academically robust to extremely long Chinese texts, but perform poorly in practical engineering applications and fail to meet application standards. Summary of the Invention

[0007] To address the issue of low accuracy in classifying extremely long Chinese texts using existing models, this invention proposes a method for classifying extremely long Chinese texts based on a large language model, aiming to improve the accuracy of such classification in engineering applications.

[0008] To achieve the above objectives, the method described in this invention includes the following steps:

[0009] (1) Obtain Chinese long text classification data and corresponding classification labels;

[0010] (2) Preprocessing of long Chinese text data;

[0011] (3) Define the downstream task model of the large language model;

[0012] (4) Fine-tuning of long Chinese texts using the large language model BERT;

[0013] (5) Predict the category of Chinese long texts using the trained model weights.

[0014] Furthermore, step (1) specifically includes:

[0015] Obtain all Chinese data for the text categories, then label all the data for each category, place them in the designated folder, and clean the data.

[0016] Furthermore, step (2) specifically includes:

[0017] Each categorized PDF file is converted into an image and its resolution is increased. Finally, it is recognized as text using OCR.

[0018] When loading data, each line of data is truncated into text of less than or equal to the maximum length and assigned the same label as the original text. The overlap between each line of text and its context is set to half the length of the truncated text. Then, each truncated segment is tokenized to generate the following three inputs:

[0019] Each character in the fragment corresponds to a unique identifier in the vocabulary;

[0020] The position of the first sentence and special symbols is 0, and the position of the second sentence is 1;

[0021] Positions that are less than the maximum length of each row are assigned a value of 0, while other positions are assigned a value of 1. Positions with a value of 1 are included in the attention calculation.

[0022] Furthermore, step (3) specifically includes:

[0023] Define a fully connected layer in the downstream task model to convert the vector dimension of BERT output into the number of categories;

[0024] The embeddings of multiple clauses truncated in the long text are looped, and the BERT language model is used to extract features for each clause to obtain the CLS. The CLS is the word vector of the first word in the input text. It is added to the set of CLS features of the truncated clauses in the long text. Then, the average of the CLS sets of all clauses in the long text is taken as the final CLS output of the long text.

[0025] The formula for taking the average of the CLS set of all clauses is as follows:

[0026]

[0027] In the formula:

[0028] cls i The word vector representing the first word of each clause in each long sentence;

[0029] l represents the length of all clauses;

[0030] The feature vector corresponding to the CLS identifier at the beginning of the long text is input into a binary or n-class output layer and softmax is used for classification.

[0031] The softmax classification formula is as follows:

[0032] p(c|h) = softmax(Wh)

[0033] In the formula:

[0034] c is a tag;

[0035] h represents the final hidden state of CLS;

[0036] W is the parameter matrix for the classification task;

[0037] The Softmax function is used to convert the output values ​​of multi-class classification into a probability distribution ranging from [0,1] to 1.

[0038] Furthermore, step (4) specifically includes:

[0039] Load the weights of the large language model, input the preprocessed dataset, and in each epoch (an epoch refers to training all data once), take the average of the CLS set of clauses of the long text in each iteration as the final CLS output of the long text. Then, output the feature vector corresponding to the CLS identifier at the beginning of each long text and input it into a binary or n-classification output layer to perform softmax classification.

[0040] In classification tasks, the parameters of the additional fully connected layers are learned from scratch during training, while all parameters in the pre-trained BERT model are fine-tuned.

[0041] Calculate the loss function value and accuracy for each epoch of training and validation data. After multiple epochs, save the model weight with the smallest loss function value. This optimal model weight can be used directly for prediction.

[0042] Furthermore, step (5) specifically includes:

[0043] Obtain the weights of the classification model trained in step (4), classify the preprocessed prediction set, and calculate the average accuracy of the dataset. The formula for accuracy is as follows:

[0044] sum(T′=Tl ) / L

[0045] In the formula:

[0046] T' represents the predicted data label;

[0047] Tl is the initial label for the data;

[0048] L represents the total number of labels in the predicted data.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1. When training a large language model with ultra-long text data, each line of data is truncated into text smaller than the maximum length of the model and assigned the same label as the original text. BERT extracts features from all clauses of the long text. Compared with previous models, it can mine all features of long text data instead of discarding many important features, which greatly improves the accuracy of ultra-long Chinese text classification.

[0051] 2. The text is input into the large language model BERT. The CLS identifier at the beginning of the sentence is output as the corresponding feature vector. This feature vector is then input into a binary (or n-class) output layer for softmax classification. In classification tasks, the parameters of fully connected layers are learned from scratch, while all parameters in the pre-trained BERT model are fine-tuned. Attached Figure Description

[0052] Figure 1 This is a flowchart of the Chinese long text classification method provided in the embodiments of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0054] like Figure 1 As shown in the figure, this invention proposes a method for classifying extremely long Chinese texts based on a large language model. The method includes the following steps:

[0055] 1) Obtain Chinese long text classification data and corresponding classification labels;

[0056] Step 1) specifically includes the following: obtaining all Chinese data for all text categories, labeling all the data for each category, placing them in the designated folder, and cleaning the data;

[0057] 2) Preprocessing of extremely long Chinese text data;

[0058] Step 2) specifically includes the following: First, each categorized PDF file is converted into an image and the image resolution is improved. Finally, it is recognized as text using OCR.

[0059] Secondly, training ultra-long text data directly using a large language model results in low accuracy. Each line of text is several thousand characters long, but the maximum length of the large language model BERT is relatively short. Therefore, data exceeding the maximum length of each line will be lost, significantly reducing the accuracy of the training model. So, when loading data, each line of data is truncated into text of less than or equal to the maximum length and assigned the same label as the original text. The overlap length between each line of text and its context is set to half the truncated text length. Then, each truncated segment is tokenized, generating three inputs: a unique identifier for each character in the segment corresponding to the vocabulary; the position of the first sentence and special symbols is 0, and the position of the second sentence is 1; positions that are less than the maximum length of each line are 0, and other positions are 1. Positions with a value of 1 participate in attention calculation.

[0060] 3) Define the downstream classification task model of the large language model;

[0061] Step 3) specifically includes the following:

[0062] In the downstream task model, a fully connected layer is defined to convert the vector dimension of the BERT output into the number of categories. The embeddings of multiple clauses after truncation in the long text are iterated, and the BERT language model is used to extract features for each clause to obtain CLS (CLS is the word vector of the first word in the input text). The CLS is added to the set of CLS features of the truncated clauses in the long text. Then, the average of the CLS sets of all clauses in the long text is taken as the final CLS output of the long text.

[0063] The formula for taking the average of all clause CLS sets is as follows:

[0064]

[0065] In the formula:

[0066] cls i The word vector representing the first word of each clause in each long sentence;

[0067] l represents the length of all clauses;

[0068] Finally, the feature vector corresponding to the CLS identifier at the beginning of the long text is input into a binary (or n-class) output layer for softmax classification.

[0069] The softmax classification formula is as follows:

[0070] p(c|h) = softmax(Wh)

[0071] In the formula:

[0072] c is a tag;

[0073] h represents the final hidden state of CLS;

[0074] W is the parameter matrix for the classification task.

[0075] The Softmax function described above can be used to convert the output values ​​of multi-class classification into a probability distribution with a range of [0,1] and a value of 1.

[0076] 4) Input the data into the large language model BERT to train on long Chinese texts;

[0077] In step 4), the long Chinese text is fine-tuned using the large language model BERT, specifically including the following:

[0078] Load the weights of the large language model, input the preprocessed dataset, and in each epoch (an epoch refers to training all data once), take the average of the CLS set of clauses of the long text in each iteration, and use it as the final CLS output of the long text. Then, output the feature vector corresponding to the CLS identifier at the beginning of each long text and input it into a binary (or n-class) output layer to perform softmax classification.

[0079] In classification tasks, the parameters of the additional fully connected layers are learned from scratch during training, while all parameters in the pre-trained BERT model are fine-tuned.

[0080] Calculate the loss function value and accuracy for each epoch of training and validation data. After multiple epochs, save the model weight with the smallest loss function value. This optimal model weight can be used directly for prediction.

[0081] 5) Finally, use the trained model weights to predict the category of the long Chinese text.

[0082] Step 5) specifically includes the following steps:

[0083] Obtain the weights of the classification model trained in step 4), classify the preprocessed prediction set, and calculate the average accuracy of the dataset. The formula for accuracy is as follows:

[0084] sum(T′=T l ) / L

[0085] In the formula:

[0086] T' represents the predicted data label;

[0087] Tl is the initial label for the data;

[0088] L represents the total number of labels in the predicted data.

[0089] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made under the inventive concept of the present invention using the contents of the present invention specification and drawings, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A classification method for extremely long Chinese texts based on a large language model, characterized in that, Includes the following steps: (1) Obtain Chinese long text classification data and corresponding classification labels; (2) Preprocessing of long Chinese text data; (3) Define the downstream task model of the large language model; (4) Fine-tuning of long Chinese texts using the large language model BERT; (5) Predict the category of the long Chinese text using the trained model weights; Step (2) includes: converting each categorized PDF file into an image and increasing the image resolution, then recognizing it as text using OCR; when loading data, truncating each line of data into text of less than or equal to the maximum length and assigning it the same label as the original text, setting the overlap length between each line of text and its context to half the length of the truncated text; then tokenizing each truncated segment to generate the following three inputs: Each character in the fragment corresponds to a unique identifier in the vocabulary; The position of the first sentence and special symbols is 0, and the position of the second sentence is 1; Positions shorter than the maximum length of each line are assigned a value of 0, while other positions are assigned a value of 1. Positions with a value of 1 are included in the attention calculation. Step (3) includes: defining a fully connected layer in the downstream task model to convert the vector dimension of BERT output into the number of categories; looping through the embeddings of multiple clauses after truncation in the long text, extracting features for each clause using the BERT language model, obtaining CLS, where CLS is the word vector of the first word in the input text, adding it to the set of CLS features of the truncated clauses of the long text, and then taking the average of the CLS sets of all clauses in the long text as the final CLS output of the long text; The formula for taking the average of the CLS set of all clauses is as follows: ； In the formula: The word vector representing the first word of each clause in each long sentence; The length of all clauses; The feature vector corresponding to the CLS identifier at the beginning of the long text is input into a binary or n-class output layer and softmax is used for classification. The softmax classification formula is as follows: ; In the formula: For tags; This represents the final hidden state of CLS; It is the parameter matrix for the classification task; The Softmax function is used to convert the output values ​​of multi-class classification into a probability distribution ranging from [0, 1] to 1. Step (4) includes: loading the weights of the large language model, inputting the preprocessed dataset, in each epoch (an epoch refers to training all data once), taking the average of the CLS set of clauses of the long text in each iteration as the final CLS output of the long text, and then inputting the feature vector corresponding to the CLS identifier at the beginning of each long text into a binary or n-classification output layer to perform softmax classification. In classification tasks, the parameters of the additional fully connected layers are learned from scratch during training, while all parameters in the pre-trained BERT model are fine-tuned. Calculate the loss function value and accuracy for each epoch of training and validation data. After multiple epochs, save the model weight with the smallest loss function value. This optimal model weight can be used directly for prediction.

2. The classification method according to claim 1, characterized in that, Step (1) includes: Obtain all Chinese data for the text categories, then label all the data for each category, place them in the designated folder, and clean the data.

3. The classification method according to claim 1, characterized in that, Step (5) includes: Obtain the weights of the classification model trained in step (4), classify the preprocessed prediction set, and calculate the average accuracy of the dataset. The formula for accuracy is as follows: In the formula: Indicates the predicted data labels; Initial labels for the data; This represents the total number of labels for the predicted data.

Images