A data classification method and system based on a pre-trained language model and a large language model

Abstract

The application relates to the technical field of natural language processing and artificial intelligence, and discloses a data classification method and system based on a pre-training language model and a large language model, which comprises the following steps: constructing a training data set and fine-tuning a pre-training language model (PLM) to obtain a PLM classifier; screening a high-quality sample subset from the training data set; performing preliminary classification on new text data to be classified by a large language model (LLM) based on the high-quality sample subset and a prompt word template; combining the classification results of the PLM classifier and the LLM to output a final classification result, and the collaborative decision-making stage comprises arbitration processing logic when the classification results are inconsistent. The application significantly improves the classification accuracy and reliability.

Description

Technical Field

[0001] This application relates to the fields of natural language processing and artificial intelligence, and in particular to a data classification method and system based on pre-trained language models and large language models. Background Technology

[0002] Text classification is a core task in natural language processing, widely used in scenarios such as sentiment analysis, news classification, spam filtering, and intent recognition. Traditional text classification methods heavily rely on large-scale, high-quality labeled data to train classification models (such as SVM and Naive Bayes). However, in practical applications, acquiring large amounts of labeled data is costly and time-consuming, especially in emerging scenarios with rapidly changing professional fields or categories (such as news classification of emerging topics), where traditional methods face serious challenges.

[0003] In recent years, pre-trained language models have learned rich linguistic knowledge through self-supervised pre-training on a large amount of unlabeled text. They can achieve excellent performance in downstream tasks with only a small amount of labeled data for fine-tuning, thus alleviating the dependence on labeled data. However, these models have limited performance in zero-shot scenarios where no samples are provided, and their knowledge is fixed at training time, making it difficult to adapt to emerging concepts not covered by the training data.

[0004] Large language models exhibit powerful general knowledge, emergent capabilities, and contextual learning abilities, enabling them to complete classification tasks with minimal or even zero-sample prompts and demonstrating good openness to new categories. However, their inference costs are high, their speed is slow, and their judgments in specific professional domains may not be accurate enough, posing a risk of "illusion."

[0005] Therefore, the existing technology has the following problems: 1) Insufficient classification accuracy in small sample or zero sample scenarios; 2) Difficulty in a single model to balance versatility and specialization; 3) Weak adaptability in dynamic classification scenarios. Summary of the Invention

[0006] To address the shortcomings in accuracy and adaptability of existing technologies in scenarios involving small samples, zero samples, and dynamic classification, this application provides a data classification method and system based on pre-trained language models and large language models.

[0007] This application discloses a data classification method based on a pre-trained language model and a large language model, which includes an initialization stage, a high-quality demonstration sample screening stage, a large language model context classification stage, and a collaborative decision-making stage. The initialization phase involves: constructing a training dataset and fine-tuning the pre-trained language model PLM to obtain the PLM classifier; The high-quality demonstration sample selection stage involves selecting a subset of high-quality demonstration samples from the training dataset. The context classification stage of the large language model: Based on a high-quality subset of demonstration samples and prompt word templates, the new text data to be classified is initially classified using the large language model LLM; The collaborative decision-making stage combines the classification results of the PLM classifier and the LLM classifier to output the final classification result. The collaborative decision-making stage also includes arbitration processing logic when the classification results are inconsistent.

[0008] Furthermore, the initialization phase includes: S101. Manually label the positive and negative samples in the historical data of the enterprise's internal data security inspection to construct a training dataset. ,in For text data, The positive and negative category labels are assigned to them; the positive samples are the samples corresponding to the target objects identified or detected in the historical data of the enterprise's internal data security inspection, and the negative samples are the samples in the historical data of the enterprise's internal data security inspection that do not contain the target objects; when constructing the training dataset, positive samples are randomly selected according to the number of negative samples to ensure the sample balance of the training dataset; S102. Select the pre-trained language model PLM and use the training dataset. The parameters are input into the PLM for full parameter fine-tuning to obtain the fine-tuned PLM classifier. .

[0009] Furthermore, the full parameter fine-tuning includes: Loading the pre-trained model: Use the functions provided by the framework to load the pre-trained model and pre-trained weights, modify the output layer of the pre-trained model, and replace it with a new output layer that matches the number of categories in the classification task. Configure training hyperparameters: Set the differential learning rate, using the first learning rate for the bottom layer of the model and the second learning rate for the top layer of the model; select Adam or SGD as the optimizer; set the appropriate number of training epochs and use early stopping techniques to prevent overfitting; the second learning rate should be greater than the first learning rate. Execute training loop: Iterate through the training dataset multiple times, and perform forward propagation, loss calculation, backpropagation, and weight update operations on each batch of samples. After each training loop, evaluate the model performance on the validation set and save the checkpoint of the model with the best performance on the validation set. In this process, forward propagation involves inputting training sample data into the model to obtain the model's prediction results; loss calculation involves comparing the model's prediction results with the true labels of the samples to calculate the loss value, which uses cross-entropy loss; backpropagation involves calculating the gradient of the model parameters based on the loss value, where the gradient represents the direction and magnitude of the model parameters that need to be adjusted; and weight update involves the optimizer updating the model parameters based on the calculated gradient.

[0010] Furthermore, the high-quality demonstration sample screening stage includes: S201, Prepare the training dataset Samples in Input to the fine-tuned PLM classifier Perform inference, obtain and record the prediction confidence. and prediction labels ; S202, Comparison Sample Manually labeled authentic labels With predictive labels Using a pre-defined filtering strategy, from the training dataset Selecting a high-quality sample subset The preset screening strategies include high-confidence screening, consistency screening, and category balance screening. The high-confidence screening involves selecting prediction confidence levels. Above the threshold For the samples, the consistency screening involves selecting predicted labels. With real labels Consistent samples, the class-balanced screening is to ensure that the filtered samples are consistent. The number of samples in each category is relatively balanced to avoid category bias.

[0011] Further, S201 includes: Model forward propagation: passing samples Enter to The semantic feature vectors of the samples are extracted through the Transformer layer inside the model, and then the semantic feature vectors are mapped to a value equal to the total number of categories in the classification task through the classification head at the end of the model. ; Obtain the raw score Logits: The length of the classification header output is... The vector is used as the original fraction Logits, where Logits is a vector composed of any real numbers; Logits normalization: Apply the Softmax function to the Logits to transform them into a probability distribution vector. The formula for calculating the Softmax function is as follows: ,in For the first Logits values ​​for each category; Extracting prediction results: Select the category corresponding to the maximum value in the probability distribution vector as the prediction label. The maximum value is selected as the prediction confidence level. .

[0012] Furthermore, the context classification stage of the large language model includes: S301. Construct a prompt word template, the prompt word template including a task description, a category list, and a subset of high-quality demonstration samples. Several sample examples were selected, and the format of the sample examples is "input text -> category"; S302, The new text data to be classified Fill in the prompt word template to form a complete prompt word. ; S303, Prompt words Input to the large language model LLM Perform context learning and output the correct values. Classification results and its confidence level ; The context learning specifically includes the following sub-steps: Word segmentation and embedding: Word segmentation is... Using its predefined vocabulary The process involves segmenting the word into a sequence of tokens; embedding involves converting each token into a high-dimensional word embedding vector, and the entire sequence of tokens forms a vector sequence for input into the model. Forward computation: sequentially pass through the vector sequence All Transformer decoder layers use a causal self-attention mechanism to calculate the relationship between each word in the sequence and all previous words, understanding the task format and the input-output mapping pattern. When processing the last position of the sequence, the final hidden state vector is output, which is then fed into the language model head, i.e., the linear layer, with an output length equal to the vocabulary size. The Logits vector, where each value represents the original score by which the model considers the next word to be the corresponding word in the vocabulary; Generating the next word and calculating the confidence score: Apply the Softmax function to the Logits vector to convert it into a probability distribution, where each value represents the probability that the corresponding word will appear as the next word. The calculation formula is as follows: The token is the minimum semantic length unit set according to the text encoding; a greedy decoding strategy is adopted to select the word with the highest probability in the probability distribution as the classification result. The probability value corresponding to the term is used as the confidence level. .

[0013] Furthermore, the collaborative decision-making stage includes: S401. The new text data to be classified... Input to the fine-tuned PLM classifier Perform predictions to obtain PLM prediction labels. and its confidence level ; S402, based on , , , New text data to be classified is generated using a pre-defined collaborative decision-making strategy. Final classification results The preset collaborative decision-making strategy includes at least a confidence-weighted voting strategy, a confidence-threshold arbitration strategy, and an inconsistency handling strategy.

[0014] Furthermore, the confidence-weighted voting strategy and the confidence-threshold arbitration strategy are specifically as follows: The calculation formula for the confidence-weighted voting strategy is as follows: ,in and These are the weighting coefficients. For indicator functions, ( )=δ( ={1,0} ( )=δ( ={1,0}; The confidence threshold arbitration strategy is as follows: if Higher than the preset threshold Then As the final classification result ;like Not higher than Then As the final classification result .

[0015] Furthermore, the inconsistency handling strategy specifically includes: when At this time, a lightweight arbitrator is introduced, which is based on the training dataset. The loss function of the logistic regression model obtained from the training is: ; in, This is the real label; a value of 1 indicates selection. The classification result, a value of 0 indicates selection. The classification results; Predicting Arbitrator Selection The probability of the classification result; The number of training samples; , , , The input is fed into the arbitrator, which then outputs the final classification result. ; The method also includes an incremental model update step: During the periodic collection and classification process, if the confidence level is less than or equal to a preset threshold... The sample data and their manually labeled true categories are used to form an incremental dataset, which is then input into the fine-tuned PLM classifier. Perform incremental fine-tuning and update. The model parameters; and the pre-trained language model PLM is a BERT, RoBERTa, or ALBERT model.

[0016] This application also discloses a data classification system based on a pre-trained language model and a large language model, which implements the above-described method and includes: The initialization module is used to build the training dataset and fine-tune the pre-trained language model PLM to obtain the PLM classifier. The high-quality demonstration sample selection module is used to select a subset of high-quality demonstration samples from the training dataset; The Large Language Model Context Classification Module is used to perform preliminary classification of new text data to be classified based on a high-quality subset of demonstration samples and prompt word templates using the Large Language Model (LLM). The collaborative decision-making module combines the classification results of the PLM classifier and the LLM classifier to output the final classification result. The collaborative decision-making stage includes arbitration processing logic when the classification results are inconsistent.

[0017] Due to the adoption of the above technical solution, this application has the following advantages: 1. High accuracy and high robustness: By combining and verifying LLM and PLM, the shortcomings of a single model are compensated, especially when the classification boundary is ambiguous or the sample is scarce, which significantly improves the classification accuracy and reliability.

[0018] 2. High efficiency: Most simple and clear samples can be quickly classified by LLM, and only difficult samples trigger PLM for in-depth analysis, which realizes the optimized allocation of computing resources and high overall processing efficiency.

[0019] 3. Strong adaptability: LLM's zero-shot capability allows it to easily handle new categories. Combined with the subsequent incremental model update mechanism, the system can continuously learn and adapt to dynamically changing data distributions.

[0020] 4. Cost-effectiveness: It reduces the reliance on large-scale labeled data, making it particularly suitable for application scenarios where it is difficult to obtain labeled data or where classification requirements change frequently. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0022] Figure 1 This is a flowchart illustrating a data classification method based on a pre-trained language model and a large language model, according to an embodiment of this application. Detailed Implementation

[0023] The present application will be further described in conjunction with the accompanying drawings and embodiments. The described embodiments are only some, not all, of the embodiments of the present application. All other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of the present application.

[0024] See Figure 1 This application provides an embodiment of a data classification method based on a pre-trained language model and a large language model, comprising four main stages: initialization, high-quality demonstration sample selection, contextual classification using the large language model, and collaborative decision-making. It also includes an incremental model update step. This method can achieve accurate classification of text data and, when the initial classification confidence of the large language model is insufficient, can combine the accurate classification results of the pre-trained language model for collaborative decision-making to improve classification accuracy. Furthermore, it achieves continuous model optimization through incremental fine-tuning. The pre-trained language model PLM used in this method is a BERT, RoBERTa, or ALBERT model fine-tuned with domain-specific text data, ensuring the model's classification performance in that specific domain.

[0025] I. Initialization Phase The core objective of the initialization phase is to construct a training dataset and fine-tune the pre-trained language model PLM with all parameters to obtain a PLM classifier adapted to the enterprise's internal data security inspection and classification task. Specifically, it includes two steps: S101 and S102. S101: Constructing the training dataset Manual annotation was performed on positive and negative samples from historical data of internal data security inspections. Positive samples consisted of text data corresponding to target objects identified or detected in the historical data, while negative samples consisted of text data that did not contain target objects. Since the number of negative samples is usually small in real-world scenarios, positive samples were randomly selected based on the number of negative samples to ensure the balance of the training dataset. This resulted in the final training dataset. The dataset contains This is text data (sample). The text data is labeled with corresponding positive and negative categories, with each label corresponding to a specific category, providing a basis for subsequent model fine-tuning.

[0026] S102: Fine-tuning of all parameters of a pre-trained language model Select a pre-trained language model PLM (BERT, RoBERTa, or ALBERT model), and use the training dataset constructed above. The input is fed into the PLM for full parameter fine-tuning, updating all parameters of the model, and finally obtaining the fine-tuned PLM classifier. The fine-tuning process is divided into three sub-steps: loading the pre-trained model, configuring the training hyperparameters, and executing the training loop. (1) Load the pre-trained model The framework provides functions to load the selected pre-trained model and its corresponding pre-trained weights. Since the output layer of the pre-trained model is designed for the original task (e.g., ImageNet has 1000 categories), it cannot be adapted to the classification task of internal data security inspection. Therefore, the key operation is to modify the output layer (header) of the pre-trained model and replace it with a new output layer that matches the number of categories in this classification task, so that the model output matches the task requirements.

[0027] (2) Configure training hyperparameters The configuration of training hyperparameters directly affects the model fine-tuning effect. The specific configuration rules are as follows: Learning rate: Set a differential learning rate, with a low learning rate for the lower layers of the model that extracts general features and a higher learning rate for the upper layers of the model that extracts specific features. This setting method can avoid drastically changing the general features already learned by the pre-trained model, and only make small adjustments to the specific feature layers to adapt to specific classification tasks. Optimizer: Select Adam or SGD as the model optimizer; Training rounds: Set a sufficient number of training rounds, and use early stopping techniques to prevent overfitting. Sufficient training rounds ensure that the model fully fits the training data, while early stopping techniques can terminate training in advance when the model performance on the validation set no longer improves, thus avoiding model overfitting.

[0028] (3) Execute the training loop The training dataset is iterated through multiple rounds. For each batch of text data, forward propagation, loss calculation, back propagation, and weight update operations are performed sequentially. After each training round, model performance is evaluated and the optimal model is saved. The specific implementation logic of each operation is as follows: Forward propagation: Inputting training text data into the model, and obtaining the model's prediction results through feature extraction and calculation; Loss calculation: The model prediction results are compared with the true labels of the samples to calculate the loss value. In this implementation, cross-entropy loss is used as the loss calculation index to quantify the deviation between the model prediction results and the true results. Backpropagation: Calculates the gradient of the model parameters based on the calculated loss value. This gradient represents the direction and magnitude of the model parameters that need to be adjusted, providing a basis for parameter updates. Weight update: The optimizer updates the parameters of the model based on the gradient calculated by backpropagation, so that the model iterates in the direction of fitting the training data. Model Evaluation and Saving: After each training epoch, the validation set is input into the model to evaluate its performance in terms of accuracy, loss, and other dimensions. The best-performing model checkpoint on the validation set is saved as the final PLM classifier. .

[0029] II. High-quality demonstration sample selection stage The core of the high-quality demonstration sample selection phase is to select samples from the training dataset. Selecting a high-quality subset of demonstration samples This provides reliable demonstration samples for subsequent contextual learning in large language models, specifically including two steps, S201 and S202. The selected samples must meet three requirements: high confidence, consistency between predictions and true labels, and class balance. S201: PLM classifier inference and record results training dataset Text data in Input one by one into the fine-tuned PLM classifier Inference is performed, and the prediction confidence level corresponding to each text data is calculated and recorded through the model. and prediction labels The reasoning process is specifically divided into four sub-steps, realizing the transformation from text input to prediction output: (1) Model forward propagation Text data Input to PLM classifier The model first extracts semantic feature vectors (hidden states) from the text data through its internal Transformer layer. These feature vectors are rich in semantic information about the text. Then, through the Classifier Head (linear / fully connected layer) at the end of the model, the extracted semantic feature vectors are mapped to a value with a length equal to the total number of categories in the classification task. (For example, in a sentiment analysis task, C=2, corresponding to "positive" and "negative") vectors whose dimensions match the number of categories in a classification task.

[0030] (2) Obtain the raw score Logits The length output by the classification head is The vector is defined as the raw scores Logits (logic values ​​or logical scores). Logits is a vector of arbitrary real numbers (positive, negative, or zero), each value representing the raw score by which the model believes the input text data belongs to the corresponding category. The higher the Logits value for a category, the more likely the model is to believe the text data belongs to that category. Logits itself is not a probability, as it can be negative, and the sum of all values ​​is not 1.

[0031] (3) Logits normalization To give the model output probabilistic meaning, the Logits are normalized using the Softmax function, transforming them into a probability distribution vector where each value is between 0 and 1, and the sum of all values ​​is 1. The formula for calculating the Softmax function is: ,in For the first Logits values ​​for each category, The function is an exponential function, ensuring all calculated values ​​are positive. The denominator is the sum of the exponents of all categories, achieving normalization. After Softmax processing, a probability distribution vector is obtained, for example, [0.05, 0.15, 0.80]. Each element of this vector represents the probability that the model predicts a sample belongs to a certain category.

[0032] (4) Extract the prediction results From the probability distribution vector processed by the Softmax function, the category corresponding to the largest probability value is selected as the predicted label for the text data. ,Right now Simultaneously, the highest probability value is selected as the prediction confidence level for the text data. ,Right now This enables accurate extraction of predicted labels and confidence levels. For example, if the probability distribution vector is [0.05, 0.15, 0.80], then... It's the third category (index 2). = 0.80.

[0033] S202: Selecting a high-quality subset of exemplary samples Compare each text data Manually labeled authentic labels With model prediction labels Combined with a preset filtering strategy, from the training dataset Selecting a high-quality sample subset The preset screening strategy includes high-confidence screening, consistency screening, and category balance screening. All three screening strategies must be satisfied simultaneously. The specific screening rules are as follows: (1) High confidence screening: Selecting the PLM classifier Output prediction confidence Above the threshold The text data ensures that the selected samples are those predicted by the model with high confidence. (2) Consistency screening: Ensure the PLM classifier Output predicted labels Human-labeled real labels for text data Completely consistent, ensuring the accuracy of the prediction results for the selected samples; (3) Class-balanced screening: Ensure that the final selected sample subset is of high quality. In this model, the amount of text data corresponding to each category is relatively balanced, avoiding category bias and ensuring that the large language model can access samples from each category during subsequent context learning.

[0034] III. Context Classification Stage of Large Language Model The core of the context classification stage in large language models is based on a high-quality subset of demonstration samples. Construct prompt word templates and use the Large Language Model (LLM) New text data to be classified Perform context learning and preliminary classification, and output classification results. and its confidence level Specifically, it includes three steps: S301, S302, and S303. This stage utilizes the zero-shot / few-shot learning capability of the large language model to achieve rapid preliminary classification. S301: Constructing a prompt word template Develop a prompt template adapted to the enterprise's internal data security audit categorization tasks. This template includes three core components: task description, category list, and a subset of high-quality sample demos. The sample text is a collection of examples, arranged in the format of "input text -> category", such as: document content inspection, internal enterprise information, internal network policy, and specific document examples.

[0035] S302: Generate complete prompt words New text data to be classified Fill in the above-constructed prompt template to form a complete prompt. .

[0036] S303: Context Learning and Classification of Large Language Models Complete prompt words Input to the large language model LLM In the middle, by Based on the example samples in the prompts, the system learns the context and understands the "input-output" mapping pattern of the classification task, then outputs new text data to be classified. Classification results and its confidence level The context learning process is specifically divided into three sub-steps: word segmentation and embedding, forward computation (obtaining Logits), and generating the next lexical unit and calculating confidence. (1) Word segmentation and embedding Word segmentation: Large language model Use its predefined vocabulary to provide complete prompt words. This long string is segmented into a sequence of tokens, where a token is the smallest semantic unit defined by the text encoding. For example, it can be segmented into a token sequence of ["task", ":", "file attributes", "time", ... , "profession", ":"]. Embedding: Each segmented word is converted into a high-dimensional word embedding vector. The embedding vectors of all words are arranged in order to form a vector sequence, which serves as the input data for the model, thus completing the transformation from text to vector.

[0037] (2) Forward computation (obtaining Logits) The generated vector sequences are then sequentially passed through a large language model. In all Transformer decoder layers, each layer uses a causal self-attention mechanism to calculate the relationship between each word in the sequence and all words preceding it (including instructions and all example samples), enabling the model to fully understand the classification task format and the "input-output" mapping pattern. When the model processes the last position of the sequence, i.e., "sensitive: after (currently still blank)," the model has "understood" the entire prompt. It outputs a final hidden state vector for this position, which is then fed into the language model head (a linear layer) of the large language model. The language model head outputs a length equal to the vocabulary size. The Logits vector, where each value represents the original score by which the model considers the next lexicon to be the corresponding lexicon in the vocabulary.

[0038] (3) Generate the next word and calculate the confidence score This step is the core step in the large language model outputting classification results, assuming the classification labels... (For example, "positive") is itself an independent word unit in the vocabulary, specifically implemented as follows: Applying the Softmax function to the Logits vector obtained from the forward computation transforms it into a probability distribution, where each value represents the probability of the corresponding word appearing as the next word. The calculation formula is as follows: ; A basic text unit is the minimum semantic length unit defined by the text encoding. A greedy decoding strategy is used as the word selection strategy, directly selecting the word with the highest probability in the probability distribution as the new text data to be classified. Classification results In the probability distribution, the model may assign the highest probability (e.g., 0.7) to "positive", 0.25 to "neutral", and 0.05 to "negative".

[0039] The probability value corresponding to the word with the highest probability (i.e., y_llm = "positive") is taken as the generation probability (probability value corresponding to the word) for this classification, i.e., the confidence level. (0.7), complete the preliminary classification of the large language model.

[0040] IV. Collaborative Decision-Making Stage The core of the collaborative decision-making phase is to simultaneously utilize the PLM classifier. and large language models New text data to be classified The classification process is performed, and the final classification result is generated by combining the classification results and confidence levels from both methods through a pre-defined collaborative decision-making strategy. This fully leverages the accuracy of the PLM classifier and the flexibility of the large language model, specifically including two steps: S401 and S402. S401: PLM Classifier Prediction New text data to be classified Input to the fine-tuned PLM classifier In the process of prediction, the predicted label output by the PLM classifier is obtained through the aforementioned model inference process. and its confidence level This provides a precise classification reference for subsequent collaborative decision-making.

[0041] S402: Multi-strategy collaborative decision-making Classification results based on large language models Confidence level And the predicted labels of the PLM classifier Confidence level New text data to be classified is generated using a pre-defined collaborative decision-making strategy. Final classification results The preset collaborative decision-making strategy includes at least a confidence-weighted voting strategy, a confidence-threshold arbitration strategy, and an inconsistency handling strategy. These three strategies are flexibly applied based on the actual classification results. The specific implementation logic is as follows: (1) Confidence-weighted voting strategy This strategy combines the classification results of the two methods using a weighted calculation, and the calculation formula is as follows: ; in, and These are weighting coefficients, which can be adjusted based on the classification performance of the PLM classifier and the large language model in the actual scenario. For indicator functions, ( )=δ( ={1,0} ( )=δ( ={1,0} ( )and ( The value of ) is either 1 or 0, taking the value 1 when the condition within the parentheses is true and 0 when the condition is false; finally, the category with the highest confidence after weighted calculation is selected as the final classification result. .

[0042] (2) Confidence threshold arbitration strategy Preset extremely high threshold (For example, 0.95), leveraging the high reliability of the PLM classifier's classification results under high confidence conditions, a fast arbitration is performed: if the confidence level output by the PLM classifier is... Above this extremely high threshold Then directly use the predicted labels of the PLM classifier As the final classification result ;like Not higher than Then the classification results of the large language model will be used. As the final classification result This strategy can reduce the cost of invalid calls to large language models and the uncertainty of classification results while ensuring classification accuracy.

[0043] (3) Inconsistency handling strategy When the PLM classifier predicts the label Classification results of large language models In case of inconsistency, a lightweight arbitrator is introduced to make the final decision. This arbitrator is based on the training dataset. The loss function of the logistic regression model obtained from the training is: ; in, The true labels used for training the arbitrator; a value of 1 indicates the selection of a large language model. The classification result, a value of 0 indicates that the PLM classifier is selected. The classification results; Choosing a large language model for arbitrator prediction The probability of the classification result; The number of samples used to train the arbitrator; In practical applications, , , , The input is fed into the arbitrator, which then outputs the final classification result based on the trained model logic. This resolves the inconsistency between the two classification results.

[0044] V. Model Incremental Update Steps To achieve continuous model optimization and improve classification accuracy over long-term use, this method also includes an incremental model update step: during daily classification, all classification results with confidence levels less than or equal to an extremely high threshold are periodically collected. The text data is manually labeled to supplement its true categories, forming an incremental dataset; this incremental dataset is then input into a fine-tuned PLM classifier. Perform incremental fine-tuning and update. The model parameters enable the model to continuously learn new sample features, adapt to constantly changing classification scenarios, and ensure that the model's classification performance remains at a high level.

[0045] Furthermore, this method can optimize the preliminary classification process of the large language model, and in the large language model When performing the initial classification, the configuration model simultaneously outputs keywords or phrases that support its initial classification judgment. When the initial classification category of the large language model is inconsistent with the accurate classification category of the pre-trained language model, the consistency of the keywords or phrases can be verified with the accurate classification results of the pre-trained language model. The final classification category can be further determined through verification, thereby improving the reliability of the classification results.

[0046] The core idea of ​​this application is to construct a two-stage collaborative classification pipeline of "LLM preliminary screening + PLM precise verification". This method fully utilizes the versatility and zero-sample capability of LLM for rapid and broad preliminary judgment, while using domain-specific PLM for in-depth and precise analysis, and combines the advantages of both through an intelligent decision-making mechanism.

[0047] This application constructs a hierarchical collaborative classification framework: First, a large-scale, generalizable language model with a large parameter size is used as a preliminary classifier with zero or few samples to generate preliminary classification results and keywords with confidence. Second, a lightweight pre-trained language model, fine-tuned for a specific domain or task and with faster inference speed, is used as the precise classifier. The system automatically decides whether to activate the precise classifier for secondary analysis and verification based on the confidence level of the preliminary results, and intelligently fuses the outputs of both to generate the final classification result. This application significantly improves the accuracy, robustness, and efficiency of data classification, and is particularly suitable for scenarios where labeled data is scarce or classification categories change dynamically.

[0048] This application combines the accurate classification capabilities of pre-trained language models with the zero-shot / few-shot learning capabilities of large language models. It achieves efficient and accurate classification of text data through a four-stage classification process. At the same time, it addresses the limitations of single-model classification through collaborative decision-making strategies and enables continuous model optimization through incremental fine-tuning. It is particularly suitable for text classification scenarios in enterprise internal data security inspections, and can effectively improve the efficiency and accuracy of data classification.

[0049] This application also provides an embodiment of a data classification system based on a pre-trained language model and a large language model, which implements the method described in the above embodiment, and includes: The initialization module is used to build the training dataset and fine-tune the pre-trained language model PLM to obtain the PLM classifier. The high-quality demonstration sample selection module is used to select a subset of high-quality demonstration samples from the training dataset; The Large Language Model Context Classification Module is used to perform preliminary classification of new text data to be classified based on a high-quality subset of demonstration samples and prompt word templates using the Large Language Model (LLM). The collaborative decision-making module combines the classification results of the PLM classifier and the LLM classifier to output the final classification result. The collaborative decision-making stage includes arbitration processing logic when the classification results are inconsistent.

[0050] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and not to limit them. Although this application has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of this application. Any modifications or equivalent substitutions that do not depart from the spirit and scope of this application should be covered within the protection scope of the claims of this application.

Claims

1. A data classification method based on pre-trained language models and large language models, characterized in that, It includes the initialization phase, the high-quality demonstration sample selection phase, the large language model context classification phase, and the collaborative decision-making phase; The initialization phase involves: constructing a training dataset and fine-tuning the pre-trained language model PLM to obtain the PLM classifier; The high-quality demonstration sample selection stage involves selecting a subset of high-quality demonstration samples from the training dataset. The context classification stage of the large language model: Based on a high-quality subset of demonstration samples and prompt word templates, the new text data to be classified is initially classified using the large language model LLM; The collaborative decision-making stage combines the classification results of the PLM classifier and the LLM classifier to output the final classification result. The collaborative decision-making stage also includes arbitration processing logic when the classification results are inconsistent.

2. The method according to claim 1, characterized in that, The initialization phase includes: S101. Manually label the positive and negative samples in the historical data of the enterprise's internal data security inspection to construct a training dataset. ,in For text data, The positive and negative category labels are assigned to them; the positive samples are the samples corresponding to the target objects identified or detected in the historical data of the enterprise's internal data security inspection, and the negative samples are the samples in the historical data of the enterprise's internal data security inspection that do not contain the target objects; when constructing the training dataset, positive samples are randomly selected according to the number of negative samples to ensure the sample balance of the training dataset; S102. Select the pre-trained language model PLM and use the training dataset. The parameters are input into the PLM for full parameter fine-tuning to obtain the fine-tuned PLM classifier. .

3. The method according to claim 2, characterized in that, The full parameter fine-tuning includes: Loading the pre-trained model: Use the functions provided by the framework to load the pre-trained model and pre-trained weights, modify the output layer of the pre-trained model, and replace it with a new output layer that matches the number of categories in the classification task. Configure training hyperparameters: Set the differential learning rate, using the first learning rate for the bottom layer of the model and the second learning rate for the top layer of the model; select Adam or SGD as the optimizer; set the appropriate number of training epochs and use early stopping techniques to prevent overfitting; the second learning rate should be greater than the first learning rate. Execute training loop: Iterate through the training dataset multiple times, and perform forward propagation, loss calculation, backpropagation, and weight update operations on each batch of samples. After each training loop, evaluate the model performance on the validation set and save the checkpoint of the model with the best performance on the validation set. In this process, forward propagation involves inputting training sample data into the model to obtain the model's prediction results; loss calculation involves comparing the model's prediction results with the true labels of the samples to calculate the loss value, which uses cross-entropy loss; backpropagation involves calculating the gradient of the model parameters based on the loss value, where the gradient represents the direction and magnitude of the model parameters that need to be adjusted; and weight update involves the optimizer updating the model parameters based on the calculated gradient.

4. The method according to claim 1, characterized in that, The high-quality demonstration sample screening phase includes: S201, Prepare the training dataset Samples in Input to the fine-tuned PLM classifier Perform inference, obtain and record the prediction confidence. and prediction labels ; S202, Comparison Sample Manually labeled authentic labels With predictive labels Using a pre-defined filtering strategy, from the training dataset Selecting a high-quality sample subset The preset screening strategies include high-confidence screening, consistency screening, and category balance screening. The high-confidence screening involves selecting prediction confidence levels. Above the threshold For the samples, the consistency screening involves selecting predicted labels. With real labels Consistent samples, the class-balanced screening is to ensure that the filtered samples are consistent. The number of samples in each category is relatively balanced to avoid category bias.

5. The method according to claim 4, characterized in that, S201 includes: Model forward propagation: passing samples Enter to The semantic feature vectors of the samples are extracted through the Transformer layer inside the model, and then the semantic feature vectors are mapped to a value equal to the total number of categories in the classification task through the classification head at the end of the model. ; Obtain the raw score Logits: The length of the classification header output is... The vector is used as the original fraction Logits, where Logits is a vector composed of any real numbers; Logits normalization: Apply the Softmax function to the Logits to transform them into a probability distribution vector. The formula for calculating the Softmax function is as follows: ,in For the first Logits values ​​for each category; Extracting prediction results: Select the category corresponding to the maximum value in the probability distribution vector as the prediction label. The maximum value is selected as the prediction confidence level. .

6. The method according to claim 1, characterized in that, The context classification stage of the large language model includes: S301. Construct a prompt word template, the prompt word template including a task description, a category list, and a subset of high-quality demonstration samples. Several sample examples were selected, and the format of the sample examples is "input text -> category"; S302, The new text data to be classified Fill in the prompt word template to form a complete prompt word. ; S303, Prompt words Input to the large language model LLM Perform context learning and output the correct values. Classification results and its confidence level ; The context learning specifically includes the following sub-steps: Word segmentation and embedding: Word segmentation is... Using its predefined vocabulary The process involves segmenting the word into a sequence of tokens; embedding involves converting each token into a high-dimensional word embedding vector, and the entire sequence of tokens forms a vector sequence for input into the model. Forward computation: sequentially pass through the vector sequence All Transformer decoder layers use a causal self-attention mechanism to calculate the relationship between each word in the sequence and all previous words, understanding the task format and the "input-output" mapping pattern. When processing the last position of the sequence, the final hidden state vector is output, which is then fed into the language model head, i.e., the linear layer, with an output length equal to the vocabulary size. The Logits vector, where each value represents the original score by which the model considers the next word to be the corresponding word in the vocabulary; Generating the next word and calculating the confidence score: Apply the Softmax function to the Logits vector to convert it into a probability distribution, where each value represents the probability that the corresponding word will appear as the next word. The calculation formula is as follows: The token is the minimum semantic length unit set according to the text encoding; a greedy decoding strategy is adopted to select the word with the highest probability in the probability distribution as the classification result. The probability value corresponding to the term is used as the confidence level. .

7. The method according to claim 1, characterized in that, The collaborative decision-making phase includes: S401. The new text data to be classified... Input to the fine-tuned PLM classifier Perform predictions to obtain PLM prediction labels. and its confidence level ; S402, based on , , , New text data to be classified is generated using a pre-defined collaborative decision-making strategy. Final classification results The preset collaborative decision-making strategy includes at least a confidence-weighted voting strategy, a confidence-threshold arbitration strategy, and an inconsistency handling strategy.

8. The method according to claim 7, characterized in that, The confidence-weighted voting strategy and the confidence threshold arbitration strategy are specifically as follows: The calculation formula for the confidence-weighted voting strategy is as follows: ,in and These are the weighting coefficients. For indicator functions, ( )=δ( ={1,0} ( )=δ( ={1,0}; The confidence threshold arbitration strategy is as follows: if Higher than the preset threshold Then As the final classification result ;like Not higher than Then As the final classification result .

9. The method according to claim 7, characterized in that, The inconsistency handling strategy is specifically as follows: when At this time, a lightweight arbitrator is introduced, which is based on the training dataset. The loss function of the logistic regression model obtained from the training is: ； in, This is the real label; a value of 1 indicates selection. The classification result, a value of 0 indicates selection. The classification results; Predicting Arbitrator Selection The probability of the classification result; The number of training samples; , , , The input is fed into the arbitrator, which then outputs the final classification result. ; The method also includes an incremental model update step: During the periodic collection and classification process, if the confidence level is less than or equal to a preset threshold... The sample data and their manually labeled true categories are used to form an incremental dataset, which is then input into the fine-tuned PLM classifier. Perform incremental fine-tuning and update. The model parameters; and the pre-trained language model PLM is a BERT, RoBERTa, or ALBERT model.

10. A data classification system based on a pre-trained language model and a large language model, implementing the method described in any one of claims 1-9, characterized in that, include: The initialization module is used to build the training dataset and fine-tune the pre-trained language model PLM to obtain the PLM classifier. The high-quality demonstration sample selection module is used to select a subset of high-quality demonstration samples from the training dataset; The Large Language Model Context Classification Module is used to perform preliminary classification of new text data to be classified based on a high-quality subset of demonstration samples and prompt word templates using the Large Language Model (LLM). The collaborative decision-making module combines the classification results of the PLM classifier and the LLM classifier to output the final classification result. The collaborative decision-making stage includes arbitration processing logic when the classification results are inconsistent.

Images