A method for improving transfer learning text classification accuracy by using small sample data

Abstract

The application discloses a method for improving the accuracy of text classification of transfer learning by using small sample data, comprising the following steps: S1, using industrial internet data as original training data; S2, using a LaserTagger data enhancement method to enhance the original training data to generate more training data; S3, using the enhanced data to train a BERT model, specifically using BERT for text classification, and BERT is based on Masked Language Model (MLM) as a new pre-training target language model. The application can improve the text classification and recognition effect in the field of industrial internet by using the model with small sample corpus, increase the amount of text data in the field of industrial internet and improve the robustness of the model through the mode of semantics and model, and then recognize by using the transfer learning method, reduce the cost of text labeling data, and improve the text classification effect and model recognition accuracy in the field of industrial internet.

Description

Technical Field

[0001] This invention relates to a method for improving the accuracy of text classification in transfer learning using small sample data, and belongs to the fields of natural language processing and industrial internet. Background Technology

[0002] Text classification, as one of the fundamental and important tasks in the field of natural language processing, has received close attention from researchers for many years. Its research can be traced back to the mid-20th century. Text classification techniques have gradually evolved and matured from early methods based on rules defined by domain experts to today's advancements in deep learning. Text classification techniques can be mainly divided into three categories: dictionary-based and rule-based methods, statistical model-based methods, and deep learning-based methods.

[0003] Natural language processing (NLP) techniques based on statistical machine learning rely on rule coverage and accuracy to classify text. However, rule formulation heavily depends on expert experience and is limited by the expressive power of rules, resulting in high manpower costs. Therefore, many machine learning algorithms based on rule coverage and accuracy have emerged, such as Naive Bayes, K-Nearest Neighbors, SVM, Decision Trees, and Random Forests. These algorithms have reduced the cost of building classifiers and improved classification accuracy to some extent. With the emergence of word vector training models that have achieved success in text representation, more text classification models based on deep neural networks have also emerged, such as convolutional neural networks and recurrent neural networks. Currently, there is still a large amount of key information similar to n-grams within sentences, which can be obtained through local feature extraction in convolution, achieving better local semantic relevance. LSTM and GRU models, which simplify recurrent neural networks, have also been introduced, solving the gradient vanishing problem in RNN sequences to some extent while achieving good results in text classification tasks. By 2017, the model also proved that increasing the network depth could solve the problem of long-distance text dependencies and achieve significantly better accuracy.

[0004] Traditional entity extraction methods based on dictionaries and rules have high accuracy within the scope of the dictionary and rules, but poor generalization effect, which cannot meet the requirements of high-accuracy text classification. Deep learning-based knowledge extraction models have good generalization effect, but require a large amount of training data.

[0005] The Industrial Internet field is characterized by the diversity and differentiation of domain knowledge. Classifying textual information related to the Industrial Internet can help to categorize and organize this information. However, its data suffers from problems such as small scale and lack of labeled data.

[0006] Due to the unique industrial environment and industry characteristics, the Industrial Internet field often faces the following small-sample problems: Insufficient data volume: In some cases, the dataset is very small, such as in unconventional scenarios within certain industries or with specific equipment. This may result in the dataset not covering all scenarios or failing to reflect the diversity of the real world when building machine learning models. Poor data quality: In some cases, data may be subject to varying degrees of noise interference. For example, sensor data may be affected by transmission errors, acquisition noise, or equipment malfunctions, leading to poor data quality. These interfering factors can negatively impact model training. Class imbalance: In some cases, there may be a severe imbalance in the number of samples from different classes in the dataset. For example, there may be very few samples of certain faults, while the number of normal samples is very large. This may cause the model to perform poorly in classifying a minority of classes during prediction. Summary of the Invention

[0007] The purpose of this invention is to propose a method for improving the accuracy of text classification through transfer learning using small sample data, thereby addressing the problems existing in the prior art. By using semantics and models, the method increases the amount of text data and improves the robustness of the model for samples in the industrial internet field without increasing the cost of manual labeling. Then, transfer learning is used for recognition, reducing the cost of text annotation data and improving text classification performance in the industrial internet field, thus enhancing the model recognition accuracy in this area.

[0008] A method for improving the accuracy of transfer learning text classification using small sample data is described below:

[0009] Step S1: Use industrial internet data as the original training data.

[0010] Step S2, using a LaserTagger-based (e.g., Figure 1 The data augmentation method (shown) enhances the original training data to generate more training data. The specific process is as follows:

[0011] S201. Split each data sample in the original data into an input sentence and a label sequence.

[0012] S202. Enhance the input sentence based on LaserTagger. This can be done by using methods such as synonym replacement, antonym replacement, random word insertion, and word order reversal to generate a new input sentence.

[0013] S203. Enhance the label sequence accordingly to ensure that it corresponds to the enhanced input sentence.

[0014] S204. Add the enhanced input sentence from step S202 and the enhanced label sequence from step S203 as new training data to the original training data.

[0015] Step S3: Train the BERT model using the augmented data. Specifically, use BERT for text classification. BERT is based on the Masked Language Model (MLM) as a new pre-trained target language model. The specific steps are as follows:

[0016] S301. Divide the enhanced dataset into training set, validation set and test set.

[0017] S302. When using the BERT model for training, you can choose a pre-trained base model for transfer learning.

[0018] S303. During training, various techniques can be used to improve model performance, such as learning rate decay, early stopping, dropout, etc.

[0019] S304. After the model training is completed, the model can be evaluated using a test set to assess its performance.

[0020] S305. If the model's performance is poor, you can adjust the model parameters, increase the training data, or try other techniques to further improve the model's performance.

[0021] The present invention provides a method for improving the accuracy of transfer learning text classification using small sample data. Its advantages and effects are as follows: it can improve the text classification and recognition effect in the industrial internet field by using models with small sample corpora. Through semantics and models, it increases the amount of text data and improves the robustness of the model for samples in the industrial internet field without increasing the cost of manual labeling. Then, it uses transfer learning methods for recognition, reduces the cost of text labeling data, improves the text classification effect in the industrial internet field, and improves the model recognition accuracy in the industrial internet field. Attached Figure Description

[0022] Figure 1 The diagram shows the process of processing Chinese characters using LaserTagger.

[0023] Figure 2 The diagram shows the basic structure of the transformer model.

[0024] Figure 3 The diagram shows the process flow for transfer learning based on a small sample size.

[0025] The specific implementation method is as follows:

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been presented in the various embodiments of the present invention to enable the reader to better understand the present invention. However, the technical solutions claimed in the present invention can be implemented even without these technical details and various changes and modifications based on the following embodiments.

[0027] This invention proposes a method to improve the accuracy of transfer learning text classification using small sample data, primarily targeting knowledge recognition tasks in the industrial internet field. Traditional data augmentation methods generally focus on semantic enhancement at the word level. Specifically, this method includes: replacing non-entity words in the labeled data with synonyms based on a thesaurus; random insertion, randomly inserting words into the non-entity parts to introduce noise into the text data and increase system robustness; entity replacement, randomly replacing several named entities in the labeled data with other named entities of the same type to enrich the diversity of entity representations; and masking core material information, such as material names, to ensure that core material information is not lost and to maintain the integrity of core information in the material recognition corpus during the conversion process.

[0028] Example 1

[0029] This invention proposes a data augmentation method based on LaserTagger. LaserTagger (proposed by Google) was developed based on the following situation: for many text generation tasks, there is often overlap between input and output. For example, in text processing tasks, most of the input text can remain unchanged, requiring only modification of a small subset of words. LaserTagger can perform a series of editing operations on the original sentence, such as keep (copying words directly to the output), delete (deleting words), keep-addx (adding the phrase X before the tag), and delete-addx (deleting the tagged word).

[0030] LaserTagger adds only words from the vocabulary to the target text, thus avoiding the problem of the model adding arbitrary words and reducing the occurrence of outputs that are not supported by the input text. Furthermore, LaserTagger can predict editing operations in parallel with high accuracy, achieving end-to-end speedup compared to models that perform predictions sequentially.

[0031] LaserTagger technology is used to model parallel corpus text generation, which is based on the transformer model (e.g., ...). Figure 2As shown, Transformer is a model based on a multi-head attention mechanism. Attention can solve the long-distance dependency problem of RNNs and their variants. In other words, the attention mechanism can have better memory and can remember information over a longer distance. In addition, most importantly, attention supports parallel computation, and Transformer has a great advantage in parallel corpus modeling.

[0032]

[0033] Table 1

[0034] Then, BERT is used for text classification. BERT uses Masked Language Model (MLM) as a new pre-trained target language model, and the model structure is shown in Table 1 above. MLM randomly selects and masks some words, aiming to predict the masked part based on the context before and after the masked words. Therefore, the fine-tuning method is different from the unidirectional language model from left to right. MLM combines the context, that is, the context on both sides. MLM reflects the relationship at the word granularity more.

[0035] BERT also introduces a method for predicting the next sentence, the task of which is to determine whether two sentences are consecutive. During training, the BERT model randomly selects a pair of adjacent sentences and then predicts their order. If the two sentences are consecutive, the label is "consecutive"; otherwise, the label is "discontinuous." This ensures that pre-training maintains good contextual relationships at the sentence level. Using BERT transfer learning for entity recognition on a corpus can demonstrate its advantages with a small number of samples.

[0036] This invention relates to two technologies: LaserTagger and BERT. The following will detail the application steps of these two technologies in the augmentation of raw training data and model training (e.g., ...). Figure 3 (As shown).

[0037] 1. Augmentation of raw training data:

[0038] First, industrial internet data is used as the raw training data. Then, LaserTagger technology is used to augment the raw training data to generate more training data. The specific steps are as follows:

[0039] • Split each data sample in the original data into an input sentence and a label sequence.

[0040] • LaserTagger-based enhancements can be applied to the input sentence, using methods such as synonym replacement, antonym replacement, random word insertion, and word order reversal to generate new input sentences.

[0041] • Enhance the label sequence accordingly to ensure it corresponds to the enhanced input sentence.

[0042] • Add the enhanced input sentence and label sequence as new training data to the original training data.

[0043] 2. Model Training:

[0044] The specific steps for training the BERT model using the augmented data are as follows:

[0045] • Divide the enhanced dataset into training, validation, and test sets.

[0046] • When training with the BERT model, you can choose a pre-trained base model for transfer learning.

[0047] During training, various techniques can be used to improve model performance, such as learning rate decay, early stopping, dropout, etc.

[0048] • After the model has been trained, it can be evaluated using a test set to assess its performance.

[0049] If the model's performance is poor, you can adjust the model parameters, increase the training data, or try other techniques to further improve the model's performance.

[0050] In summary, using LaserTagger to augment the original training data and then using the augmented data to train the BERT model can improve the performance and robustness of machine learning models.

[0051] This invention presents a method for data augmentation in the industrial internet field through rules, semantic technology, and LaserTagger technology. This process integrates traditional semantic methods and model-based modeling techniques, which can solve the problem of small sample size in knowledge recognition in the industrial internet field and efficiently improve the text classification effect in the industrial internet field.

[0052] The core of this embodiment lies in using LaserTagger for data augmentation to improve the effectiveness and capability of small samples.

[0053] Example 2

[0054] Semantic enhancement of manually labeled data can be achieved using other traditional techniques, such as Word2vec. Transformer models can be replaced with other networks, such as LSTM-based seq2seq methods, to model parallel corpora.

[0055] Word2vec is a widely used natural language processing technique that converts words in natural language into vector representations. This technique is commonly used for semantic analysis of vocabulary and recommendation of related words, and can also be used for data augmentation at the sentence level in Chinese.

[0056] Chinese sentence-level data augmentation refers to using natural language processing techniques to expand and transform Chinese sentences to increase the diversity and quantity of the dataset. In the process, LaserTagger can be used for the data augmentation sub-step, or Word2vec technology can be used. We can convert each word in the original sentence into a vector representation, and then perform various transformations on the vectors to generate new sentences.

[0057] Specifically, the method for generating new sentences using Word2vec technology is as follows: By replacing certain words in a sentence with their synonyms, a new sentence is generated. For example, "Learning is important" can be replaced with "Learning is crucial." This can increase the diversity and quantity of datasets, improving the accuracy and robustness of machine learning models.

Claims

1.A method for improving the accuracy of transfer learning text classification using small sample data, characterized in that: The specific process of the method is as follows: Step S1, using industrial internet data as original training data; Step S2, using a data enhancement method based on LaserTagger to enhance the original training data to generate more training data; Step S3, using the enhanced data to train the BERT model, specifically using BERT for text classification, BERT based on Masked Language Model (MLM) as a new pre-training target language model; The specific process of step 2 is as follows: S201, split each data sample in the original data into an input sentence and a label sequence; S202, perform LaserTagger-based enhancement on the input sentence, using synonym replacement, antonym replacement, random word insertion, and word order reversal to generate a new input sentence; S203, enhance the label sequence accordingly to ensure that it corresponds to the enhanced input sentence; S204, add the enhanced input sentence of step S202 and the enhanced label sequence of step S203 to the original training data as new training data; The specific process of step 3 is as follows: S301, divide the enhanced data set into training set, validation set and test set; S302, use BERT model for training, select pre-trained base model for transfer learning; S303, in the training process, use learning rate decay, early stopping, or dropout method to improve the performance of the model. 2.The method of claim 1, wherein: The step 3 also includes: S304, after the model training is completed, use the test set to evaluate the model, evaluate the performance of the model; S305, if the performance of the model is not good, adjust the model parameters, increase the training data to further improve the performance of the model.

Images