Data augmentation method for small sample events based on semi-supervised meta-pseudo-label learning method

By adopting semi-supervised meta-pseudo-label learning method and unsupervised data enhancement method in the field of event extraction, the problem of scarcity of labeled data is solved, the accuracy and robustness of the model are improved, and high-quality training is achieved under a small amount of labeled data is achieved. event extraction model.

CN118194964BActive Publication Date: 2024-11-29杨思慧
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202410392595.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-02
Publication Date
2024-11-29
Estimated Expiration
2044-04-02

AI Technical Summary

Technical Problem

The scarcity of labeled data in the event extraction field has led to poor performance of machine learning models during training and prediction.

Method used

The semi-supervised meta-pseudo-label learning method combined with the unsupervised data augmentation method is used to improve the prediction effect of the model by minimizing the KL divergence between the augmented data of labelless data and labelless data in the teacher network as a loss function.

Benefits of technology

With a small number of labeled samples, the accuracy and robustness of the event extraction model are improved, and a high-precision, high-rootability and high-quality event extraction model is trained under a small number of labeled data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118194964B_ABST
    Figure CN118194964B_ABST
Patent Text Reader

Abstract

The present invention is a data augmentation method for small sample events based on a semi-supervised meta-pseudo-label learning method. The data augmentation method for small sample events based on a semi-supervised meta-pseudo-label learning method is: using a semi-supervised meta-pseudo-label learning method for data augmentation; wherein, when the semi-supervised meta-pseudo-label learning method is used to train the teacher network, minimizing the KL divergence between unlabeled data and augmented data of unlabeled data is added as a loss function. The data augmentation method for small sample events based on a semi-supervised meta-pseudo-label learning method described in the present invention effectively solves the problem of data scarcity in event extraction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of electronic information technology, and in particular relates to a data augmentation method for small sample events based on a semi-supervised meta-pseudo-label learning method. Background Art

[0002] In today's Internet age, a large amount of information resources are generated every moment, most of which are non-standard structured data, such as text and dynamic video information. These big data provide users with rich and diverse information in various application scenarios in our lives. Therefore, how to mine and process these massive amounts of data to meet the information acquisition needs of all walks of life has become a very important research direction. The goal of information extraction technology is to extract standard event information from unstructured text, so that people can retrieve event information and analyze behavior.

[0003] Event extraction technology is an important and challenging task in information extraction technology. Events are a structured representation of information that describes things that one or more participants participate in, and can usually be described as a change in state. Event information is considered to be an important type of information due to its real-time, dynamic and rich nature. Event extraction has become an important technology in information extraction technology. Event extraction includes two main tasks: event detection and event argument extraction. The first task of event detection is to extract the trigger words in a sentence. Trigger words are generally verbs or nouns. There may be multiple trigger words in a sentence; the second task is to classify the event type represented by each trigger word. The event argument extraction task is first to identify the arguments, that is, the elements involved in an event, and then to classify the argument roles of these arguments.

[0004] Machine learning and deep learning technologies have achieved good application results in the field of natural language processing. Supervised learning is a method adopted by most researchers in event extraction tasks. By learning from labeled data, a network model that can extract events from unlabeled data is trained. An important challenge of this type of method is the scarcity of labeled data: due to the complexity of event structure, manual labeling is time-consuming and labor-intensive, and the existing labeled data scale is difficult to meet the needs of machine learning models.

[0005] As the driving force of deep learning, data is crucial for model training. Sufficient training data can not only alleviate the overfitting problem of the model during training, but also further expand the parameter search space and help the model further optimize towards the global optimal solution. However, in many fields or tasks, it is very difficult and costly to obtain sufficient training samples. Therefore, data augmentation has become a common means of increasing training samples.

[0006] In view of this, the present invention proposes a new data augmentation method for small sample events, which aims at the data scarcity problem of event extraction and uses other learning methods to improve the prediction effect of the model from a modeling perspective. Summary of the invention

[0007] The object of the present invention is to provide a data augmentation method for small sample events based on a semi-supervised meta-pseudo-label learning method. From a modeling perspective, the data augmentation method adds the UDA unsupervised data augmentation method on the basis of the meta-pseudo-label method, and has high feasibility.

[0008] In order to achieve the above purpose, the technical solution adopted is:

[0009] The data augmentation method for small sample events based on semi-supervised meta-pseudo-label learning method is: using semi-supervised meta-pseudo-label learning method to perform data augmentation;

[0010] Among them, when the semi-supervised meta-pseudo-label learning method is used to train the teacher network, minimizing the KL divergence between unlabeled data and augmented data of unlabeled data is added as a loss function.

[0011] Furthermore, the formula of the loss function between the new data generated by the data enhancement method and the unlabeled data is as follows:

[0012]

[0013] In the formula, pU(x) represents the unlabeled dataset; and are the same teacher network model parameters; represents a data augmentation transformation.

[0014] Furthermore, in the semi-supervised meta-pseudo-label learning method, when the teacher network updates itself according to the feedback of the student network, the formula of the overall loss function is expressed as follows:

[0015] Where pL(x) represents the labeled dataset and θ is the student network parameter.

[0016] Furthermore, the steps of the semi-supervised pseudo-label learning method are as follows:

[0017] First, the teacher network is obtained under the training of labeled data. On the pseudo-labeled dataset (xu, qΦ(xu)) generated by the teacher network, the student network uses SGD to update the weight parameter θ t :

[0018] Then, using the labeled dataset (xl, yl) as the validation data, calculate yl and the student network θ after the previous update t +1 the resulting label prediction The loss function between them, the teacher network uses SGD to update the weight parameters ;

[0019] Finally, the entire learning optimization is performed.

[0020] Furthermore, the weight parameter θ t The formula is as follows:

[0021]

[0022] Where, L CE Represents the cross-entropy loss function, η θ Represents the learning rate of the student network.

[0023] Furthermore, the weight parameter The formula is as follows:

[0024]

[0025] In the formula, represents the learning rate of the teacher network.

[0026] Furthermore, the formula of the optimization process is as follows:

[0027]

[0028] Another object of the present invention is to provide application of the above-mentioned data augmentation method in small sample event extraction.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The technical solution of the present invention is mainly aimed at the problem of scarcity of labeled data in the field of event extraction. Based on the modeling idea, the present invention innovatively applies the semi-supervised and unsupervised learning methods such as pseudo-label learning method, meta-pseudo-label learning method and unsupervised data enhancement originally used in the field of computer vision to the field of event extraction, and integrates these methods with the event extraction model proposed by the present invention to train an event extraction model that can still predict accurately with a small number of sample inputs, and conducts experiments on these models to verify the feasibility of the method. Therefore, a small sample event extraction model can be constructed from the perspective of data augmentation, so that a high-precision, high-robustness, and high-quality event extraction model can be trained based on a small number of labeled samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1Learn an update process for the original pseudo-labels;

[0032] Figure 2 Learning update process for meta pseudo labels;

[0033] Figure 3 UDA+ meta-pseudo-label training model architecture for the teacher network;

[0034] Figure 4 The trigger word classification accuracy of the three experimental paradigms;

[0035] Figure 5 The trigger word classification accuracy of the three experimental paradigms;

[0036] Figure 6 Trigger word accuracy of meta-pseudo-label semi-supervised learning and supervised learning under different proportions of labeled data. DETAILED DESCRIPTION

[0037] In order to further illustrate the data augmentation method for small sample events based on the semi-supervised meta-pseudo-label learning method of the present invention and achieve the intended purpose of the invention, the data augmentation method for small sample events based on the semi-supervised meta-pseudo-label learning method proposed in the present invention is described in detail below in combination with the preferred embodiments, its specific implementation method, structure, features and efficacy are described as follows. In the following description, different "one embodiment" or "embodiment" do not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0038] Before elaborating on the data augmentation method for small sample events based on the semi-supervised meta-pseudo-label learning method of the present invention, it is necessary to further explain the relevant background mentioned in the present invention to achieve better results.

[0039] 1. Event extraction field

[0040] There are two main types of research methods in the field of event extraction. The first type is to apply pattern matching to event extraction tasks, and the second type is to introduce machine learning methods into event extraction tasks. The latter can be divided into methods based on traditional machine learning methods and methods based on deep learning.

[0041] The core idea of ​​the pattern matching method is to create templates based on a manually annotated corpus through regular expressions, syntax trees, and semantic constraints, and then match them. Its representative research work is the AutoSlog system proposed in 1993. This system can build syntax trees and establish template matching rules based on the semantic information of the corpus, thereby automatically generating expression templates for a specific event type. Experiments have shown that the dictionary created by this system is more effective than the manually created dictionary. The performance of the pattern matching method depends largely on the form and expression of the text, as well as the quality of the manually annotated corpus, so it will only perform well in specific fields, but the migration ability between different event type fields will be relatively poor.

[0042] Then, based on the statistical model, the researchers used the classification idea of ​​machine learning methods to realize event extraction. The key points are the construction of the classifier and the selection of features. The main machine learning models include support vector machines, GBDT, etc. There are two main paradigms of this type of method. One is to regard the event extraction task as a multi-stage classification task, and perform trigger word recognition, event classification and argument classification in sequence; the other is to aggregate several tasks together and obtain prediction results at the same time.

[0043] After 2015, more and more researchers have tried to use deep learning methods to build end-to-end event extraction models, such as CNN and RNN to achieve event extraction tasks. The advantage is that it reduces the dependence on external NLP tools and the tedious work of manual feature design. Similar to machine learning methods, the mainstream research paradigms of deep learning methods are divided into multi-stage classification models and joint classification models. The more classic networks include DMCNN proposed in 2015 and JRNN proposed in 2016.

[0044] 2. Challenge of event extraction data sparsity

[0045] In order to solve the data sparsity problem in the field of event extraction and achieve the goal of training high-quality models with a small number of manually labeled samples, researchers mainly solve this problem from the perspectives of data augmentation and modeling. The data augmentation perspective can be divided into generating data using external semantic knowledge frameworks and semi-supervised learning strategies. In 2015, it was proposed to use the English vocabulary database FrameNet based on frame semantics and Wikipedia data to align large-scale unlabeled data sets and knowledge graph entities through remote supervision methods to determine labels in order to expand the amount of labeled data. In 2018, it was proposed that event types can be clustered, and events can be automatically labeled by scanning the clusters where the events are located, and the original data set can be expanded to retrain the model. This is the idea of ​​semi-supervised learning.

[0046] Some researchers have tried to improve the model's prediction and generalization capabilities for unlabeled events from a modeling perspective. Representative methods include few-shot learning (FSL) and zero-shot transfer learning. In 2020, it was proposed to transform the event extraction problem into a few-shot learning problem, so that the model can predict unseen event types with only a small amount of labeled data during training. In 2017, it was proposed to use the similarity in the structure of sentences and event types and use a zero-shot transfer model to solve the event extraction task.

[0047] After understanding the relevant background mentioned in the present invention, the data augmentation method for small sample events based on the semi-supervised meta-pseudo-label learning method of the present invention will be further introduced in detail below in combination with specific embodiments:

[0048] From the perspective of modeling, the present invention proposes the MPL semi-supervised meta-pseudo-label learning method based on the original pseudo-label semi-supervised learning method and applies it to the event extraction model. Then, the UDA unsupervised data enhancement method is added on the basis of the meta-pseudo-label method. The experimental results show that the accuracy of the semi-supervised meta-pseudo-label learning method can be improved by about 11% compared with the supervised learning trigger word. After integrating the UDA method, the accuracy can be improved by about 6%. The technical solution of the present invention is:

[0049] The data augmentation method for small sample events based on semi-supervised meta-pseudo-label learning method is: using semi-supervised meta-pseudo-label learning method to perform data augmentation;

[0050] Among them, when the semi-supervised meta-pseudo-label learning method is used to train the teacher network, minimizing the KL divergence between unlabeled data and augmented data of unlabeled data is added as a loss function.

[0051] Preferably, the formula of the loss function between the new data generated by the data enhancement method and the unlabeled data is as follows:

[0052]

[0053] In the formula, pU(x) represents the unlabeled dataset; and are the same teacher network model parameters; represents a data augmentation transformation.

[0054] Further preferably, in the semi-supervised pseudo-label learning method, when the teacher network is self-updated according to the feedback of the student network, the overall loss function is expressed as:

[0055] Where pL(x) represents the labeled dataset and θ is the student network parameter.

[0056] Preferably, the steps of the semi-supervised pseudo-label learning method are:

[0057] First, the teacher network is obtained under the training of labeled data. On the pseudo-labeled dataset (xu, qΦ(xu)) generated by the teacher network, the student network uses SGD to update the weight parameter θ t :

[0058] Then, using the labeled dataset (xl, yl) as the validation data, calculate yl and the student network θ after the previous update t +1 the resulting label prediction The loss function between them, the teacher network uses SGD to update the weight parameters ;

[0059] Finally, the entire learning optimization is performed.

[0060] Further preferably, the weight parameter The formula is as follows:

[0061]

[0062] Where, L CE Represents the cross-entropy loss function, η θ Represents the learning rate of the student network.

[0063] Further preferably, the weight parameter The formula is as follows:

[0064]

[0065] In the formula, represents the learning rate of the teacher network.

[0066] Further preferably, the formula of the optimization process is as follows:

[0067]

[0068] This invention mainly solves the problem of scarcity of labeled data in the field of event extraction from the perspective of modeling, and studies how to use semi-supervised and unsupervised learning paradigms to replace common supervised learning methods, so as to achieve a high-performance model trained on the premise of a small number of event samples. This chapter first proposes the MPL meta-pseudo-label semi-supervised learning method based on the original pseudo-label semi-supervised learning method and applies it to our event extraction model. Then, the UDA unsupervised data enhancement method is added on the basis of the meta-pseudo-label method. Finally, experiments are conducted on the data set to verify the feasibility of the model method and the influence of other experimental variables on the results. Specific embodiments are as follows:

[0069] Example 1.

[0070] The specific steps are as follows:

[0071] 1. Method Overview

[0072] (1) Original pseudo-label semi-supervised learning method

[0073] In order to solve the problem of insufficient neural network training data, a pseudo-label semi-supervised learning method was proposed in 2013. The main idea is to improve the model performance in the supervised learning process through unlabeled data. The algorithm flow is as follows Figure 1 As shown:

[0074] First, a network model M is trained with supervised labelled data, and the trained model M is used to predict the unlabelled data, and the one with the highest prediction probability is selected as the pseudo label of this sample, as shown in the following formula:

[0075]

[0076] After generating the pseudo labels for the unlabeled data, the pseudo labels are regarded as the ground truth and input into the model again together with the labeled data for retraining to obtain the new model M'. The overall loss function is expressed as follows:

[0077]

[0078] Where n and n ’ is the batch size, C is the number of categories, y represents the true label, y' is the pseudo label, f is the output of the neural network, α(t) is a function of the training time t, which is used to balance the training weights between labeled data and pseudo-label training data. At the beginning of training, α(t) is 0, that is, the model only trains labeled samples so that it can learn quickly; then α(t) slowly increases linearly, training labeled samples and pseudo-label samples at the same time; when the training iteration time reaches a certain number of training times, the growth of α(t) stops, and α(t) is expressed as follows:

[0079]

[0080] (2) Semi-supervised pseudo-label learning method

[0081] A meta-pseudo label (MPL) technique is used for semi-supervised image classification problems, and has achieved good results on CIFAR10, SVHN and ImageNet datasets. The meta-pseudo label learning method adopts the idea of ​​adaptive generation of meta-learning target distribution. The difference between it and the traditional pseudo-label learning method is that the teacher network in the traditional pseudo-label learning method will not be updated to guide the student network to learn and optimize; however, the teacher network in the meta-pseudo label learning method will receive feedback from the student network on the performance of the labeled dataset and update it, so that it can generate better pseudo-labels to guide the student network training.

[0082] This paper innovatively applies semi-supervised meta-pseudo-label learning to the event extraction model, and can train a powerful event extraction model with a small number of labeled sample inputs. There are two networks in meta-pseudo-label semi-supervised learning, the teacher network is represented by q Φ , the student network is represented by p θ , with labeled data representation (x l ,y l ), the unlabeled data is represented as (x u ,y u ), the algorithm steps are as follows Figure 2 As shown:

[0083] First, a teacher network is obtained by training with labeled data, and then the dataset with pseudo labels (x u ,q Φ (x u )) The student network uses SGD to update the weight parameter θ t :

[0084]

[0085] Where, L CE Represents the cross-entropy loss function, η θ Represents the learning rate of the student network.

[0086] Then, using the labeled dataset (x l ,y l ) as the validation data, calculate y l and the student network θ after the previous update t+1 Generated label predictions The loss function between them, the teacher network uses SGD to update the weight parameters :

[0087]

[0088] In the formula, represents the learning rate of the teacher network.

[0089] Finally, we perform learning optimization. The entire learning optimization process is very similar to meta-learning, and the overall optimization process can be written as follows:

[0090]

[0091] (3) Combination of semi-supervised pseudo-labeling method and unsupervised data enhancement method

[0092] The present invention combines the data enhancement method from the data perspective with the semi-supervised meta-pseudo-labeling method from the modeling perspective. The combination method is to convert the original data enhancement method based on supervised data into a data enhancement method based on unsupervised data. This is an unsupervised data enhancement method UDA (Unsupervised Data Augmentation). When the semi-supervised meta-pseudo-labeling is used to train the teacher network, the present invention adds the KL divergence between the unlabeled data and the augmented data of the unlabeled data as the loss function to achieve the combination of the two methods. The overall idea of ​​the algorithm is as follows Figure 3 As shown:

[0093] Figure 3 The left half is the stage of pre-training the teacher network using labeled data in the semi-supervised meta-pseudo-label learning method; the right half is the process in which the teacher network generates pseudo-labels to guide the student network, and the teacher network updates itself according to the feedback of the student network. However, it can be clearly found that compared with the meta-pseudo-label learning method, the loss function here has an additional UDA loss function, that is, the UDA loss function between the new data generated by the data augmentation method and the unlabeled data, which is expressed as follows:

[0094]

[0095] In the formula, pU(x) represents the unlabeled dataset; and are the same teacher network model parameters; It represents data enhancement transformation, such as word vector-based replacement method. Since data enhancement is time-consuming, all data enhancement samples in the experiment conducted in this embodiment are generated offline in advance.

[0096] By adding equation (8) and the loss function of the semi-supervised meta-pseudo-label learning method together, the overall loss function when the teacher network updates itself according to the feedback of the student network is:

[0097] In the formula, pL(x) represents the labeled data set, and θ is the student network parameter. The first term of the loss function is the cross entropy loss between the teacher network's prediction of the labeled data based on the student network and the true label; the second term of the loss function is the UDA loss in formula 8.

[0098] 2. Experimental results and analysis

[0099] (1) Effectiveness of modeling method

[0100] In order to verify the effectiveness of the semi-supervised Meta Pseudo Labels (MPL) meta pseudo label learning method and the unsupervised data augmentation UDA method proposed in the present invention, this example uses the ACE 2005 dataset to compare the model performance effects under the three training paradigms, as shown in the following table:

[0101] Paradigm 1: Supervised learning is performed on the RoBERTa-based event extraction model, and the labeled training set is 40% of the documents in the ACE 2005 training set.

[0102] Paradigm 2: Use semi-supervised meta-pseudo-label learning method MPL, where the teacher network is the model saved after 30 epochs of training in Paradigm 1; the architecture of the student network and the teacher network is the same. The labeled dataset in semi-supervised learning is 40% of the documents in the ACE 2005 training set; the unlabeled dataset is the remaining 60% of the documents in the ACE 2005 training set.

[0103] Paradigm 3: Based on Paradigm 2, unsupervised data augmentation (UDA) loss is added. The labeled dataset is 40% of the documents in the ACE2005 training set; the unlabeled dataset is the remaining 60% of the documents in the ACE 2005 training set plus an equal amount of unlabeled data generated by the TF-IDF-word vector replacement method.

[0104] ① This embodiment selects the trigger word classification accuracy F as the main evaluation indicator and compares the experimental results of the three paradigms, such as Figure 4 As shown:

[0105] Depend on Figure 4 It can be seen from the experimental results that the semi-supervised meta-pseudo-label learning method can improve the performance of the model by generating pseudo-labels under the premise of the same labeled data. The trigger word classification accuracy can be improved by about 10%, but it is about 9% lower than the 67.74% trigger word classification accuracy of supervised learning using all training set label data.

[0106] Depend on Figure 4The experimental results in the same section also show that the addition of unsupervised data enhancement (UDA) can also improve the performance of the model to a certain extent. This embodiment selects the best supervised data enhancement method from a data perspective to generate augmented unlabeled data based on TF-IDF and word vector replacement. The addition of the UDA loss function triggers an increase in the word classification accuracy by about 5% on the basis of MPL, reaching 63.90%.

[0107] ② In order to verify the coupling relationship of mutual feedback between the teacher network and the student network in the semi-supervised pseudo-label learning method, this embodiment observed the trigger word classification accuracy curves of the teacher network and the student network during the training process, and compared them with the accuracy change curve of supervised learning. The results are as follows: Figure 5 As shown:

[0108] pass Figure 5 It can be found that the accuracy of the student network and the teacher network in MPL is always lower than that of supervised learning. The main reason is that supervised learning uses all the training data, while MPL only uses 40% of the labeled data. In the end, the two can converge to a difference of about 5%, which is already a very good performance.

[0109] It can also be observed that the convergence of the teacher network and the student network is relatively slow. This is because the accuracy of the student network in MPL is evaluated by the gap between it and the pseudo-labels generated by the teacher network. Therefore, the low accuracy of the student network means that it is difficult for the student network to fit the pseudo-labels generated by the teacher network. This gap is fed back to the teacher network, making it constantly update itself to generate better pseudo-labels, making it more difficult for the student network to fit. This adversarial generation mechanism avoids premature convergence of the teacher network and the student network, achieving the goal of continuous learning and improvement with a small amount of labeled data to improve model performance.

[0110] (2) The impact of the amount of labeled data on the meta-pseudo-label learning method

[0111] In order to study the impact of semi-supervised meta-pseudo-label learning on learning effect under different amounts of labeled data, this embodiment selects an event extraction model based on RoBERTa pre-training as the teacher network, uses ACE 2005 as the corpus, changes the ratio of labeled data to unlabeled data, and observes the performance parameters of the network model under different ratios.

[0112] In this embodiment, 10%, 20%, 40% and 80% of the 529 documents in the ACE 2005 training set are selected as labeled data for the semi-supervised pseudo-label learning method, and the remaining documents in the training set are used as unlabeled data for experiments. Other experimental parameters are shown in the following table:

[0113] Table 1 Explanation of main parameters of the experiment

[0114]

[0115] In order to verify the performance of MPL, this example also performs supervised learning under the same ratio of labeled data and compares the performance of the two learning paradigms. Figure 6 This is a curve chart of the trigger word classification accuracy P under different data ratios:

[0116] from Figure 6 The results show that compared with supervised learning, the meta-pseudo-labeling method can bring greater performance benefits when there is less labeled data, thanks to its semi-supervised learning paradigm, which can make full use of limited labeled data for learning. We can also observe that as the amount of labeled data increases, the performance brought by MPL will gradually stabilize, which shows that the paradigm of semi-supervised learning is very effective for the problem of sparse data resources.

[0117] 3. Conclusion

[0118] The embodiment of the present invention solves the problem of scarcity of labeled data in the field of event extraction from the perspective of modeling, uses semi-supervised and unsupervised learning methods, and proposes two learning paradigms: MPL meta-pseudo-label semi-supervised learning paradigm and UDA unsupervised data enhancement. The difference between the semi-supervised meta-pseudo-label learning method and the traditional pseudo-label learning method is that the teacher network in MPL will receive performance feedback from the student network in a labeled data set for update, so that it can generate better pseudo-labels to guide the training of the student network. In the present invention, the idea of ​​UDA unsupervised data enhancement is included. In the training stage of MPL for the teacher network, the KL divergence between unlabeled data and augmented data of unlabeled data is minimized as a loss function, thereby realizing the combination of the two methods.

[0119] This embodiment uses these two learning paradigms and supervised learning for experimental comparison. Compared with the 67.74% trigger word classification accuracy of supervised learning using all ACE 2005 training set labeled data, the trigger word accuracy of the MPL method using only 40% labeled data in the training set reached 58%. After adding UDA loss, the accuracy was further improved by nearly 6% to 63.90%.

[0120] This example also studies the effect of semi-supervised meta-pseudo-label learning on learning results under different amounts of labeled data. The experimental results show that the meta-pseudo-label method can bring greater performance benefits when there is less labeled data. As the amount of labeled data increases, the performance brought by MPL will gradually stabilize, which shows that the paradigm of semi-supervised learning is very effective for the problem of sparse data resources.

[0121] The above is only a preferred embodiment of the embodiment of the present invention, and does not impose any form of limitation on the embodiment of the present invention. Any simple modification, equivalent changes and modifications made to the above embodiment based on the technical essence of the embodiment of the present invention are still within the scope of the technical solution of the embodiment of the present invention.

Claims

1. A data augmentation method for small sample events based on semi-supervised meta-pseudo-label learning method, characterized in that: The data augmentation method is: using a semi-supervised pseudo-label learning method to perform data augmentation; The small sample event is standard event information extracted from unstructured text; The steps of the semi-supervised pseudo-label learning method are: First, a teacher network is obtained by training with labeled data. The dataset (x u ,q Φ (x u )) The student network uses SGD to update the weight parameter θ t : Then, using the labeled dataset (x l ,y l ) as the validation data, calculate y l and the student network θ after the previous update t +1 the resulting label prediction The loss function between them, the teacher network uses SGD to update the weight parameters ; The weight parameter θ t The formula is: Where, L CE Represents the cross-entropy loss function, η θ Represents the learning rate of the student network; the weight parameter The formula is: In the formula, represents the learning rate of the teacher network; Finally, the entire learning optimization is performed, and the formula for the optimization process is as follows: Wherein, when the semi-supervised meta-pseudo-label learning method is used to train the teacher network, minimizing the KL divergence between the unlabeled data and the augmented data of the unlabeled data is added as a loss function; The formula for the loss function between new data generated by data augmentation methods and unlabeled data is: In the formula, pU(x) represents the unlabeled dataset; and are the same teacher network model parameters; represents data augmentation transformation; In the semi-supervised meta-pseudo-label learning method, the overall loss function when the teacher network updates itself according to the feedback of the student network is expressed as: Where pL(x) represents the labeled dataset and θ is the student network parameter.

Citation Information

Patent Citations

  • Small sample text classification method based on semi-supervised teacher-student model

    CN117150021A