Small sample learning method and system based on data enhancement

Abstract

The invention discloses a small sample learning method and system based on data enhancement. Collecting electric power operation and maintenance text data from a plurality of data sources, preprocessing and labeling the data, and constructing an electric power field corpus; performing fine tuning on a predefined pre-training language model by using the power field corpus, training power field word vectors and entity vectors from the power field corpus, and fusing the power field word vectors and entity vectors into the pre-training language model; performing named entity recognition and multi-task learning of power operation and maintenance knowledge based on a pre-training language model, constructing a multi-task adversarial learning model in combination with an adversarial learning algorithm, training the multi-task adversarial learning model by adopting multiple data enhancement strategies, and performing data prediction through integrated learning and self-training by utilizing the multi-task adversarial learning model; and evaluating a model prediction result, and dynamically adjusting the weight of each data enhancement strategy based on an evaluation result. The small sample learning performance is improved.

Description

Technical Field

[0001] The present invention belongs to the field of electric power operation and maintenance and machine learning, and in particular relates to a small sample learning method and system based on data enhancement. Background Art

[0002] With the trend of big data training models, machine learning and deep learning have achieved success in many fields. However, in many application scenarios in the real world, the sample size is small or the labeled samples are small, and labeling a large number of unlabeled samples will consume a lot of manpower. How to learn with a small number of samples has become a problem that needs attention. Small sample learning aims to learn a model that solves the problem through a small number of samples. At present, the relevant technologies of small sample learning mainly include model fine-tuning, data enhancement, and transfer learning. With the continuous increase in the scale of unstructured text data in power operation and maintenance, and the grammatical and syntactic structure of Chinese is more special than other languages, the difficulty of named entity recognition of Chinese text has increased, and the small sample scenario of power operation and maintenance has further increased the difficulty of entity recognition. Small sample learning technology has made great progress, but there is still a big gap compared with human classification accuracy. For example, the method based on data enhancement is to expand the data or enhance the features of the small sample data set. This method can avoid adjusting the parameters of the model, but it is easy to introduce noise data. Summary of the invention

[0003] In order to address the deficiencies in the prior art, the present invention provides a small sample learning method and system based on data enhancement to improve performance in a few-sample scenario and achieve or approach the performance of a supervised learning model under sufficient samples.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions.

[0005] The present invention first discloses a small sample learning method based on data enhancement, which comprises the following steps:

[0006] Step 1: Collect power operation and maintenance text data from multiple data sources and preprocess them, and annotate the preprocessed data, including named entity recognition, relationship extraction and text classification, to build a corpus in the power field;

[0007] Step 2: fine-tune the predefined pre-trained language model using the power domain corpus, use the model to train power domain word vectors and entity vectors from the power domain corpus, and integrate the power domain word vectors and entity vectors into the pre-trained language model;

[0008] Step 3: Perform multi-task learning of named entity recognition and power operation and maintenance knowledge based on the pre-trained language model, build a multi-task adversarial learning model in combination with the adversarial learning algorithm, adopt multiple data enhancement strategies to train the multi-task adversarial learning model, and use the trained multi-task adversarial learning model to perform data prediction through integrated learning and self-training;

[0009] Step 4: Evaluate the model prediction results and dynamically adjust the weights of each data enhancement strategy based on the evaluation results.

[0010] The present invention further includes the following preferred embodiments:

[0011] The data sources include the power company's internal operation and maintenance database, power operation and maintenance documents and Internet public data; the power company's internal operation and maintenance database includes equipment inspection records, fault diagnosis reports, and operation and maintenance procedures; the power operation and maintenance documents include power industry standards, academic papers, and technical blogs; the Internet public data includes power forums and news websites.

[0012] The labeling of the preprocessed data further includes:

[0013] Named entity recognition is used to mark out entities in the text, including power equipment, fault types, and operating status. Relationship extraction is used to mark out the relationships between entities, including the relationship between equipment and faults, and the relationship between faults and causes. Text classification is used to add category labels to the text, including fault diagnosis reports, inspection records, and safety regulations.

[0014] The method of using the model to train electric power domain word vectors and entity vectors from the electric power domain corpus further includes:

[0015] Collect text data related to the power sector, build a corpus in the power sector, use word vector training methods to train word vectors in the power sector; use word similarity and word analogy to evaluate the quality of word vectors; build knowledge graphs of entities such as power equipment, fault types, and operating status. Use knowledge representation learning methods to train entity vectors in the power sector; use link prediction and triple classification to evaluate the quality of entity vectors.

[0016] The step of integrating the electric power domain word vector and entity vector into the pre-trained language model further includes:

[0017] Select a suitable pre-trained language model as the basic model, use the constructed power field corpus to fine-tune the pre-trained language model, and update the model parameters; concatenate the trained power field word vectors and entity vectors with the word embedding matrix of the pre-trained language model as the input of the model.

[0018] The adopting of multiple data enhancement strategies to train the multi-task adversarial learning model further includes:

[0019] Match each word in the sample with the dictionary in the field of power operation and maintenance to determine the synonym set of the word; select synonyms for replacement based on the frequency, part of speech or contextual semantic factors of the synonyms;

[0020] Use the pre-trained language model to generate a list of candidate replacement words for each word in the sample based on the context; use the power operation and maintenance knowledge base to filter the list of candidate replacement words and remove words that are not related to the power operation and maintenance field; set different replacement probabilities based on the semantic similarity or part of speech consistency between the candidate word and the word to be replaced;

[0021] Select technical languages ​​related to the field of power operation and maintenance for back-translation, and then select the neural machine translation model with the best performance for back-translation. Fine-tune the model according to the power operation and maintenance text, and screen the samples generated by the back-translation to remove samples with grammatical errors and incoherent semantics.

[0022] Using entity recognition and entity linking technology, the entities in the sample are linked to the corresponding entities in the power operation and maintenance knowledge graph; based on the relationship and attribute information in the knowledge graph, a list of candidate replacement entities is generated for each entity; based on the semantic similarity and contextual relevance between the candidate entity and the replaced entity, the entity is selected for replacement; and a threshold for the entity replacement ratio is set;

[0023] Use the predicted probability entropy or maximum probability difference indicator to measure the uncertainty of the model's sample prediction; sort the unlabeled samples according to the uncertainty measurement value; select samples ranked before a predefined number for manual labeling, and use the newly labeled samples to update the model.

[0024] The dynamically adjusting the weights of each data enhancement strategy further includes:

[0025] Increase the weight of synonym replacement based on domain dictionary and entity replacement strategy based on knowledge graph;

[0026] Increase the weight of the back-translation-based sentence enhancement strategy; or

[0027] Added enhanced samples for specific types of entities.

[0028] The present invention also discloses a data-enhanced small sample learning system using the data-enhanced small sample learning method, comprising:

[0029] A data set construction module is used to collect and preprocess power operation and maintenance text data from multiple data sources, and to annotate the preprocessed data, wherein the annotation includes named entity recognition, relationship extraction, and text classification to construct a corpus in the power field;

[0030] A vector integration module is used to fine-tune a predefined pre-trained language model using the power domain corpus, use the model to train power domain word vectors and entity vectors from the power domain corpus, and integrate the power domain word vectors and entity vectors into the pre-trained language model;

[0031] A multi-task adversarial learning module is used to perform multi-task learning of named entity recognition and power operation and maintenance knowledge based on the pre-trained language model, build a multi-task adversarial learning model in combination with an adversarial learning algorithm, adopt a variety of data enhancement strategies to train the multi-task adversarial learning model, and use the trained multi-task adversarial learning model to perform data prediction through integrated learning and self-training;

[0032] The model evaluation module is used to evaluate the model prediction results and dynamically adjust the weights of various data enhancement strategies based on the evaluation results.

[0033] Accordingly, the present application also discloses a terminal, including a processor and a storage medium;

[0034] The storage medium is used to store instructions;

[0035] The processor is used to operate according to the instructions to execute the steps of the aforementioned small sample learning method based on data enhancement.

[0036] Accordingly, the present application also discloses a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the aforementioned small sample learning method based on data enhancement are implemented.

[0037] The beneficial effect of the present invention is that, compared with the prior art, the present invention provides a small sample learning method and system based on data enhancement, which adopts data enhancement technology to effectively expand the scale and diversity of training data and reduce the model's dependence on large-scale labeled data. Through the multi-task learning framework, the professional knowledge in the field of power operation and maintenance is integrated into the training process, and even in the case of a small number of samples, a stronger generalization ability can be obtained. The multi-task learning framework is used to enhance the expression of domain knowledge and improve the model's ability to understand power operation and maintenance data. The adversarial learning mechanism is introduced to promote the generation of high-quality enhanced samples and enhance the generalization ability of the model. Through the adaptive multi-strategy data enhancement method, the enhancement strategy is dynamically adjusted according to the model learning situation, the sample diversity is improved, and the model generalization ability is further enhanced. Experimental results show that the present invention effectively improves the model learning performance in the small sample scenario of power operation and maintenance, and under the condition of limited labeled data, it can achieve performance close to or even exceed the performance of the supervised learning model under sufficient samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a diagram of the information extraction solution in a small sample scenario in the present invention.

[0039] Figure 2 It is a data enhancement flow chart in the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present invention more clear, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0041] The embodiments described in this application are only some embodiments of the present invention, not all embodiments. Based on the spirit of the present invention, other embodiments obtained by ordinary technicians in this field without creative work are all within the protection scope of the present invention.

[0042] In view of the shortcomings of the prior art, the present invention proposes a small sample learning method and system based on data enhancement. Sample enhancement is to generate multiple new samples with the same / similar semantics as the original samples in a certain way without changing the semantics of the original samples, and the labels of the new samples are the same as the original samples. Commonly used sample enhancement methods are: Obviously, the new samples obtained by sample enhancement contain certain noise, and the new samples even have opposite semantics to the original samples. Therefore, the present invention adopts the following learning strategy to process these new samples with noise. First, the model is trained with standard data; after the model converges to a certain degree, certain enhanced data is added to the standard data to continue training; when the model converges to a certain degree again, certain enhanced data is continued to be added to continue training, and so on. Combine multi-task adversarial learning with adaptive multi-strategy data enhancement strategy to effectively utilize knowledge and data in the field of power operation and maintenance and improve model performance. According to the characteristics of power operation and maintenance text data, a data enhancement method is designed, and dynamically adjusted in combination with the model prediction results to improve the generalization ability and accuracy of the model. The adversarial learning mechanism is introduced to further improve the stability of the model, so that it has stronger adaptability to data noise and sample distribution changes.

[0043] See also Figure 1 As shown, the small sample learning method based on data enhancement disclosed in the present invention includes the following steps:

[0044] Step 1: Collect power operation and maintenance text data from multiple data sources and preprocess them, and annotate the preprocessed data. The annotation includes named entity recognition, relationship extraction and text classification to build a corpus in the power field.

[0045] The step 1 is used to construct a power field corpus for the power operation and maintenance field through data collection and preprocessing operations, so as to prepare for subsequent model training.

[0046] First, the power operation and maintenance text data is collected from heterogeneous data sources. The heterogeneous data sources include the internal operation and maintenance database of the power enterprise, the public power operation and maintenance documents, and the public data on the Internet. The internal operation and maintenance database of the power enterprise includes equipment inspection records, fault diagnosis reports, operation and maintenance procedures, etc. The power operation and maintenance documents include power industry standards, academic papers, technical blogs, etc. The public data on the Internet includes power forums, news websites, etc.

[0047] Preprocess the collected raw data, including text cleaning, such as removing irrelevant information, removing HTML tags, special symbols, converting case, spelling correction, etc., to unify the text format; filter out texts related to power operation and maintenance, and remove irrelevant content; identify and delete duplicate data to ensure data uniqueness. Remove noise data, such as punctuation marks, special characters, etc. Remove common but meaningless stop words, such as "de", "le", "shi", etc. Use a professional word segmentation tool in the power field to segment the text, such as the Jieba word segmentation tool, and load the power field dictionary to improve the accuracy of word segmentation. Build a vocabulary based on the training set, convert the text into a digital sequence, and use it as the input of the model. According to the characteristics of power operation and maintenance text data, extract key features, such as equipment names, fault types, operation steps, etc., and use a word vector model to convert the text features into vector representations.

[0048] According to the characteristics of the power operation and maintenance field and the specific task requirements, design a reasonable annotation system, including named entity recognition, relation extraction, and text classification. Through named entity recognition, annotate entities such as power equipment, fault types, and operating states in the text. Through relation extraction, annotate the relationships between entities, such as "equipment - fault" relationship, "fault - cause" relationship, etc. Through text classification, add category labels to the text, such as fault diagnosis reports, inspection records, safety regulations, etc. Select a suitable text annotation tool, such as Label Studio, etc., to improve the annotation efficiency and accuracy. To ensure the annotation quality, multiple annotators annotate the same data, and improve the annotation consistency through cross - validation, or have power field experts review the annotation results to ensure the accuracy and professionalism of the annotation.

[0049] Divide the annotated dataset into a training set, a validation set, and a test set, with a general ratio of 8:1:1.

[0050] Step 2: Fine - tune the predefined pre - trained language model using the power field corpus, and use this model to train power field word vectors and entity vectors from the power field corpus, and integrate the power field word vectors and entity vectors into the pre - trained language model.

[0051] Collect power field - related text data to build a power field corpus. Use word vector training methods such as Word2Vec and GloVe to train power field word vectors; use tasks such as word similarity and word analogy to evaluate the quality of word vectors. Build a knowledge graph of entities such as power equipment, fault types, and operating states. Use knowledge representation learning methods such as TransE and RotatE to train power field entity vectors. Use tasks such as link prediction and triple classification to evaluate the quality of entity vectors.

[0052] Select a suitable pre-trained language model, such as BERT, ChatGLM3, etc., as the basic model. Use the constructed power field corpus to fine-tune the pre-trained language model and update the model parameters to make it more suitable for the language characteristics and knowledge distribution of power field texts.

[0053] The trained electric power domain word vectors and entity vectors are concatenated with the word embedding matrix of the pre-trained language model as the input of the model. For example, in the input layer of the BERT model, the electric power domain word vectors are concatenated with the word embedding vector of BERT to generate a new input vector E(x i ):

[0054] E(x i )=[E BERT (x i );E domain (x i )]

[0055] E BERT (x i ) represents the input word x in the BERT model i The word embedding vector E domain (x i ) represents the electric power domain word vector, and (;) represents the concatenation operation of the vector. The electric power domain word vector and entity vector are injected into the input of the Transformer layer as additional features. For example, in each Transformer layer of the BERT model, the electric power domain word vector is concatenated with the hidden state vector of the corresponding word:

[0056] H' l (x i )=[H l (x i );E entity (x i )]; H' l (x i ) means that in the Transformer of the BERT layer l, word x i The hidden state vector of E entity (x i ) represents the entity vector of the power field; the new hidden state H' l (x i ) introduces entity vectors in the power domain as additional features.

[0057] Step 3: Perform multi-task learning of named entity recognition and power operation and maintenance knowledge based on the pre-trained language model, build a multi-task adversarial learning model in combination with the adversarial learning algorithm, adopt a variety of data enhancement strategies to train the multi-task adversarial learning model, and use the trained multi-task adversarial learning model to perform data prediction through integrated learning and self-training.

[0058] Multi-task learning improves the generalization ability of the model by sharing information between different tasks. In the field of power operation and maintenance, named entity recognition, relationship extraction, text classification and other tasks can be jointly trained to improve model performance by using the correlation information between tasks. In the field of power operation and maintenance, multi-task learning can make full use of the correlation information between different tasks and improve data utilization, especially when labeled data is scarce; it helps the model learn more general feature representations, improves the generalization ability of the model on different tasks, and trains multiple models at the same time to reduce model training costs.

[0059] Adversarial learning can generate more realistic data and improve the stability of the model by training the discriminator and generator networks to compete with each other. Generative adversarial networks consist of two main parts: the generator and the discriminator. The generator receives random noise as input and generates samples that are similar to the distribution of real samples. The discriminator determines whether the input sample is from the real data set or the sample generated by the generator. The generator and the discriminator compete with each other during the training process. The generator tries to generate fake samples to deceive the discriminator, while the discriminator tries to accurately distinguish between real samples and generated samples. Through continuous adversarial training, the generator can generate more and more realistic samples.

[0060] A multi-task adversarial learning model is constructed, which simultaneously performs named entity recognition and power operation and maintenance related knowledge learning, and introduces an adversarial learning mechanism to improve model performance.

[0061] Use pre-trained language models, such as BERT, and combine them with the BiLSTM-CRF network for named entity recognition. Use the power operation and maintenance knowledge base, such as equipment information, fault type, etc., to build a knowledge graph and incorporate it into the model training process as auxiliary information. Introduce a discriminator network to distinguish between real data and enhanced data, so that the generator network can generate more realistic data and improve the generalization ability of the model.

[0062] According to the characteristics of power operation and maintenance text data, an adaptive multi-strategy data enhancement method is designed, including:

[0063] 3.1: Use the power operation and maintenance domain dictionary to perform synonym replacement to ensure that the data enhancement process does not change the semantics of the text. First, match each word in the sample with the power operation and maintenance domain dictionary to determine the synonym set of the word. Select appropriate synonyms for replacement based on factors such as the frequency, part of speech, and contextual semantics of the synonyms. For example, give priority to synonyms with higher frequency or synonyms with the same part of speech as the replaced word. Set a synonym replacement ratio threshold, such as 20%, to avoid excessive replacement that affects the fluency and readability of the sample.

[0064] 3.2: Use pre-trained language models to replace words, and screen according to the characteristics of power operation and maintenance texts to avoid introducing irrelevant words. Specifically, use the pre-trained language model to generate a list of candidate replacement words for each word in the sample based on the context. Use the power operation and maintenance knowledge base to filter the list of candidate replacement words and remove words that are not related to the power operation and maintenance field. Set different replacement probabilities based on factors such as the semantic similarity and part-of-speech consistency between the candidate word and the replaced word. For example, the higher the semantic similarity and the more consistent the part-of-speech, the greater the replacement probability.

[0065] 3.3: Combine the knowledge in the field of power operation and maintenance for targeted back translation to generate samples that are closer to real scenarios. Specifically, select technical languages ​​related to the field of power operation and maintenance for back translation, such as English and German, which can better retain professional terminology and semantic information, and then select the neural machine translation model with the best performance for back translation, such as the Transformer model, and fine-tune the model according to the characteristics of the power operation and maintenance text to improve the quality of back translation. Screen the samples generated by back translation to remove samples with grammatical errors and incoherent semantics.

[0066] 3.4: Use the power operation and maintenance knowledge graph to replace entities, generate new samples, and maintain semantic consistency. Specifically, use entity recognition and entity linking technology to link the entities in the sample to the corresponding entities in the power operation and maintenance knowledge graph. Generate a list of candidate replacement entities for each entity based on the relationships and attribute information in the knowledge graph. For example, for the "transformer" entity, other transformer entities with similar attributes can be determined as candidate replacement entities based on attribute information such as "model" and "voltage level"; select appropriate entities for replacement based on factors such as the semantic similarity and contextual relevance between the candidate entity and the replaced entity; set an entity replacement ratio threshold to avoid excessive replacement that affects the semantic integrity of the sample.

[0067] 3.5: Using the prediction results of the model, select samples with uncertainty higher than the threshold for manual labeling to further optimize model training. Specifically, use indicators such as prediction probability entropy and maximum probability difference to measure the uncertainty of the model's prediction of samples; sort the unlabeled samples according to the uncertainty measurement value; select samples ranked before a predefined number for manual labeling, for example, you can select samples with the highest prediction probability entropy, samples with a relatively uniform distribution of prediction results between different categories, etc.; use the newly labeled samples to update the model to improve the model's prediction ability.

[0068] The predicted probability entropy is used to measure the uncertainty of the model when predicting the category; if the output of the model is a probability distribution p(y|x), where p(y i |x) is the probability that the model predicts that sample x belongs to category i, then the formula for entropy is:

[0069]

[0070] Where H(p(y|x)) represents the predicted probability entropy of sample x; C is the number of categories, p(y i |x) is the model's predicted probability for category i. The larger the entropy, the more uncertain the model's prediction of sample x is.

[0071] The maximum probability difference is used to measure the confidence difference of the model prediction category, which represents the difference between the model's highest and second highest probability predictions. The calculation formula is:

[0072] Δp(x)=p(y max |x)-p(y second |x)

[0073] Where p(y max |x) is the model for category y max The maximum probability of prediction; p(y second |x) is the model's prediction for the second highest category y second The smaller Δp(x) is, the higher the prediction uncertainty of the model for this sample is.

[0074] For unlabeled samples, sort them according to uncertainty metrics (such as entropy or maximum probability difference) and select the samples with the highest entropy or the smallest maximum probability difference:

[0075] or

[0076] The data enhancement flow chart is as follows Figure 2 shown.

[0077] Ensemble learning is based on the idea that "most decisions are better than a single decision", that is, the joint reasoning results of multiple models are more reliable and stable than the reasoning results of a single model. By integrating multiple different models, the variance of the model can be effectively reduced and the generalization ability of the model on unseen data can be improved. Self-training is a semi-supervised learning method that combines a small amount of labeled data with a large amount of unlabeled data. The specific steps are as follows:

[0078] 1. Use a small amount of labeled data to perform preliminary training on the model;

[0079] 2. Use the trained model to predict the unlabeled data and select the prediction results with higher confidence as pseudo labels;

[0080] 3. Add the unlabeled data with pseudo labels to the training set and continue training the model;

[0081] 4. Repeat steps 2 and 3 to iteratively optimize the model.

[0082] Through self-training, the training data set can be gradually expanded when labeled data is scarce, thereby improving the learning ability and generalization performance of the model.

[0083] Combining ensemble learning with self-training, multiple models are trained using a small amount of labeled data (and data augmentation), and these models jointly infer the unlabeled data to generate pseudo labels for the next iteration. In this process, ensemble learning can ensure that the pseudo labels are closer to the real labels, improving the reliability of the self-training stage.

[0084] Step 4: Evaluate the model prediction results and dynamically adjust the weights of each data enhancement strategy based on the evaluation results.

[0085] In order to further improve the performance of small sample learning, the present invention also combines data enhancement strategies such as semi-supervised learning and active learning. Use preprocessed data and enhanced samples to train a multi-task adversarial learning model. A joint training strategy is used to jointly optimize named entity recognition tasks, power operation and maintenance related knowledge learning tasks, and adversarial learning tasks. Using an adaptive multi-strategy data enhancement strategy, the data enhancement method and weights are dynamically adjusted according to the prediction results of the model to improve the generalization ability and stability of the model. The various parts and their processes are as follows:

[0086] At time step t=0, the system enters the training phase.

[0087] The small amount of collected domain annotated data and domain-related unlabeled data are stored in the sample pool.

[0088] Use domain-relevant unlabeled data for unsupervised pre-training.

[0089] Multi-task learning using pre-trained language models.

[0090] In the process of domain feature learning, data with domain annotations are generated.

[0091] Multi-task training using a small number of labeled samples.

[0092] In some cases, manually annotated samples are needed to enhance the dataset.

[0093] Generate data using a generative adversarial network to increase data samples in order to increase sample diversity and improve the generalization ability of the model.

[0094] Through active learning, samples with higher uncertainty are selected for further annotation, which makes more effective use of limited annotation resources and improves learning efficiency.

[0095] The entire process emphasizes collecting and enhancing data in a variety of ways, as well as continuously optimizing the model through iterative cycles to improve the performance and adaptability of the model.

[0096] For example, when the model has poor recognition effect on a specific type of entity, add enhanced samples of this type of entity. First, use indicators such as accuracy, recall, and F1 value to evaluate the performance of the model on the validation set; dynamically adjust the weights of different data enhancement strategies based on the performance of the model on different tasks and different types of data.

[0097] For example, comparative experiments are conducted to verify the effectiveness of multi-task adversarial learning and adaptive multi-strategy data enhancement methods. The multi-task adversarial learning objective function L is used to simultaneously optimize the named entity recognition task, the power operation and maintenance knowledge learning task, and the adversarial learning task:

[0098] L=LNER+λ1*LKG+λ2*LGAN

[0099] Among them, LNER is the loss function of the named entity recognition task, LKG is the loss function of the power operation and maintenance knowledge learning task, LGAN is the loss function of the adversarial learning task, and λ1 and λ2 are weight coefficients.

[0100]

[0101] N is the number of words in the sentence; C is the number of entity categories; y i,c is an indicator that the i-th word belongs to the c-th category, which is 1 if the word belongs to that category and 0 otherwise; is the probability that the model predicts that the i-th word belongs to the c-th category.

[0102]

[0103] P(t|h,r) is the probability that the model predicts t given h and r, normalized using the softmax function:

[0104]

[0105] h is the entity head, t is the entity tail, and r is the relationship; f(h,r,t) is the score of the relationship r between the head entity h and the tail entity t predicted by the model. τ is the set of all knowledge triples. Among them, ε is the entity set, which represents all possible tail entities.

[0106] LGAN=-Ε x～pdata [logD(x)]-Ε z～pz [log(1-D(G(z)))]-Ε z～pz [logD(G(z))]

[0107] x is a real sample, G(z) is a generated sample; p data is the distribution of the real data, p z is the noise distribution; D(x) and D(G(z)) are the predicted probabilities of the discriminator for real samples and generated samples, respectively.

[0108] For named entity recognition (NER) strategy, it directly affects the performance of the model on entity recognition tasks. No additional weighting is performed, and LNER is directly optimized.

[0109] For the power operation and maintenance knowledge learning strategy (KG), the weight λ1 determines the contribution of the power operation and maintenance knowledge learning task in the overall optimization process of the model. During the training process, the power operation and maintenance knowledge is enhanced, and the weight λ1 controls the proportion of its gradient update in the total loss. The update gradient formula is:

[0110] ▽θ LKG ∝λ1·▽LKG

[0111] For the adversarial learning strategy (GAN), its contribution is controlled by the weight λ2. In the process of adversarial sample generation and discrimination, the weight λ2 affects the update of the model parameters by this part of the loss. The updated gradient formula is:

[0112] ▽θ LGAN ∝λ2·▽LGAN

[0113] In each training iteration, the total loss (L) of the model is the weighted sum of the three losses. The specific gradient update formula is as follows:

[0114] ▽θ←▽L=▽LNER+λ1·▽LKG+λ2·▽LGAN

[0115] If λ1 is large, the weight of the power operation and maintenance knowledge learning task in the gradient update will be greater, and the model will tend to optimize this task faster; the larger λ2 is, the more significant the contribution of the adversarial learning task. During the training process, the weights λ1 and λ2 can be dynamically adjusted to make the model pay more attention to the current task. For example: in the early stage, λ1 can be increased to make the model more focused on learning power operation and maintenance knowledge; as the model masters the knowledge, λ2 can be increased to improve the stability of the model against noise and abnormal data. This weight adjustment mechanism allows the model to make full use of multi-task learning and adaptive enhancement strategies during the training process at different stages.

[0116] Adaptive data augmentation strategies dynamically adjust data augmentation methods and weights based on the model’s prediction results, for example:

[0117] D′=A(D,M,p)

[0118] Among them, D is the original data set, M is the model, D' is the enhanced sample set, p is the model's prediction result for the original data, and A is an adaptive data enhancement function, which dynamically selects different data enhancement methods and adjusts their weights according to the model's prediction results.

[0119] For example, when the overall performance of the model in entity recognition tasks is low, the weight of synonym replacement based on domain dictionaries and entity replacement strategies based on knowledge graphs can be appropriately increased to expand the entity vocabulary and improve the model's ability to recognize entities. When the model performs poorly in relationship classification tasks, the weight of the sentence enhancement strategy based on back translation can be appropriately increased to enrich the language expression of the samples and improve the model's ability to understand semantic relationships. When the model has a poor recognition effect on a specific type of entity, the enhanced samples of that type of entity can be increased, such as increasing the number of samples containing that type of entity, increasing the replacement ratio of that type of entity, etc.

[0120] The beneficial effect of the present invention is that, compared with the prior art, the present invention provides a small sample learning method and system based on data enhancement, which adopts data enhancement technology to effectively expand the scale and diversity of training data and reduce the model's dependence on large-scale labeled data. Through the multi-task learning framework, the professional knowledge in the field of power operation and maintenance is integrated into the training process, and even in the case of a small number of samples, a stronger generalization ability can be obtained. The multi-task learning framework is used to enhance the expression of domain knowledge and improve the model's ability to understand power operation and maintenance data. The adversarial learning mechanism is introduced to promote the generation of high-quality enhanced samples and enhance the generalization ability of the model. Through the adaptive multi-strategy data enhancement method, the enhancement strategy is dynamically adjusted according to the model learning situation, the sample diversity is improved, and the model generalization ability is further enhanced. Experimental results show that the present invention effectively improves the model learning performance in the small sample scenario of power operation and maintenance, and under the condition of limited labeled data, it can achieve performance close to or even exceed the performance of the supervised learning model under sufficient samples.

[0121] The present invention may be a system, a method and / or a computer program product. The present invention also discloses a data-enhanced small sample learning system based on the aforementioned data-enhanced small sample learning method, comprising:

[0122] A data set construction module is used to collect and preprocess power operation and maintenance text data from multiple data sources, and to annotate the preprocessed data, wherein the annotation includes named entity recognition, relationship extraction, and text classification to construct a corpus in the power field;

[0123] A vector integration module is used to fine-tune a predefined pre-trained language model using the power domain corpus, use the model to train power domain word vectors and entity vectors from the power domain corpus, and integrate the power domain word vectors and entity vectors into the pre-trained language model;

[0124] A multi-task adversarial learning module is used to perform multi-task learning of named entity recognition and power operation and maintenance knowledge based on the pre-trained language model, build a multi-task adversarial learning model in combination with an adversarial learning algorithm, adopt a variety of data enhancement strategies to train the multi-task adversarial learning model, and use the trained multi-task adversarial learning model to perform data prediction through integrated learning and self-training;

[0125] The model evaluation module is used to evaluate the model prediction results and dynamically adjust the weights of various data enhancement strategies based on the evaluation results.

[0126] Based on the spirit of the present invention, those skilled in the art can easily think that a computer program product can be obtained based on the aforementioned data-enhanced small sample learning method. The computer program product may include a computer-readable storage medium on which computer-readable program instructions for enabling a processor to implement various aspects of the present disclosure are loaded. That is, the present application also includes a terminal, including a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to perform the steps of the aforementioned data-enhanced small sample learning method.

[0127] Computer readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device. Computer readable storage medium can be, for example, - but not limited to - electrical storage device, magnetic storage device, optical storage device, electromagnetic storage device, semiconductor storage device or any suitable combination of the above. More specific examples (non-exhaustive list) of computer readable storage medium include: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, for example, punch card or groove protrusion structure with instructions stored thereon and any suitable combination of the above. Computer readable storage medium used here is not interpreted as instantaneous signal itself, such as radio wave or other free propagating electromagnetic wave, electromagnetic wave propagated by waveguide or other transmission medium (for example, light pulse by optical fiber cable) or electrical signal transmitted by wire.

[0128] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0129] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages-such as Smalltalk, C++, etc., and conventional procedural programming languages-such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network-including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may be personalized by utilizing the state information of a computer-readable program instruction, and the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the present disclosure.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A small sample learning method based on data enhancement, characterized in that: The following steps are involved: Step 1: Collect power operation and maintenance text data from multiple data sources and preprocess them, and annotate the preprocessed data. The annotation includes named entity recognition, relationship extraction and text classification to build a corpus in the power field. Step 2: fine-tune the predefined pre-trained language model using the power domain corpus, use the model to train power domain word vectors and entity vectors from the power domain corpus, and integrate the power domain word vectors and entity vectors into the pre-trained language model; Step 3: Perform multi-task learning of named entity recognition and power operation and maintenance knowledge based on the pre-trained language model, build a multi-task adversarial learning model in combination with the adversarial learning algorithm, adopt multiple data enhancement strategies to train the multi-task adversarial learning model, and use the trained multi-task adversarial learning model to perform data prediction through integrated learning and self-training; Step 4: Evaluate the model prediction results and dynamically adjust the weights of each data enhancement strategy based on the evaluation results.

2. The small sample learning method based on data enhancement according to claim 1 is characterized in that: The data sources include the power enterprise's internal operation and maintenance database, power operation and maintenance documents, and Internet public data; the power enterprise's internal operation and maintenance database includes equipment inspection records, fault diagnosis reports, and operation and maintenance procedures; the power operation and maintenance documents include power industry standards, academic papers, and technical blogs; The Internet public data includes power forums and news websites.

3. The small sample learning method based on data enhancement according to claim 2 is characterized in that: The labeling of the preprocessed data further includes: Named entity recognition is used to mark out entities in the text, including power equipment, fault types, and operating status. Relationship extraction is used to mark out the relationships between entities, including the relationship between equipment and faults, and the relationship between faults and causes. Text classification is used to add category labels to the text, including fault diagnosis reports, inspection records, and safety regulations.

4. The small sample learning method based on data enhancement according to claim 3 is characterized in that: The method of using the model to train electric power domain word vectors and entity vectors from the electric power domain corpus further includes: Collect text data related to the power sector, build a corpus in the power sector, use word vector training methods to train word vectors in the power sector; use word similarity and word analogy to evaluate the quality of word vectors; build knowledge graphs of entities such as power equipment, fault types, and operating status. Use knowledge representation learning methods to train entity vectors in the power sector; use link prediction and triple classification to evaluate the quality of entity vectors.

5. The small sample learning method based on data enhancement according to claim 4 is characterized in that: The step of integrating the electric power domain word vector and entity vector into the pre-trained language model further includes: Select a suitable pre-trained language model as the basic model, use the constructed power field corpus to fine-tune the pre-trained language model, and update the model parameters; concatenate the trained power field word vectors and entity vectors with the word embedding matrix of the pre-trained language model as the input of the model.

6. The small sample learning method based on data enhancement according to claim 5 is characterized in that: The adopting of multiple data enhancement strategies to train the multi-task adversarial learning model further includes: Match each word in the sample with the dictionary in the field of power operation and maintenance to determine the synonym set of the word; select synonyms for replacement based on the frequency, part of speech or contextual semantic factors of the synonyms; Use the pre-trained language model to generate a list of candidate replacement words for each word in the sample based on the context; use the power operation and maintenance knowledge base to filter the list of candidate replacement words and remove words that are not related to the power operation and maintenance field; set different replacement probabilities based on the semantic similarity or part of speech consistency between the candidate word and the word to be replaced; Select technical languages ​​related to the field of power operation and maintenance for back-translation, and then select the neural machine translation model with the best performance for back-translation. Fine-tune the model according to the power operation and maintenance text, and screen the samples generated by the back-translation to remove samples with grammatical errors and incoherent semantics. Using entity recognition and entity linking technology, the entities in the sample are linked to the corresponding entities in the power operation and maintenance knowledge graph; based on the relationship and attribute information in the knowledge graph, a list of candidate replacement entities is generated for each entity; based on the semantic similarity and contextual relevance between the candidate entity and the replaced entity, the entity is selected for replacement; and a threshold for the entity replacement ratio is set; Use the predicted probability entropy or maximum probability difference indicator to measure the uncertainty of the model's sample prediction; sort the unlabeled samples according to the uncertainty measurement value; select samples ranked before a predefined number for manual labeling, and use the newly labeled samples to update the model.

7. The small sample learning method based on data enhancement according to claim 6 is characterized in that: The dynamically adjusting the weights of each data enhancement strategy further includes: Increase the weight of synonym replacement based on domain dictionary and entity replacement strategy based on knowledge graph; Increase the weight of the back-translation-based sentence enhancement strategy; or Added enhanced samples for specific types of entities.

8. A small sample learning system based on data enhancement, characterized in that: include: A data set construction module is used to collect and preprocess power operation and maintenance text data from multiple data sources, and to annotate the preprocessed data, wherein the annotation includes named entity recognition, relationship extraction, and text classification to construct a corpus in the power field; A vector integration module is used to fine-tune a predefined pre-trained language model using the power domain corpus, use the model to train power domain word vectors and entity vectors from the power domain corpus, and integrate the power domain word vectors and entity vectors into the pre-trained language model; A multi-task adversarial learning module is used to perform multi-task learning of named entity recognition and power operation and maintenance knowledge based on the pre-trained language model, build a multi-task adversarial learning model in combination with an adversarial learning algorithm, adopt a variety of data enhancement strategies to train the multi-task adversarial learning model, and use the trained multi-task adversarial learning model to perform data prediction through integrated learning and self-training; The model evaluation module is used to evaluate the model prediction results and dynamically adjust the weights of various data enhancement strategies based on the evaluation results.

9. A terminal comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps of the small sample learning method based on data enhancement according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the small sample learning method based on data enhancement described in any one of claims 1 to 7 are implemented.

Images