Small sample entity identification method and system based on pseudo tag and cross validation

Abstract

The invention provides a small sample entity recognition method based on pseudo labels and cross validation, which comprises the following steps: processing labeled data based on an improved K-fold cross validation algorithm to obtain a training set and a test set to train and verify a baseline model UIE to obtain a fine tuning model; performing random batch selection and processing in the unlabeled data based on a preset rule to obtain selected pseudo data; based on the selected pseudo data and the labeled data, training the baseline model UIE by using a semi-supervised algorithm and an improved K-fold cross validation algorithm to obtain a final model; and processing the input text based on the final model to obtain an entity recognition result. The purpose of training the model by adopting small sample data of unlabeled data is achieved, meanwhile, fine adjustment on one model is achieved by adopting an improved K-fold cross validation algorithm, the generalization ability and index performance of the small sample entity recognition model are improved, and accurate recognition of entities under the small sample condition is completed.

Description

Technical Field

[0001] The present invention relates to the technical field of small sample entity recognition, and in particular to a small sample entity recognition method and system based on pseudo-labeling and cross-validation. Background Art

[0002] Named entity recognition is an important upstream task in the field of natural language processing. Its goal is to identify entities with specific meanings from text, including people, places, organizations, etc. In recent years, named entity recognition technology based on deep learning models has achieved outstanding results, but there are still problems such as relying on large-scale labeled data and being unable to incrementally learn new types, making it difficult for most existing models and methods to meet actual needs. At present, the named entity recognition method is generally a supervised fine-tuning training with labeled data based on open source models, so as to obtain a model that can solve the task, but truly high-quality labeled data is often very rare, and unlabeled data is easier to obtain. This type of method is powerless for small sample labeled data and fully utilizing labeled data and high-quality unlabeled data, so it cannot effectively obtain high-quality results in small sample entity recognition. Therefore, the present invention proposes a small sample entity recognition method and system based on pseudo-labeling and cross-validation. Summary of the invention

[0003] The present invention provides a small sample entity recognition method and system based on pseudo-labeling and cross-validation to solve the above problems.

[0004] The present invention provides a small sample entity recognition method based on pseudo labels and cross-validation, comprising:

[0005] The labeled data is processed based on the improved K-fold cross-validation algorithm to obtain the training set and the test set to train and verify the baseline model UIE to obtain the fine-tuning model;

[0006] After randomly selecting and processing unlabeled data in batches in the fine-tuning model based on preset rules, selected pseudo data are obtained;

[0007] Based on the selected pseudo data and labeled data, the baseline model UIE is trained using a semi-supervised algorithm and an improved K-fold cross-validation algorithm to obtain the final model;

[0008] The input text is processed based on the final model to obtain entity recognition results.

[0009] Preferably, in a small sample entity recognition method based on pseudo labels and cross-validation, after randomly selecting and processing unlabeled data in a fine-tuning model based on preset rules, the selected pseudo data is obtained, including:

[0010] According to the preset step size and sliding window size, batches of unlabeled data are selected to obtain batches of unlabeled data;

[0011] Predict batches of unlabeled data based on the fine-tuned model to obtain primary pseudo data;

[0012] Mix primary pseudo data and labeled data to obtain training data;

[0013] The baseline model UIE is trained based on the training data to obtain a test model;

[0014] The test data is tested by the test model to obtain the test result;

[0015] Get the test score corresponding to each test result, and regard the data with the highest test score as pseudo data.

[0016] Preferably, in a small sample entity recognition method based on pseudo labels and cross-validation, an improved K-fold cross-validation algorithm specifically includes:

[0017] Based on the improved K-fold cross-validation algorithm, K cycles are set and the labeled data are divided into K parts. Any one of the subsets is used as the test set, and the remaining K-1 subsets are used as the training set.

[0018] Use K-1 of the K parts as training sets and 1 part as validation set for training to finally get a model.

[0019] Preferably, in a small sample entity recognition method based on pseudo labels and cross-validation, based on the selected pseudo data and the labeled data, a semi-supervised algorithm and an improved K-fold cross-validation algorithm are used to train the baseline model UIE to obtain the final model, including:

[0020] Semi-supervised data is obtained based on the selected pseudo data and real data, and the semi-supervised data is divided into K parts based on the improved K-fold cross-validation algorithm to obtain a model training set and a model validation set;

[0021] Based on the model training set and model validation set, a semi-supervised algorithm is used to train the baseline model UIE to obtain the final model.

[0022] Preferably, in a small sample entity recognition method based on pseudo labels and cross-validation, a semi-supervised algorithm is used to train the baseline model UIE to obtain a final model, including:

[0023] A dynamic parameter α is introduced into the loss function of the semi-supervised algorithm to adjust the semi-supervised loss of the selected pseudo data and real data in the data set. The specific formula of the semi-supervised loss function is:

[0024] Losslabel =α*BCELoss(label(Fake))+(1-α)*BCELoss(label(True))

[0025] Among them, Loss label represents the semi-supervised loss, α is a dynamic parameter with a value range of (0, 1); BCEloss(label(Fake)) represents the binary cross entropy loss of fake data, BCE(logits, label(True)) represents the binary cross entropy loss of labeled data; label(Fake) represents the label of fake data; label(True) represents the label of labeled data;

[0026] Add the dynamic parameter R-Drop algorithm to the output of the baseline model UIE to perform two Dropouts on the model output and obtain the KL discreteness between the two Dropout processes:

[0027]

[0028] Among them, Loss R-Drop Represents the KL discreteness between two Dropout treatments; P1 w (y|x) represents the sub-model corresponding to the first Dropout process; P2 w (y|x) represents the sub-model corresponding to the second Dropout process; Represents the difference between the probability distribution of the first Dropout process and the probability distribution of the second Dropout process; Represents the difference between the probability distribution of the second Dropout process and the probability distribution of the first Dropout process;

[0029] Generate the total loss function Loss for training the semi-supervised algorithm based on the dynamic parameter β, semi-supervised loss and R-Drop loss 总 :

[0030] Loss 总 =Loss label +β(Loss R-Drop,start +Loss R-Drop,end )

[0031] Among them, β is a dynamic parameter with a value range of (0, 1]; Loss R-Drop,start Indicates the KL divergence at the beginning of the entity; Loss R-Drop,end Indicates the KL divergence of the entity end; Loss R-Drop,start +Loss R-Drop,end represents R-Drop loss;

[0032] Based on the total loss function Loss 总 The model parameters are adjusted, and the baseline model UIE is semi-supervised trained by combining the model training set and the model validation set to obtain the final model.

[0033] Preferably, in a small sample entity recognition method based on pseudo labels and cross-validation, the input text is processed based on the final model to obtain an entity recognition result, including:

[0034] Encode the input text based on the fine-tuned model and extract the text features of the input text to obtain the word vector;

[0035] Based on the final model, the linear layer is passed through the activation function layer Sigmoid to obtain the probability of each word vector, and then the beginning and end of the entity segment are obtained through the set threshold comparison;

[0036] Entity fragments are extracted based on the start position, end position, and category label and output as entity recognition results.

[0037] Preferably, in a small sample entity recognition method based on pseudo labels and cross-validation, it also includes:

[0038] Add a domain recognition model to the input layer of the baseline model UIE, perform domain recognition on the input text based on the domain recognition model, and determine the description domain of the used text;

[0039] And after the input text is field-labeled based on the described field, entity recognition processing is performed.

[0040] Preferably, in a small sample entity recognition method based on pseudo-labeling and cross-validation, a domain recognition model is added to the input layer of the baseline model UIE, and domain recognition is performed on the input text based on the domain recognition model to determine the description domain of the used text, including:

[0041] Based on the domain recognition model, the input text is recognized to determine the location of the text punctuation marks. Based on the location of the text punctuation marks, the input text is divided into multiple text segments, and each text segment is marked with keywords.

[0042] Generate corresponding input text vectors based on each text segment, compare each text vector with the database text vector, and obtain the best matching text vector for each text vector based on the comparison result;

[0043] Determine the description field corresponding to each input text according to the description field corresponding to the best matching text vector, and obtain the keyword similarity based on the subvector corresponding to the keyword tag result of the text segment and the keyword subvector of its corresponding best matching text vector;

[0044] When the keyword similarity is greater than or equal to a preset value, the current text segment is determined to be a key field, and a description field corresponding to the key field is obtained;

[0045] If the description fields corresponding to all the key fields are consistent, determining that the description field is the description field of the current input text;

[0046] Otherwise, the ratio of the total data volume corresponding to the key field to the total data volume of the input text is obtained. If the ratio is greater than or equal to a preset threshold, the description field corresponding to the adjacent text segment of the input text corresponding to each key field is obtained. If the description field corresponding to the adjacent text segment is consistent with the input text segment, or the description field corresponding to the adjacent text segment is a common field, the key field is used as the target text segment;

[0047] Respectively obtain the description fields corresponding to each target text segment and the first quantity proportions corresponding to them, and use the description field corresponding to the largest first quantity proportion as the description field of the current input text;

[0048] If the ratio is less than a preset threshold, the description fields corresponding to each valid text segment and their corresponding second quantity proportions are obtained respectively, and the description field corresponding to the largest second quantity proportion is used as the description field of the current input text.

[0049] Preferably, in a small sample entity recognition method based on pseudo labels and cross-validation, it also includes:

[0050] Based on the domain recognition model, determine the keyword tagging results and key field determination results corresponding to the entity fragment, and record them;

[0051] Before outputting the entity recognition result, the type tag corresponding to the entity output result is detected based on the optional tag category table corresponding to the description field to determine whether the tag is used correctly. If the tag is used correctly, it is determined whether the key field and its corresponding keyword have corresponding type tags. If so, the entity output result is output;

[0052] Otherwise, the entity fragment is sent to the final model for secondary entity recognition.

[0053] The present invention provides a small sample entity recognition system based on pseudo labels and cross-validation for executing any one of the small sample entity recognition methods based on pseudo labels and cross-validation, characterized in that it comprises:

[0054] The fine-tuning model training module is used to process the labeled data based on the improved K-fold cross-validation algorithm to obtain the training set and the test set to train and verify the baseline model UIE to obtain the fine-tuning model;

[0055] A data selection module is used to randomly select and process unlabeled data in batches in a fine-tuning model based on preset rules to obtain selected pseudo data;

[0056] The final model training module is used to train the baseline model UIE based on the selected pseudo data and labeled data using a semi-supervised algorithm and an improved K-fold cross-validation algorithm to obtain the final model;

[0057] The recognition result acquisition module is used to process the input text based on the final model to obtain the entity recognition result.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] The present invention adopts sliding window search in the selection of unlabeled data to select the local optimal unlabeled data for the next step of training. In the model training stage, for entity recognition of small samples, and in order to make full use of labeled data, an improved K-fold cross-validation method is designed to make full use of labeled data. Compared with the traditional training method, the training data is increased. Compared with the traditional K-fold cross-validation, K-fold is used cyclically to fine-tune a model, overcoming the problem that the traditional K-fold cross-validation generates K models and trains on different K-1 training sets. For named entity recognition, different threshold scores of entities and entity fragments may be identified due to different training sets, and then the weighted average thereof may be used for the actual scene. There may be errors and the calculation of P in the calculation of macro-F1 is too small (the prediction weighting of each model makes the predicted number of entities too large), thereby reducing the overall index. Compared with traditional semi-supervised learning, the R-Drop method based on dynamic parameters is added to enhance its semantic understanding ability, and the loss of joint semi-supervision constitutes the overall loss of semi-supervised learning. Dynamic parameters are used to control the loss of R-Drop. Since semi-supervised learning is the main purpose, the dynamic parameters of R-Drop are controlled in the interval (0,1]. Compared with the traditional R-Drop, the fixed parameters are changed to dynamic parameters, which enables it to learn by itself during the model training process, enhances the diversity of parameters, and indirectly improves the overall ability of the model.

[0060] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures specifically pointed out in this application document.

[0061] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0063] Figure 1 A flowchart of a small sample entity recognition method based on pseudo labels and cross-validation according to the present invention;

[0064] Figure 2 This is a flowchart of step 1 of a small sample entity recognition method based on pseudo labels and cross-validation in the present invention;

[0065] Figure 3 This is a flowchart of step 2 of a small sample entity recognition method based on pseudo labels and cross-validation in the present invention;

[0066] Figure 4 Schematic diagram of the dynamic parameter R-Drop algorithm of the present invention;

[0067] Figure 5 This is a flowchart of step 3 of a small sample entity recognition method based on pseudo labels and cross-validation in the present invention;

[0068] Figure 6 This is a structural diagram of a small sample entity recognition system based on pseudo-labeling and cross-validation in the present invention. DETAILED DESCRIPTION

[0069] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0070] Embodiment 1:

[0071] The present invention provides a small sample entity recognition method based on pseudo labeling and cross-validation. Figure 1 As shown, including:

[0072] Step 1: Process the labeled data based on the improved K-fold cross-validation algorithm to obtain the training set and test set to train and verify the baseline model UIE to obtain the fine-tuning model;

[0073] Step 2: After randomly selecting and processing the unlabeled data in batches in the fine-tuning model based on the preset rules, the selected pseudo data is obtained;

[0074] Step 3: Based on the selected pseudo data and labeled data, the baseline model UIE is trained using a semi-supervised algorithm and an improved K-fold cross-validation algorithm to obtain the final model;

[0075] Step 4: Process the input text based on the final model to obtain entity recognition results.

[0076] In this embodiment, the preset rules include a step size and a sliding window size, wherein the step size is 10% of the total amount of unlabeled data, and the sliding window size is 20% of the total amount of unlabeled data. For example, the amount of unlabeled data may be 5000, and then a sliding window is used to select data on the randomly selected batch data in the form of a sliding block with a step size of 500 and a total length of 1000.

[0077] Both the labeled data and the unlabeled data used in the present invention include multiple description fields.

[0078] In this embodiment, the baseline model UIE refers to a grand unified model for information extraction.

[0079] The beneficial effects of the above technical solution are as follows: the present invention realizes processing of labeled data based on an improved K-fold cross-validation algorithm, obtains training sets and test sets to train and verify the baseline model UIE to obtain a fine-tuning model, makes full use of labeled data, and provides a basis for the prediction of unlabeled data; after random batch selection and processing of unlabeled data in the fine-tuning model based on preset rules, selected pseudo data is obtained, and the selection of local optimal unlabeled data is realized, which effectively improves the quality of semi-supervised data and provides a good basis for the training of the final model; based on the selected pseudo data and labeled data, the baseline model UIE is trained using a semi-supervised algorithm and an improved K-fold cross-validation algorithm to obtain the final model, which realizes the purpose of using unlabeled data and small sample data to train the model, and at the same time uses the improved K-fold cross-validation algorithm to realize fine-tuning on a model, thereby improving the generalization ability and index performance of the small sample entity recognition model; based on the final model, the input text is processed to obtain the entity recognition result, and the accurate recognition of the small sample entity is completed.

[0080] Embodiment 2:

[0081] Based on Example 1, step 2: after randomly selecting and processing the unlabeled data in batches in the fine-tuning model based on the preset rules, the selected pseudo data is obtained, such as Figure 2 As shown, including:

[0082] Step 201: selecting unlabeled data in batches according to a preset step size and a sliding window size to obtain batches of unlabeled data;

[0083] Step 202: Predicting the batch of unlabeled data based on the fine-tuned model to obtain primary pseudo data;

[0084] Step 203: Mixing the primary pseudo data and the labeled data to obtain training data;

[0085] Step 204: training the baseline model UIE based on the training data to obtain a test model;

[0086] Step 205: Testing the test data using the test model until a test result is obtained;

[0087] Step 206: Repeat the above steps until all test data are tested, and select the data with the highest score in the test set as the selected pseudo data.

[0088] In this embodiment, the primary pseudo data refers to pseudo label data of the selected pseudo annotated data predicted based on the fine-tuning model.

[0089] In this embodiment, the score of the test set data is the macro-F1 index score, and the calculation formula of macro-F1 is as follows:

[0090]

[0091] Among them, macro-F1 represents the macro-F1 index score of the test data; N represents the total number of annotated label types, F1 i It refers to the F1 value of the i-th annotation label type;

[0092] Among them, the F1 value of each annotation label type is calculated as follows;

[0093]

[0094] Among them, P i = the number of correctly predicted entities corresponding to the i-th label type in the test set / the number of entities predicted by the i-th label type in the test set * 100%; R i = The number of entities predicted correctly by the i-th annotation label type in the test set / the number of entities in the test set * 100%.

[0095] The beneficial effects of the above technical solution are as follows: the present invention first selects unlabeled data in batches according to a preset step size and a sliding window size to obtain batch unlabeled data, performs optimization on the unlabeled data, and predicts the batch unlabeled data based on a fine-tuning model to obtain primary pseudo data, thereby realizing the prediction of pseudo-labeled data; and mixes the primary pseudo data with the labeled data to obtain training data, thereby realizing full utilization of the labeled data; and then trains the baseline model UIE based on the training data to obtain a test model; tests the test data through the test model until a test result is obtained; and repeats the above steps until all the test data are tested, and selects the data with the highest score in the test set as the selected pseudo data to realize the automatic selection of unlabeled data.

[0096] Embodiment 3:

[0097] Based on Example 1, the improved K-fold cross validation algorithm specifically includes:

[0098] Set K cycles and divide the labeled data into K parts, and use any one of the subsets as the test set and the remaining K-1 subsets as the training set;

[0099] In each cycle, the training set in each cycle is used for training, and the validation set is used for validation, where the subset corresponding to the test set selected each time is different.

[0100] The beneficial effects of the above technical solution: The present invention proposes an improved K-fold cross-validation method. Compared with the traditional K-fold cross-validation, the K-fold is used cyclically to perform fine-tuning on a model, which can overcome the problem that the traditional K-fold cross-validation generates K models and trains them on different K-1 training sets. For named entity recognition, different threshold scores of entities and entity fragments may be identified due to different training sets, and then the weighted average is used, which may be wrong for actual scenarios, and the calculation of P in the macro-F1 calculation is too small (the prediction weighting of each model makes the predicted number of entities too large), thereby reducing the overall index. The index performance of the model is effectively improved.

[0101] Embodiment 4:

[0102] On the basis of Example 1, step 3: Based on the selected pseudo data and the labeled data, the baseline model UIE is trained using a semi-supervised algorithm and an improved K-fold cross-validation algorithm to obtain the final model, such as Figure 3 As shown, including:

[0103] Step 301: obtaining semi-supervised data based on the selected pseudo data and real data, and dividing the semi-supervised data into K parts based on an improved K-fold cross-validation algorithm to obtain a model training set and a model validation set;

[0104] Step 302: Based on the model training set and the model validation set, a semi-supervised algorithm is used to train the baseline model UIE to obtain the final model.

[0105] The beneficial effects of the above technical solution: the present invention mixes the selected pseudo data and real data to obtain semi-supervised data, and divides the semi-supervised data into K parts based on the improved K-fold cross-validation algorithm to obtain a model training set and a model verification set, and uses the semi-supervised algorithm to train the baseline model UIE to obtain the final model, which effectively improves the generalization ability of the small sample entity recognition model and improves the recognition degree of small sample entity recognition.

[0106] Embodiment 5:

[0107] Based on Example 4, a semi-supervised algorithm is used to train the baseline model UIE to obtain a final model, including:

[0108] A dynamic parameter α is introduced into the loss function of the semi-supervised algorithm to adjust the semi-supervised loss of the selected pseudo data and real data in the data set. The specific formula of the semi-supervised loss function is:

[0109] Loss label =α*BCELoss(label(Fake))+(1-α)*BCELoss(label(True))

[0110] Among them, Loss label represents the semi-supervised loss, α is a dynamic parameter with a value range of (0, 1); BCEloss(label(Fake)) represents the binary cross entropy loss of fake data, BCE(logits, label(True)) represents the binary cross entropy loss of labeled data; label(Fake) represents the label of fake data; label(True) represents the label of labeled data;

[0111] like Figure 4 As shown in the figure, the dynamic parameter R-Drop algorithm is added to the output of the baseline model UIE to perform two Dropouts on the model output, and the KL discreteness between the two Dropout processes is obtained:

[0112]

[0113] Among them, Loss R-Drop Represents the KL discreteness between two Dropout treatments; P1 w (y|x) represents the sub-model corresponding to the first Dropout process; P2 w (y|x) represents the sub-model corresponding to the second Dropout process; Represents the difference between the probability distribution of the first Dropout process and the probability distribution of the second Dropout process; Represents the difference between the probability distribution of the second Dropout process and the probability distribution of the first Dropout process;

[0114] Generate the total loss function Loss for training the semi-supervised algorithm based on the dynamic parameter β, semi-supervised loss and R-Drop loss 总 :

[0115] Loss 总 =Loss label +β(Loss R-Drop,start +Loss R-Drop,end )

[0116] Among them, β is a dynamic parameter with a value range of (0, 1]; Loss R-Drop,start Indicates the KL divergence at the beginning of the entity; Loss R-Drop,end Indicates the KL divergence of the entity end; Loss R-Drop,start +Loss R-Drop,end represents R-Drop loss;

[0117] Based on the total loss function Loss 总 The model parameters are adjusted, and the baseline model UIE is semi-supervised trained by combining the model training set and the model validation set to obtain the final model.

[0118] In this embodiment, the dynamic parameters α and β are both adjustable parameters and can be changed automatically during the model training process.

[0119] In this embodiment, the output probabilities of the two Dropouts remain consistent.

[0120] In this embodiment, the semi-supervised data (mixed data of pseudo data and labeled data) used for model training is small sample data, for example, 10 entity labels, each of which is labeled with 50 orders of magnitude, and the total sample labeling amount is less than 100.

[0121] The beneficial effects of the above technical solution are as follows: the present invention introduces a dynamic parameter α into the loss function of the semi-supervised algorithm to adjust the semi-supervised loss of the selected pseudo data and the real data in the data set, which can effectively solve the problem that the selected pseudo data in the semi-supervised data has low confidence and is prone to errors, and effectively improve the accuracy of the training data. The dynamic parameter R-Drop added at the output end of the baseline model UIE performs two Dropouts on the model output, which can effectively improve the generalization ability and semantic understanding ability of the model, reduce the learning of erroneous data and prevent the model from overfitting. The total loss function Loss of the training semi-supervised algorithm is generated based on the dynamic parameter β, the semi-supervised loss and the R-Drop loss. 总 And based on the total loss function Loss 总 The model parameters are adjusted, and the baseline model UIE is semi-supervised trained in combination with the model training set and the model validation set to obtain the final model. The dynamic parameter R-Drop is used to facilitate model learning during training, enhance the diversity of parameters, and indirectly improve the overall capability of the model.

[0122] Embodiment 6:

[0123] Based on Example 1, step 4 processes the input text based on the final model to obtain entity recognition results, such as Figure 5 As shown, including:

[0124] Step 401: Encode the input text based on the fine-tuning model, and extract text features of the input text to obtain a word vector;

[0125] Step 402: The probability of each word vector is obtained through the activation function layer Sigmoid, and then compared with the set threshold to obtain the beginning and end of the entity segment;

[0126] Step 403: Perform model decoding based on the beginning and the end and classify the input text category to obtain a type label;

[0127] Step 404: extract entity segments according to the start position, the end position and the category label, and output them as entity recognition results.

[0128] In this embodiment, the input text may be a paragraph, a sentence or a document described in a natural language.

[0129] In this embodiment, Linear generally refers to a linear layer, also known as a fully connected layer (Fully Connected Layer) or a dense layer (Dense Layer), which is used to generate a vector of a fixed size for each character through a linear layer. This vector can represent the characteristics of the input sequence.

[0130] The Sigmoid function is a commonly used activation function that maps the linear value to the (0,1) interval and is used to predict the starting position of each token.

[0131] The word vectors with a threshold greater than 0.5 are selected as the beginning and end of the entity segment.

[0132] The beneficial effects of the above technical solution are as follows: the present invention first encodes the input text based on the fine-tuning model, and extracts the text features of the input text to obtain a word vector; based on the final model, the linear layer is passed, and then the activation function layer Sigmoid is passed to obtain the probability of each word vector, and the beginning and end of the entity fragment are obtained; then the model is decoded based on the beginning and the end, and the input text category is classified to obtain a type label; finally, the entity fragment and its category are extracted according to the beginning position, the ending position and the category label, and output as an entity recognition result, which can effectively improve the accuracy of entity recognition and improve the recognition precision.

[0133] Embodiment 7:

[0134] Based on Example 1, a small sample entity recognition method based on pseudo labels and cross-validation also includes:

[0135] Add a domain recognition model to the input layer of the baseline model UIE, perform domain recognition on the input text based on the domain recognition model, and determine the description domain of the used text;

[0136] And after the input text is field-labeled based on the described field, entity recognition processing is performed.

[0137] The beneficial effects of the above technical solution are as follows: the present invention adds a domain recognition model to the input layer of the baseline model UIE, performs domain recognition on the input text based on the domain recognition model, and determines the description domain of the text; and after domain annotation of the input text based on the description domain, entity recognition processing is performed, and the domain of the text to be recognized is identified and determined before entity recognition, which is conducive to broadening the scope of entity content that can be recognized by the model, and improving the accuracy of the model entity recognition, thereby realizing the improvement of the generalization ability of the final model.

[0138] Embodiment 8:

[0139] On the basis of Example 7, a domain recognition model is added to the input layer of the baseline model UIE, and domain recognition is performed on the input text based on the domain recognition model to determine the description domain of the used text, including:

[0140] Based on the domain recognition model, the input text is recognized to determine the location of the text punctuation marks. Based on the location of the text punctuation marks, the input text is divided into multiple text segments, and each text segment is marked with keywords.

[0141] Generate corresponding input text vectors based on each text segment, compare each text vector with the database text vector, and obtain the best matching text vector for each text vector based on the comparison result;

[0142] Determine the description field corresponding to each input text according to the description field corresponding to the best matching text vector, and obtain the keyword similarity based on the subvector corresponding to the keyword tag result of the text segment and the keyword subvector of its corresponding best matching text vector;

[0143] When the keyword similarity is greater than or equal to a preset value, the current text segment is determined to be a key field, and a description field corresponding to the key field is obtained;

[0144] If the description fields corresponding to all the key fields are consistent, determining that the description field is the description field of the current input text;

[0145] Otherwise, the ratio of the total data volume corresponding to the key field to the total data volume of the input text is obtained. If the ratio is greater than or equal to a preset threshold, the description field corresponding to the adjacent text segment of the input text corresponding to each key field is obtained. If the description field corresponding to the adjacent text segment is consistent with the input text segment, or the description field corresponding to the adjacent text segment is a common field, the key field is used as the target text segment;

[0146] Respectively obtain the description fields corresponding to each target text segment and the first quantity proportions corresponding to them, and use the description field corresponding to the largest first quantity proportion as the description field of the current input text;

[0147] If the ratio is less than a preset threshold, the description fields corresponding to each valid text segment and their corresponding second quantity proportions are obtained respectively, and the description field corresponding to the largest second quantity proportion is used as the description field of the current input text.

[0148] In this embodiment, the database processes templates of multiple types of entities describing multiple fields and their corresponding text vectors.

[0149] In this embodiment, the description field refers to the field corresponding to the object described by the input text content.

[0150] In this embodiment, the general domain refers to a word or sentence that cannot define the description domain without being used with other words in the description, for example, today, tomorrow, the weather is very good today, etc.

[0151] In this embodiment, the first quantity ratio refers to the ratio of the data volume of a certain target text segment to the total data volume of all target text segments.

[0152] In this embodiment, the second quantity ratio refers to the ratio of the data volume of a certain valid text segment to the total data volume of all valid text segments.

[0153] In this embodiment, the valid text segment refers to a text segment whose description domain is identified as a non-general domain.

[0154] The beneficial effects of the above technical solution are as follows: the present invention confirms the description field of the amount u text before performing entity recognition, which provides a basis for the model to realize multi-field entity recognition, improves the generalization ability of the final model, and expands the scope of application of the final model.

[0155] Embodiment 9:

[0156] Based on Example 1, a small sample entity recognition method based on pseudo labels and cross-validation also includes:

[0157] Based on the domain recognition model, determine the keyword tagging results and key field determination results corresponding to the entity fragment, and record them;

[0158] Before outputting the entity recognition result, the type tag corresponding to the entity output result is detected based on the optional tag category table corresponding to the description field to determine whether the tag is used correctly. If the tag is used correctly, it is determined whether the key field and its corresponding keyword have corresponding type tags. If so, the entity output result is output;

[0159] Otherwise, the entity fragment is sent to the final model for secondary entity recognition.

[0160] In this embodiment, the available tag type table is obtained after big data processing to determine the tag type based on the named entity content included in each description field text description, and is pre-stored in the database.

[0161] The beneficial effects of the above technical solution are as follows: the present invention determines the keyword tagging results and key field determination results corresponding to the entity fragment based on the domain recognition model, and records them; and before the entity recognition result is output, the type label corresponding to the entity output result is detected based on the optional label category table corresponding to the description field to determine whether the label is used correctly, thereby realizing the verification of the description field in the entity model recognition process; when the label is used correctly, it is determined whether the key field and its corresponding keyword have corresponding type labels, and if so, the entity output result is output; otherwise, the entity fragment is sent to the final model for secondary entity recognition, thereby realizing automatic verification of the entity recognition result, ensuring that the important content of the input text is successfully identified and marked, which is conducive to improving the accuracy of the output result and providing a reliable reference for users.

[0162] Embodiment 10:

[0163] The present invention provides a small sample entity recognition system based on pseudo labels and cross-validation for executing a small sample entity recognition method based on pseudo labels and cross-validation described in any one of embodiments 1-9, such as Figure 6 As shown, including:

[0164] The fine-tuning model training module is used to process the labeled data based on the improved K-fold cross-validation algorithm to obtain the training set and the test set to train and verify the baseline model UIE to obtain the fine-tuning model;

[0165] A data selection module is used to randomly select and process unlabeled data in batches in a fine-tuning model based on preset rules to obtain selected pseudo data;

[0166] The final model training module is used to train the baseline model UIE based on the selected pseudo data and labeled data using a semi-supervised algorithm and an improved K-fold cross-validation algorithm to obtain the final model;

[0167] The recognition result acquisition module is used to process the input text based on the final model to obtain the entity recognition result.

[0168] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A small sample entity recognition method based on pseudo labeling and cross-validation, characterized in that: include: The labeled data is processed based on the improved K-fold cross-validation algorithm to obtain the training set and the test set to train and verify the baseline model UIE to obtain the fine-tuning model; After randomly selecting and processing unlabeled data in batches in the fine-tuning model based on preset rules, the selected pseudo data is obtained; Based on the selected pseudo data and labeled data, the baseline model UIE is trained using a semi-supervised algorithm and an improved K-fold cross-validation algorithm to obtain the final model; The input text is processed based on the final model to obtain entity recognition results.

2. According to claim 1, a small sample entity recognition method based on pseudo labeling and cross-validation is characterized in that: After randomly selecting and processing unlabeled data in batches in the fine-tuning model based on preset rules, the selected pseudo data is obtained, including: According to the preset step size and sliding window size, batches of unlabeled data are selected to obtain batches of unlabeled data; Predict batches of unlabeled data based on the fine-tuned model to obtain primary pseudo data; Mix primary pseudo data and labeled data to obtain training data; The baseline model UIE is trained based on the training data to obtain a test model; The test data is tested by the test model to obtain the test result; Get the test score corresponding to each test result, and regard the data with the highest test score as pseudo data.

3. According to the method of claim 1, the method is characterized in that: The improved K-fold cross-validation algorithm includes: Based on the improved K-fold cross-validation algorithm, K cycles are set and the labeled data are divided into K parts. Any one of the subsets is used as the test set, and the remaining K-1 subsets are used as the training set. Use K-1 of the K parts as training sets and 1 part as validation set for training to finally get a model.

4. The method for small sample entity recognition based on pseudo labeling and cross-validation according to claim 1, characterized in that: Based on the selected pseudo data and labeled data, the baseline model UIE is trained using a semi-supervised algorithm and an improved K-fold cross-validation algorithm to obtain the final model, including: Semi-supervised data is obtained based on the selected pseudo data and real data, and the semi-supervised data is divided into K parts based on the improved K-fold cross-validation algorithm, which are used as model training sets and model validation sets; Based on the model training set and model validation set, a semi-supervised algorithm is used to train the baseline model UIE to obtain the final model.

5. According to the method for small sample entity recognition based on pseudo labeling and cross-validation in claim 4, a semi-supervised algorithm is used to train the baseline model UIE to obtain a final model, comprising: A dynamic parameter α is introduced into the loss function of the semi-supervised algorithm to adjust the semi-supervised loss of the selected pseudo data and real data in the data set. The specific formula of the semi-supervised loss function is: Loss label =α*BCELoss(label(Fake))+(1-α)*BCELoss(label(True)) Among them, Loss label represents the semi-supervised loss, α is a dynamic parameter with a value range of (0, 1); BCEloss(label(Fake)) represents the binary cross entropy loss of fake data, BCE(logits, label(True)) represents the binary cross entropy loss of labeled data; label(Fake) represents the fake data label; label(True) represents the labeled data label Add the dynamic parameter R-Drop algorithm to the output of the baseline model UIE to perform two Dropouts on the model output and obtain the KL discreteness between the two Dropout processes: Among them, Loss R-Drop Represents the KL discreteness between two Dropout treatments; P1 w (y|x) represents the sub-model corresponding to the first Dropout process; P2 w (y|x) represents the sub-model corresponding to the second Dropout process; Represents the difference between the probability distribution of the first Dropout process and the probability distribution of the second Dropout process; Represents the difference between the probability distribution of the second Dropout process and the probability distribution of the first Dropout process; Generate the total loss function Loss for training the semi-supervised algorithm based on the dynamic parameter β, semi-supervised loss and R-Drop loss 总 : Loss 总 =Loss label +β(Loss R-Drop,start +Loss R-Drop,end ) Among them, β is a dynamic parameter with a value range of (0, 1]; Loss R-Drop,start Indicates the KL divergence at the beginning of the entity; Loss R-Drop,end Indicates the KL divergence of the entity end; Loss R-Drop,start +Loss R-Drop,end represents R-Drop loss; Based on the total loss function Loss 总 The model parameters are adjusted, and the baseline model UIE is semi-supervised trained by combining the model training set and the model validation set to obtain the final model.

6. The method for small sample entity recognition based on pseudo labeling and cross-validation according to claim 1, characterized in that: The input text is processed based on the final model to obtain entity recognition results, including: Encode the input text based on the fine-tuned model and extract the text features of the input text to obtain the word vector; Based on the final model, the probability of each word vector is obtained through the Linear layer and then the Sigmoid activation function layer to obtain the beginning and end of the entity segment; Based on the beginning and the end, the model is decoded and the input text category is classified to obtain a type label; Entity fragments are extracted based on the start position, end position, and category label and output as entity recognition results.

7. The method for small sample entity recognition based on pseudo labeling and cross-validation according to claim 1, characterized in that: Also includes: Add a domain recognition model to the input layer of the baseline model UIE, perform domain recognition on the input text based on the domain recognition model, and determine the description domain of the used text; And after the input text is field-labeled based on the described field, entity recognition processing is performed.

8. The method for small sample entity recognition based on pseudo labeling and cross-validation according to claim 7, characterized in that: A domain recognition model is added to the input layer of the baseline model UIE. Based on the domain recognition model, the domain of the input text is recognized to determine the description domain of the used text, including: Based on the domain recognition model, the input text is recognized to determine the location of the text punctuation marks. Based on the location of the text punctuation marks, the input text is divided into multiple text segments, and each text segment is marked with keywords. Generate corresponding input text vectors based on each text segment, compare each text vector with the database text vector, and obtain the best matching text vector for each text vector based on the comparison result; According to the description field corresponding to the best matching text vector, the description field corresponding to each input text is determined, and the keyword similarity is obtained based on the subvector corresponding to the keyword tag result of the text segment and the keyword subvector of its corresponding best matching text vector; When the keyword similarity is greater than or equal to a preset value, the current text segment is determined to be a key field, and a description field corresponding to the key field is obtained; If the description fields corresponding to all the key fields are consistent, determining that the description field is the description field of the current input text; Otherwise, the ratio of the total data volume corresponding to the key field to the total data volume of the input text is obtained. If the ratio is greater than or equal to a preset threshold, the description field corresponding to the adjacent text segment of the input text corresponding to each key field is obtained. If the description field corresponding to the adjacent text segment is consistent with the input text segment, or the description field corresponding to the adjacent text segment is a common field, the key field is used as the target text segment; Respectively obtain the description fields corresponding to each target text segment and the first quantity proportions corresponding to them, and use the description field corresponding to the largest first quantity proportion as the description field of the current input text; If the ratio is less than a preset threshold, the description fields corresponding to each valid text segment and their corresponding second quantity proportions are obtained respectively, and the description field corresponding to the largest second quantity proportion is used as the description field of the current input text.

9. The method for small sample entity recognition based on pseudo labeling and cross-validation according to claim 8, characterized in that: Also includes: Based on the domain recognition model, determine the keyword tagging results and key field determination results corresponding to the entity fragment, and record them; Before outputting the entity recognition result, the type tag corresponding to the entity output result is detected based on the optional tag category table corresponding to the description field to determine whether the tag is used correctly. If the tag is used correctly, it is determined whether the key field and its corresponding keyword have corresponding type tags. If so, the entity output result is output; Otherwise, the entity fragment is sent to the final model for secondary entity recognition.

10. A small sample entity recognition system based on pseudo labeling and cross-validation, characterized in that: include: The fine-tuning model training module is used to process the labeled data based on the improved K-fold cross-validation algorithm to obtain the training set and the test set to train and verify the baseline model UIE to obtain the fine-tuning model; A data selection module is used to randomly select and process unlabeled data in batches in a fine-tuning model based on preset rules to obtain selected pseudo data; The final model training module is used to train the baseline model UIE based on the selected pseudo data and labeled data using a semi-supervised algorithm and an improved K-fold cross-validation algorithm to obtain the final model; The recognition result acquisition module is used to process the input text based on the final model to obtain the entity recognition result.

Images