Method for annotating data, method and apparatus for annotating data by a model

Abstract

The application provides a method for labeling data, a method and device for labeling data by a model, the method comprising: obtaining a first data set of part sample data labeled by a human; performing feature extraction on the data in the first data set to obtain a labeling template; labeling unlabeled sample data by using the labeling template to obtain a second data set. Further comprising: obtaining a text to be labeled; labeling the text to be labeled by using a preset automatic labeling model to obtain a labeling result. The method can improve the efficiency of labeling data.

Description

Technical Field

[0001] This application relates to the field of data processing, and more specifically, to methods for labeling data, methods and apparatus for labeling data using models. Background Technology

[0002] Currently, filtering data from large amounts of system data requires manual data annotation or the use of fixed annotation templates.

[0003] The above-mentioned data annotation methods have significant limitations. Manual annotation wastes a lot of time, while template annotation can lead to errors in the annotated data and also requires a lot of time.

[0004] Therefore, how to improve the efficiency of labeled data is a technical problem that needs to be solved. Summary of the Invention

[0005] The purpose of this application is to provide a method for labeling data, and the technical solution of this application can improve the efficiency of labeling data.

[0006] In a first aspect, embodiments of this application provide a method for labeling data, including: obtaining a first data set of manually labeled sample data; extracting features from the data in the first data set to obtain a labeling template; and using the labeling template to label the unlabeled sample data to obtain a second data set.

[0007] In the above embodiments, this application obtains a labeling template from a portion of the sample data, and automatically labels the unlabeled sample data using the labeling template. This enables rapid labeling of unlabeled sample data and improves the efficiency of labeling data.

[0008] In some embodiments, after labeling the unlabeled sample data using a labeling template to obtain a second dataset, the method further includes:

[0009] The first and second data sets are combined to obtain a mixed data sample.

[0010] The basic bidirectional annotation model, the basic local feature annotation model, and the basic machine annotation model were trained using mixed data samples to obtain the initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model.

[0011] The initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model were updated using standard data samples in the validation set, resulting in the bidirectional annotation model, the local feature annotation model, and the machine annotation model.

[0012] By fusing the bidirectional annotation model, the local feature annotation model, and the machine annotation model, an automatic annotation model is obtained.

[0013] In the above embodiments, this application trains the basic model using manually annotated data and data annotated using annotation templates to obtain an automatic annotation model. This model can be used to directly annotate text data, thereby improving the efficiency of annotated data.

[0014] In some embodiments, feature extraction is performed on the data in the first dataset to obtain a labeled template, including:

[0015] Feature extraction is performed on data with the same or similar annotation structure in the first dataset to obtain annotation templates.

[0016] In the above embodiments, this application can obtain a labeling template for data annotation by extracting features from identical or similar data. The text data can be quickly annotated using the labeling template.

[0017] Secondly, embodiments of this application provide a method for annotating data using a model, comprising: acquiring text to be annotated; and annotating the text to be annotated using a preset automatic annotation model to obtain annotation results. The automatic annotation model is obtained by fusing a bidirectional annotation model, a local feature annotation model, and a machine annotation model. The bidirectional annotation model, the local feature annotation model, and the machine annotation model are obtained by updating the initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model respectively using standard annotation samples from a validation set. The initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model are obtained by training the basic bidirectional annotation model, the basic local feature annotation model, and the basic machine annotation model with hybrid samples obtained by mixing manually annotated sample data and preset template annotated template data samples.

[0018] In the above embodiments, this application trains the basic model using manually annotated data and data annotated using annotation templates to obtain an automatic annotation model. This model can be used to directly annotate the text to be annotated, thereby improving the efficiency of annotation data.

[0019] In some embodiments, the method further includes, before obtaining the text to be annotated:

[0020] Obtain manually annotated sample data and template data samples annotated with preset templates, where the manually annotated sample data is much smaller than the template data sample;

[0021] The artificial sample data and the template data sample are mixed to obtain a mixed sample;

[0022] The basic bidirectional annotation model, the basic local feature annotation model, and the basic machine annotation model were trained using mixed samples to obtain the initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model.

[0023] The initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model were updated using standard labeled samples in the validation set, resulting in the bidirectional annotation model, the local feature annotation model, and the machine annotation model.

[0024] Calculate the prediction weight of the labeled data from the bidirectional annotation model, the local feature annotation model, and the machine annotation model;

[0025] The model parameters from the bidirectional annotation model, the local feature annotation model, and the machine annotation model are retrieved and fused based on the predicted weight to obtain the automatic annotation model.

[0026] In the above embodiments, the basic model is trained using manually annotated data and data annotated using annotation templates, and the model is further updated using samples from the validation set. By fusing the three models, an automatic annotation model can be obtained. This model can be used to directly annotate text data, thereby improving the efficiency of annotated data.

[0027] In some embodiments, the initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model are updated using standard labeled samples in the validation set to obtain the bidirectional annotation model, the local feature annotation model, and the machine annotation model, including:

[0028] Input a text data from a standard labeled sample into the initial bidirectional labeling model, the initial local feature labeling model, and the initial machine labeling model respectively to obtain the first labeling result, the second labeling result, and the third labeling result respectively.

[0029] If two of the first, second, and third annotation results are the same, then the annotation result other than the two annotation results is replaced with one of the two annotation results randomly selected. The model corresponding to the annotation result other than the two annotation results is trained using the replaced annotation result and text data to obtain the bidirectional annotation model, the local feature annotation model, and the machine annotation model.

[0030] If the first, second, and third annotation results are all different, then the text data will be deleted.

[0031] In the above embodiments, this application further updates the training samples of the three models by using sample data from the validation set, which can enable the training samples of the training models to train more accurate automatic labeling models.

[0032] Thirdly, embodiments of this application provide an apparatus for labeling data, comprising:

[0033] The acquisition module is used to acquire the first dataset of manually labeled sample data;

[0034] The feature extraction module is used to extract features from the data in the first dataset to obtain a label template;

[0035] The annotation module is used to annotate unannotated sample data using annotation templates to obtain a second dataset.

[0036] Optionally, the device further includes:

[0037] The training module is used by the annotation module to annotate the unannotated sample data using the annotation template to obtain a second data set, and then mix the first data set and the second data set to obtain a mixed data sample.

[0038] The basic bidirectional annotation model, the basic local feature annotation model, and the basic machine annotation model were trained using mixed data samples to obtain the initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model.

[0039] The initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model were updated using standard data samples in the validation set, resulting in the bidirectional annotation model, the local feature annotation model, and the machine annotation model.

[0040] By fusing the bidirectional annotation model, the local feature annotation model, and the machine annotation model, an automatic annotation model is obtained.

[0041] Optionally, the feature extraction module is specifically used for:

[0042] Feature extraction is performed on data with the same or similar annotation structure in the first dataset to obtain annotation templates.

[0043] Fourthly, embodiments of this application provide an apparatus for labeling data using a model, comprising:

[0044] The acquisition module is used to acquire the text to be labeled;

[0045] The annotation module is used to annotate the text to be annotated using a preset automatic annotation model to obtain annotation results. The automatic annotation model is obtained by fusing a bidirectional annotation model, a local feature annotation model, and a machine annotation model. These models are updated using standard annotated samples from a validation set. The initial bidirectional annotation model, local feature annotation model, and machine annotation model are trained using hybrid samples obtained by mixing manually annotated sample data with preset template annotated sample data.

[0046] Optionally, the device further includes:

[0047] The training module is used to obtain manually annotated sample data and preset template annotated sample data before the acquisition module obtains the text to be annotated, wherein the manually annotated sample data is much smaller than the template sample data.

[0048] The artificial sample data and the template data sample are mixed to obtain a mixed sample;

[0049] The basic bidirectional annotation model, the basic local feature annotation model, and the basic machine annotation model were trained using mixed samples to obtain the initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model.

[0050] The initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model were updated using standard labeled samples in the validation set, resulting in the bidirectional annotation model, the local feature annotation model, and the machine annotation model.

[0051] Calculate the prediction weight of the labeled data from the bidirectional annotation model, the local feature annotation model, and the machine annotation model;

[0052] The model parameters from the bidirectional annotation model, the local feature annotation model, and the machine annotation model are retrieved and fused based on the predicted weight to obtain the automatic annotation model.

[0053] Optionally, the training module is specifically used for:

[0054] Input a text data from a standard labeled sample into the initial bidirectional labeling model, the initial local feature labeling model, and the initial machine labeling model respectively to obtain the first labeling result, the second labeling result, and the third labeling result respectively.

[0055] If two of the first, second, and third annotation results are the same, then the annotation result other than the two annotation results is replaced with one of the two annotation results randomly selected. The model corresponding to the annotation result other than the two annotation results is trained using the replaced annotation result and text data to obtain the bidirectional annotation model, the local feature annotation model, and the machine annotation model.

[0056] If the first, second, and third annotation results are all different, then the text data will be deleted.

[0057] Fifthly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing computer-readable instructions, which, when executed by the processor, perform the steps of the methods provided in the first or second aspect above.

[0058] In a sixth aspect, embodiments of this application provide a readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the methods provided in the first or second aspect above.

[0059] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0060] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 A flowchart illustrating a method for labeling data provided in an embodiment of this application;

[0062] Figure 2 A flowchart illustrating a method for labeling data using a model, as provided in this application embodiment;

[0063] Figure 3 A flowchart illustrating a method for training an automatically labeled model, as provided in this application embodiment;

[0064] Figure 4 A flowchart illustrating a method for manually labeling samples provided in this application embodiment;

[0065] Figure 5 A flowchart illustrating a method for annotating data using an annotation template, as provided in this application embodiment;

[0066] Figure 6 A flowchart illustrating a method for updating an automatically labeled model, as provided in this application embodiment;

[0067] Figure 7 A schematic block diagram of a data annotation device provided in an embodiment of this application;

[0068] Figure 8 A schematic block diagram of an apparatus for annotating model data according to an embodiment of this application;

[0069] Figure 9 A schematic block diagram of a data annotation device provided in this application embodiment;

[0070] Figure 10 This is a schematic block diagram of a device for annotating data using a model, as provided in an embodiment of this application. Detailed Implementation

[0071] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0072] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0073] First, some of the terms used in the embodiments of this application will be explained to facilitate understanding by those skilled in the art.

[0074] Txt:txt is a text format included with Microsoft operating systems and is one of the most common file formats.

[0075] MySQL: A relational database management system developed by the Swedish company MySQL AB and a product of Oracle. MySQL is one of the most popular relational database management systems, and in web applications, it is one of the best RDBMS (Relational Database Management System) application software.

[0076] CRF: Conditional Random Field, the most common algorithm is the Conditional Random Field model.

[0077] Semi-supervised learning (SSL) is a key research area in pattern recognition and machine learning. It is a learning method that combines supervised and unsupervised learning. SSL uses a large amount of unlabeled data, as well as labeled data simultaneously, to perform pattern recognition tasks.

[0078] BERT consists of two parts: pre-training and fine-tuning. In the pre-training stage, the model is trained on unlabeled data and different pre-training tasks. In the fine-tuning stage, BERT is initialized based on the parameters of the pre-trained model and then fine-tuned on labeled data from downstream tasks.

[0079] Long Short-Term Memory (LSTM) is a type of recurrent neural network designed to address the long-term dependency problem inherent in general RNNs. All RNNs have a chain-like structure of repeating neural network modules.

[0080] This application is applied to data processing scenarios, specifically data annotation using annotation templates or automatic annotation using automatic annotation templates.

[0081] However, currently, filtering data from large amounts of system data requires manual data annotation or the use of fixed annotation templates. These methods have significant limitations: manual annotation wastes a lot of time, and template-based annotation can lead to errors and also requires considerable time.

[0082] To this end, this application obtains a first dataset of manually labeled sample data; extracts features from the data in the first dataset to obtain a labeling template; and uses the labeling template to label the unlabeled sample data to obtain a second dataset. It also includes obtaining the text to be labeled; and using a preset automatic labeling model to label the text to obtain the labeling results. This method can improve the efficiency of data labeling.

[0083] In this embodiment of the application, the executing entity can be the annotation data device in the annotation data system. In practical applications, the annotation data device can be electronic devices such as terminal devices and servers, and there are no restrictions here.

[0084] The following is combined Figure 1 The method for annotating data in the embodiments of this application will be described in detail.

[0085] Please refer to Figure 1 , Figure 1 A flowchart of a method for labeling data provided in an embodiment of this application is shown below. Figure 1 The methods for labeling data shown include:

[0086] Step 110: Obtain the first dataset of manually labeled sample data.

[0087] Some sample data constitute only a small portion of the total sample data. The first dataset includes multiple labeled sample data sets. This application uses the commonly used BMEO four-labeling method to label organizational structures: B represents the start of the current organizational structure; M represents the continuation of the current organizational structure; E represents the end; and O represents any non-entity. Entity relationships are distinguished by R1, R2, ..., where R1 represents a sub-unit, R2 represents a sub-product, and so on. An example of labeling is shown below:

[0088] Original text: "Sunshine Life Insurance is a subsidiary of Sunshine Insurance Group."

[0089] Annotated text: Yangguang Life Insurance is a subsidiary of Yangguang Insurance Group. ||E1.

[0090] Relationship extraction: (Sunshine Life Insurance) (Sunshine Insurance Group) [subsidiary unit].

[0091] Original text: "Worry-free driving insurance is a preferential product launched by Sunshine Property Insurance in 2022 for ordinary office workers."

[0092] Labeled text: "Driver / B Passenger / M Worry-Free / M Insurance / E is a 2022 / O year-end product launched by Sunshine Property Insurance / E specifically for / O ordinary / O working / O workers / O, offering / O preferential / O benefits." ||E2.

[0093] Relationship extraction: (Worry-free driving insurance) (Sunshine Property Insurance) [sub-product].

[0094] Step 120: Extract features from the data in the first dataset to obtain the annotation template.

[0095] Feature extraction includes format feature extraction, which is used to extract the annotation format features from sample data with the same or similar annotation formats to obtain annotation templates with the same or similar annotation formats.

[0096] In some embodiments of this application, feature extraction is performed on the data in the first data set to obtain a labeling template, including: feature extraction is performed on the data in the first data set with the same or similar labeling structure to obtain a labeling template.

[0097] In the above process, this application can obtain a labeling template for data annotation by extracting features from the same or similar data. The text data can be quickly annotated using the labeling template.

[0098] In some embodiments of this application, feature extraction is performed on data with the same or similar labeled structure in the first data set to obtain a labeling template, including: selecting manually labeled data with similar structure from the manually labeled sample data; and performing template analysis on the manually labeled data with similar structure to obtain a preset template.

[0099] Step 130: Use the annotation template to annotate the unannotated sample data to obtain the second dataset.

[0100] The number of unlabeled sample data is much larger than the number of sample data in step 110.

[0101] In some embodiments of this application, after labeling the unlabeled sample data using a labeling template to obtain a second dataset, Figure 1The method further includes: mixing the first data set and the second data set to obtain a mixed data sample; training the basic bidirectional annotation model, the basic local feature annotation model, and the basic machine annotation model using the mixed data sample to obtain an initial bidirectional annotation model, an initial local feature annotation model, and an initial machine annotation model; updating the initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model using standard data samples in the validation set to obtain a bidirectional annotation model, a local feature annotation model, and a machine annotation model; and fusing the bidirectional annotation model, the local feature annotation model, and the machine annotation model to obtain an automatic annotation model.

[0102] In the above process, this application trains the basic model using manually annotated data and data annotated using annotation templates to obtain an automatic annotation model. This model can be used to directly annotate text data, thereby improving the efficiency of annotated data.

[0103] The basic bidirectional annotation model can be obtained by automatically annotating entities and relations using popular industry models such as BERT, bidirectional LSTM, and Conditional Random Field (CRF). The basic local feature annotation model can be obtained by automatically annotating entities and relations using Convolutional Neural Networks (TextCNN) and Hidden Markov Models (HMM) for extracting local features. The basic machine annotation model can be obtained by automatically annotating entities and relations using traditional machine learning models such as Document Object Model (TF-IDF) and Maximum Entropy Markov Model (MHMM). The automatic annotation model can be used as a semi-supervised learning SSL device for the task of automatically extracting entity relations. Employing a periodic offline update method can improve the model's adaptability.

[0104] In the above Figure 1 In the process shown, this application obtains a labeling template through a portion of the sample data, and automatically labels the unlabeled sample data using the labeling template, which can quickly label the unlabeled sample data and improve the efficiency of labeling data.

[0105] The following is combined Figure 2 The method of using model-annotated data according to embodiments of this application will be described in detail.

[0106] Please refer to Figure 2 , Figure 2 A flowchart illustrating a method for labeling data using a model, as provided in this application embodiment, is shown below. Figure 2 The methods for labeling data shown include:

[0107] Step 210: Obtain the text to be annotated.

[0108] The text to be annotated can be the text data of any article or document.

[0109] In some embodiments of this application, before obtaining the text to be annotated, Figure 2 The method further includes: acquiring manually annotated sample data and template data samples with preset template annotations, wherein the number of manually annotated sample data is much smaller than that of template data samples; mixing the manually annotated sample data and template data samples to obtain a mixed sample; training the basic bidirectional annotation model, the basic local feature annotation model, and the basic machine annotation model using the mixed sample to obtain the initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model; updating the initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model using standard annotation samples in the validation set to obtain the bidirectional annotation model, the local feature annotation model, and the machine annotation model; calculating the prediction weight of the annotation data of the bidirectional annotation model, the local feature annotation model, and the machine annotation model; and retrieving and fusing the model parameters from the bidirectional annotation model, the local feature annotation model, and the machine annotation model based on the prediction weight to obtain an automatic annotation model.

[0110] In the above process, the basic model is trained using manually annotated data and data annotated using annotation templates, and the model is further updated using samples from the validation set. By fusing the three models, an automatic annotation model can be obtained. This model can be used to directly annotate text data, thereby improving the efficiency of annotated data.

[0111] The following is combined Figure 3 The method for training an automatically labeled model according to the embodiments of this application will be described in detail.

[0112] Please refer to Figure 3 , Figure 3 A flowchart of a method for training an automatically labeled model provided in an embodiment of this application is shown below. Figure 3 The methods for training an automatic annotation model shown include:

[0113] Step 310: Manually label the samples.

[0114] Specifically, the manual annotation of sample streams includes the collection and standardization of raw data, sequence and relation annotation, data storage and other processing procedures.

[0115] Step 320: Label the sample using the label template.

[0116] Specifically, the process involves template discovery and automatic data annotation, including identifying common templates from the annotated data, automatically annotating the data using the templates, and manually checking and correcting the annotation results.

[0117] Step 330: Train the automatic annotation model.

[0118] Specifically, the base model is trained using manually labeled samples and labeled samples from labeled templates to obtain an automatically labeled model.

[0119] Step 340: Automatically label the model and label the samples.

[0120] Specifically, the samples in the training set are labeled using an automatic labeling model.

[0121] Step 350: Update the automatically labeled model.

[0122] Specifically, a text data from a standard labeled sample is input into the initial bidirectional labeling model, the initial local feature labeling model, and the initial machine labeling model, respectively, to obtain the first labeling result, the second labeling result, and the third labeling result. The model is then updated based on the first labeling result, the second labeling result, and the third labeling result.

[0123] Step 360: Fusion Model.

[0124] Specifically, the model parameters from the bidirectional annotation model, the local feature annotation model, and the machine annotation model are retrieved and fused according to the predicted weight to obtain the automatic annotation model.

[0125] also, Figure 3 Please refer to the specific methods and steps shown. Figure 1 or Figure 2 This will not be elaborated upon further here.

[0126] The following is combined Figure 4 The method for manually annotating samples according to embodiments of this application will be described in detail.

[0127] Please refer to Figure 4 , Figure 4 A flowchart of a method for manually labeling samples provided in this application embodiment is shown below. Figure 4 The methods for manually labeling samples shown include:

[0128] Step 410: Collect raw data.

[0129] Specifically, data is collected from customer service robot dialogue data, internal product introduction documents, and industry-standard documents.

[0130] Step 420: Data normalization.

[0131] Specifically, the collected data is integrated according to the format required by the algorithm. From the obtained data, text statements rich in entities and relationships are selected, organized into a unified txt format, and stored in the database for later use.

[0132] Step 430: Sequence labeling and relation labeling.

[0133] Specifically, data with the same format and related data in the sample data are labeled.

[0134] Step 440: Review the unusual data.

[0135] Specifically, the three types of data with different labels were reviewed and a unified standard was established.

[0136] Step 450: Save the annotation data.

[0137] Specifically, it is stored in a MySQL database for later use.

[0138] also, Figure 4 Please refer to the specific methods and steps shown. Figure 1 or Figure 2 This will not be elaborated upon further here.

[0139] The following is combined Figure 5 The method for annotating data using an annotation template according to embodiments of this application will be described in detail.

[0140] Please refer to Figure 5 , Figure 5 A flowchart illustrating a method for annotating data using an annotation template, as provided in this application embodiment, is shown below. Figure 5 The methods for annotating data using the annotation template shown include:

[0141] Step 510: Template annotation data.

[0142] Specifically, the labeled data is organized by retrieving the entire labeled data from MySQL for later use.

[0143] Step 520: Relationship clustering.

[0144] Specifically, data with the same relationship in the labeled data are clustered together.

[0145] Step 530: Rule template writing.

[0146] Specifically, the clustered data is organized to identify commonalities and abstract rule templates.

[0147] Step 540: Template expansion.

[0148] Specifically, the templates are expanded by incorporating industry expertise, increasing the number of templates available. The original templates are then merged with the expanded templates, and unannotated data is automatically annotated.

[0149] Step 550: Save the template data.

[0150] Specifically, it is stored in a MySQL database for later use.

[0151] also, Figure 5 Please refer to the specific methods and steps shown. Figure 1 or Figure 2 This will not be elaborated upon further here.

[0152] The following is combined Figure 6 The method for updating the automatic annotation model according to the embodiments of this application will be described in detail.

[0153] Please refer to Figure 6 , Figure 6 A flowchart of a method for updating an automatically labeled model provided in an embodiment of this application is shown below. Figure 6 The methods shown for updating the automatic annotation model include:

[0154] Step 610: Obtain sample data for the updated template.

[0155] Specifically, unlabeled data obtained periodically.

[0156] Step 620: Fusion Model.

[0157] Specifically, unlabeled data is labeled using models, and the models are then merged.

[0158] Step 630: Match template.

[0159] Specifically, it matches the corresponding annotation template to the unannotated data.

[0160] Step 640: Analyze and update the model.

[0161] Specifically, the model output results are analyzed, and the updated data is stored in real time to update the model.

[0162] also, Figure 6 Please refer to the specific methods and steps shown. Figure 1 or Figure 2 This will not be elaborated upon further here.

[0163] In some embodiments of this application, after retrieving and fusing model parameters from the bidirectional annotation model, the local feature annotation model, and the machine annotation model based on the predicted weight to obtain the automatic annotation model, Figure 2 The method also includes: calculating the cross-entropy of the automatic annotation model using model parameters in the bidirectional annotation model, local feature annotation model, and machine annotation model of the predicted weight; and adjusting the automatic annotation model based on the cross-entropy until the cross-entropy meets the preset value.

[0164] The cross-entropy can be calculated using the following formula:

[0165] L=αH a+βH b +γH c

[0166]

[0167] Where the prediction weights are α, β, and γ, respectively, and L is the cross-entropy H. a H b H c The sum of p(x) k p(x) represents the probability that each sub-label is correct. m ) represents the probability of correctly labeling entity relationships, and λ and μ represent the proportions of the loss functions for sequence labeling and relationship labeling, respectively.

[0168] In some embodiments of this application, the initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model are updated using standard annotation samples in the validation set to obtain the bidirectional annotation model, the local feature annotation model, and the machine annotation model, respectively. This includes: inputting a piece of text data from a standard annotation sample into the initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model to obtain a first annotation result, a second annotation result, and a third annotation result, respectively; if two of the first, second, and third annotation results are the same, then the annotation result other than the two annotation results is replaced with one of the two annotation results randomly, and the model corresponding to the annotation result other than the two annotation results is trained using the replaced annotation result and the text data to obtain the bidirectional annotation model, the local feature annotation model, and the machine annotation model; if the first, second, and third annotation results are all different, then the text data is deleted.

[0169] In the above process, this application further updates the training samples of the three models by using sample data from the validation set, which can enable the training samples of the training models to train more accurate automatic labeling models.

[0170] Furthermore, if the first, second, and third annotation results are the same, the standard annotation sample can be used to further train the initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model.

[0171] Step 220: Use the preset automatic annotation model to annotate the text to be annotated and obtain the annotation results.

[0172] The automatic annotation model is obtained by fusing the bidirectional annotation model, the local feature annotation model, and the machine annotation model. The bidirectional annotation model, the local feature annotation model, and the machine annotation model are obtained by updating the initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model respectively using standard annotation samples in the validation set. The initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model are obtained by training the basic bidirectional annotation model, the basic local feature annotation model, and the basic machine annotation model with hybrid samples obtained by mixing manually annotated sample data and template data samples annotated with preset templates.

[0173] In the above Figure 2 In the process shown, this application trains the basic model using manually annotated data and data annotated using annotation templates to obtain an automatic annotation model. This model can be used to directly annotate the text to be annotated, thereby improving the efficiency of annotation data.

[0174] The previous text passed Figures 1-6 The methods for labeling data and training and updating automatic labeling models are described below. Figures 7-10 A device for describing labeled data.

[0175] Please refer to Figure 7 This is a schematic block diagram of a data annotation device 700 provided in an embodiment of this application. The device 700 can be a module, program segment, or code on an electronic device. This device 700 is related to the above... Figure 1 The method implementation corresponds to this and can be executed. Figure 1 The various steps involved in the method embodiments and the specific functions of the device 700 can be found in the following description. To avoid repetition, detailed descriptions are omitted here.

[0176] Optionally, the device 700 includes:

[0177] The acquisition module 710 is used to acquire the first dataset of manually labeled sample data;

[0178] The feature extraction module 720 is used to extract features from the data in the first dataset to obtain a label template;

[0179] The annotation module 730 is used to annotate unannotated sample data using an annotation template to obtain a second dataset.

[0180] Optionally, the device further includes:

[0181] The training module is used by the annotation module to annotate unannotated sample data using annotation templates to obtain a second data set, then mixes the first and second data sets to obtain a mixed data sample; the mixed data sample is used to train the basic bidirectional annotation model, the basic local feature annotation model, and the basic machine annotation model to obtain an initial bidirectional annotation model, an initial local feature annotation model, and an initial machine annotation model; the initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model are updated using standard data samples in the validation set to obtain a bidirectional annotation model, a local feature annotation model, and a machine annotation model; the bidirectional annotation model, the local feature annotation model, and the machine annotation model are then fused to obtain an automatic annotation model.

[0182] Optionally, the feature extraction module is specifically used for:

[0183] Feature extraction is performed on data with the same or similar annotation structure in the first dataset to obtain annotation templates.

[0184] Please refer to Figure 8 This is a schematic block diagram of a device 800 for annotating model data according to an embodiment of this application. The device 800 can be a module, program segment, or code on an electronic device. This device 800 is related to the above... Figure 2 The method implementation corresponds to this and can be executed. Figure 2 The various steps involved in the method embodiment, and the specific functions of the device 800, can be found in the description below. To avoid repetition, detailed descriptions are appropriately omitted here.

[0185] Optionally, the device 800 includes:

[0186] Module 810 is used to acquire the text to be annotated;

[0187] The annotation module 820 is used to annotate the text to be annotated using a preset automatic annotation model to obtain annotation results. The automatic annotation model is obtained by fusing a bidirectional annotation model, a local feature annotation model, and a machine annotation model. These models are updated using standard annotated samples from a validation set. The initial bidirectional annotation model, local feature annotation model, and machine annotation model are trained using hybrid samples obtained by mixing manually annotated sample data and template data samples annotated with preset templates.

[0188] Optionally, the device further includes:

[0189] The training module is used by the acquisition module to acquire manually annotated sample data and template data samples with preset template annotations before acquiring the text to be annotated, wherein the number of manually annotated sample data is much smaller than that of template data samples; the manually annotated sample data and template data samples are mixed to obtain a mixed sample; the mixed sample is used to train the basic bidirectional annotation model, the basic local feature annotation model and the basic machine annotation model respectively to obtain the initial bidirectional annotation model, the initial local feature annotation model and the initial machine annotation model; the initial bidirectional annotation model, the initial local feature annotation model and the initial machine annotation model are updated using standard annotation samples in the validation set respectively to obtain the bidirectional annotation model, the local feature annotation model and the machine annotation model; the prediction weight of the annotation data of the bidirectional annotation model, the local feature annotation model and the machine annotation model is calculated; the model parameters in the bidirectional annotation model, the local feature annotation model and the machine annotation model are retrieved according to the prediction weight and fused to obtain the automatic annotation model.

[0190] Optionally, the training module is specifically used for:

[0191] A text data from a standard labeled sample is input into the initial bidirectional labeling model, the initial local feature labeling model, and the initial machine labeling model, respectively, to obtain the first labeling result, the second labeling result, and the third labeling result, respectively. If two of the first, second, and third labeling results are the same, the labeling result other than the two labels is replaced with one of the two labels randomly selected. The model corresponding to the labeling result other than the two labels is trained using the replaced labeling result and the text data to obtain the bidirectional labeling model, the local feature labeling model, and the machine labeling model. If the first, second, and third labeling results are all different, the text data is deleted.

[0192] Please refer to Figure 9 This is a schematic block diagram of a data annotation device 900 provided in an embodiment of this application. The device may include a memory 910 and a processor 920. Optionally, the device may further include a communication interface 930 and a communication bus 940. This device is similar to the one described above. Figure 1 The method implementation corresponds to this and can be executed. Figure 1 The specific functions of the device involved in the method embodiments can be found in the following description.

[0193] Specifically, memory 910 is used to store computer-readable instructions.

[0194] Processor 920 is used to process readable instructions stored in memory and is capable of executing... Figure 1 Each step in the method.

[0195] The communication interface 930 is used for signaling or data communication with other node devices. For example, it is used for communication with a server or terminal, or for communication with other device nodes, but the embodiments of this application are not limited thereto.

[0196] Communication bus 940 is used to enable direct communication between the above components.

[0197] In this embodiment, the communication interface 930 of the device is used for signaling or data communication with other node devices. The memory 910 can be high-speed RAM or non-volatile memory, such as at least one disk storage device. Optionally, the memory 910 can also be at least one storage device located remotely from the aforementioned processor. The memory 910 stores computer-readable instructions, which, when executed by the processor 920, enable the electronic device to perform the aforementioned... Figure 1 The method process is shown. Processor 920 can be used on device 700 and is used to perform the functions described in this application. Exemplarily, the processor 920 described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components; the embodiments of this application are not limited thereto.

[0198] Please refer to Figure 10 This is a schematic block diagram of a device 1000 for annotating model data according to an embodiment of this application. The device may include a memory 1010 and a processor 1020. Optionally, the device may further include a communication interface 1030 and a communication bus 1040. This device is similar to the one described above. Figure 2 The method implementation corresponds to this and can be executed. Figure 2 The specific functions of the device involved in the method embodiments can be found in the following description.

[0199] Specifically, memory 1010 is used to store computer-readable instructions.

[0200] Processor 1020 is used to process readable instructions stored in memory and is capable of executing... Figure 2 Each step in the method.

[0201] The communication interface 1030 is used for signaling or data communication with other node devices. For example, it is used for communication with a server or terminal, or for communication with other device nodes, but the embodiments of this application are not limited thereto.

[0202] The communication bus 1040 is used to enable direct communication between the above components.

[0203] In this embodiment, the communication interface 1030 of the device is used for signaling or data communication with other node devices. The memory 1010 can be high-speed RAM or non-volatile memory, such as at least one disk storage device. Optionally, the memory 1010 can also be at least one storage device located remotely from the aforementioned processor. The memory 1010 stores computer-readable instructions, which, when executed by the processor 1020, enable the electronic device to perform the aforementioned... Figure 2 The method process is shown. The processor 1020 can be used on the device 800 and is used to perform the functions in this application. Exemplarily, the processor 1020 described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, and the embodiments of this application are not limited thereto.

[0204] This application embodiment also provides a readable storage medium, wherein when the computer program is executed by a processor, it performs the following... Figure 1 or Figure 2 The method process executed by the electronic device in the illustrated method embodiment.

[0205] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0206] In summary, this application provides a method for labeling data, a method for labeling data using a model, and an apparatus. The method includes: acquiring a first dataset of manually labeled sample data; extracting features from the data in the first dataset to obtain a labeling template; and using the labeling template to label unlabeled sample data to obtain a second dataset. It also includes: acquiring text to be labeled; and using a preset automatic labeling model to label the text to be labeled to obtain a labeling result. This method can improve the efficiency of data labeling.

[0207] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0208] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0209] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0210] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0211] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0212] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for labeling data using a model, characterized in that, include: Obtain manually annotated sample data and template data samples with preset template annotations; The artificial sample data and the template data sample are mixed to obtain a mixed sample; The basic bidirectional annotation model, the basic local feature annotation model, and the basic machine annotation model are trained using the mixed samples to obtain the initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model. The initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model are updated using standard labeled samples in the validation set to obtain the bidirectional annotation model, the local feature annotation model, and the machine annotation model. Calculate the prediction weight of the labeled data from the bidirectional annotation model, the local feature annotation model, and the machine annotation model; Based on the predicted weight, the model parameters from the bidirectional annotation model, the local feature annotation model, and the machine annotation model are retrieved and fused to obtain an automatic annotation model; Get the text to be annotated; The text to be annotated is annotated using an automatic annotation model to obtain annotation results. The automatic annotation model is obtained by fusing a bidirectional annotation model, a local feature annotation model, and a machine annotation model. The bidirectional annotation model, the local feature annotation model, and the machine annotation model are obtained by updating the initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model respectively using standard annotation samples in the validation set. The initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model are obtained by training the basic bidirectional annotation model, the basic local feature annotation model, and the basic machine annotation model with hybrid samples obtained by mixing manually annotated sample data and template data samples annotated with preset templates. The bidirectional annotation model, the local feature annotation model, and the machine annotation model are updated through the following steps: A text data from the standard annotation sample is input into the initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model, respectively, to obtain a first annotation result, a second annotation result, and a third annotation result; if two of the first, second, and third annotation results are the same, the annotation result other than the two specified annotation results is replaced with one of the two specified annotation results randomly, and the model corresponding to the annotation result other than the two specified annotation results is trained using the replaced annotation result and the text data to obtain the bidirectional annotation model, the local feature annotation model, and the machine annotation model; if the first, second, and third annotation results are all different, the text data is deleted. The basic bidirectional annotation model includes models obtained by automatically annotating entities and relations using BERT, bidirectional LSTM, and conditional random field models; the basic local feature annotation model includes models obtained by automatically annotating entities and relations using convolutional neural networks and hidden Markov models that extract local features; the basic machine annotation model includes models obtained by automatically annotating entities and relations using machine learning models such as document object models and maximum entropy Markov models.

2. A device for labeling data using a model, characterized in that, include: The acquisition module is used to acquire manually annotated sample data and template data samples with preset template annotations; The artificial sample data and the template data samples are mixed to obtain a hybrid sample; the hybrid sample is used to train the basic bidirectional annotation model, the basic local feature annotation model, and the basic machine annotation model to obtain the initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model; the initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model are updated using standard annotation samples in the validation set to obtain the bidirectional annotation model, the local feature annotation model, and the machine annotation model; the prediction weight of the annotation data of the bidirectional annotation model, the local feature annotation model, and the machine annotation model is calculated; the model parameters in the bidirectional annotation model, the local feature annotation model, and the machine annotation model are retrieved according to the prediction weight and fused to obtain the automatic annotation model; the text to be annotated is obtained. The annotation module is used to annotate the text to be annotated using an automatic annotation model to obtain annotation results. The automatic annotation model is obtained by fusing a bidirectional annotation model, a local feature annotation model, and a machine annotation model. The bidirectional annotation model, the local feature annotation model, and the machine annotation model are obtained by updating the initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model respectively using standard annotation samples from a validation set. The initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model are obtained by training the basic bidirectional annotation model, the basic local feature annotation model, and the basic machine annotation model with hybrid samples obtained by mixing manually annotated sample data and template data samples annotated with preset templates. The bidirectional annotation model, the local feature annotation model, and the machine annotation model are updated through the following steps: inputting one piece of text data from the standard annotation sample into the initial bidirectional annotation model, the initial local feature annotation model, and the initial machine annotation model respectively. The process involves obtaining a first annotation result, a second annotation result, and a third annotation result, respectively. If two of the first, second, and third annotation results are the same, the annotation result other than the two specified annotation results is replaced with a random one of the two specified annotation results. The model corresponding to the annotation result other than the two specified annotation results is then trained using the replaced annotation result and the text data to obtain the bidirectional annotation model, the local feature annotation model, and the machine annotation model. If the first, second, and third annotation results are all different, the text data is deleted. The basic bidirectional annotation model includes models obtained by automatically annotating entities and relations using BERT, bidirectional LSTM, and conditional random field models. The basic local feature annotation model includes models obtained by automatically annotating entities and relations using convolutional neural networks and hidden Markov models that extract local features. The basic machine annotation model includes models obtained by automatically annotating entities and relations using machine learning models such as document object models and maximum entropy Markov models.

3. An electronic device, characterized in that, include: A memory and a processor, the memory storing computer-readable instructions that, when executed by the processor, perform the steps of the method as described in claim 1.

4. A computer-readable storage medium, characterized in that, include: A computer program that, when run on a computer, causes the computer to perform the method as described in claim 1.

Images