Text data labeling method and system, electronic device, and storage medium

Abstract

A text data annotation method, system, electronic device, and storage medium are disclosed. The text data annotation method includes: preprocessing the text to be annotated; collecting data from the dataset and extracting various features; determining whether labeled data exists, and if so, training and predicting a model; selecting a sampling strategy and extracting data for annotation. This invention, based on active learning technology, selects the most representative and informative data for manual annotation, eliminating restrictions on label sets and seed data, improving the efficiency of manual annotation, and effectively lowering the barrier to entry.

Description

Technical Field

[0001] This invention belongs to the field of machine learning technology, specifically relating to a text data annotation method and system, electronic device and storage medium. Background Technology

[0002] With the development of the internet and artificial intelligence, the demand for data annotation services is increasing. Data annotation has also evolved from purely manual annotation to machine annotation that combines manual annotation with active learning. Currently available intelligent data annotation platforms are typically based on active learning, sampling unlabeled data according to uncertainty and diversity to improve annotation efficiency. However, because the sampling methods are too simplistic, many constraints are imposed on users, such as a known label set, a certain amount of seed data for each label, and a relatively balanced amount of data across labels. However, in complex datasets, such as unknown label sets, no seed data, or imbalanced categories, simple sampling strategies are insufficient. Conversely, complex sampling strategies require users to possess specialized knowledge to understand when to switch from strategy A to strategy B, or to combine multiple sampling strategies, all of which result in a high barrier to entry. Summary of the Invention

[0003] In view of this, the main objective of the present invention is to provide a text data annotation method and system, electronic device and storage medium, in order to at least partially solve at least one of the above-mentioned technical problems.

[0004] To achieve the above objectives, as a first aspect of the present invention, a method for recommending sampling strategies for a text data annotation platform is provided, comprising the following steps:

[0005] Use the text to be labeled as the current dataset;

[0006] Based on the current dataset, extract selected features from the dataset;

[0007] Determine whether there is labeled data in the current dataset, select a sampling strategy based on the determination result, and use it to extract and label data in the current dataset.

[0008] As a second aspect of the present invention, a text data annotation system is also provided, comprising:

[0009] The feature statistics extraction module is used to extract selected features from the text to be labeled as the current dataset.

[0010] The judgment module is used to determine whether there is labeled data in the current dataset;

[0011] The strategy sampling module is used to select a sampling strategy based on the judgment result of the judgment module and use it to extract and label data from the current dataset.

[0012] As a third aspect of the present invention, an electronic device is also provided, including a processor and a memory, the memory being used to store a computer-executable program, wherein when the computer-executable program is executed by the processor, the processor executes the text data annotation platform sampling strategy recommendation method as described above.

[0013] As a fourth aspect of the present invention, a computer-readable medium is also provided, storing a computer-executable program, which, when executed, implements the text data annotation platform sampling strategy recommendation method as described above.

[0014] Based on the above technical solution, it can be seen that the text data annotation method and system of the present invention have at least one of the following beneficial effects compared with the prior art:

[0015] Based on artificial intelligence technologies such as text clustering, self-learning, and active learning, this invention can select the most representative and informative data for manual annotation even in complex datasets. During the annotation process, it takes into account the expansion of historical label data and the discovery of new labels, removes the restrictions on label sets and seed data, improves the efficiency of manual annotation, and effectively lowers the threshold for use.

[0016] This invention is particularly suitable for complex datasets. Its sampling strategy can reduce the amount of manual annotation, make the annotation platform adaptable to various situations, even zero-seed data, and reduce the reliance on professional technicians, making it more accessible. Attached Figure Description

[0017] Figure 1 This is a flowchart of the sampling strategy recommendation method for the text data annotation platform of the present invention.

[0018] Figure 2 This is a schematic diagram of the framework of the text data annotation system of the present invention;

[0019] Figure 3 This is a schematic diagram of the structure of the electronic device of the present invention;

[0020] Figure 4 This is a schematic diagram of the storage medium of the present invention;

[0021] Figure 5 This is a flowchart illustrating the sampling strategy recommendation method for a text data annotation platform according to Embodiment 1 of the present invention. Detailed Implementation

[0022] In the description of specific embodiments, detailed descriptions of structures, performance, effects, or other features are provided to enable those skilled in the art to fully understand the embodiments. However, this does not preclude those skilled in the art from implementing the present invention with technical solutions that do not contain the aforementioned structures, performance, effects, or other features under specific circumstances.

[0023] The flowchart in the accompanying drawings is merely an exemplary process demonstration and does not imply that the solution of this invention must include all the content, operations, and steps in the flowchart, nor does it imply that they must be executed in the order shown in the diagram. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. Without departing from the inventive spirit of this invention, the execution order shown in the flowchart can be changed according to the actual situation.

[0024] The box in the attached diagram Figure 1 Generally, these refer to functional entities, and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processing unit devices and / or microcontroller devices.

[0025] The same reference numerals in the accompanying drawings denote the same or similar elements, components, or parts, and therefore, repeated descriptions of the same or similar elements, components, or parts may be omitted below. It should also be understood that although terms such as first, second, third, etc., indicating numbers may be used herein to describe various devices, elements, components, or parts, these devices, elements, components, or parts should not be limited by these terms. That is, these terms are only used to distinguish one from another. For example, a first device may also be referred to as a second device, without departing from the essential technical solution of the invention. Furthermore, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0026] The meanings of some technical terms in this manual are as follows:

[0027] Clustering

[0028] The process of dividing a collection of physical or abstract objects into multiple classes composed of similar objects is called clustering. Traditional clustering analysis computation methods mainly include: partitioning methods (such as the k-means algorithm), hierarchical methods, density-based methods, grid-based methods, and model-based methods.

[0029] Cluster

[0030] A set of samples generated by clustering. Samples within the same cluster are similar to each other, but different from samples in other clusters.

[0031] Uncertainty i )

[0032] In machine learning problems, models often need to learn good representations from massive amounts of training data. A major challenge is that manually labeled data often contains semantic ambiguity. Therefore, some papers use uncertainty to describe and correct labeling errors, allowing the model to reduce the weight given to learning uncertain samples and instead focus on learning more accurate labels.

[0033] positive examples of labels

[0034] Positive examples are also called "affirmative examples." Every concept has appropriate examples and inappropriate examples, the latter being called "counterexamples" or "negative examples." For example, elephants, lions, tigers, cats, dogs, and whales are all positive examples of the concept of mammals. Labeled positive examples represent appropriate examples that satisfy this label.

[0035] Recall

[0036] Recall is a measure of coverage, representing the number of correctly predicted data points out of those that were true positives. Recall = TP / (TP + FN) = TP / P = sensitive. As you can see, recall and sensitivity are the same. In the above formula, TP stands for True Positive and FN stands for False Negative.

[0037] Sentence embedding

[0038] Representing sentences of variable length using fixed-length vectors provides services for downstream NLP tasks. For word embeddings, after training, each word corresponds to a vector, allowing for an intuitive assessment of embedding quality. However, for sentence embeddings, there is no ground truth for evaluation. The quality is assessed by feeding the sentence embedding into downstream tasks and evaluating its performance there.

[0039] Cosine similarity

[0040] Cosine distance, also known as cosine similarity, is a measure of the difference between two vectors in a vector space, using the cosine of the angle between them. The closer the cosine value is to 1, the closer the angle is to 0 degrees, meaning the two vectors are more similar; this is called "cosine similarity".

[0041] Diversity Sampling

[0042] Diversity sampling is used to ensure the diversity of samples. It is mainly based on two dimensions: clusters and effective words. Priority is given to selecting data whose clusters have low labeling rates or contain effective words that have not yet been covered.

[0043] Uncertainty sampling

[0044] The data are sorted in reverse order based on the uncertainty of the unlabeled data, and data with greater uncertainty are sampled first.

[0045] Stratified sampling

[0046] Each label is stratified according to its predicted score, and a certain amount of data is randomly drawn from each stratum.

[0047] Minority sampling

[0048] For labels with limited labeled data, the data is arranged in reverse order and layered according to the predicted labels and scores of the data based on the model. The width of each layer increases exponentially, and a certain amount of data is extracted from each layer.

[0049] mop-up sampling

[0050] Based on the cosine similarity of sentence vectors, data with a similarity to labeled data less than a threshold is extracted from unlabeled data, while ensuring that the similarity between the extracted data is greater than the threshold.

[0051] While there has been considerable research on automatic annotation methods based on active learning, these methods often require seed data or rely on the expertise of sampling personnel. This invention proposes a text data annotation method for complex datasets. This method expands the candidate set of sampling strategies for the annotation platform to include diversity sampling, uncertainty sampling, probabilistic stratified sampling, minority class sampling, and / or tail-sweep sampling. During the annotation process, statistical annotation data is collected and features are constructed. Then, based on these features, the optimal sampling strategy or combination of strategies is recommended to extract data for manual annotation. This reduces the amount of manual annotation while broadening the scope of the annotation platform and lowering its barrier to entry.

[0052] like Figure 1 As shown, the sampling strategy recommendation method for the text data annotation platform of the present invention includes the following steps:

[0053] Use the text to be labeled as the current dataset;

[0054] Based on the current dataset, extract selected features from the dataset;

[0055] Determine whether there is labeled data in the current dataset, select a sampling strategy based on the determination result, and use it to extract and label data in the current dataset.

[0056] The sampling strategy recommendation method may further include:

[0057] The process involves determining whether the labeled dataset has reached the target coverage. If so, the text to be labeled is completed. If not, the process repeats the steps of extracting selected features from the dataset, determining whether there is already labeled data in the current dataset, selecting a sampling strategy based on the determination result, and using it to extract and label data from the current dataset, until all steps of determining whether the target coverage has been reached are repeated. The purpose of this step is to repeat the above sampling / labeling steps of this invention several times in order to better and more completely label a dataset.

[0058] The sampling strategy recommendation method, after selecting a sampling strategy and labeling the data, and before determining whether the target coverage has been achieved, may include steps of summarizing the dataset and post-processing the strategy. The summarization aims to ensure data accuracy by accumulating all labeled data from multiple datasets into the final coverage metric. Post-processing refers to the calculation and processing of data after extraction for some sampling strategies. For example, minority class sampling strategies require recall estimation and calculation of other relevant parameters. Recall estimation is already included in the minority class sampling strategy steps, while other calculation and processing steps, which are not essential but only necessary in specific situations, can be placed within post-processing.

[0059] The text to be labeled may contain a certain amount of pre-labeled data, or it may be completely unlabeled data without seeds.

[0060] The method further includes a text preprocessing step, which may include, for example:

[0061] Hierarchical clustering of text data is performed based on sentence vectors, recording the cluster center (C1, C2, ..., C6) to which each data point belongs. k );

[0062] The data is segmented using a word segmentation tool, and the document frequency of each word is counted. Words with a document frequency greater than a first threshold (e.g., 2) are recorded as the set of valid words.

[0063] The purpose of clustering is to group words with the same or similar meanings together for processing, thereby reducing the labor cost of repeated labeling.

[0064] The text preprocessing steps may include, for example, one or more of the following steps: initial screening of text information, removal of invalid text, word segmentation of text information, and removal of stop words. These steps depend on the state of the initial text data; if relevant conditions exist, they can be processed. The purpose of preprocessing is to ensure that the retained data is valid, thereby reducing interference and bias during later sampling and annotation.

[0065] The selected features of the dataset can be chosen from the total number of data entries N and the number of labeled entries N. label Label coverage Number of manually labeled entries N manual Number of machine-expanded labels N model Number of tags M tag , Number of data entries for each label N tag Uncertainty of each data point i The number of data entries N with uncertainty greater than the threshold uncertain One or more of the following: historical sampling strategies and annotation information. It is necessary to determine, in turn, which feature parameters need to be calculated based on the sampling strategies that might be used with the input text data, and which sampling strategy corresponds to which feature parameters need to be calculated.

[0066] Uncertainty is defined as follows: x represents text, tag i For the i-th label, P θ (tag i |x) represents the model's prediction of text x as a tag. i The probability of.

[0067] The steps of determining whether labeled data exists, selecting a sampling strategy based on the determination result, and labeling the data include, for example, the following logical branches:

[0068] If no labeled data exists, directly select a sampling strategy and use it to extract and label data in the current dataset;

[0069] If labeled data exists, a model is trained based on the labeled data and labels and used to make predictions on the current dataset. Then, a sampling strategy is selected and used to extract and label data from the current dataset.

[0070] The step of training a model and making predictions based on labeled data and labels includes:

[0071] A text classifier model is trained based on labeled data and tags, and predictions are made on unlabeled data. The prediction result and tag for each data point are recorded. i and score i;

[0072] The text classifier model is selected from, for example, LSTM, TextCNN, or BERT models.

[0073] The selection sampling strategy satisfies the following conditions:

[0074] The selected sampling strategies conform to policy disabling, policy voting, and policy mutual exclusion;

[0075] Prioritize one or more strategies that score higher.

[0076] The policy disabling refers to:

[0077] If N label =N, disable all sampling strategies;

[0078] If N label ≤1 or M tag ≤1, stratified sampling, minority class sampling, and uncertainty sampling are disabled;

[0079] The strategy recommendation refers to:

[0080] If the coverage ratio is marked label If the score is below the threshold, stratified probability sampling will award extra points.

[0081] If the number of data points with high uncertainty exceeds the threshold, bonus points will be awarded for uncertainty sampling.

[0082] If the cluster centers or effective vocabulary are not fully covered, the diversity sampling will receive bonus points.

[0083] If the minimum number of tags for a single label If the score is below the threshold, minority class sampling receives bonus points;

[0084] Label coverage ratio label If the value exceeds the threshold, bonus points are awarded for the final sampling.

[0085] Wherein, the mutual exclusion of strategies means:

[0086] Minority sampling strategies cannot coexist with other strategies.

[0087] If the estimated recall rate of the minority class in the previous round is greater than the threshold, it will not be selected again in this round.

[0088] In the step of selecting a sampling strategy, the sampling strategy is selected from one or more of the following: diversity sampling, uncertainty sampling, probability stratified sampling, minority class sampling, and sweeping sampling strategies.

[0089] The diversity sampling strategy refers to prioritizing data with low labeling rates or containing uncovered effective words, based at least on two dimensions: clustering and effective words.

[0090] The uncertainty sampling strategy refers to: sampling based on the uncertainty of unlabeled data. i Sort the data in reverse order, prioritizing data with high uncertainty;

[0091] The probability stratified sampling strategy refers to: based on the predicted label and score of unlabeled data, for each label, the unlabeled data predicted as the label are stratified according to the predicted score scorei, and a certain amount of data is randomly extracted from each stratum;

[0092] In the probability stratified sampling strategy, after the current round of manual annotation is completed, based on the manual annotation results, the score layers with an accuracy rate higher than the second threshold (e.g., 0.8) for each label are counted, and the data located in the layer are labeled with this label and added to the labeled set as machine-amplified data.

[0093] The minority class sampling strategy refers to the following: for labels with few labeled data, based on the model's predicted labels and scores, the data is arranged in reverse order and layered according to the scores, with the width of each layer increasing exponentially, and a certain amount of data is extracted from each layer.

[0094] For minority class sampling strategies, after the current round of manual labeling is completed, the concentration of positive examples of the label in each layer needs to be estimated based on the manual labeling results, so as to obtain the recall rate estimate of the label.

[0095] The tail sampling strategy refers to extracting data from unlabeled data that has a similarity to labeled data less than a third threshold (e.g., less than 0.15) based on the cosine similarity of sentence vectors, while ensuring that the similarity between the extracted data is less than the third threshold (e.g., less than 0.15).

[0096] like Figure 2 As shown, the present invention also discloses a text data annotation system, comprising:

[0097] The feature statistics extraction module is used to extract selected features from the text to be labeled as the current dataset.

[0098] The judgment module is used to determine whether there is labeled data in the current dataset;

[0099] The strategy sampling module is used to select a sampling strategy based on the judgment result of the judgment module and use it to extract and label data from the current dataset.

[0100] The text data annotation system also includes a target coverage detection module, which is used to detect whether the target coverage of the annotated text processed by the sampling annotation module meets the requirements. If not, the feature statistics extraction module, machine learning module and sampling annotation module are called to continuously annotate the remaining unannotated text.

[0101] The text to be labeled may contain a certain amount of pre-labeled data, or it may be completely unlabeled data without seeds.

[0102] The text data annotation system may further include a preprocessing module for preprocessing the current dataset, such as the following operations:

[0103] Hierarchical clustering of text data is performed based on sentence vectors, recording the cluster center (C1, C2, ..., C6) to which each data point belongs. k );

[0104] The data is segmented using a word segmentation tool, and the document frequency of each word is counted. Words with a document frequency greater than a first threshold (e.g., 2) are recorded as the set of valid words.

[0105] The text preprocessing operations of the preprocessing module may include, for example, one or more of the following steps: initial screening of text information, removal of invalid text, word segmentation of text information, and removal of stop words.

[0106] The selected features of the dataset are chosen from the total number of data entries N and the number of labeled entries N. label Label coverage Number of manually labeled entries N manual Number of machine-expanded labels N model Number of tags M tag , Number of data entries for each label N tag Uncertainty of each data point i The number of data entries N with uncertainty greater than the threshold uncertain One or more of the following: historical sampling strategies and labeling information.

[0107] Uncertainty is defined as follows: x represents text, tag i For the i-th label, P θ (tag i |x) represents the model's prediction of text x as a tag. i The probability of.

[0108] The text data annotation system may further include a machine learning module for performing operations such as training a model based on the labeled data and labels and making predictions, including, for example, the following steps:

[0109] A text classifier model is trained based on labeled data and tags, and predictions are made on unlabeled data. The prediction result and tag for each data point are recorded. i and score i ;

[0110] The text classifier model is selected from, for example, LSTM, TextCNN, or BERT models.

[0111] The selection sampling strategy satisfies the following conditions:

[0112] The selected sampling strategies conform to policy disabling, policy voting, and policy mutual exclusion;

[0113] Prioritize one or more strategies that score higher.

[0114] The policy disabling refers to:

[0115] If N label =N, disable all sampling strategies;

[0116] If N label ≤1 or M tag ≤1, stratified sampling, minority class sampling, and uncertainty sampling are disabled;

[0117] The strategy recommendation refers to:

[0118] If the coverage ratio is marked label If the score is below the threshold, stratified probability sampling will award extra points.

[0119] If the number of data points with high uncertainty exceeds the threshold, bonus points will be awarded for uncertainty sampling.

[0120] If the cluster centers or effective vocabulary are not fully covered, the diversity sampling will receive bonus points.

[0121] If the minimum number of tags for a single label If the score is below the threshold, minority class sampling receives bonus points;

[0122] Label coverage ratio label If the value exceeds the threshold, bonus points are awarded for the final sampling.

[0123] Wherein, the mutual exclusion of strategies means:

[0124] Minority sampling strategies cannot coexist with other strategies.

[0125] If the estimated recall rate of the minority class in the previous round is greater than the threshold, it will not be selected again in this round.

[0126] Among them, the various thresholds in the strategy recommendation rules and strategy mutual exclusion rules are all determined based on practical experience and can be fine-tuned for different application scenarios.

[0127] In the step of selecting a sampling strategy, the sampling strategy is selected from one or more of the following: diversity sampling, uncertainty sampling, probability stratified sampling, minority class sampling, and sweeping sampling strategies.

[0128] The diversity sampling strategy refers to prioritizing data with low labeling rates or containing uncovered effective words, based at least on two dimensions: clustering and effective words.

[0129] The uncertainty sampling strategy refers to: sampling based on the uncertainty of unlabeled data. i Sort the data in reverse order, prioritizing data with high uncertainty;

[0130] The probability stratified sampling strategy refers to: based on the predicted labels and scores of unlabeled data, for each label, unlabeled data predicted to be that label are sorted according to the predicted score. i The data is divided into layers, and a certain amount of data (e.g., 5 or 10) is randomly extracted from each layer. After the current round of manual annotation is completed, based on the manual annotation results, the score layers with an accuracy rate higher than the second threshold (e.g., 0.8) for each label are counted, and the data in the layer are labeled with this label and added to the labeled set as machine-amplified data.

[0131] The minority class sampling strategy refers to the following: for labels with few labeled data, based on the model's predicted labels and scores, the data is arranged in reverse order and layered according to the scores, with the width of each layer increasing exponentially, and a certain amount of data is extracted from each layer.

[0132] For the minority class sampling strategy, after the current round of manual labeling is completed, the concentration of positive examples of the label in each layer needs to be estimated based on the manual labeling results, so as to obtain the recall rate estimate of the label.

[0133] The tail sampling strategy refers to extracting data from unlabeled data that has a similarity to labeled data less than a third threshold (e.g., less than 0.15) based on the cosine similarity of sentence vectors, while ensuring that the similarity between the extracted data is less than the third threshold (e.g., less than 0.15).

[0134] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0135] Example 1

[0136] like Figure 5 As shown, the intelligent text annotation method in this embodiment includes the following steps:

[0137] Step 1: Use the text to be labeled as the current dataset and perform text preprocessing on it;

[0138] Step 2: Based on the current data annotation, statistically analyze the various features required for the recommendation strategy;

[0139] Step 3: Determine whether there is labeled data. If so, train a text classifier model based on the labeled data and labels using the BERT algorithm, and use the text classifier model to make predictions on the current dataset.

[0140] Step 4: Recommend a sampling strategy, which is selected based on the following conditions:

[0141] It conforms to the three categories of rules: policy disabling, policy voting, and policy mutual exclusion;

[0142] Prioritize one or more strategies with higher scores;

[0143] Step 5: Extract data according to the sampling strategy;

[0144] Step 6: Manually label the extracted data;

[0145] Step 7: Data aggregation and strategy post-processing;

[0146] Step 8: Determine whether the data annotation for the current dataset has reached the target coverage. If not, proceed to step 2 above; if yes, complete the text data annotation method.

[0147] The code for the above method was uploaded to the company platform and used with actual annotations. The data came from a VOC questionnaire survey, with a data volume of 278,566 entries and unknown initial tags.

[0148] The labeling was carried out in 5 rounds. After each round, the number of labels was 46, 75, 76, 87, and 91 respectively (that is, 46, 29, 1, 11, and 4 new labels appeared in each round of labeling).

[0149] The sampling strategies used in each round were diversity sampling, probability stratification + diversity + uncertainty sampling, minority sampling, probability stratification + diversity sampling, and diversity + cleanup sampling. The number of manually labeled records in each round were 400, 2226, 1289, 2521, and 1542, respectively. In the end, the human and machine labels covered a total of 251,277 data records.

[0150] Therefore, the method of the present invention can effectively perform data annotation scenarios where there is no seed data and the labels are unknown, thus expanding the applicable scope of conventional machine annotation.

[0151] The present invention also discloses an electronic device, Figure 3 This is a schematic diagram of the structure of the electronic device of the present invention, as shown below. Figure 3 As shown, the electronic device includes a processor and a memory, the memory being used to store a computer-executable program, wherein when the computer-executable program is executed by the processor, the processor executes the text data annotation platform sampling strategy recommendation method as described above.

[0152] The electronic device of this invention is embodied in the form of a general-purpose computing device. There may be one or more processors working collaboratively. This invention also does not preclude distributed processing, meaning that processors may be distributed across different physical devices. The electronic device of this invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0153] The memory stores a computer-executable program, typically machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least some steps of the method.

[0154] The memory includes volatile memory, such as random access memory (RAM) and / or cache memory, and may also be non-volatile memory, such as read-only memory (ROM).

[0155] Optionally, in this embodiment, the electronic device further includes an I / O interface for exchanging data with external devices. The I / O interface can represent one or more of several bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0156] It should be understood that Figure 3 The electronic device shown is merely one example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as displays, and some electronic devices also include human-computer interaction elements such as buttons and keyboards. Any electronic device capable of executing a computer-readable program in memory to implement the method of the present invention or at least some steps of the method can be considered as an electronic device covered by the present invention.

[0157] The present invention also discloses a storage medium, Figure 4 This is a schematic diagram of the storage medium of the present invention. (As shown) Figure 4As shown, the storage medium stores a computer-executable program, which, when executed, implements the text data annotation platform sampling strategy recommendation method as described above. The storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The storage medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0158] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Python, Java, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0159] From the above description of the embodiments, those skilled in the art will readily understand that the present invention can be implemented by hardware capable of executing specific computer programs, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. included in the system. The present invention can also be implemented by other electronic devices that include at least a part of the above-mentioned systems or components, such as communication electronic devices, entertainment electronic devices, learning electronic devices, etc. The present invention can also be implemented by computer software that executes the methods of the present invention, for example, by control software executed by the microprocessor of the client, electronic control unit, client, server, etc. However, it should be noted that the computer software that executes the methods of the present invention is not limited to execution in one or a specific hardware entity; it can also be implemented in a distributed manner by unspecified hardware. For example, some method steps executed by the computer program can be executed at the locomotive end, while another part can be executed in the mobile terminal or smart helmet, etc. For computer software, the software product can be stored in a computer-readable storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) or distributed storage on a network, as long as it enables electronic devices to execute the methods according to the present invention.

[0160] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the present invention is not inherently related to any specific computer, virtual device, or electronic device, and various general-purpose devices can also implement the present invention. The above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A text data labeling method, characterized by, The method comprises the following steps: The text to be labeled is taken as a current data set, wherein the text to be labeled contains a certain amount of pre-labeled data or completely unlabeled data without seeds; Selected features of the data set are extracted based on the current data set; It is judged whether there is labeled data in the current data set, and a sampling strategy is selected according to the judgment result, including: if there is no labeled data, the sampling strategy is directly selected; if there is labeled data, a model is trained based on the labeled data and labels, and the current data set is predicted by using the model, and then the sampling strategy is selected; wherein the selection sampling strategy is selected from one or more of the following: diversity sampling, uncertainty sampling, probability stratified sampling, minority class sampling and sweep tail sampling strategy; the selected sampling strategy is used to extract data from the current data set and label the data; It is judged whether the labeled data set reaches a target coverage rate, if yes, the labeling of the text to be labeled is completed, and if no, the selected features of the data set are repeatedly extracted, it is judged whether there is labeled data in the current data set, the sampling strategy is selected according to the judgment result, and the selected sampling strategy is used to extract data from the current data set and label the data, until all steps of judging whether the target coverage rate is reached.

2. The method of claim 1, wherein, It also includes text preprocessing: Hierarchical clustering is performed on the text data based on the sentence vectors, cluster centers (C1, C2, …, C k ) to which each piece of data belongs are recorded, the data is segmented into words, and the document frequency of each word is counted; words with a document frequency greater than a first threshold value are recorded as an effective word set; And / or, One or more of the following steps is performed on the text information: preliminary screening, elimination of invalid text, word segmentation and stop word removal.

3. The method of claim 1, wherein, The selected features of the data set include: selected from a total number of data N, a number of labeled data N label , a labeling coverage a number of artificial labeled data N manual , a number of machine expanded labeled data N model , a number of labels M tag , a number of data labeled by each label N tag , an uncertainty of each data i , a number of data whose uncertainty is greater than a threshold value N uncertain , one or more of a historical sampling strategy set and a labeling situation where the uncertainty is defined as x is the text, tag i is the i-th label, P θ (tag i |x) is the probability that the model predicts the text x as tag i .

4. The method according to any one of claims 1 to 3, characterized in that, The step of training a model based on the labeled data and labels and using the model to predict the current data set includes: Training a text classifier model based on the labeled data and labels, and using it to make predictions on the current dataset includes: making predictions on the unlabeled data, recording the predicted result label tag and score for each piece of data i i ;​ And / or, the text classifier model is selected from LSTM, TextCNN or BERT model.

5. The method of claim 4, wherein, The selection sampling strategy satisfies the following conditions: The selected sampling strategy meets three types of rules: strategy disabling, strategy voting and strategy mutual exclusion; and one or more strategies with higher scores are preferentially selected; Wherein, the policy disabling refers to: if N label =N, all sampling policies are disabled; if N label ≤1 or M tag ≤1, probability stratified sampling, minority class sampling and uncertainty sampling are disabled. And / or, Strategy voting refers to: if the annotation coverage ratio label is less than a threshold, then probability stratified sampling is added, if the number of data with high uncertainty is higher than a threshold, then uncertainty sampling is added, if the cluster center or the effective word table is not completely covered, then diversity sampling is added, if the minimum number of single label annotations is less than a threshold, then minority class sampling is added, and if the annotation coverage ratio label is greater than a threshold, then sweep tail sampling is added. And / or, The strategy mutual exclusion means that the minority class sampling strategy and other strategies cannot exist at the same time, and if the estimated recall rate is greater than a threshold in the previous round of minority class sampling, the minority class sampling is not selected in the current round.

6. The method of claim 1, wherein The diversity sampling strategy means that at least based on two dimensions of clustering cluster and effective word, data with low annotation rate of the clustering cluster or containing effective words not covered are preferentially selected; and / or The uncertainty sampling strategy refers to: according to the uncertainty uncertainty of the unlabeled data i The data is arranged in reverse order, and the data with large uncertainty is preferentially selected; and / or, The probability stratified sampling strategy is: based on the predicted label and score of the unlabeled data, for each label, the unlabeled data predicted as the label is stratified according to the predicted score score i , and a certain amount of data is randomly selected from each layer; after the current round of manual labeling is completed, according to the manual labeling result, the score layer with an accuracy rate higher than the second threshold value for each label is counted, and the data located in the layer is marked with this label as machine expansion label data and added to the labeled set; and / or, The minority class sampling strategy means that for labels with less labeled data, based on the predicted label and score of the data by the model, the data is inversely arranged and stratified according to the score, the width of each layer is increased according to an exponential function, and a certain amount of data is extracted from each layer; and / or The sweep tail sampling strategy means that according to the cosine similarity of sentence vectors, data with a similarity less than a third threshold to the labeled data is extracted from the unlabeled data, while the similarity between the extracted data is required to be less than the third threshold.

7. The method of claim 6, wherein, For the minority class sampling strategy, it also includes: after the current round of manual labeling is completed, the label positive concentration in each layer is estimated according to the manual labeling result, so as to obtain the recall rate estimation of the label.

8. A text data labeling system, characterized by, It includes: The feature statistics extraction module is configured to extract selected features of the current data set by taking the text to be labeled as the current data set, wherein the text to be labeled contains a certain amount of pre-labeled data or completely unlabeled data without seeds. The judgment module is configured to judge whether the current data set contains labeled data. The strategy sampling module is configured to select a sampling strategy according to the judgment result of the judgment module, including directly selecting a sampling strategy if there is no labeled data, and selecting a sampling strategy after predicting the current data set based on a model trained by the labeled data and labels if there is labeled data, wherein the selection of the sampling strategy includes one or more of the following: diversity sampling, uncertainty sampling, probability stratified sampling, minority class sampling, and sweep tail sampling strategy, and the selected sampling strategy is used to label the data extracted from the current data set, and it is judged whether the labeled data set reaches the target coverage rate, if yes, the text to be labeled is labeled, if not, the selected sampling strategy is repeatedly selected according to the judgment result of the judgment module, and the data extracted from the current data set is labeled until all steps of judging whether the target coverage rate is reached. 9.An electronic device comprising a processor and a memory, the memory being configured to store a computer executable program, characterized in that: When the computer executable program is executed by the processor, the processor executes the text data labeling method according to any one of claims 1-7.

10. A computer readable medium storing a computer executable program, characterized in that, When the computer executable program is executed, the text data labeling method according to any one of claims 1-7 is implemented.

Images