Text data labeling method and device, electronic equipment, and storage medium
By combining diversity and similarity sampling with self-learning and active learning methods, the data annotation problem under unlabeled sets and seedless data is solved, achieving efficient text data annotation and expanding the application scenarios of the data annotation platform.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING XUENA BAICHUAN EDUCATION TECHNOLOGY CO LTD
- Filing Date
- 2021-05-24
- Publication Date
- 2026-06-09
AI Technical Summary
Existing intelligent data annotation platforms are limited in use when the label set is unknown or there is no seed data, making them unable to effectively annotate data and limiting their application scenarios.
By combining diversity sampling and similarity sampling strategies with self-learning and active learning, representative data is selected for manual annotation through text clustering, preprocessing, and model training. This removes restrictions on the label set and seed data, thereby improving annotation efficiency.
It achieves efficient data annotation in the absence of labeled sets and seed data, expands the applicable scope of machine annotation, lowers the threshold for use, and improves the efficiency of manual annotation.
Smart Images
Figure CN113297351B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of machine learning technology, specifically relating to a text data annotation method and apparatus, electronic device and storage medium. Background Technology
[0002] With the development of the internet and artificial intelligence, the demand for data annotation services is increasing. Data annotation has also evolved from purely manual annotation to machine annotation that combines manual annotation with active learning. Currently available intelligent data annotation platforms generally have certain usage conditions, such as a known label set and a certain amount of seed data for each label. However, these limitations do not hold true in many situations. For example, with entirely new data, users may not know the label set and seed data before completing annotation, which to some extent limits the application scenarios of the annotation platform. Summary of the Invention
[0003] In view of this, the main objective of the present invention is to provide a text data annotation method and apparatus, electronic device and storage medium, in order to at least partially solve at least one of the above-mentioned technical problems.
[0004] To achieve the above objectives, as a first aspect of the present invention, a text data annotation method is provided, comprising the following steps:
[0005] Use the text to be labeled as the current dataset;
[0006] Determine if labeled data exists in the current dataset; if so, expand the labeling of the labeled data.
[0007] Data is extracted and labeled from the current dataset using diversity sampling and similarity sampling strategies;
[0008] Calculate the coverage of the labeled text in the current dataset and compare it with the target coverage. If the target coverage is not reached, repeat the above expansion and labeling operations.
[0009] As a second aspect of the present invention, a text data annotation apparatus is also provided, comprising the following steps:
[0010] The preprocessing module is used to preprocess the text to be labeled as the current dataset;
[0011] The label expansion module is used to determine whether there is already labeled data in the current dataset. If so, it expands the label data with high confidence.
[0012] The sampling and labeling module is used to process the current dataset according to preset dimensions, and based on the processing results, to extract and label data from the current dataset using diversity sampling and similarity sampling strategies.
[0013] The target coverage detection module is used to detect the target coverage of the current dataset. If the target coverage is not reached, the expansion and labeling modules are called to expand and label the remaining unlabeled data.
[0014] As a third aspect of the present invention, an electronic device is also provided, including a processor and a memory, the memory being used to store a computer-executable program, wherein when the computer-executable program is executed by the processor, the processor performs the text data annotation method as described above.
[0015] As a fourth aspect of the present invention, a computer-readable medium is also provided, which stores a computer-executable program that, when executed, implements the text data annotation method described above.
[0016] Based on the above technical solution, it can be seen that the text data annotation method and apparatus of the present invention have at least one of the following beneficial effects compared with the prior art:
[0017] This invention is based on artificial intelligence technologies such as text clustering, self-learning, and active learning. It selects the most representative and informative data and assigns it to manual annotation. During the annotation process, it takes into account the expansion of historical label data and the discovery of new labels. It removes the restrictions on label sets and seed data, improves the efficiency of manual annotation, and effectively lowers the threshold for use.
[0018] The method of this invention can perform data annotation scenarios where there is no seed data and the labels are unknown, thus expanding the applicable scope of conventional machine annotation. Attached Figure Description
[0019] Figure 1 This is a block flowchart of the text data annotation method of the present invention;
[0020] Figure 2 This is a schematic diagram of the framework of the text data annotation device of the present invention;
[0021] Figure 3 This is a schematic diagram of the structure of the electronic device of the present invention;
[0022] Figure 4 This is a schematic diagram of the storage medium of the present invention;
[0023] Figure 5 This is a flowchart of the text data annotation method according to Embodiment 1 of the present invention. Detailed Implementation
[0024] In the description of specific embodiments, detailed descriptions of structures, performance, effects, or other features are provided to enable those skilled in the art to fully understand the embodiments. However, this does not preclude those skilled in the art from implementing the present invention with technical solutions that do not contain the aforementioned structures, performance, effects, or other features under specific circumstances.
[0025] The flowchart in the accompanying drawings is merely an exemplary process demonstration and does not imply that the solution of this invention must include all the content, operations, and steps in the flowchart, nor does it imply that they must be executed in the order shown in the diagram. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. Without departing from the inventive spirit of this invention, the execution order shown in the flowchart can be changed according to the actual situation.
[0026] The box in the attached diagram Figure 1 Generally, these refer to functional entities, and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processing unit devices and / or microcontroller devices.
[0027] The same reference numerals in the accompanying drawings denote the same or similar elements, components, or parts, and therefore, repeated descriptions of the same or similar elements, components, or parts may be omitted below. It should also be understood that although terms such as first, second, third, etc., indicating numbers may be used herein to describe various devices, elements, components, or parts, these devices, elements, components, or parts should not be limited by these terms. That is, these terms are only used to distinguish one from another. For example, a first device may also be referred to as a second device, without departing from the essential technical solution of the invention. Furthermore, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.
[0028] The meanings of some technical terms in this manual are as follows:
[0029] Clustering is the process of dividing a collection of physical or abstract objects into multiple classes composed of similar objects. Traditional clustering analysis methods mainly include: partitioning methods (such as the K-MEANS algorithm), hierarchical methods, density-based methods, grid-based methods, and model-based methods. Of course, there are also other clustering methods such as transitive closure, Boolean matrix method, direct clustering, correlation analysis clustering, and statistical clustering methods.
[0030] Stratified sampling is a type of probability sampling that includes four types: simple random sampling, systematic sampling, stratified sampling, and cluster sampling.
[0031] Text similarity, as the name suggests, refers to the degree of similarity between two texts. For example, in a question-answering system, the system prepares some classic questions and corresponding answers. When a user's question is very similar to a classic question, the system directly returns the prepared answer. During corpus preprocessing, it's necessary to identify and remove duplicate texts based on text similarity. In short, text similarity is a very useful tool. Measuring text similarity includes three main methods: first, traditional methods based on keyword matching, such as N-gram similarity; second, mapping text to a vector space and then using methods like cosine similarity; and third, deep learning methods, such as the deep learning semantic matching model DSSM based on user click data, ConvNet based on convolutional neural networks, and the currently state-of-the-art Siamese LSTM.
[0032] In active learning machine annotation methods, seed data refers to pre-annotated data that can be used as seeds for imitation and expansion.
[0033] Document frequency (DF) refers to the number of texts in the entire dataset that contain a particular word. DF measures the number of documents by calculating a linear approximation of the number of training documents. Its low computational complexity makes it applicable to any corpus, thus it is a common method for feature dimensionality reduction.
[0034] There has been considerable research on automatic annotation methods based on active learning, but these methods all require certain seed data or can only be used in specific scenarios. This invention proposes a text intelligent annotation method that can select the most representative and informative data for manual annotation based on artificial intelligence technologies such as text clustering, self-learning, and active learning. During the annotation process, it takes into account both the expansion of historical label data and the discovery of new labels, eliminates the restrictions on label sets and seed data, improves the efficiency of manual annotation, and effectively lowers the threshold for use.
[0035] like Figure 1 As shown, the text data annotation method based on self-training and active learning of the present invention includes the following steps:
[0036] Use the text to be labeled as the current dataset;
[0037] Determine if labeled data exists in the current dataset; if so, expand the labeling of the labeled data.
[0038] Data is extracted and labeled from the current dataset using diversity sampling and similarity sampling strategies;
[0039] Calculate the coverage of the labeled text in the current dataset and compare it with the target coverage. If the target coverage is not reached, repeat the above expansion and labeling operations.
[0040] The text to be labeled can be various types of text data, such as the question and answer text in an after-sales response system, text in news or social media articles, text in personal blogs and microblogs related to personality profiling, and so on. The text to be labeled can contain a certain amount of pre-labeled data, or it can be completely unlabeled data without any seed data.
[0041] The method further includes a preprocessing step for the current dataset, such as initial screening of text data to remove invalid text. The purpose of this preprocessing is to reduce the interference of invalid text on clustering and labeling operations, thereby improving labeling accuracy.
[0042] Before the steps of extracting and labeling data using diversity sampling and similarity sampling strategies, the dataset is further processed according to preset dimensions, for example, including:
[0043] The preprocessed text is segmented, stop words are removed, and a valid vocabulary is obtained.
[0044] The preprocessed texts are clustered to obtain the similarity between texts and the representative texts of each cluster.
[0045] These two steps also need to be selected based on the state of the input text data itself. If it is raw chat logs or Weibo / WeChat text, it needs to be processed through this preset dimension to obtain the final dataset that retains only the effective words from the clustering. The purpose of word segmentation and stop word removal is to reduce the interference of invalid words and improve the accuracy of annotation. The purpose of clustering is to classify the same or similar words, reducing the repetitive work of annotation.
[0046] The clustering step is implemented, for example, through the following steps:
[0047] The sentence vector (e1, e2, ..., e) of each text is calculated using a sentence vector model pre-trained on a domain-specific corpus. k );
[0048] Text-based sentence vector cosine distance Perform hierarchical clustering and save the cluster center (C1, C2, ..., C6) of each sample in the clustering results. k ); where x i Dist(x) represents the i-th text. i ,x j ) represents x i x j The distance between them;
[0049] The cluster partitioning requires that the average cosine distance between two clusters be greater than a first threshold (e.g., greater than 0.15). The average distance between two clusters is defined as follows: |C i | Represents cluster center C i Cluster size:
[0050]
[0051] The sentence vector model pre-trained on domain-specific corpora can be FastText, BERT, or similar models. Domain-specific corpora include corpora from service communication dialogues, online education, secondary school knowledge, and university knowledge.
[0052] The steps for obtaining a valid vocabulary are implemented as follows:
[0053] Perform word segmentation on the text data, remove stop words, and record the word segmentation results (words). i ;
[0054] Document frequency (DF) of words and tuples;
[0055] Record words and pairs with DF>2 as the valid word set Vocab.
[0056] The steps for expanding the labels on already labeled data include, for example:
[0057] Train the model and make predictions on the remaining unlabeled data;
[0058] Data were manually verified using probability stratified sampling.
[0059] Based on the results of manual verification, high-confidence data expansion is performed, and the expanded data is added to the labeled set.
[0060] The models used here are, for example, TextCNN, LSTM, or BERT text classification models.
[0061] Considering the difference in accuracy between manually labeled and machine-labeled data, the machine-labeled data needs to be downweighted during training. The specific value can be determined based on experience from multiple trials.
[0062] In the step of predicting the remaining data, the prediction result label y for each data point is recorded. i and score i ;
[0063] The probability stratified sampling refers to:
[0064] For each label, the unlabeled data predicted as that label is determined based on the prediction score obtained when the model predicts the unlabeled data. i The data is stratified, and n_samples are randomly selected from each stratum and added to the annotation set.
[0065] The high-confidence scaling refers to:
[0066] After each round of annotation, for the probability stratified sampling data, based on the manual annotation results, the layers whose prediction accuracy meets the third threshold (e.g., 0.8) are statistically analyzed.
[0067] Unlabeled data that are predicted to be the label and whose predicted scores are located in the layer are labeled with the label and added to the labeled set as machine-labeled data.
[0068] Here, diversity sampling refers to:
[0069] Diversity sampling is based on two dimensions: clustering and effective words. It counts the cluster centers C covered by the currently labeled data. covered And the effective word set Vocabulary covered ;
[0070]
[0071]
[0072] Where, x i Let be the i-th text, and let word be the words contained in that text;
[0073] The clusters are arranged in reverse order of cluster size. Data to be labeled is extracted based on the following two rules, and a certain amount of data that meets the conditions is added to the set to be labeled:
[0074] The data belongs to a cluster where the label coverage is less than 0.5.
[0075] The data includes data not belonging to Vocabul. covered Effective words;
[0076] The similarity sampling refers to:
[0077] Considering that the number of samples when new labels appear is extremely small, the data cannot be effectively expanded by probability stratified sampling. Therefore, for labels with fewer than the preset number of labels, unlabeled data located in the same cluster or with a cosine similarity greater than the second threshold (e.g., greater than 0.7) are selected and added to the set to be labeled.
[0078] If, at the end of the method, the coverage of the annotated text reaches the target coverage, the method further includes: a step of selecting representative text from the remaining small amount of unannotated data for annotation based on clustering and / or effective word information (tail representative sample sampling).
[0079] The step of selecting representative samples from the remaining unlabeled data for labeling follows two rules:
[0080] The data contains valid words that were not covered;
[0081] The maximum cosine similarity between the sentence vectors of the data and the labeled data is below the fourth threshold (e.g., below 0.5).
[0082] The steps of determining whether labeled data exists and extracting and labeling data using diversity sampling and similarity sampling strategies can be executed simultaneously using parallel algorithms, so the two steps have no sequential order. Therefore, a data aggregation step is required to summarize the processing results of the two steps before the step of calculating the coverage of labeled text in the current dataset is executed.
[0083] Following the step of summarizing the results of the above steps, a post-policy processing step can be performed. The purpose of summarizing is to ensure data accuracy by accumulating all labeled data into the final coverage metric. The post-policy processing mainly refers to the calculation and processing of data after extraction for some sampling strategies. For example, minority class sampling strategies require recall estimation and calculation of other relevant parameters. The recall estimation is already included in the minority class sampling strategy steps, while other calculation and processing steps are not essential but only necessary in specific situations, and therefore can be placed within the post-policy processing.
[0084] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.
[0085] Example 1
[0086] like Figure 5 As shown, the text intelligent annotation method based on self-training and active learning in this embodiment includes the following steps:
[0087] Step 1: Perform an initial screening of the text that needs to be annotated, and remove invalid text;
[0088] Step 2: Perform word segmentation and stop word removal on the text obtained in Step 1, and compile a list of effective words.
[0089] Step 3: Cluster the text obtained in Step 1, and statistically analyze the similarity between texts and the representative text of each cluster;
[0090] Step 4: Execute the following two steps in parallel:
[0091] If labeled data exists, proceed to sub-steps 4.1-4.3;
[0092] Sub-step 4.1: Train the model and make predictions on the remaining data;
[0093] Sub-step 4.2: Use probability stratified sampling to extract data for manual verification;
[0094] Sub-step 4.3: Based on the manual verification results, perform high-confidence expansion and add the expanded data to the labeled set;
[0095] Two strategies, diversity sampling and similarity sampling, were used to extract data, which were then manually labeled.
[0096] Step 6: Summarize the label and annotation results generated by the two parallel branches in Step 4;
[0097] Step 7: By repeating steps 4 and 6, continuously annotate the remaining unannotated text until the target coverage rate is achieved, then proceed to step 8.
[0098] Step 8: Select representative texts from the remaining small amount of unlabeled data and label them based on information such as clusters and effective words.
[0099] The code for the above method was uploaded to the platform and used with actual annotations. The data came from the VOC questionnaire survey, with a data volume of 245,051 entries and unknown initial tags.
[0100] The labeling was carried out in 4 rounds. After each round, the number of labels was 34, 69, 89, and 92 respectively (that is, 34, 35, 20, and 3 new labels appeared in each round of labeling).
[0101] The number of manually labeled records in each round were 400, 1459, 2450, and 2484, respectively, with a total of 229,816 records covered by human and machine labels.
[0102] Example 2
[0103] The text intelligent annotation method based on self-training and active learning in this embodiment includes the following steps:
[0104] Step 1: Perform an initial screening of the text that needs to be annotated, and remove invalid text;
[0105] Step 2: Perform word segmentation and stop word removal on the text obtained in Step 1, and compile a list of effective words.
[0106] Step 3: Cluster the text obtained in Step 1, and statistically analyze the similarity between texts and the representative text of each cluster;
[0107] Step 4: If labeled data exists, proceed to sub-steps 4.1-4.3;
[0108] Sub-step 4.1: Train the model and make predictions on the remaining data;
[0109] Sub-step 4.2: Use probability stratified sampling to extract data for manual verification;
[0110] Sub-step 4.3: Based on the manual verification results, perform high-confidence expansion and add the expanded data to the labeled set;
[0111] Step 5: Extract data using both diversity sampling and similarity sampling strategies and manually label the data;
[0112] Step 6: Summarize the labels and annotations generated in Steps 4 and 5;
[0113] Step 7: By repeating steps 4, 5, and 6, continuously annotate the remaining unannotated text until the target coverage rate is achieved, then proceed to step 8.
[0114] Step 8: Select representative texts from the remaining small amount of unlabeled data and label them based on information such as clusters and effective words.
[0115] Therefore, it can be seen that the difference between Example 2 and Example 1 lies only in the expansion and annotation steps (steps 4 and 5). One involves parallel algorithms executing simultaneously, while the other involves sequential execution. Both approaches can achieve the purpose of this invention. Furthermore, the method of this invention can effectively perform data annotation scenarios where there is no seed data and the labels are unknown, thus expanding the applicable scope of conventional machine annotation.
[0116] This invention also discloses a text data annotation system based on self-training and active learning, comprising:
[0117] The preprocessing module is used to preprocess the text to be labeled as the current dataset as needed;
[0118] The label expansion module is used to determine whether there is already labeled data in the current dataset. If so, it expands the label of the already labeled data.
[0119] The sampling and labeling module uses diversity sampling and similarity sampling strategies to extract and label data from the current dataset.
[0120] The target coverage detection module is used to detect the target coverage of the current dataset. If the target coverage is not reached, the expansion and labeling modules are called to expand and label the remaining unlabeled data.
[0121] The text to be labeled contains a certain amount of pre-labeled data, or it may be completely unlabeled data without seeds.
[0122] The preprocessing module performs preprocessing operations on the current dataset, including, for example, initial screening of text data to remove invalid text.
[0123] The sampling and labeling module, before the steps of extracting and labeling data using diversity sampling and similarity sampling strategies, also includes operations that process the current dataset according to preset dimensions, such as:
[0124] The preprocessed text is segmented, stop words are removed, and a valid vocabulary is obtained.
[0125] The preprocessed texts are clustered to obtain the similarity between texts and the representative texts of each cluster.
[0126] The clustering step is implemented, for example, through the following steps:
[0127] The sentence vector (e1, e2, ..., e) of each text is calculated using a sentence vector model pre-trained on a domain-specific corpus. k );
[0128] Text-based sentence vector cosine distance Perform hierarchical clustering and save the cluster center (C1, C2, ..., C6) of each sample in the clustering results. k ); where x i Dist(x) represents the i-th text. i ,x j ) represents x i x j The distance between them;
[0129] The cluster partitioning requires that the average cosine distance between two clusters be greater than a first threshold (e.g., greater than 0.15). The average distance between two clusters is defined as follows: |C i | Represents cluster center C i Cluster size:
[0130]
[0131] The sentence vector model pre-trained on domain-specific corpora can be FastText, BERT, or similar models. Domain-specific corpora include corpora from service communication dialogues, online education, secondary school knowledge, and university knowledge.
[0132] The steps for obtaining a valid vocabulary are implemented as follows:
[0133] Perform word segmentation on the text data, remove stop words, and record the word segmentation results (words). i ;
[0134] Document frequency (DF) of words and tuples;
[0135] Record words and pairs with DF>2 as the valid word set Vocab.
[0136] The steps for expanding the labels on already labeled data include, for example:
[0137] Train the model and make predictions on the remaining unlabeled data;
[0138] Data were manually verified using probability stratified sampling.
[0139] Based on the results of manual verification, high-confidence data expansion is performed, and the expanded data is added to the labeled set.
[0140] The models trained here are, for example, TextCNN, LSTM, or BERT text classification models.
[0141] In particular, considering the difference in accuracy between manually labeled and machine-labeled data, the machine-labeled data needs to be downweighted during the training process.
[0142] In the step of predicting the remaining data, the prediction result label y for each data point is recorded. i and score i ;
[0143] The probability stratified sampling refers to:
[0144] For each label, the unlabeled data predicted as that label is determined based on the prediction score obtained when the model predicts the unlabeled data. i The data is stratified, and n_samples are randomly selected from each stratum and added to the annotation set.
[0145] The high-confidence scaling refers to:
[0146] After each round of annotation, for the probability stratified sampling data, based on the manual annotation results, the layers whose prediction accuracy meets the third threshold (e.g., 0.8) are statistically analyzed.
[0147] Unlabeled data that are predicted to be the label and whose predicted scores are located in the layer are labeled with the label and added to the labeled set as machine-labeled data.
[0148] Here, diversity sampling refers to:
[0149] Diversity sampling is based on two dimensions: clustering and effective words. It counts the cluster centers C covered by the currently labeled data. covered And the effective word set Vocabulary covered ;
[0150]
[0151]
[0152] Where, x i Let be the i-th text, and let word be the words contained in that text;
[0153] The clusters are arranged in reverse order of cluster size. Data to be labeled is extracted based on the following two rules, and a certain amount of data that meets the conditions is added to the set to be labeled:
[0154] The data belongs to a cluster where the label coverage is less than 0.5.
[0155] The data includes data not belonging to Vocabul. covered Effective words;
[0156] The similarity sampling refers to:
[0157] Considering that the number of samples when new labels appear is extremely small, the data cannot be effectively expanded by probability stratified sampling. Therefore, for labels with fewer than the preset number of labels, unlabeled data located in the same cluster or with a cosine similarity greater than the second threshold (e.g., greater than 0.7) are selected and added to the set to be labeled.
[0158] When the target coverage detection module calls the expansion module and the sampling annotation module to perform expansion and annotation, it can, for example, call the expansion module first to perform expansion and then call the sampling annotation module to perform annotation operations; or it can call the two modules to perform operations simultaneously through a parallel algorithm.
[0159] If the coverage of the labeled text calculated by the target coverage detection module has reached the target coverage, the sampling labeling module will also select representative text from the remaining small amount of unlabeled data for labeling based on cluster and / or effective word information (tail representative sample sampling).
[0160] The selection of representative samples from the remaining unlabeled data for labeling follows two rules:
[0161] The data contains valid words that were not covered;
[0162] The maximum cosine similarity between the sentence vectors of the data and the labeled data is below the fourth threshold (e.g., below 0.5).
[0163] The present invention also discloses an electronic device comprising a processor and a memory for storing a computer-executable program, wherein when the computer-executable program is executed by the processor, the processor performs the text data annotation method as described above.
[0164] Figure 3 This is a schematic diagram of the structure of the electronic device of the present invention, as shown below. Figure 3 As shown, the electronic device is embodied in the form of a general-purpose computing device. There can be one or more processors working collaboratively. This invention also does not preclude distributed processing, meaning that processors can be distributed across different physical devices. The electronic device of this invention is not limited to a single entity, but can also be the sum of multiple physical devices.
[0165] The memory stores a computer-executable program, typically machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least some steps of the method.
[0166] The memory includes volatile memory, such as random access memory (RAM) and / or cache memory, and may also be non-volatile memory, such as read-only memory (ROM).
[0167] Optionally, in this embodiment, the electronic device further includes an I / O interface for exchanging data with external devices. The I / O interface can represent one or more of several bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.
[0168] It should be understood that Figure 3 The electronic device shown is merely one example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as displays, and some electronic devices also include human-computer interaction elements such as buttons and keyboards. Any electronic device capable of executing a computer-readable program in memory to implement the method of the present invention or at least some steps of the method can be considered as an electronic device covered by the present invention.
[0169] The present invention also discloses a storage medium storing a computer-executable program thereon, wherein when the computer-executable program is executed, it implements the text data annotation method described above. Figure 4 This is a schematic diagram of the storage medium of the present invention. (As shown) Figure 4 As shown, a computer-executable program is stored in the storage medium. When executed, the computer-executable program implements the text data annotation method of the present invention as described above. The storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The storage medium may also be any readable medium other than a readable storage medium, which can send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.
[0170] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages such as Python, Java, and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0171] From the above description of the embodiments, those skilled in the art will readily understand that the present invention can be implemented by hardware capable of executing specific computer programs, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. included in the system. The present invention can also be implemented by other electronic devices that include at least a part of the above-mentioned systems or components, such as communication electronic devices, entertainment electronic devices, learning electronic devices, etc. The present invention can also be implemented by computer software that executes the methods of the present invention, for example, by control software executed by the microprocessor of the client, electronic control unit, client, server, etc. However, it should be noted that the computer software that executes the methods of the present invention is not limited to execution in one or a specific hardware entity; it can also be implemented in a distributed manner by unspecified hardware. For example, some method steps executed by the computer program can be executed at the locomotive end, while another part can be executed in the mobile terminal or smart helmet, etc. For computer software, the software product can be stored in a computer-readable storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) or distributed storage on a network, as long as it enables electronic devices to execute the methods according to the present invention.
[0172] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the present invention is not inherently related to any specific computer, virtual device, or electronic device, and various general-purpose devices can also implement the present invention. The above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A text data annotation method, characterized in that, Includes the following steps: The text to be labeled is used as the current dataset, and the current dataset is preprocessed. Determine whether there is labeled data in the current dataset. If so, perform high-confidence expansion on the labeled data, including: training the model and predicting the remaining unlabeled data, using probability stratified sampling to extract data for manual verification, performing high-confidence expansion based on the manual verification results, and adding the expanded data to the labeled set. The current dataset is processed according to preset dimensions. Based on the processing results, diversity sampling and similarity sampling strategies are used to extract and label data from the current dataset, including: The preprocessed text is segmented and stop words are removed to obtain an effective vocabulary. The preprocessed text is also clustered to obtain the similarity between texts and the representative text of each cluster. The clustering includes: calculating the sentence vector for each text using a sentence vector model pre-trained on a domain-specific corpus. Text-based sentence vector cosine distance Perform hierarchical clustering and save the cluster center of each sample in the clustering results. ,in, Indicates the first Item text, express x i , x j The distance between the two clusters must be greater than a first threshold, and the average distance between the two clusters must be greater than a first threshold. The average distance between the two clusters is defined as follows: Indicates cluster center Cluster size: ； The diversity sampling includes: based on two dimensions, clustering and effective words, the diversity sampling counts the cluster centers covered by the currently labeled data. and the set of effective words ; ； ； in, For the words contained in this text, Indicates the cluster center. Indicates the cluster centers covered by the currently labeled data; The clusters are arranged in reverse order of cluster size, based on the data's cluster coverage being less than 0.5 and the data containing elements that do not belong to the cluster. These two rules for valid words are used to extract the data to be labeled, and a certain amount of data that meets the conditions is added to the set to be labeled. as well as, The similarity sampling refers to: for labels with fewer than a preset number of labels, selecting unlabeled data that are located in the same cluster or have a cosine similarity greater than a second threshold and adding them to the unlabeled set; Specifically, the judgment and the data extraction and labeling are performed simultaneously using a parallel algorithm, and the data is summarized after execution. Calculate the coverage of labeled text in the current dataset and compare it with the target coverage. If the target coverage is not reached, repeat the above expansion and labeling operations. Otherwise, select representative texts from the remaining small amount of unlabeled data for labeling based on cluster and / or effective word information.
2. The method according to claim 1, characterized in that, The text to be labeled includes: a certain amount of pre-labeled data, or completely unlabeled data without seeds.
3. The method according to claim 1, characterized in that, Preprocessing of the current dataset includes: Perform initial screening on the text data to remove invalid text.
4. The method according to claim 1, characterized in that, Also includes: The sentence vector model pre-trained on the domain-specific corpus is either the FastText model or the BERT model; and / or, the domain-specific corpus includes corpus from service communication dialogues, online education, secondary school knowledge and / or university knowledge domains.
5. The method according to any one of claims 1-4, characterized in that, The statistically obtained effective vocabulary includes: Perform word segmentation and stop word removal on the text data, and record the segmentation results. ; Document frequency (DF) of words and tuples; Record The words and tuples are used as the effective word set. .
6. The method according to claim 1, characterized in that, Also includes: The trained model is a TextCNN, LSTM, or BERT text classification model; And / or, During training, it is necessary to reduce the weight of the data that has been augmented by the machine.
7. The method according to claim 1, characterized in that, Also includes: In the step of predicting the remaining data, the prediction result label for each data point is recorded. and scores .
8. The method according to claim 1, characterized in that, Also includes: The probabilistic stratified sampling refers to: for each label, using the prediction score obtained when predicting unlabeled data that is predicted to be that label based on the model. Divide into layers and randomly select from each layer. Add the item to the set to be labeled; and / or, The high-confidence amplification refers to the following: after each round of labeling, for the data sampled by probability stratification, based on the results of manual labeling, statistically analyze the layers whose prediction accuracy meets the third threshold; the unlabeled data that are predicted to be the label and whose prediction scores are located in the layer are labeled with the label and added to the labeled set as machine-amplified data.
9. The method according to claim 1, characterized in that, Also includes: The step of selecting representative samples from the remaining unlabeled data for labeling follows two rules: the data contains valid words that have not been covered; and the sentence vectors of the data have a maximum cosine similarity to the labeled data that is below a fourth threshold.
10. A text data annotation device, characterized in that, include: The preprocessing module is used to preprocess the text to be labeled as the current dataset; The label expansion module is used to determine whether there is labeled data in the current dataset. If so, it expands the labeled data with high confidence, including: training the model, predicting the remaining unlabeled data, using probability stratified sampling to extract data for manual verification, expanding the label with high confidence based on the manual verification results, and adding the expanded data to the labeled set. The sampling and labeling module is used to process the current dataset according to preset dimensions, and based on the processing results, extract and label data from the current dataset using diversity sampling and similarity sampling strategies, including: The preprocessed text is segmented and stop words are removed to obtain an effective vocabulary. The preprocessed text is also clustered to obtain the similarity between texts and the representative text of each cluster. The clustering includes: calculating the sentence vector for each text using a sentence vector model pre-trained on a domain-specific corpus. Text-based sentence vector cosine distance Perform hierarchical clustering and save the cluster center of each sample in the clustering results. ,in, Indicates the first Item text, express x i , x j The distance between the two clusters must be greater than a first threshold, and the average distance between the two clusters must be greater than a first threshold. The average distance between the two clusters is defined as follows: Indicates cluster center Cluster size: ； The diversity sampling includes: based on two dimensions, clustering and effective words, the diversity sampling counts the cluster centers covered by the currently labeled data. and the set of effective words ; ； ； in, For the words contained in this text, Indicates the cluster center. Indicates the cluster centers covered by the currently labeled data; The clusters are arranged in reverse order of cluster size, based on the data's cluster coverage being less than 0.5 and the data containing elements that do not belong to the cluster. Two rules are used to extract data to be labeled, and a certain amount of data that meets the conditions is added to the set to be labeled. as well as, The similarity sampling refers to: for labels with fewer than a preset number of labels, selecting unlabeled data that are located in the same cluster or have a cosine similarity greater than a second threshold and adding them to the unlabeled set; The expansion and sampling labeling modules are processed simultaneously using parallel algorithms, and the data from both modules are aggregated after each processing step. The target coverage detection module is used to detect the target coverage of the current dataset. If the target coverage is not reached, the expansion and sampling annotation modules are called to expand and annotate the remaining unlabeled data. Otherwise, representative texts are selected from the remaining small amount of unlabeled data for annotation based on cluster and / or effective word information.
11. An electronic device comprising a processor and a memory, the memory being used to store a computer-executable program, characterized in that: When the computer executable program is executed by the processor, the processor performs the text data annotation method as described in any one of claims 1-9.
12. A computer-readable medium storing a computer-executable program, characterized in that, When the computer-executable program is executed, it implements the text data annotation method as described in any one of claims 1-9.