A data semantic labeling method, device and medium

Abstract

The application relates to the technical field of natural language processing, and discloses a data semantic labeling method and device and a medium, the method comprising training a classification model by using data samples of a training set to predict labels of data samples of a test set; calculating the confidence of the labels of the data samples in the test set; and according to the confidence of the labels of the data samples, updating the abnormal sample cache list through an abnormal sample cache list updating mechanism, so that a pre-trained semantic labeler learns the labeling preference knowledge of people for abnormal samples online, whether a data sample is an abnormal sample can be judged according to the uncertainty of the predicted labels of the data sample by the classification model, and abnormal samples can be added or deleted to a preset abnormal sample cache list, data samples that are difficult to process or are incorrectly processed by the classification model can be effectively screened out, a large language model is used to perform semantic labeling on the data samples, text is generated in a given context, and a semantic label is output, and the accuracy and efficiency of the abnormal sample cache list can be improved.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a data semantic annotation method, device, and medium. Background Technology

[0002] Text classification is a crucial task in natural language processing, with widespread practical applications. It primarily faces two challenges: unclear categories in the data source and insufficient labeled data. To address these issues, a common approach is to extract a portion of the data from the data source, manually label it, and then initially categorize the data based on the labeling results. This process is repeated until a sufficient amount of data is available for each category, which can then be used to train a relatively good text classification model. While this method is simple to implement, it is inefficient, requires significant human resources, and can delay model deployment.

[0003] To overcome this difficulty, few-shot learning and active learning methods are commonly used to label data samples. Few-shot learning methods use special algorithms to train a data labeling model with a small amount of data to achieve a high accuracy rate; active learning methods continuously and rapidly update the data labeling model, enabling it to reach a better performance in a shorter time.

[0004] However, while active learning uses metric algorithms to filter data to be labeled, it still requires multiple rounds of manual labeling, which is inefficient. Few-shot learning uses a small amount of labeled data to automatically label unlabeled data, but the model accuracy is low and it also requires a lot of manual review. Summary of the Invention

[0005] To overcome the shortcomings of low accuracy and low efficiency in existing data semantic annotation methods, this invention proposes the following technical solution:

[0006] In the first aspect, this invention proposes a data semantic annotation method, comprising:

[0007] The dataset is divided into training and testing sets according to a preset ratio.

[0008] A classification model is trained using data samples from the training set, and then the data samples from the test set are input into the trained classification model to predict the labels of the data samples.

[0009] Calculate the confidence score of the label for each data sample in the test set, and determine whether the confidence score of the label for the data sample is greater than the threshold.

[0010] When the confidence level of a data sample's label is greater than the threshold, it is determined whether the data sample exists in the abnormal sample cache list. If it does, the data sample is deleted from the abnormal sample cache list and updated to the dataset. If not, the data sample is directly updated to the dataset.

[0011] When the confidence level of the label of a data sample is not greater than the threshold, the data sample is added to the abnormal sample cache list. It is then determined whether the number of times the data sample exists consecutively in the abnormal sample cache list is less than the threshold. If so, the data sample is semantically labeled using a pre-trained semantic labeler and updated in the dataset. If not, manually labeled the data sample is semantically labeled and updated in the dataset.

[0012] Repeat the above steps until the abnormal sample cache list is empty, and complete the semantic annotation of all data samples.

[0013] In a second aspect, the present invention also proposes a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, performs operations performed by the data semantic annotation method as described in any of the embodiments of the first aspect.

[0014] Thirdly, the present invention also proposes a computer-readable storage medium storing a program that is executed by a processor as performed by the data semantic annotation method described in any of the embodiments of the first aspect.

[0015] The beneficial effects of the present invention include at least the following:

[0016] (1) This invention uses a pre-trained semantic labeler to perform semantic labeling on data samples. It can utilize massive amounts of unlabeled data for pre-training and fine-tuning according to different tasks and domains to generate diverse and rich texts in a given context and output semantic labels. This can effectively reduce the workload and cost of manual labeling and improve the efficiency of data semantic labeling.

[0017] (2) This invention uses an abnormal sample cache list update mechanism to allow the semantic labeler to learn online the labeling preference knowledge of humans for abnormal samples. It can determine whether a data sample is an abnormal sample based on the uncertainty of the classification model's prediction of the data sample label, and add or delete abnormal samples to the preset abnormal sample cache list. This can effectively filter out data samples that are difficult or incorrectly processed by the classification model, dynamically adjust the quantity and quality of data samples in the abnormal sample cache list, and improve the accuracy and efficiency of the abnormal sample cache list. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the data semantic annotation method provided in the embodiments of this application.

[0019] Figure 2 This is an architecture diagram of the recurrent annotation framework based on active learning in the embodiments of this application.

[0020] Figure 3 This is a schematic diagram illustrating the working principle of label prediction using a large language model in the embodiments of this application.

[0021] Figure 4 This is an architecture diagram of the circular annotation framework that combines active learning, semantic annotators, and an abnormal data cache list in the embodiments of this application.

[0022] Figure 5 This is an architectural diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0023] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred technical solutions. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred technical solutions are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0024] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0025] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0026] Specifically, Figure 1 This is a flowchart illustrating the data semantic annotation method provided in the embodiments of this application.

[0027] like Figure 1 As shown, this data semantic annotation method includes the following steps:

[0028] S10: Divide the dataset into training and test sets according to a preset ratio.

[0029] Optionally, in one embodiment of this application, such as Figure 2 As shown, the dataset includes initially labeled data samples and initially unlabeled data samples. When training the classification model, in the first round of training, the initially labeled data samples are used as the training set, and the initially unlabeled data samples are used as the test set for training and prediction respectively. After the first round of training, 10% of the data samples are randomly selected as the training set, and 90% of the data samples are used as the test set for training and prediction respectively, until the loss function of the classification model converges, and the trained classification model is obtained.

[0030] In the specific implementation process, such as Figure 2 The diagram shown illustrates the architecture of the recurrent annotation framework based on active learning in this embodiment. The initial dataset is divided into two parts: initially labeled data samples and initially unlabeled data samples. The labeled data samples are manually labeled data with 100% accuracy. After the first round of training, these labeled data are mixed, and 10% of the data samples are randomly selected as the training set, while the remaining 90% are used as the test set for training and prediction, respectively, to train a basic classifier. Then, the classifier is used to re-predict the test set data. Using the framework-defined metrics and confidence thresholds, samples with a high probability of incorrect labeling are selected, and re-labeled using the labeler (i.e., the labeler predicts the label for the sample). The re-predicted label then replaces the current label of the data sample. The framework repeats this process until no incorrectly labeled samples are found.

[0031] S20: Train a classification model using data samples from the training set, and input the data samples from the test set into the trained classification model to predict the labels of the data samples.

[0032] Optionally, in one embodiment of this application, the dataset includes initially labeled data samples and initially unlabeled data samples; when training the classification model, in the first round of training, the initially labeled data samples are used as the training set, and the initially unlabeled data samples are used as the test set for training and prediction respectively; after the first round of training, data samples are randomly selected according to a preset ratio, and the dataset is divided into training set and test set for training and prediction respectively, until the loss function of the classification model converges.

[0033] Optionally, in one embodiment of this application, the classification model is the BERT classification model.

[0034] S30: Calculate the confidence level of the label for each data sample in the test set, and determine whether the confidence level of the label for the data sample is greater than the threshold.

[0035] Optionally, in one embodiment of this application, a query value is calculated for each data sample and its corresponding label using a query function in active learning, serving as the confidence level of the label for each data sample. The query function typically includes the following types:

[0036] 1. Uncertainty Sampling Queries;

[0037] 2. Query-by-Committee

[0038] 3. Queries based on expected model change

[0039] 4. Expected Error Reduction (EER) based queries

[0040] 5. Variance Reduction-based queries;

[0041] 6. Density-Weighted Methods.

[0042] In this embodiment, the expression of the query function used to calculate the confidence score of the label for each data sample in the test set is as follows:

[0043]

[0044] Wherein, P(y i (x; θ) represents a data sample of x and a label of y. i The probability value output by the classification model. Let θ represent the sum of the entropies corresponding to all labels of data sample x, where i represents the sample index and θ represents the parameters of the classification model.

[0045] S40: When the confidence level of the label of a data sample is greater than the threshold, determine whether the data sample exists in the abnormal sample cache list. If it does, delete the data sample from the abnormal sample cache list and update the data sample to the dataset. If not, update the data sample directly to the dataset.

[0046] S50: When the confidence level of the label of a data sample is not greater than the threshold, the data sample is added to the abnormal sample cache list. It is then determined whether the number of times the data sample exists consecutively in the abnormal sample cache list is less than the threshold. If so, the data sample is semantically labeled using a pre-trained semantic labeler and updated in the dataset. If not, the data sample is semantically labeled using a human label and updated in the dataset.

[0047] In this embodiment, the semantic annotator is built based on a large language model, which is ChatGpt or Llama.

[0048] Optionally, in one embodiment of this application, a semantic annotator is used to semantically annotate the data samples, specifically including:

[0049] Obtain example samples related to the data sample task to be semantically labeled.

[0050] The example sample is used as context information and concatenated with the data sample to be semantically labeled. The concatenated data sample is then input into the semantic labeler. The semantic labeler learns based on the context information, predicts the label of the data sample, and uses the label to semantically label the data sample.

[0051] In this embodiment, the similarity between the data sample to be semantically labeled and the existing data samples in the dataset is calculated, and several data samples with the highest similarity are selected as example samples.

[0052] This invention combines active learning and large language model techniques to construct an automatic semantic annotation tool. Specifically, it uses a large language model combined with In-Context Learning Prompt technology to build an annotation module. Because In-Context Learning Prompt has the ability to learn from examples and generalize, it can act as an autonomous agent to replace human annotation work in the active learning framework, freeing annotators from tedious annotation tasks. Figure 3 As shown, this is a diagram illustrating the working principle of a large language model for label prediction. K existing data samples are selected from the dataset and their corresponding labels are obtained. The data samples to be semantically labeled are then input into a large language model with fixed parameters for learning, and the label "positive (0)" is output for the data samples to be semantically labeled. Figure 4 As shown, Figure 4To illustrate the architecture of a recurrent annotation framework combining active learning and a large language model, and to teach the large language model's annotation agent how to handle samples that are very difficult to classify, this invention designs an outlier cache list, dividing samples into basic and rare examples. During automatic recurrent annotation, the cache records data that has been judged as difficult to classify in multiple consecutive rounds. Annotators can set intervals for specified rounds to manually annotate these samples. The annotated samples are simultaneously updated in both the database and the large language model's Prompt outlier cache list. In this way, both the prediction model and the annotation agent learn the human classification preferences for this type of data.

[0053] In this embodiment, the threshold for the number of consecutive occurrences of anomaly samples in the cache list is 3.

[0054] Optionally, in one embodiment of this application, in order to control the length of the Prompt, the abnormal sample cache list is designed as a first-in-first-out queue with a fixed length.

[0055] As an example, here is a sample In-Context Prompt, where the contents of the curly braces and square brackets can be replaced according to the specific task:

[0056] Prompt = "You are a professional category agent. Your job is to categorize the content following Input: into the specified categories. If..."

[0057] If you are unsure which category to classify your application to, return None. Please do not classify your application arbitrarily. The available categories are:

[0058] {Navigation, News, Phone Calls, Translation, Weather Forecast, Encyclopedia Search, Radio}

[0059] Below are examples of classifications.

[0060] {navigation:

[0061] Navigate to Yunmeng Rugu B&B

[0062] What are the road conditions ahead?

[0063] Old Sichuan Restaurant

[0064] [Outlierlst]

[0065] news:

[0066] Playing news from XX Daily

[0067] XX incident

[0068] XXXX Information

[0069] [Outlierlst]

[0070] Call up:

[0071] Call Liu X and dial Li XX

[0072] Calling Li X

[0073] [Outlierlst]

[0074] translate:

[0075] How do you pronounce "music" in English? How do you say "Li Bai" in Japanese? How do you say "criminal" in English? [Outlierlst]

[0076] Weather forecast:

[0077] Will it rain today? What's the temperature today?

[0078] Songyuan Weather

[0079] [Outlierlst]

[0080] Encyclopedia Search:

[0081] Let me introduce Daoxiang and Liu Xihua.

[0082] Who is Ma Xiteng?

[0083] What is collective economy?

[0084] [Outlierlst]

[0085] Radio station:

[0086] Playing 101.7 XX Traffic Radio [Outlierlst]

[0087] }

[0088] User input: {input}

[0089] Predicted tags:

[0090] ```

[0091] During the labeling process, the data samples replace the {input} placeholders in the Prompt and are then input into the large language model, ultimately predicting new labels.

[0092] Understandably, the In-context Learning prompt of large language models is a learning paradigm that allows large language models to learn tasks by organizing several examples or instructions in the form of demonstrations. The core of In-context Learning lies in learning from task-related example samples. It requires several examples to be demonstrated in a specific format, and then the current input data sample is concatenated with these example samples using the prompt as input to the large language model. Essentially, it uses a trained language model to estimate the probability of candidate answers based on the demonstrated examples. Simply put, it allows the language model to better understand the current task through several complete examples, thereby making more accurate predictions. Compared to other model solutions based on small samples, large language models have the following advantages: the demonstrations composed of several examples are written in natural language, providing an interpretable means of communicating with the large language model. These examples and templates make it easier for the language model to utilize human knowledge. Large language models are similar to the human analogical learning decision-making process, enabling them to generalize from one example to another. Compared to supervised learning, large language models do not require model training, reducing the computational cost of adapting the model to new tasks and making it easier to apply to more real-world scenarios.

[0093] S60: Repeat S10 to S50 until the abnormal sample cache list is empty, and complete the semantic annotation of all data samples.

[0094] Understandably, this invention uses a pre-trained semantic annotator to semantically annotate data samples. It can utilize massive amounts of unlabeled data for pre-training, fine-tuning it according to different tasks and domains to generate diverse and rich text within a given context, outputting semantic labels. This effectively reduces the workload and cost of manual annotation, improving the efficiency of data semantic annotation. Through an anomaly sample cache list update mechanism, the semantic annotator learns online about human annotation preferences for anomaly samples. It can determine whether a data sample is an anomaly based on the uncertainty of the classification model's predicted label, adding or deleting anomaly samples to a pre-set anomaly sample cache list. This effectively filters out data samples that are difficult for the classification model to handle or that are incorrectly handled, dynamically adjusting the quantity and quality of data samples in the anomaly sample cache list, thus improving the accuracy and efficiency of the anomaly sample cache list.

[0095] Experimental analysis shows that, in the process of automatic data semantic annotation, this invention requires manual annotation only once every 10 rounds of automatic execution on average. After approximately 60 rounds of manual annotation, the annotation query function cannot find any uncertain samples, and the annotation process ends. The final manually evaluated annotation accuracy reaches 99.89%.

[0096] Figure 5This is a schematic diagram of the structure of a computer device 100 provided in an embodiment of this application. The computer device 100 includes: a memory 101, a processor 102, and a computer program stored in the memory 101 and executable on the processor 102.

[0097] When the processor 102 executes the program, it implements the data semantic annotation method provided in the above embodiments.

[0098] Furthermore, the computer device 100 also includes a communication interface 103 for communication between the memory 101 and the processor 102.

[0099] The memory 101 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0100] If the memory 101, processor 102, and communication interface 103 are implemented independently, then the communication interface 103, memory 101, and processor 102 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0101] Optionally, in a specific implementation, if the memory 101, processor 102, and communication interface 103 are integrated on a single chip, then the memory 101, processor 102, and communication interface 103 can communicate with each other through an internal interface.

[0102] The processor 102 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0103] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the data semantic annotation method provided in the above embodiments.

[0104] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0105] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0106] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0107] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0108] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

[0109] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A data semantic annotation method, characterized in that, include: The dataset is divided into a training set and a test set according to a preset ratio; A classification model is trained using data samples from the training set, and data samples from the test set are input into the trained classification model to predict the labels of the data samples. Calculate the confidence score of the label for each data sample in the test set, and determine whether the confidence score of the label for the data sample is greater than the threshold. When the confidence level of the label of a data sample is greater than the threshold, it is determined whether the data sample exists in the abnormal sample cache list. If it does, the data sample is deleted from the abnormal sample cache list and updated to the dataset. If not, the data sample is directly updated to the dataset. If the confidence level of a data sample's label is not greater than a threshold, the data sample is added to the abnormal sample cache list. It is then determined whether the number of consecutive occurrences of the data sample in the abnormal sample cache list is less than a threshold. If so, example samples related to the data sample task to be semantically labeled are obtained. These example samples are then concatenated with the data sample to be semantically labeled using the example samples as context information. The concatenated data sample is then input into the semantic labeler. The semantic labeler learns based on the context information, predicts the label of the data sample, and uses this label to semantically label the data sample. If not, manually assigned labels are obtained to semantically label the data sample and update the dataset. Repeat the above steps until the abnormal sample cache list is empty, and complete the semantic annotation of all data samples.

2. The data semantic annotation method according to claim 1, characterized in that, The semantic annotator is built based on a large language model.

3. The data semantic annotation method according to claim 2, characterized in that, The large language model is either ChatGpt or Llama.

4. The data semantic annotation method according to claim 1, characterized in that, Obtain example samples related to the data sample task to be semantically labeled, specifically including: Calculate the similarity between the data sample to be semantically labeled and the existing data samples in the dataset, and select the data samples with the highest similarity as example samples.

5. The data semantic annotation method according to any one of claims 1 to 4, characterized in that, The confidence level is calculated by taking the entropy of the label for each data sample in the test set, as shown in the following expression: in, Indicates data sample as x , tag as y i The probability value output by the classification model. Represents data samples x The sum of the entropies corresponding to all labels, i Indicates the sample number. Represents the parameters of the classification model.

6. The data semantic annotation method according to claim 5, characterized in that, The dataset includes initially labeled data samples and initially unlabeled data samples. When training the classification model, in the first round of training, the initially labeled data samples are used as the training set, and the initially unlabeled data samples are used as the test set for training and prediction, respectively. After the first round of training, data samples are randomly selected according to a preset ratio, and the dataset is divided into training set and test set for training and prediction respectively.

7. The data semantic annotation method according to claim 5, characterized in that, The abnormal sample cache list is a first-in-first-out queue with a fixed length.

8. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to perform the operations performed by the data semantic annotation method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that is executed by a processor as described in any one of claims 1 to 7, and the program performs the operations performed by the data semantic annotation method.

Images