Automatic annotation methods and electronic devices for sample data

Abstract

This application relates to an automatic sample data labeling method and electronic device. The method includes: training multiple target classification models using fewer labeled samples than unlabeled samples; inputting unlabeled samples into each target classification model to obtain pseudo-label data; performing a voting operation on the pseudo-label data to obtain candidate label data; calculating the confidence score of each label category output by the target classification model corresponding to the candidate label data; determining whether the confidence score meets pre-configured conditions; if so, using the candidate label data as the labeling data for the unlabeled samples; and repeating the above steps until all unlabeled samples are labeled. This application can automatically label unlabeled samples in the original sample set, completing the automatic labeling of large amounts of data in the original sample set, replacing manual review and screening of labeled data. Especially when there are no labels and the data volume is large, it can save a lot of manpower and time.

Description

Technical Field

[0001] This application relates to the field of data annotation technology, and in particular to an automatic sample data annotation method and electronic device. Background Technology

[0002] Currently, sample annotation mainly utilizes manual or semi-automatic methods to label sample data. Traditional manual annotation methods require manual selection of feature information, which can lead to incomplete feature extraction and poor generalization ability of models trained using these samples. While semi-automatic methods can label data, manual review and correction are still required after the data is labeled. When the amount of data to be labeled is large, this requires a significant amount of manpower and time, resulting in low annotation efficiency. Summary of the Invention

[0003] In view of the above, this application provides a method and electronic device for automatic annotation of sample data, the purpose of which is to solve the above-mentioned technical problems.

[0004] Firstly, this application provides a method for automatically labeling sample data, the method comprising:

[0005] Obtain a first number of labeled samples from the original sample set, and train n target classification models based on the labeled samples, where n is a positive integer greater than 1. The original sample set consists of a first number of labeled samples and a second number of unlabeled samples to be labeled, and the first number is less than the second number.

[0006] The unlabeled samples are input into each of the target classification models to obtain the pseudo-label data predicted by each target classification model for the unlabeled samples;

[0007] A voting operation is performed on the pseudo-label data to obtain candidate label data;

[0008] Calculate the confidence score of each label category output by the target classification model corresponding to the candidate label data, and determine whether the confidence score meets the pre-configured conditions. If so, use the candidate label data as the labeled data of the unlabeled sample; otherwise, put the unlabeled sample back into the original sample set.

[0009] The labeled data is placed into the original sample set as the third number of labeled samples. The steps of training n target classification models based on the labeled samples and judging whether the confidence level meets the pre-configured conditions are repeated until the unlabeled samples in the original sample set are labeled.

[0010] Secondly, this application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0011] Memory, used to store computer programs;

[0012] When a processor executes a program stored in memory, it implements the automatic sample data annotation method described in any embodiment of the first aspect.

[0013] The technical solutions provided in this application have the following advantages compared with the prior art:

[0014] This application trains multiple target classification models using labeled samples (fewer than unlabeled samples) in the original sample set. The unlabeled samples to be labeled are input into each target classification model, resulting in pseudo-label data predicted by each model. Candidate labels are obtained by voting on these pseudo-label data. Since the candidate labels are derived from these pseudo-label data, they can represent the true labels of the unlabeled data to a certain extent. The confidence score of each label category output by the target classification model corresponding to the candidate label data is then calculated. If the confidence score meets pre-configured conditions, it further indicates that the candidate label data can accurately represent the labeled data of the unlabeled samples. The candidate label data is then used as the labeled data for the unlabeled samples, and the labeled data is added to the original sample set as labeled samples. This process is repeated until all unlabeled samples in the original sample set are labeled. This allows for the automatic labeling of unlabeled samples in the original sample set, completing the automatic labeling of large amounts of data in the original sample set. This replaces manual review and screening of labeled data, especially in cases with no labels and a large amount of data, saving significant manpower and time and improving labeling efficiency. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a preferred embodiment of the automatic annotation method for sample data in this application;

[0018] Figure 2This is a schematic diagram of a preferred embodiment of the automatic sample data labeling device of this application;

[0019] Figure 3 This is a schematic diagram of a preferred embodiment of the electronic device of this application;

[0020] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0022] It should be noted that the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed in this application.

[0023] This application provides a method for automatic annotation of sample data. (Refer to...) Figure 1 The diagram shown is a flowchart illustrating an embodiment of the automatic sample data annotation method of this application. This method can be executed by an electronic device, which can be implemented in software and / or hardware. The automatic sample data annotation method includes:

[0024] Step S10: Obtain the first number of labeled samples in the original sample set, and train n target classification models based on the labeled samples, where n is a positive integer greater than 1. The original sample set consists of the first number of labeled samples and the second number of unlabeled samples to be labeled, and the first number is less than the second number.

[0025] Step S20: Input the unlabeled samples into each of the target classification models to obtain the pseudo-label data of the unlabeled samples predicted by each target classification model;

[0026] Step S30: Perform a voting operation on the pseudo-label data to obtain candidate label data;

[0027] Step S40: Calculate the confidence score of each label category output by the target classification model corresponding to the candidate label data, and determine whether the confidence score meets the pre-configured conditions. If yes, use the candidate label data as the labeled data of the unlabeled sample; otherwise, put the unlabeled sample back into the original sample set.

[0028] Step S50: Place the labeled data into the original sample set as the third number of labeled samples, and repeat the steps of training n target classification models based on the labeled samples until the confidence level meets the pre-configured conditions, until the unlabeled samples in the original sample set are labeled.

[0029] In this embodiment, the sample can be image sample data, such as training samples required for image recognition models applied to scenarios such as security, chemical industry, and gas stations. The solution of this application is described using image data as an example. It is understood that the actual application scenarios of this application are not limited to this; the sample can also be text data, such as training samples required for text classification models that need to identify whether the emotion expressed in the text is positive, neutral, or negative.

[0030] The original sample set contains a small number of labeled samples. The original sample set consists of a first number of labeled samples and a second number of unlabeled samples to be labeled. The first number can be 5%-10% of the original sample set, and the second number can be 90%-95% of the original sample set. A small number of labeled samples are obtained from the original sample set. The original sample set refers to the set of training samples needed to train the image classification model. The labeled samples can be samples labeled using automatic labeling technology or samples labeled manually. For example, if the image classification model is a three-class classification model and the recognition result may be A, B, or C, then 5%-10% of the samples in the original sample set can be labeled as A, B, or C, which is the first number of labeled samples.

[0031] Multiple target classification models (e.g., image classification models) are trained based on a first number of labeled samples. For example, n (n is a positive integer greater than 1) image classification models are trained. For example, a small number of labeled samples are divided into training and validation sets. The training set is used to train the model, and the validation set is used to validate the trained model. The specific training process is roughly the same as the training process of traditional deep learning models, and will not be described in detail here.

[0032] Suppose we train 8 image classification models, denoted as Model 1, Model 2, Model 3... Model 8. We input the unlabeled samples to be labeled into each image classification model. Each image classification model makes a prediction on the unlabeled samples, and we obtain the pseudo-label data of the unlabeled samples predicted by each image classification model. That is, Model 1, Model 2, Model 3... Model 8 all have corresponding pseudo-label data.

[0033] After obtaining the pseudo-label data for the unlabeled samples predicted by each image classification model, a voting operation is performed on the pseudo-label data of the eight image classification models to obtain candidate label data. Specifically, a voting operation is performed on the pseudo-label data, and the number of votes for the pseudo-label data in each label category is counted. The pseudo-label data with the highest number of votes is selected as the candidate label data. Voting refers to counting the pseudo-label with the most votes. For example, the pseudo-label data for the unlabeled samples predicted by models 1 to 4 is A, the pseudo-label data predicted by models 5 and 6 is B, and the pseudo-label data predicted by models 7 and 8 is C. Pseudo-label data A has the highest number of votes, so pseudo-label data A is selected as the candidate label.

[0034] Calculate the confidence score of each label category output by the image classification model corresponding to the candidate label data. That is, calculate the confidence score of each label category output by the image classification model for all images predicted as candidate label data. Calculate the average confidence score corresponding to each label category. Determine whether the calculated confidence score meets the pre-configured conditions. The pre-configured conditions are three conditions that can be configured based on the average confidence score corresponding to the candidate label data, the difference between the maximum confidence score and the second largest confidence score corresponding to the candidate label data, and the entropy of the unlabeled samples. If all three pre-configured conditions are met, the candidate label data is used as the labeled data for the unlabeled samples. That is, the unlabeled samples are labeled using the candidate label data to obtain labeled data. When the newly added labeled data reaches a certain number m (m is a positive integer greater than or equal to 1), the labeled data is put into the original sample set as the newly added third number of labeled samples. At this time, the original sample set contains the first and third number of labeled samples. Repeat the above steps S10 to S40 until all unlabeled samples in the original sample set have been labeled. If the confidence level does not meet any of the above three conditions, it is considered that the confidence level does not meet the pre-configured conditions. The unlabeled sample is then reinserted into the original sample set as an unlabeled sample to be labeled, and steps S10 to S40 are repeated until all unlabeled samples in the original sample set are labeled. This allows for the automatic labeling of unlabeled samples in the original sample set, thus completing the automatic labeling of large amounts of data in the original sample set. This replaces manual review of the labeled data, saving significant manpower and time, especially in cases where there are no labels and the data volume is large.

[0035] In one embodiment, obtaining a first number of labeled samples in the original sample set and training n target classification models based on the labeled samples includes:

[0036] A predetermined proportion of samples are obtained from the original sample set and labeled to obtain the first number of labeled samples;

[0037] The weights of the n initial classification models are initialized using the weights of the pre-trained models.

[0038] Based on the first number of labeled samples, each initialized initial classification model is trained to obtain n target classification models.

[0039] A predetermined proportion (e.g., 10%) of samples is obtained from the original sample set and labeled. Labeling can be done manually or automatically, resulting in a small number of labeled samples (the first number of labeled samples). To prevent model overfitting and accelerate model convergence, the weights of n initial classification models are initialized using the weights of the pre-trained model. Each initialized initial classification model is then trained using the small number of labeled samples, resulting in n target classification models (e.g., image classification models).

[0040] Further, the step of obtaining a preset proportion of samples from the original sample set and labeling them to obtain the first number of labeled samples includes:

[0041] A predetermined proportion of samples are randomly obtained from the original sample set;

[0042] From the preset proportion of samples, select the same number of samples for each label category of the preset proportion of samples;

[0043] The same number of samples are labeled to obtain the first number of labeled samples.

[0044] To ensure that the acquired sample data covers all scenarios of the original data, an equal number of samples for each label category are selected from the randomly drawn sample data. For example, 2000 samples with label category A, 2000 samples with label category B, and 2000 samples with label category C are selected from the randomly drawn sample data. Then, the selected samples are labeled to obtain the first number of labeled samples.

[0045] In one embodiment, the step of inputting the unlabeled samples into each of the target classification models to obtain pseudo-label data predicted by each target classification model for the unlabeled samples includes:

[0046] The unlabeled samples are input into each of the target classification models, and the confidence of each target classification model in predicting each label category of the unlabeled sample is calculated.

[0047] The label category with the highest confidence level corresponding to each target classification model is used as the pseudo label for the unlabeled sample predicted by the target classification model, thus obtaining the pseudo label data for the unlabeled sample predicted by each target classification model.

[0048] Suppose we train eight target classification models (e.g., image classification models), denoted as Model 1, Model 2, Model 3...Model 8. We input the unlabeled samples to be labeled into each model and use the Softmax formula to calculate the confidence score of each model for predicting each label category of the unlabeled sample. Specifically, we calculate the confidence scores for Model 1, Model 2, Model 3...Model 8 for predicting unlabeled samples as label category A, label category B, and label category C. The label category with the highest confidence score for each model is then used as the pseudo-label for that model's predicted unlabeled sample. For example, if Model 1 has a confidence score of 0.8 for predicting unlabeled sample A, 0.1 for predicting unlabeled sample B, and 0.1 for predicting unlabeled sample C, then label category A is used as the pseudo-label for Model 1's predicted unlabeled sample. This yields the pseudo-label data for each model's predicted unlabeled sample. The Softmax formula is as follows:

[0049]

[0050] S(z i ) represents the confidence level of the i-th category, k is the total number of label categories, and z i The scores for each label category.

[0051] In one embodiment, using the pseudo-label data with the highest number of votes as the candidate label data includes:

[0052] If the pseudo-label data with the highest number of votes is a label category, then the pseudo-label data with the highest number of votes will be used as the candidate label data;

[0053] If the pseudo-label data with the highest number of votes belongs to two or more label categories, calculate the average confidence of the pseudo-label category output by the target classification model for each pseudo-label data with the highest number of votes, and use the pseudo-label data corresponding to the label category with the highest average confidence as the candidate label data.

[0054] If the pseudo-label data with the most votes is a label category, then the pseudo-label data with the most votes is used as the candidate label data. For example, the pseudo-label data for the unlabeled samples predicted by models 1 to 6 is A, the pseudo-label data predicted by model 7 is B, the pseudo-label data predicted by model 8 is C, and the pseudo-label data with the most votes is a label category A, then the pseudo-label data A is used as the candidate label.

[0055] If the pseudo-label data with the highest number of votes has two or more label categories, for example, the pseudo-label data predicted by models 1 to 3 for unlabeled samples is A, the pseudo-label data predicted by models 4 to 6 is B, and the pseudo-label data predicted by models 7 and 8 is C, and the pseudo-label data A and B both have 3 votes, then we calculate the average confidence of the pseudo-label data predicted by models 1 to 3 for unlabeled samples as A, and calculate the average confidence of the pseudo-label data predicted by models 4 to 6 for unlabeled samples as B. Assuming that the confidence of models 1 to 3 predicting A is 0.9 (i.e., the average confidence is 0.9), and the confidence of models 4 to 6 predicting B is 0.8 (i.e., the average confidence is 0.8), we select the pseudo-label data corresponding to the label category with the highest average confidence as the candidate label data, that is, we select pseudo-label data A as the candidate label data.

[0056] In one embodiment, calculating the confidence score of each label category output by the target classification model corresponding to the candidate label data, and determining whether the confidence score meets pre-configured conditions, includes:

[0057] Calculate the confidence score of each label category output by the target classification model corresponding to the candidate label data, and calculate the average confidence score of each label category;

[0058] Sort the average values ​​from largest to smallest, and calculate the difference between the average values ​​of the top two values ​​in the sort;

[0059] Based on the average confidence level of the candidate label data, the difference, and the entropy of the unlabeled sample, it is determined whether the confidence level meets the pre-configured conditions.

[0060] For example, if the target classification model for predicting candidate label data is Model 1 to Model 4, then the confidence scores for each label category output by Model 1, Model 2, Model 3, and Model 4 are calculated. Assuming Model 1 has a confidence level of 0.8 for output A, 0.1 for output B, and 0.1 for output C; Model 2 has a confidence level of 0.7 for output A, 0.1 for output B, and 0.2 for output C; Model 3 has a confidence level of 0.6 for output A, 0.2 for output B, and 0.2 for output C; and Model 4 has a confidence level of 0.8 for output A, 0.1 for output B, and 0.1 for output C, then the average confidence level for label category A is 0.725, the average confidence level for label category B is 0.125, and the average confidence level for label category C is 0.15.

[0061] The average confidence scores are sorted from largest to smallest, resulting in scores of 0.725, 0.15, and 0.125. The difference between the average scores of the top two scores is 0.575. Based on the average confidence score of the candidate labeled data (0.725), the difference (0.575), and the entropy of the unlabeled samples, it is determined whether the confidence scores meet the pre-configured conditions (e.g., whether the three values ​​fall within a certain interval).

[0062] Further, determining whether the confidence level meets the pre-configured conditions based on the average confidence level corresponding to the candidate label data, the difference, and the entropy of the unlabeled sample includes:

[0063] Determine whether the average confidence level corresponding to the candidate label data is greater than a first threshold;

[0064] If so, determine whether the difference is greater than the second threshold;

[0065] If so, determine whether the entropy of the unlabeled sample is less than the third threshold;

[0066] If so, then the confidence level satisfies the pre-configured conditions.

[0067] If the average confidence score of the candidate label data is greater than the first threshold, it indicates that the candidate label data can accurately represent the labeled data of the unlabeled samples. If the difference is greater than the second threshold, it indicates that the difference in the average confidence scores of the top two categories is large, ensuring that the target classification model accurately distinguishes categories with similar features. Since the smaller the entropy, the more ideal the pattern of the confidence scores of each category, if the entropy of the unlabeled samples is less than the third threshold, it indicates that the classifier has high stability. That is, when all three conditions are met, it means that the confidence scores of each label category output by the target classification model corresponding to the candidate label data meet the conditions, and the candidate labels can be used as labeled data for unlabeled samples. The values ​​of the first, second, and third thresholds can all be in the range (0, 1), and the entropy calculation formula is as follows:

[0068]

[0069] Where x represents the unlabeled sample to be labeled, y i Let P(y) represent the i-th label category. i |x) indicates that the label category of x is y. i The confidence level.

[0070] Furthermore, the step of determining whether the confidence level meets the pre-configured conditions based on the average confidence level of the candidate label data, the difference, and the entropy of the unlabeled sample further includes:

[0071] If the average confidence level of the candidate label data is less than or equal to the first threshold, or if the difference is less than or equal to the second threshold, or if the entropy of the unlabeled sample is greater than or equal to the third threshold, then the confidence level does not meet the pre-configured conditions, and the unlabeled sample is put back into the original sample set.

[0072] Repeat the steps of training n target classification models based on the labeled samples to determine whether the confidence level meets the pre-configured conditions, until the unlabeled samples in the original sample set are labeled.

[0073] If the average confidence level of the candidate label data is less than or equal to the first threshold, or the above difference is less than or equal to the second threshold, or the entropy of the unlabeled sample is greater than or equal to the third threshold, it means that the confidence level of each label category output by the target classification model corresponding to the candidate label data does not meet the conditions. In this case, the unlabeled sample is put back into the original sample set, and the above steps S10 to S40 are repeated until the unlabeled samples in the original sample set are labeled.

[0074] Reference Figure 2 The diagram shown is a functional module schematic of the sample data automatic labeling device 100 of this application.

[0075] The automatic sample data labeling device 100 described in this application can be installed in an electronic device. Depending on the functions implemented, the automatic sample data labeling device 100 may include a training module 110, a prediction module 120, a voting module 130, a labeling module 140, and an insertion module 150. The module described in this application can also be referred to as a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device.

[0076] In this embodiment, the functions of each module / unit are as follows:

[0077] Training module 110: used to obtain a first number of labeled samples in the original sample set, and train n target classification models based on the labeled samples, where n is a positive integer greater than 1. The original sample set consists of a first number of labeled samples and a second number of unlabeled samples to be labeled, and the first number is less than the second number.

[0078] Prediction module 120: used to input the unlabeled samples into each of the target classification models respectively, and obtain the pseudo-label data of the unlabeled samples predicted by each target classification model;

[0079] Voting module 130: used to perform voting operations on the pseudo-label data to obtain candidate label data;

[0080] Labeling module 140: is used to calculate the confidence level of each label category output by the target classification model corresponding to the candidate label data, and determine whether the confidence level meets the pre-configured conditions. If so, the candidate label data is used as the label data of the unlabeled sample; otherwise, the unlabeled sample is put back into the original sample set.

[0081] Insertion module 150: is used to insert the labeled data into the original sample set as the third number of labeled samples, and repeatedly execute the step of training n target classification models based on the labeled samples to determine whether the confidence level meets the pre-configured conditions, until the unlabeled samples in the original sample set are labeled.

[0082] In one embodiment, obtaining a first number of labeled samples in the original sample set and training n target classification models based on the labeled samples includes:

[0083] A predetermined proportion of samples are obtained from the original sample set and labeled to obtain the first number of labeled samples;

[0084] The weights of the n initial classification models are initialized using the weights of the pre-trained models.

[0085] Based on the first number of labeled samples, each initialized initial classification model is trained to obtain n target classification models.

[0086] In one embodiment, obtaining a preset proportion of samples from the original sample set and labeling them to obtain the first number of labeled samples includes:

[0087] A predetermined proportion of samples are randomly obtained from the original sample set;

[0088] From the preset proportion of samples, select the same number of samples for each label category of the preset proportion of samples;

[0089] The same number of samples are labeled to obtain the first number of labeled samples.

[0090] In one embodiment, the step of inputting the unlabeled samples into each of the target classification models to obtain pseudo-label data predicted by each target classification model for the unlabeled samples includes:

[0091] The unlabeled samples are input into each of the target classification models, and the confidence of each target classification model in predicting each label category of the unlabeled sample is calculated.

[0092] The label category with the highest confidence level corresponding to each target classification model is used as the pseudo label for the unlabeled sample predicted by the target classification model, thus obtaining the pseudo label data for the unlabeled sample predicted by each target classification model.

[0093] In one embodiment, performing a voting operation on the pseudo-label data to obtain candidate label data includes:

[0094] Perform a voting operation on the pseudo-label data and count the number of votes for each pseudo-label data category;

[0095] The pseudo-label data with the highest number of votes is used as the candidate label data.

[0096] In one embodiment, using the pseudo-label data with the highest number of votes as the candidate label data includes:

[0097] If the pseudo-label data with the highest number of votes is a label category, then the pseudo-label data with the highest number of votes will be used as the candidate label data;

[0098] If the pseudo-label data with the highest number of votes belongs to two or more label categories, calculate the average confidence of the pseudo-label category output by the target classification model for each pseudo-label data with the highest number of votes, and use the pseudo-label data corresponding to the label category with the highest average confidence as the candidate label data.

[0099] In one embodiment, calculating the confidence score of each label category output by the target classification model corresponding to the candidate label data, and determining whether the confidence score meets pre-configured conditions, includes:

[0100] Calculate the confidence score of each label category output by the target classification model corresponding to the candidate label data, and calculate the average confidence score of each label category;

[0101] Sort the average values ​​from largest to smallest, and calculate the difference between the average values ​​of the top two values ​​in the sort;

[0102] Based on the average confidence level of the candidate label data, the difference, and the entropy of the unlabeled sample, it is determined whether the confidence level meets the pre-configured conditions.

[0103] In one embodiment, determining whether the confidence level meets pre-configured conditions based on the average confidence level of the candidate label data, the difference, and the entropy of the unlabeled samples includes:

[0104] Determine whether the average confidence level corresponding to the candidate label data is greater than a first threshold;

[0105] If so, determine whether the difference is greater than the second threshold;

[0106] If so, determine whether the entropy of the unlabeled sample is less than the third threshold;

[0107] If so, then the confidence level satisfies the pre-configured conditions.

[0108] In one embodiment, determining whether the confidence level meets the pre-configured conditions based on the average confidence level of the candidate label data, the difference, and the entropy of the unlabeled sample further includes:

[0109] If the average confidence level of the candidate label data is less than or equal to the first threshold, or if the difference is less than or equal to the second threshold, or if the entropy of the unlabeled sample is greater than or equal to the third threshold, then the confidence level does not meet the pre-configured conditions, and the unlabeled sample is put back into the original sample set.

[0110] Repeat the steps of training n target classification models based on the labeled samples to determine whether the confidence level meets the pre-configured conditions, until the unlabeled samples in the original sample set are labeled.

[0111] Reference Figure 3 The diagram shown is a schematic diagram of a preferred embodiment of the electronic device 1 of this application.

[0112] The electronic device 1 includes, but is not limited to, a memory 11, a processor 12, a display 13, and a communication interface 14. The electronic device 1 can connect to a network via the communication interface 14. The network can be an intranet, the Internet, a Global System for Mobile communication (GSM), a Wideband Code Division Multiple Access (WCDMA) network, a 4G network, a 5G network, Bluetooth, Wi-Fi, a voice communication network, or other wireless or wired networks.

[0113] The memory 11 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the electronic device 1, such as the hard disk or memory of the electronic device 1. In other embodiments, the memory 11 may also be an external storage device of the electronic device 1, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped with the electronic device 1. Of course, the memory 11 may include both the internal storage unit and its external storage device of the electronic device 1. In this embodiment, the memory 11 is typically used to store the operating system and various computer programs installed on the electronic device 1, such as the program code of the sample data automatic labeling program 10. In addition, the memory 11 can also be used to temporarily store various types of data that have been output or will be output.

[0114] In some embodiments, processor 12 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. Processor 12 is typically used to control the overall operation of the electronic device 1, such as performing data interaction or communication-related control and processing. In this embodiment, processor 12 is used to run program code stored in memory 11 or process data, such as running the program code of sample data automatic labeling program 10.

[0115] The display 13 may be referred to as a display screen or display unit. In some embodiments, the display 13 may be an LED display, a liquid crystal display, a touch liquid crystal display, or an organic light-emitting diode (OLED) touch screen, etc. The display 13 is used to display information processed in the electronic device 1 and to display a visual working interface.

[0116] The communication interface 14 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface), which is typically used to establish a communication connection between the electronic device 1 and other electronic devices.

[0117] Figure 3 Only an electronic device 1 with components 11-14 and sample data automatic annotation program 10 is shown. However, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0118] In the above embodiments, when the processor 12 executes the sample data automatic annotation program 10 stored in the memory 11, it can perform the following steps:

[0119] Obtain a first number of labeled samples from the original sample set, and train n target classification models based on the labeled samples, where n is a positive integer greater than 1. The original sample set consists of a first number of labeled samples and a second number of unlabeled samples to be labeled, and the first number is less than the second number.

[0120] The unlabeled samples are input into each of the target classification models to obtain the pseudo-label data predicted by each target classification model for the unlabeled samples;

[0121] A voting operation is performed on the pseudo-label data to obtain candidate label data;

[0122] Calculate the confidence score of each label category output by the target classification model corresponding to the candidate label data, and determine whether the confidence score meets the pre-configured conditions. If so, use the candidate label data as the labeled data of the unlabeled sample; otherwise, put the unlabeled sample back into the original sample set.

[0123] The labeled data is placed into the original sample set as the third number of labeled samples. The steps of training n target classification models based on the labeled samples and judging whether the confidence level meets the pre-configured conditions are repeated until the unlabeled samples in the original sample set are labeled.

[0124] The storage device can be the memory 11 of the electronic device 1, or it can be other storage devices that are communicatively connected to the electronic device 1.

[0125] For a detailed explanation of the above steps, please refer to the above. Figure 2 Functional block diagram of an embodiment of the automatic sample data labeling device 100 and Figure 1 A flowchart illustrating an embodiment of the automatic sample data annotation method.

[0126] Furthermore, this application embodiment also proposes a computer-readable storage medium, which can be non-volatile or volatile. The computer-readable storage medium can be any one or any combination of several of the following: hard disk, multimedia card, SD card, flash memory card, SMC, read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, etc. The computer-readable storage medium includes a data storage area and a program storage area. The program storage area stores a sample data automatic annotation program 10, which, when executed by a processor, performs the following operations:

[0127] Obtain a first number of labeled samples from the original sample set, and train n target classification models based on the labeled samples, where n is a positive integer greater than 1. The original sample set consists of a first number of labeled samples and a second number of unlabeled samples to be labeled, and the first number is less than the second number.

[0128] The unlabeled samples are input into each of the target classification models to obtain the pseudo-label data predicted by each target classification model for the unlabeled samples;

[0129] A voting operation is performed on the pseudo-label data to obtain candidate label data;

[0130] Calculate the confidence score of each label category output by the target classification model corresponding to the candidate label data, and determine whether the confidence score meets the pre-configured conditions. If so, use the candidate label data as the labeled data of the unlabeled sample; otherwise, put the unlabeled sample back into the original sample set.

[0131] The labeled data is placed into the original sample set as the third number of labeled samples. The steps of training n target classification models based on the labeled samples and judging whether the confidence level meets the pre-configured conditions are repeated until the unlabeled samples in the original sample set are labeled.

[0132] The specific implementation of the computer-readable storage medium in this application is largely the same as the specific implementation of the above-described automatic sample data annotation method, and will not be repeated here.

[0133] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0134] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware simulation platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, electronic device, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0135] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for automatically labeling sample data, characterized in that, The method includes: Obtain a first number of labeled samples in the original sample set, and train n target classification models based on the labeled samples, where n is a positive integer greater than 1. The original sample set consists of a first number of labeled samples and a second number of unlabeled samples to be labeled, and the first number is less than the second number. The samples include image sample data or text data. The unlabeled samples are input into each of the target classification models to obtain the pseudo-label data predicted by each target classification model for the unlabeled samples; A voting operation is performed on the pseudo-label data to obtain candidate label data; Calculate the confidence score of each label category output by the target classification model corresponding to the candidate label data, and determine whether the confidence score meets the pre-configured conditions. If so, use the candidate label data as the labeled data of the unlabeled sample; otherwise, put the unlabeled sample back into the original sample set. The labeled data is placed into the original sample set as the third number of labeled samples. The steps of training n target classification models based on the labeled samples and judging whether the confidence level meets the pre-configured conditions are repeated until the original sample set has no labeled samples. The step of calculating the confidence score of each label category output by the target classification model corresponding to the candidate label data, and determining whether the confidence score meets the pre-configured conditions, includes: Calculate the confidence score of each label category output by the target classification model corresponding to the candidate label data, and calculate the average confidence score of each label category; Sort the average values ​​from largest to smallest, and calculate the difference between the average values ​​of the top two values ​​in the sort; Based on the average confidence level of the candidate label data, the difference, and the entropy of the unlabeled sample, determine whether the confidence level meets the pre-configured conditions. The step of determining whether the confidence level meets the pre-configured conditions based on the average confidence level of the candidate label data, the difference, and the entropy of the unlabeled samples includes: Determine whether the average confidence level corresponding to the candidate label data is greater than a first threshold; If so, determine whether the difference is greater than the second threshold; If so, determine whether the entropy of the unlabeled sample is less than the third threshold; If so, then the confidence level satisfies the pre-configured conditions; The step of determining whether the confidence level meets the pre-configured conditions based on the average confidence level of the candidate label data, the difference, and the entropy of the unlabeled samples further includes: If the average confidence level of the candidate label data is less than or equal to the first threshold, or if the difference is less than or equal to the second threshold, or if the entropy of the unlabeled sample is greater than or equal to the third threshold, then the confidence level does not meet the pre-configured conditions, and the unlabeled sample is put back into the original sample set. Repeat the steps of training n target classification models based on the labeled samples to determine whether the confidence level meets the pre-configured conditions, until the unlabeled samples in the original sample set are labeled.

2. The automatic sample data annotation method as described in claim 1, characterized in that, The process of obtaining a first number of labeled samples from the original sample set and training n target classification models based on the labeled samples includes: A predetermined proportion of samples are obtained from the original sample set and labeled to obtain the first number of labeled samples; The weights of the n initial classification models are initialized using the weights of the pre-trained models. Based on the first number of labeled samples, each initialized initial classification model is trained to obtain n target classification models.

3. The automatic sample data annotation method as described in claim 2, characterized in that, The step of obtaining a preset proportion of samples from the original sample set and labeling them to obtain the first number of labeled samples includes: A predetermined proportion of samples are randomly obtained from the original sample set; From the preset proportion of samples, select the same number of samples for each label category of the preset proportion of samples; The same number of samples are labeled to obtain the first number of labeled samples.

4. The automatic sample data annotation method as described in claim 1, characterized in that, The step of inputting the unlabeled samples into each of the target classification models to obtain the pseudo-label data predicted by each target classification model for the unlabeled samples includes: The unlabeled samples are input into each of the target classification models, and the confidence of each target classification model in predicting each label category of the unlabeled sample is calculated. The label category with the highest confidence level corresponding to each target classification model is used as the pseudo label for the unlabeled sample predicted by the target classification model, thus obtaining the pseudo label data for the unlabeled sample predicted by each target classification model.

5. The automatic sample data annotation method as described in claim 1, characterized in that, The step of performing a voting operation on the pseudo-label data to obtain candidate label data includes: Perform a voting operation on the pseudo-label data and count the number of votes for each label category of pseudo-label data; The pseudo-label data with the highest number of votes is used as the candidate label data.

6. The automatic sample data annotation method as described in claim 5, characterized in that, The step of using the pseudo-label data with the highest number of votes as the candidate label data includes: If the pseudo-label data with the highest number of votes is a label category, then the pseudo-label data with the highest number of votes will be used as the candidate label data; If the pseudo-label data with the highest number of votes belongs to two or more label categories, calculate the average confidence of the pseudo-label category output by the target classification model for each pseudo-label data with the highest number of votes, and use the pseudo-label data corresponding to the label category with the highest average confidence as the candidate label data.

7. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the automatic sample data annotation method according to any one of claims 1 to 6.

Images