Training data labeling method and device for model training
By obtaining the initial data set, training the basic model, and using preset strategies for iterative labeling and training during the data labeling process, the problem of being unable to take into account labeling cost and quality in the existing technology is solved, and efficient and accurate data labeling and models are achieved. train.
Patent Information
- Application Number
- CN202411168099.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2025-01-21
AI Technical Summary
The prior art cannot take into account both the annotation cost and the annotation quality when performing data annotation. Traditional manual annotation costs high and the quality is difficult to guarantee. The method based on unsupervised or semi-supervised learning sacrifices the accuracy of the annotation data or the performance of the model.
By obtaining the initial dataset, select some data for initial marking, train the basic model, and use the preset strategy to mark the remaining unbelled data, iterate the training model, choose the data that is most required to reduce the cost, and at the same time through optimizing model training Ensure the quality and consistency of the marked data.
The quality and model performance of the labeling data are guaranteed while reducing the cost of labeling. The efficiency and accuracy are improved through independent selection of data and iterative optimization training processes, and the problems that cannot take into account the cost and quality of existing technologies.
Smart Images
Figure CN119337131A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data annotation, and in particular to a method and device for annotating training data for model training. Background Art
[0002] With the rapid development of artificial intelligence technology, the demand for data in machine learning models is increasing. As the basis for training machine learning models, the quality and quantity of labeled data directly affect the performance of the model. However, traditional data labeling methods have obvious shortcomings. On the one hand, manual labeling requires a lot of manpower, material resources and time, and is costly; on the other hand, the quality of labeling is affected by human factors, and it is difficult to ensure consistency and accuracy.
[0003] In the existing technology, data labeling methods based on unsupervised learning or semi-supervised learning are used. These methods reduce the labeling cost to a certain extent, but often sacrifice the accuracy of the labeled data or the performance of the model. Therefore, how to reduce the labeling cost while ensuring the quality of the labeled data has become an urgent problem to be solved. Summary of the invention
[0004] In view of this, an object of the present invention is to provide a method and device for labeling training data for model training, aiming to solve the problem in the prior art that it is impossible to take into account both labeling cost and labeling quality when performing data labeling.
[0005] The embodiment of the present invention is implemented as follows: A method for labeling training data for model training, the method comprising: Acquire an initial data set, select a portion of the data set from the initial data set according to a preset rule, and perform initial annotation on the portion of the data set to obtain an initial annotated data set; Performing model training using the initial labeled data set to obtain a basic model; The remaining unlabeled data of the initial data set are labeled using a preset strategy to obtain a new batch of labeled data, and then the basic model is iteratively trained using the new batch of labeled data.
[0006] Furthermore, in the above-mentioned method for labeling training data for model training, the step of labeling the remaining unlabeled data of the initial data set using a preset strategy to obtain a new batch of labeled data includes: Predicting the unlabeled data using the basic model, and selecting samples with uncertain prediction results from the unlabeled data for labeling; The uncertainty of the prediction result is determined based on the probability distribution, confidence or entropy output by the basic model.
[0007] Furthermore, in the above-mentioned method for labeling training data for model training, the step of labeling the remaining unlabeled data of the initial data set using a preset strategy to obtain a new batch of labeled data further includes: The similarity or distance between the unlabeled data and the labeled data is calculated to determine the data that needs to be labeled among the unlabeled data, and the data that needs to be labeled is labeled to obtain a new batch of labeled data.
[0008] Furthermore, in the above-mentioned method for labeling training data for model training, the step of labeling the remaining unlabeled data of the initial data set using a preset strategy to obtain a new batch of labeled data further includes: Calculating the similarity or distance between the unlabeled data and the labeled data to determine the data that needs to be labeled among the unlabeled data; The basic model is used to predict the data to be labeled, and samples with uncertain prediction results are selected from the data to be labeled for labeling.
[0009] Furthermore, in the above-mentioned method for labeling training data for model training, after the step of iteratively training the basic model using the new batch of labeled data, the method further comprises: The performance of the basic model is evaluated, and the preset strategy is adjusted according to the performance evaluation result of the basic model.
[0010] Furthermore, in the above-mentioned method for labeling training data for model training, after the step of iteratively training the basic model using the new batch of labeled data, the method further comprises: When a preset iteration termination condition is met, the training of the basic model is terminated; The iteration termination condition includes when the amount of labeled data reaches a preset threshold or the performance of the model reaches a preset requirement.
[0011] Another object of the present invention is to provide a device for labeling training data for model training, the device comprising: An acquisition module is used to acquire an initial data set, select a part of the data set from the initial data set according to a preset rule, and initially annotate the part of the data set to obtain an initial annotated data set; A training module, used for performing model training using the initial annotated data set to obtain a basic model; The labeling module is used to label the remaining unlabeled data of the initial data set using a preset strategy to obtain a new batch of labeled data, and then use the new batch of labeled data to iteratively train the basic model.
[0012] Furthermore, in the above-mentioned training data labeling device for model training, the labeling module includes: A first labeling unit, configured to predict the unlabeled data using the basic model, and select samples with uncertain prediction results from the unlabeled data for labeling; The uncertainty of the prediction result is determined based on the probability distribution, confidence or entropy output by the basic model.
[0013] Another object of the present invention is to provide a readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of any one of the methods described above.
[0014] Another object of the present invention is to provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.
[0015] The present invention obtains an initial data set, selects part of the data set from the initial data set according to preset rules, and performs initial annotation on the part of the data set to obtain an initial annotated data set; performs model training on the initial annotated data set to obtain a basic model; uses a preset strategy to annotate the remaining unlabeled data of the initial data set to obtain a new batch of annotated data, and then uses the new batch of annotated data to iteratively train the basic model, thereby improving efficiency and accuracy. Through the preset strategy, the model can autonomously select the data that most needs to be annotated, thereby reducing the annotation cost; at the same time, by continuously optimizing the training of the model, the quality and consistency of the annotated data can be guaranteed. The problem that the annotation cost and annotation quality cannot be taken into account when annotating data in the prior art is solved. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 It is a flowchart of a method for labeling training data for model training in the first embodiment of the present invention; Figure 2 Schematic diagram of the structure of a training data labeling device for model training in the third embodiment of the present invention.
[0017] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION
[0018] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.
[0019] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.
[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.
[0021] The following will explain in detail how to improve both the cost and quality of data annotation in conjunction with specific embodiments and drawings.
[0022] Embodiment 1 See also Figure 1 , shown is a method for labeling training data for model training in the first embodiment of the present invention, and the method includes steps S10~S12.
[0023] Step S10, obtaining an initial data set, selecting a partial data set from the initial data set according to a preset rule, and initially annotating the partial data set to obtain an initial annotated data set.
[0024] Among them, from the entire initial data set, a portion of representative data is selected for initial annotation based on factors such as data distribution, category, and diversity. This portion of data should be able to reflect the characteristics and structure of the entire data set, and the initial annotation data is obtained by annotating this portion of data.
[0025] Step S11, performing model training using the initial annotated data set to obtain a basic model.
[0026] Among them, a basic model is trained using the initial labeled dataset. Although the initial performance of this model may not be high, it is the starting point for the subsequent active learning process.
[0027] Step S12, using a preset strategy to label the remaining unlabeled data of the initial data set to obtain a new batch of labeled data, and then using the new batch of labeled data to iteratively train the basic model. Among them, a variety of strategies can be used to label the remaining unlabeled data, including uncertainty sampling strategy, representative sampling strategy and hybrid sampling strategy. Among them, the uncertainty sampling strategy is that the model predicts the unlabeled data and selects the samples with the most uncertain prediction results for labeling. Uncertainty can be measured by metrics such as probability distribution, confidence or entropy of model output. In addition to uncertainty, the model also considers the representativeness of the data. The representative sampling strategy is implemented by calculating the similarity or distance between the sample and the labeled data to ensure that the selected sample can cover the distribution of the entire data set; the hybrid sampling strategy combines uncertainty and representative sampling strategies, which not only considers the uncertainty of the model but also ensures the representativeness of the data.
[0028] Furthermore, according to the preset active learning strategy, the model selects a batch of data that needs to be labeled the most each time. This data is sent to labelers or automated tools for labeling to ensure the accuracy and consistency of the labeling. The newly labeled data is added to the training set to retrain the model. This process is iterated continuously, and as the number of labeled data increases, the performance of the model gradually improves.
[0029] In summary, the method for labeling training data for model training in the above embodiments of the present invention obtains an initial data set, selects a part of the data set from the initial data set according to preset rules, and initially labels the part of the data set to obtain an initial labeled data set; performs model training on the initial labeled data set to obtain a basic model; labels the remaining unlabeled data of the initial data set using a preset strategy to obtain a new batch of labeled data, and then iteratively trains the basic model using the new batch of labeled data, thereby improving efficiency and accuracy. Through the preset strategy, the model can autonomously select the data that most needs to be labeled, thereby reducing the labeling cost; at the same time, by continuously optimizing the training of the model, the quality and consistency of the labeled data can be guaranteed. This solves the problem in the prior art that it is impossible to take into account both the labeling cost and the labeling quality when labeling data.
[0030] Embodiment 2 The embodiment of the present invention also proposes a method for labeling training data for model training. The difference between the method for labeling training data for model training proposed in this embodiment and the method for labeling training data for model training proposed in this embodiment is that: After step S12, the following steps are also included: The performance of the basic model is evaluated, and the preset strategy is adjusted according to the performance evaluation result of the basic model.
[0031] After each iteration, the performance of the model is evaluated, including indicators such as accuracy, recall, and F1 value. This helps to understand the performance of the model under the current amount of labeled data, and adjust the active learning strategy based on the performance evaluation results of the model. For example, if the model performs poorly on certain categories, the strategy can be adjusted to select more samples of these categories for labeling.
[0032] After step S12, the following steps are also included: When a preset iteration termination condition is met, the training of the basic model is terminated; The iteration termination condition includes when the amount of labeled data reaches a preset threshold or the performance of the model reaches a preset requirement.
[0033] Among them, when the amount of labeled data reaches a preset threshold, the iteration is stopped. If the performance of the model meets the preset requirements, the iteration can also be stopped in advance.
[0034] The method for labeling training data for model training in the above-mentioned embodiment of the present invention obtains an initial data set, selects a part of the data set from the initial data set according to a preset rule, and initially labels the part of the data set to obtain an initial labeled data set; performs model training on the initial labeled data set to obtain a basic model; labels the remaining unlabeled data of the initial data set using a preset strategy to obtain a new batch of labeled data, and then iteratively trains the basic model using the new batch of labeled data, thereby improving efficiency and accuracy. Through the preset strategy, the model can autonomously select the data that most needs to be labeled, thereby reducing the labeling cost; at the same time, by continuously optimizing the training of the model, the quality and consistency of the labeled data can be guaranteed. This solves the problem in the prior art that it is impossible to take into account both the labeling cost and the labeling quality when labeling data.
[0035] Embodiment 3 See also Figure 2 , shown is a training data labeling device for model training proposed in the second embodiment of the present invention, the device comprising: The acquisition module 100 is used to acquire an initial data set, select a part of the data set from the initial data set according to a preset rule, and perform initial annotation on the part of the data set to obtain an initial annotated data set; A training module 200 is used to perform model training using the initial annotated data set to obtain a basic model; The labeling module 300 is used to label the remaining unlabeled data of the initial data set using a preset strategy to obtain a new batch of labeled data, and then use the new batch of labeled data to iteratively train the basic model.
[0036] Furthermore, in the above-mentioned training data labeling device for model training, the labeling module includes: A first labeling unit, configured to predict the unlabeled data using the basic model, and select samples with uncertain prediction results from the unlabeled data for labeling; The uncertainty of the prediction result is determined based on the probability distribution, confidence or entropy output by the basic model.
[0037] Furthermore, in the above-mentioned training data labeling device for model training, the labeling module further comprises: The second labeling unit is used to calculate the similarity or distance between the unlabeled data and the labeled data to determine the data that needs to be labeled among the unlabeled data, and label the data that needs to be labeled to obtain a new batch of labeled data. Furthermore, in the above-mentioned training data labeling device for model training, the labeling module further comprises: A third labeling unit, configured to calculate the similarity or distance between the unlabeled data and the labeled data to determine the data that needs to be labeled among the unlabeled data; The basic model is used to predict the data to be labeled, and samples with uncertain prediction results are selected from the data to be labeled for labeling.
[0038] Furthermore, the above-mentioned training data labeling device for model training, wherein the device also includes: The adjustment module is used to evaluate the performance of the basic model and adjust the preset strategy according to the performance evaluation result of the basic model.
[0039] Furthermore, the above-mentioned training data labeling device for model training, wherein the device also includes: A termination module, used to terminate the training of the basic model when a preset iteration termination condition is met; The iteration termination condition includes when the amount of labeled data reaches a preset threshold or the performance of the model reaches a preset requirement.
[0040] The functions or operation steps implemented when the above modules are executed are substantially the same as those in the above method embodiments, and will not be repeated here.
[0041] Embodiment 4 Another aspect of the present invention further provides a readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the method described in the above-mentioned embodiments 1 to 2.
[0042] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0043] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable storage medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, a "computer-readable storage medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.
[0044] More specific examples (a non-exhaustive list) of computer-readable storage media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable storage medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.
[0045] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0046] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0047] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.
Claims
1. A method for labeling training data for model training, characterized in that: The method comprises: Acquire an initial data set, select a portion of the data set from the initial data set according to a preset rule, and perform initial annotation on the portion of the data set to obtain an initial annotated data set; Performing model training using the initial labeled data set to obtain a basic model; The remaining unlabeled data of the initial data set are labeled using a preset strategy to obtain a new batch of labeled data, and then the basic model is iteratively trained using the new batch of labeled data.
2. The method for labeling training data for model training according to claim 1, characterized in that: The step of labeling the remaining unlabeled data of the initial data set using a preset strategy to obtain a new batch of labeled data includes: Predicting the unlabeled data using the basic model, and selecting samples with uncertain prediction results from the unlabeled data for labeling; The uncertainty of the prediction result is determined based on the probability distribution, confidence or entropy output by the basic model.
3. The method for labeling training data for model training according to claim 1, characterized in that: The step of labeling the remaining unlabeled data of the initial data set using a preset strategy to obtain a new batch of labeled data also includes: The similarity or distance between the unlabeled data and the labeled data is calculated to determine the data that needs to be labeled among the unlabeled data, and the data that needs to be labeled is labeled to obtain a new batch of labeled data.
4. The method for labeling training data for model training according to claim 3, characterized in that: The step of labeling the remaining unlabeled data of the initial data set using a preset strategy to obtain a new batch of labeled data also includes: Calculating the similarity or distance between the unlabeled data and the labeled data to determine the data that needs to be labeled among the unlabeled data; The basic model is used to predict the data to be labeled, and samples with uncertain prediction results are selected from the data to be labeled for labeling.
5. The method for labeling training data for model training according to claim 1, characterized in that: After the step of iteratively training the basic model using the new batch of labeled data, the following step is further included: The performance of the basic model is evaluated, and the preset strategy is adjusted according to the performance evaluation result of the basic model.
6. The method for labeling training data for model training according to claim 1, characterized in that: After the step of iteratively training the basic model using the new batch of labeled data, the following step is further included: When a preset iteration termination condition is met, the training of the basic model is terminated; The iteration termination condition includes when the amount of labeled data reaches a preset threshold or the performance of the model reaches a preset requirement.
7. A training data labeling device for model training, characterized in that: The device comprises: An acquisition module is used to acquire an initial data set, select a part of the data set from the initial data set according to a preset rule, and initially annotate the part of the data set to obtain an initial annotated data set; A training module, used for performing model training using the initial annotated data set to obtain a basic model; The labeling module is used to label the remaining unlabeled data of the initial data set using a preset strategy to obtain a new batch of labeled data, and then use the new batch of labeled data to iteratively train the basic model.
8. The training data labeling device for model training according to claim 7, characterized in that: The marking module comprises: A first labeling unit, configured to predict the unlabeled data using the basic model, and select samples with uncertain prediction results from the unlabeled data for labeling; The uncertainty of the prediction result is determined based on the probability distribution, confidence or entropy output by the basic model.
9. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.