Data annotation method and device, computer equipment and storage medium

By pre-processing and cleaning before data annotation, using multi-model annotation and real-time feedback adjustment, and dynamically optimizing the model, the problem of difficult to guarantee the labeling quality in complex data centers is solved, and high-quality annotation and model performance are improved.

CN118916733APending Publication Date: 2024-11-08CHONGQING QINGZHUO TECHNOLOGY CO LTD
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202410845079.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-27
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The prior art is difficult to effectively determine high-quality target annotation sample sets in complex and diverse, unevenly distributed or large amounts of noise.

Method used

Preprocess and clean before data annotation, use multiple teacher models for annotation, and integrate the results through integrated learning and fusion, collect feedback and evaluation of the labeling personnel in real time, dynamically adjust model parameters and structure, and continuously optimize model performance using incremental learning technology.

Benefits of technology

By removing noise and outliers, the accuracy and consistency of the annotation results are improved, the high quality of the annotation process is ensured, the performance and generalization capabilities of the model are enhanced, the complex and diverse data sets are adapted to and effectively determine the high-quality target annotation sample set.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916733A_ABST
    Figure CN118916733A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of data processing, in particular to a data annotation method and device, computer equipment and a storage medium, and the method comprises the steps: carrying out the preprocessing and cleaning of an original data set before data annotation, so as to remove noise and abnormal values; labeling the same data set by using a plurality of different teacher models to obtain a plurality of labeling results; in the data labeling process, feedback and evaluation results of labeling personnel are collected in real time; after each training period is finished, evaluating the quality of the annotated data; and dynamically adjusting parameters and structures of the teacher model and the student model according to a quality evaluation result of the annotated data. The preprocessing and cleaning steps are beneficial to removing noise and abnormal values in the data; by means of multi-model labeling and integration, the accuracy and consistency of labeling results can be improved; real-time feedback and quality control can ensure high-quality operation of the labeling process; and a high-quality target labeling sample set is effectively determined.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data labeling method, device, computer equipment and storage medium. Background Art

[0002] Labeling sample data is a common step before training a deep learning model. At present, in order to increase the efficiency of labeling sample data, an automatic labeling device may be used to automatically label sample data. However, when the existing technology automatically labels sample data, it is difficult to ensure the quality of the labeled samples.

[0003] The existing announcement number CN112200273B discloses a data labeling method, device, equipment and computer storage medium; wherein, the data labeling method includes: inputting the sample data set to be labeled into the teacher model for labeling, obtaining a first labeled sample set, and the credibility of each labeled sample in the first labeled sample set; within a training cycle, determining the credibility threshold, determining a second labeled sample set from the first labeled sample set, using the second labeled sample set to train the student model, obtaining the trained student model, and obtaining the target evaluation index of the trained student model; according to the N target evaluation indexes obtained within N training cycles, determining the target labeled sample set from the N second labeled sample sets determined within N training cycles, N is an integer greater than 1. It is possible to verify the quality of the target labeled sample set and effectively ensure the quality of the target labeled sample set.

[0004] However, for complex, diverse, unevenly distributed, or noisy datasets, this method may not be able to effectively determine a high-quality set of target annotation samples. Summary of the invention

[0005] The purpose of the present invention is to provide a data labeling method, device, computer equipment and storage medium, which solves the problem that the method cannot effectively determine a high-quality target labeling sample set for complex, diverse, unevenly distributed or noisy data sets.

[0006] To achieve the above object, the present invention provides a data labeling method, comprising the following steps:

[0007] Before data annotation, the original data set is preprocessed and cleaned to remove noise and outliers;

[0008] Use multiple different teacher models to annotate the same dataset to obtain multiple annotation results;

[0009] During the data annotation process, feedback and evaluation results from annotators are collected in real time;

[0010] After each training cycle, the quality of the labeled data is evaluated;

[0011] According to the quality assessment results of the labeled data, the parameters and structures of the teacher model and the student model are dynamically adjusted.

[0012] Before data annotation, the original data set is preprocessed and cleaned to remove noise and outliers. The steps also include:

[0013] Use statistical methods to identify and correct errors and biases in raw data sets;

[0014] Apply machine learning techniques to automatically identify and filter noisy data;

[0015] Clean the data, including removing duplicate samples, handling missing values, and eliminating outliers.

[0016] Wherein, using multiple different teacher models to annotate the same data set to obtain multiple annotation results, the step further includes:

[0017] Train multiple different teacher models;

[0018] Use these teacher models to annotate the same cleaned dataset and obtain multiple annotation results;

[0019] Ensemble learning technology is used to merge multiple annotation results into one annotation result.

[0020] In the data labeling process, feedback and evaluation results from labelers are collected in real time. The steps also include:

[0021] During the data annotation process, feedback and evaluation results from annotators are collected in real time;

[0022] Analyze feedback and evaluation results to understand problems and difficulties in the annotation process;

[0023] Adjust the annotation process in real time based on the analysis results.

[0024] Wherein, after each training cycle, the quality of the labeled data is evaluated, and the steps further include:

[0025] Use cross-validation, confusion matrix, accuracy, recall and other indicators to quantitatively evaluate the quality of labeled data;

[0026] Analyze the evaluation results to understand the deficiencies and problems in the labeled data.

[0027] According to the quality evaluation results of the labeled data, the parameters and structures of the teacher model and the student model are dynamically adjusted, and the steps further include:

[0028] Use incremental learning techniques to gradually add newly annotated high-quality data to the model to update the model and improve its performance and generalization ability;

[0029] Repeat the above steps to continuously iterate and optimize the data labeling process and model performance.

[0030] A data labeling device, comprising a data cleaning module, a labeling training module, an evaluation module and an optimization module, wherein the data cleaning module is connected to the labeling training module, the evaluation module is connected to the labeling training module, and the optimization module is connected to the evaluation module and the labeling training module respectively;

[0031] The data cleaning module is used to obtain raw data and preprocess and clean the raw data set to remove noise and outliers;

[0032] The labeling training module is used to label the same cleaned data set using multiple different teacher models to obtain multiple labeling results, merge the multiple labeling results into one labeling result, and use it as training data for the data set of the learning model;

[0033] The evaluation module is used to collect the feedback and evaluation results of the annotators in real time during the data annotation process, and to evaluate the quality of the annotated data after each training cycle;

[0034] The optimization module is used to dynamically adjust the parameters and structure of the teacher model and the student model according to the quality evaluation results of the labeled data.

[0035] A data labeling computer device executes the data labeling method.

[0036] A data labeling storage medium stores computer program instructions, and the computer program instructions implement the data labeling method.

[0037] The data labeling method, device, computer equipment and storage medium of the present invention preprocess and clean the original data set before data labeling to remove noise and outliers, use statistical methods or machine learning technology to identify and correct errors and deviations in the data set, use multiple different teacher models to label the same data set to obtain multiple labeling results, and merge the multiple labeling results into a more accurate labeling result through integrated learning technology. During the data labeling process, the feedback and evaluation results of the labeling personnel are collected in real time, and the labeling process is adjusted in real time according to the feedback and evaluation results, such as adjusting the labeling difficulty, adding labeling guidance, etc. After each training cycle, the quality of the labeled data is evaluated, and the quality of the labeled data is evaluated using cross-validation, confusion matrix, accuracy, recall rate and other indicators. According to the quality evaluation results of the labeled data, the parameters and structures of the teacher model and the student model are dynamically adjusted, and the newly labeled high-quality data are gradually added to the model using incremental learning technology to improve the performance and generalization ability of the model. Effectively handle complex and diverse data sets with uneven distribution or a large amount of noise. Preprocessing and cleaning steps help remove noise and outliers in the data; multi-model annotation and integration can improve the accuracy and consistency of annotation results; real-time feedback and quality control can ensure the high quality of the annotation process; quality assessment of the annotated data can ensure the quality of the final annotated data; incremental learning and dynamic adjustment can continuously improve the performance and generalization ability of the model. These improvements enable the method to better adapt to complex and diverse data sets and effectively determine high-quality target annotation sample sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0039] Figure 1 It is a flowchart of the steps of the data labeling method according to the first embodiment of the present invention.

[0040] Figure 2 It is a structural block diagram of a data labeling device according to a second embodiment of the present invention.

[0041] In the figure: 201-data cleaning module, 202-labeling training module, 203-evaluation module, 204-optimization module. DETAILED DESCRIPTION

[0042] Embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0043] The first embodiment of the present application is:

[0044] See also Figure 1 ,in, Figure 1 1 is a flowchart of the steps of the data labeling method of the first embodiment of the present invention. The present invention provides a data labeling method, comprising the following steps:

[0045] S101: Before data annotation, the original data set is preprocessed and cleaned to remove noise and outliers;

[0046] Specifically, load the original data set, identify and remove noise, outliers, duplicate data, etc., use statistical methods (such as box plots to detect outliers) or machine learning techniques (such as clustering algorithms to identify outliers) to assist the cleaning process and ensure the integrity and consistency of the data set. Among them, use statistical methods to identify and correct errors and deviations in the original data set, apply machine learning techniques to automatically identify and filter noise data, and for missing values ​​in the data set, select appropriate processing methods based on data characteristics and business needs, such as filling the mean, median, mode, or using machine learning algorithms for predictive filling. Finally, convert the data into a format suitable for annotation, such as converting image data into image files of a specific size, resolution, or color space.

[0047] S102: using multiple different teacher models to annotate the same data set to obtain multiple annotation results;

[0048] Specifically, multiple different teacher models are selected according to task requirements and data types. These models can be pre-trained machine learning models or models customized for different tasks. The same data set is annotated using the selected teacher model to obtain multiple annotation results. Since different teacher models may be based on different algorithms, architectures, or training data, they may produce different annotation results. The annotation results of multiple teacher models are integrated to form a data set containing multiple annotations. The methods used include voting (such as majority voting, weighted voting, etc.), averaging (such as simple averaging, weighted averaging, etc.) or other integrated learning techniques to extract a more accurate annotation result from multiple annotation results. The integrated annotated data is used as training data to prepare a data set for learning the model. According to the task requirements and data characteristics, a suitable learning model is selected. This model can be a supervised learning model, an unsupervised learning model, or a semi-supervised learning model. The selected model is trained using the prepared training data. During the training process, the model learns the ability to extract features from the data and predict the target variable.

[0049] S103: During the data annotation process, feedback and evaluation results from annotation personnel are collected in real time;

[0050] Specifically, during the data labeling process, a labeling interface is designed to allow labelers to provide real-time feedback and collect their feedback, including labeling difficulty, error reports, suggestions, etc. Labeling guidance, interface layout, task allocation, etc. are adjusted based on the feedback to improve labeling efficiency and accuracy. Labeling results are regularly spot-checked and evaluated to ensure labeling quality.

[0051] S104: After each training cycle, the quality of the labeled data is evaluated;

[0052] Specifically, after the model training is completed, the quality of the labeled data is quantitatively evaluated using indicators such as cross-validation, confusion matrix, accuracy, recall, and F1 score. Human experts can also be used to label a portion of the data and calculate the degree of consistency with the machine labeling as a reference for quality assessment.

[0053] S105: Dynamically adjust the parameters and structures of the teacher model and the student model according to the quality assessment results of the labeled data.

[0054] Specifically, according to the quality evaluation results, the parameters and structures of the teacher model and the student model are dynamically adjusted. Using incremental learning technology, newly annotated high-quality data is gradually added to the model to improve the performance and generalization ability of the model. In continuous training cycles, monitor the changes in the target evaluation indicators (such as the accuracy on the validation set), and determine when to stop training or continue to add new data based on these changes. After multiple training cycles, collect the second annotated sample set determined in each cycle (samples screened based on the credibility threshold). Use the validation set or test set to evaluate these annotated sample sets and calculate the target evaluation indicators (such as accuracy, recall, etc.). Based on the N target evaluation indicators obtained in N training cycles, comprehensively determine a target annotated sample set, which should have the highest quality evaluation score. Use the target annotated sample set for subsequent training or fine-tuning processes to further improve the performance of the model. Repeat the above steps to form an iteratively optimized data annotation and model training process.

[0055] Effectively handle complex, diverse, unevenly distributed, or noisy data sets. Preprocessing and cleaning steps help remove noise and outliers in the data; multi-model annotation and integration can improve the accuracy and consistency of annotation results; real-time feedback and quality control can ensure the high quality of the annotation process; quality assessment of the annotation data can ensure the quality of the final annotation data; incremental learning and dynamic adjustment can continuously improve the performance and generalization ability of the model. These improvements enable the method to better adapt to complex and diverse data sets and effectively determine high-quality target annotation sample sets.

[0056] The second embodiment of the present application is:

[0057] Based on the first embodiment, please refer to Figure 2 ,in, Figure 2 2 is a structural block diagram of a data labeling device according to a second embodiment of the present invention. A data labeling device according to this embodiment includes a data cleaning module 201 , a labeling training module 202 , an evaluation module 203 and an optimization module 204 .

[0058] According to this specific implementation, the data cleaning module 201 is connected to the labeling training module 202, the evaluation module 203 is connected to the labeling training module 202, and the optimization module 204 is connected to the evaluation module 203 and the labeling training module 202 respectively;

[0059] The data cleaning module 201 is used to obtain raw data and pre-process and clean the raw data set to remove noise and outliers;

[0060] The labeling training module 202 is used to label the same cleaned data set using multiple different teacher models to obtain multiple labeling results, merge the multiple labeling results into one labeling result, and use it as training data for the data set of the learning model;

[0061] The evaluation module 203 is used to collect the feedback and evaluation results of the annotators in real time during the data annotation process, and to evaluate the quality of the annotated data after each training cycle.

[0062] The optimization module 204 is used to dynamically adjust the parameters and structures of the teacher model and the student model according to the quality evaluation results of the labeled data.

[0063] Using a data labeling device of the present embodiment, the data cleaning module 201 preprocesses and cleans the original data set to remove noise and outliers, and the labeling training module 202 selects multiple different teacher models to label the same data set, obtains multiple labeling results, and integrates the labeling results of multiple teacher models to form a data set containing multiple labels. The methods used include voting (such as majority voting, weighted voting, etc.), averaging (such as simple averaging, weighted averaging, etc.) or other integrated learning techniques to extract a more accurate labeling result from multiple labeling results, use the integrated labeling data as training data, and prepare a data set for training the learning model. During the data labeling process, the evaluation module 203 collects the feedback and evaluation results of the labelers in real time, analyzes the feedback and evaluation results, understands the problems and difficulties in the labeling process, and adjusts the labeling process in real time according to the analysis results, such as adjusting the labeling difficulty, adding labeling guidance, etc., and after each training cycle, performs a quality evaluation on the labeled data, uses cross-validation, confusion matrix, accuracy, recall and other indicators to quantitatively evaluate the quality of the labeled data, analyzes the evaluation results, and understands the deficiencies and problems in the labeled data. The optimization module 204 dynamically adjusts the parameters and structures of the teacher model and the student model according to the quality evaluation results, uses incremental learning technology to gradually add the newly labeled high-quality data to the model to update the model and improve its performance and generalization ability, repeats the above steps, and continuously iterates and optimizes the data labeling process and model performance. Effectively handle complex, diverse, unevenly distributed or noisy data sets. Preprocessing and cleaning steps help remove noise and outliers in the data; multi-model annotation and integration can improve the accuracy and consistency of annotation results; real-time feedback and quality control can ensure the high quality of the annotation process; quality assessment of the annotated data can ensure the quality of the final annotated data; incremental learning and dynamic adjustment can continuously improve the performance and generalization ability of the model. These improvements enable the method to better adapt to complex and diverse data sets and effectively determine high-quality target annotation sample sets.

[0064] The third embodiment of the present application is:

[0065] Based on the first embodiment, a data labeling computer device in this embodiment is used to execute the data labeling method.

[0066] The fourth embodiment of the present application is:

[0067] Based on the first embodiment, a data annotation storage medium of this embodiment stores computer program instructions, and the computer program instructions implement the data annotation method.

[0068] What is disclosed above is only one or more preferred embodiments of the present application, and cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of implementing the above embodiments and equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data annotation method, It is characterized in that The following steps are involved: Before data annotation, the original data set is preprocessed and cleaned to remove noise and outliers; Use multiple different teacher models to annotate the same dataset to obtain multiple annotation results; During the data annotation process, feedback and evaluation results from annotators are collected in real time; After each training cycle, the quality of the labeled data is evaluated; According to the quality assessment results of the labeled data, the parameters and structures of the teacher model and the student model are dynamically adjusted.

2. The data labeling method according to claim 1, It is characterized in that Before data annotation, the original data set is preprocessed and cleaned to remove noise and outliers. The steps also include: Use statistical methods to identify and correct errors and biases in raw data sets; Apply machine learning techniques to automatically identify and filter noisy data; Clean the data, including removing duplicate samples, handling missing values, and eliminating outliers.

3. The data annotation method according to claim 2, It is characterized in that Using multiple different teacher models to annotate the same data set to obtain multiple annotation results, the step further includes: Train multiple different teacher models; Use these teacher models to annotate the same cleaned dataset and obtain multiple annotation results; Ensemble learning technology is used to merge multiple annotation results into one annotation result.

4. The data annotation method according to claim 3, It is characterized in that During the data annotation process, feedback and evaluation results from the annotators are collected in real time. The steps also include: During the data annotation process, feedback and evaluation results from annotators are collected in real time; Analyze feedback and evaluation results to understand problems and difficulties in the annotation process; Adjust the annotation process in real time based on the analysis results.

5. The data annotation method according to claim 4, It is characterized in that After each training cycle, the quality of the labeled data is evaluated, and the steps further include: Use cross-validation, confusion matrix, accuracy, recall and other indicators to quantitatively evaluate the quality of labeled data; Analyze the evaluation results to understand the deficiencies and problems in the labeled data.

6. The data labeling method according to claim 5, It is characterized in that According to the quality evaluation results of the labeled data, dynamically adjusting the parameters and structures of the teacher model and the student model, the steps further include: Use incremental learning techniques to gradually add newly annotated high-quality data to the model to update the model and improve its performance and generalization ability; Repeat the above steps to continuously iterate and optimize the data labeling process and model performance.

7. A data labeling device, applicable to the data labeling method according to any one of claims 1 to 6, It is characterized in that It includes a data cleaning module, a labeling training module, an evaluation module and an optimization module, wherein the data cleaning module is connected to the labeling training module, the evaluation module is connected to the labeling training module, and the optimization module is connected to the evaluation module and the labeling training module respectively; The data cleaning module is used to obtain raw data and preprocess and clean the raw data set to remove noise and outliers; The labeling training module is used to label the same cleaned data set using multiple different teacher models to obtain multiple labeling results, merge the multiple labeling results into one labeling result, and use it as training data for the data set of the learning model; The evaluation module is used to collect the feedback and evaluation results of the annotators in real time during the data annotation process, and to evaluate the quality of the annotated data after each training cycle; The optimization module is used to dynamically adjust the parameters and structure of the teacher model and the student model according to the quality evaluation results of the labeled data.

8. A data labeling computer device, which executes the data labeling method according to any one of claims 1 to 6.

9. A data annotation storage medium storing computer program instructions, wherein the computer program instructions implement the data annotation method according to any one of claims 1 to 6.

Citation Information

Cited By

  • Active learning-based annotation data quality evaluation method

    CN119149919A

  • Data labeling method and device, computer equipment and storage medium

    CN119720944A

  • Target recognition model sample collecting and training method for small sample problem

    CN119964062A

  • Processing method and system for collecting, cleaning and labeling culture big data and medium

    CN120296078A

  • Model application-oriented multi-modal data set labeling management method and system

    CN120724391A