Data labeling method, device and system based on incremental learning and confidence learning

Abstract

The application relates to the technical field of electric digital processing, in particular to a data labeling method, device and system based on incremental learning and confidence learning. The method comprises the following steps: training a supervised learning model in a first stage through a first data set labeled by manual labeling; inputting an unlabeled second data set into the supervised learning model trained in the first stage; mixing the first data set and the second data set to divide them into multiple data subsets; training a target model by using a part of the subsets to obtain labels and prediction probabilities of another part of the subsets; selecting data belonging to the first data set in the another part of the subsets and calculating a benchmark value of correct labels; selecting data belonging to the second data set in the another part of the subsets and retaining data with prediction probabilities greater than or equal to the benchmark value as incremental training samples. The application can reduce manual operation, ensure the accuracy and high quality of labels and improve the accuracy of model labeling.

Description

Technical Field

[0001] This application relates to the field of electronic digital processing technology, and more specifically, to a data labeling method, device, and system based on incremental learning and confidence learning. Background Technology

[0002] With the rapid development of the internet, big data, and artificial intelligence technologies, the amount of data is growing exponentially. To extract valuable information from this massive amount of data and apply it to the training of machine learning and deep learning models, accurate data labeling is essential. Data labeling provides "supervisory signals" for the model and is fundamental to building high-quality machine learning models. Current technologies primarily employ two methods: manual labeling and model labeling.

[0003] Manual annotation is the most traditional and widely used annotation method, typically involving annotators assigning labels to data samples according to predetermined standards. In this process, the annotator reviews each data sample and assigns labels based on task requirements. The annotator completes the annotation work through a specific interface and submits the label data to a database. This process usually involves viewing, analyzing, and assigning labels to each sample individually, and is suitable for tasks requiring high-precision annotation.

[0004] Model labeling, on the other hand, automatically generates labels using a trained machine learning model. This method first trains a classification or regression model on a manually labeled dataset, then uses the trained model to predict unlabeled data, automatically generating labels. These automatically generated labels are validated using predefined standards and are suitable for large-scale datasets.

[0005] Although manual and model annotation are widely used in data annotation, they still face many challenges. While manual annotation can provide high-quality labels, its biggest problems are high cost and inefficiency, especially with large datasets where the cost increases dramatically. Manual annotation requires significant time and manpower, and annotators may make errors due to fatigue or inconsistent standards, leading to inconsistent label accuracy. Especially in scenarios requiring frequent label updates, the efficiency of manual annotation cannot meet the needs of large-scale datasets. Model annotation can reduce the workload of manual annotation to some extent, but the quality of the generated labels is not consistent. Especially in scenarios with complex data and high noise levels, model-annotated labels may contain errors, requiring manual correction. Although model annotation reduces some manual annotation work, it still faces the problems of high cost and inefficiency. Therefore, how to ensure the accuracy and high quality of labels while reducing manual operations is a pressing technical challenge in the field of data annotation. Summary of the Invention

[0006] The purpose of this application is to provide a data labeling method, device, and system based on incremental learning and confidence learning, so as to ensure the accuracy and high quality of labels while reducing manual operations, thereby improving the accuracy of model labeling.

[0007] To achieve the above objectives, this application adopts the following technical solution: Firstly, this application provides a data annotation method based on incremental learning and confidence learning, including: The first stage of training for the supervised learning model was conducted using a first dataset with manually labeled data. The unlabeled second dataset is input into the supervised learning model trained in the first stage to obtain the predicted labels output by the supervised learning model, and the predicted labels are then labeled on the second dataset. The first and second datasets are combined, and the combined dataset is then divided into multiple subsets. The target model is trained using a subset of the multiple data subsets, and the labels and predicted probabilities of another subset are obtained through the trained target model. In another subset, select data belonging to the first dataset and calculate the mean of the predicted probabilities of the correct labels as a baseline. In the other subset, select data belonging to the second dataset and compare the predicted probability of the correct label with the baseline value; Data with predicted probabilities greater than or equal to the baseline value are retained as incremental training samples for the second stage of training of the supervised learning model.

[0008] Secondly, this application provides a data annotation system based on incremental learning and confidence learning, comprising: The user terminal is used to manually label the first dataset. A processing device is configured to input an unlabeled second dataset into a supervised learning model trained in the first stage, obtain the predicted labels output by the supervised learning model, and label the predicted labels onto the second dataset; mix the first and second datasets and divide the mixed dataset into multiple data subsets; train a target model using a subset of the multiple data subsets, and obtain the labels and predicted probabilities of another subset using the trained target model; select data belonging to the first dataset from the other subset and calculate the mean predicted probability of the correct label as a baseline value; select data belonging to the second dataset from the other subset and compare the predicted probability of the correct label with the baseline value; retain data with a predicted probability greater than or equal to the baseline value as incremental training samples for the second stage training of the supervised learning model; Storage device for storing the first dataset, the second dataset, the supervised learning model, and the target model.

[0009] Thirdly, this application provides an electronic device, comprising: At least one processor, and a memory communicatively connected to at least one of the processors; The memory stores instructions that can be executed by at least one of the processors, which enable the at least one processor to perform a data labeling method based on incremental learning and confidence learning.

[0010] This application has the following substantial features and significant progress compared to the prior art: 1. By automatically filtering labels through confidence learning, the process of manually filtering low-quality labels is avoided. At the same time, the model is continuously optimized through incremental learning, which improves the efficiency of data labeling and processing.

[0011] 2. Utilize incremental learning to continuously update the model, enabling it to adapt to new data and maintain high performance over the long term, thus possessing self-iterative capabilities.

[0012] 3. By combining confidence learning and incremental learning, the system can automatically identify and prioritize the use of high-confidence labels for model training, thereby improving the training efficiency and accuracy of the model.

[0013] 4. Incremental corrections are made using a small number of manually labeled problem samples, which greatly reduces the reliance on a large number of manual labels and lowers the labeling cost. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0015] Figure 1 This is a flowchart of a data annotation method based on incremental learning and confidence learning provided in this embodiment; Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0016] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0017] Figure 1 This is a flowchart illustrating a data annotation method based on incremental learning and confidence learning provided in this embodiment. This embodiment is suitable for training a supervised learning model using a small number of manually labeled samples. This embodiment does not limit the network type or function of the supervised learning model, nor does it limit the type or annotation content of the training data. See also Figure 1 The method provided in this application includes: S110. The first stage of training of the supervised learning model is carried out using the first dataset with manually labeled data.

[0018] Supervised learning is a machine learning method that trains a model using sample data with known labels, enabling the model to learn the mapping relationship between input features and output labels, thereby achieving prediction of new data. In this embodiment, the supervised learning model can be a classification model (e.g., an object recognition model), such as a support vector machine model, a random forest model, or a convolutional neural network model.

[0019] Among them, the Support Vector Machine (SVM) model is a supervised learning model based on the principle of maximizing the margin, which achieves classification tasks by constructing the optimal splitting hyperplane. Random Forest is a supervised learning algorithm based on the ensemble of multiple decision trees, which achieves classification prediction by voting or averaging the results of multiple models. Convolutional Neural Network (CNN) is a deep learning model containing convolutional layers, mainly used for feature extraction and classification of spatially structured data such as images and speech.

[0020] The first dataset can be of the type of images, text, or audio, but it can also be any other type that the model can classify. For example, the dataset includes 1000 images, each depicting a different animal, such as a cat or a dog. Users browse 100 images and label the type of animal depicted on them in the image's attributes.

[0021] In practical applications, the first dataset is displayed sequentially on the user terminal's human-computer interaction interface. The user inputs the annotation content into the user terminal through a specific interface and binds it with the currently displayed data to complete the annotation.

[0022] Then, the supervised learning model is trained using a first dataset with manually labeled data, enabling the model to learn the relationship between data features and labels. Specifically, the model is trained by inputting labeled data samples and utilizing the labels to generate a model capable of predicting unknown data. For ease of description and distinction, the stage of training using the first dataset with manually labeled data is referred to as the first stage.

[0023] Optionally, the first dataset is first preprocessed, including at least outlier removal and encoding. Outlier removal uses a method based on three standard deviations, identifying sample points that deviate from the mean by more than three standard deviations as outliers and removing them. Encoding uses one-hot encoding to convert categorical features into numerical vectors.

[0024] Then, the first dataset is input into the supervised learning model to obtain the predicted labels output by the supervised learning model. The parameters of the supervised learning model are iterated to narrow the gap between the predicted labels and the manually labeled labels. For example, the preprocessed first dataset is randomly divided into a training set and a validation set in an 8:2 ratio, and the training set is input into the supervised learning model. After forward propagation, the predicted labels are obtained. The cross-entropy loss is calculated based on the predicted labels and the manually labeled labels. The weights and bias parameters of the supervised learning model are iteratively updated using the backpropagation algorithm combined with the Adam optimizer (learning rate set to 0.001). Each iteration uses a batch of samples (batch size of 32) and monitors the loss change on the validation set until the loss no longer decreases after 5 consecutive validation rounds. This stops the iteration, thus obtaining a supervised learning model that makes the predicted labels approximate the manually labeled labels. This is the supervised learning model after the first stage of training. Due to the small number of training samples, the model still needs to undergo a second stage of training.

[0025] S120. Input the unlabeled second dataset into the supervised learning model trained in the first stage to obtain the predicted labels output by the supervised learning model, and label the predicted labels on the second dataset.

[0026] After the first stage of training of the supervised learning model is completed, the model can predict and generate predicted labels for an unlabeled second dataset (the second dataset is distinguished from the first dataset only by its name, but the data types are the same, such as images, text, or audio). These predicted labels also serve as the self-labels for the supervised learning model. For example, 900 unlabeled images can be sequentially input into the supervised learning model to predict the animal species displayed in each image. The model automatically generates labels by predicting from unlabeled sample data (e.g., images, text, audio, etc.). The goal of this step is to quickly generate labels using the model's learning results, reducing the workload of manual labeling.

[0027] For example, using a trained supervised learning model: an image recognition model classifies a new batch of unlabeled images. The model predicts the category label for each image based on the features it has learned. For example, the model automatically labels an image as "cat".

[0028] S130. Mix the first dataset and the second dataset, and divide the mixed dataset into multiple data subsets.

[0029] While automatically generated labels can reduce the workload of manual annotation, the quality of these labels may be problematic. To address this, this application introduces a confidence learning mechanism. Confidence learning assesses the reliability of each generated label by calculating its predicted probability (or confidence level), thereby filtering out labels with high confidence levels—that is, more accurate labels. Labels with low confidence levels are considered potential noise labels and require further correction or removal. Confidence learning can effectively identify and handle errors or uncertainties in predicted labels, thus improving label accuracy.

[0030] Specifically, the first dataset (with manually labeled data) and the second dataset (with predicted labels) are merged, and then the data is shuffled. The shuffled dataset is then evenly divided into five subsets, labeled S3_1, S3_2, S3_3, S3_4, and S3_5. If the first and second datasets together contain 1000 images, then each subset contains 200 images.

[0031] S140. Train the target model using a subset of multiple data subsets, and obtain the labels and predicted probabilities of another subset through the trained target model.

[0032] The target model has the same network architecture as the supervised learning model. The target model is a pre-trained model. The target model can be pre-trained on a small dataset to obtain initial model parameters; and based on the initial model parameters, the parameters can be fine-tuned using a subset of the dataset.

[0033] To better train the target model, the majority of the data subsets are used for training. For example, the target model is trained using four data subsets, and prediction is performed using one data subset. Optionally, N-1 subsets that have not been selected as a whole are selected from the N data subsets; the target model is trained using these N-1 subsets; the labels and prediction probabilities of the remaining (one) subset are obtained from the trained target model; the selection operation of the N-1 subsets is returned, until all N-1 subsets have been selected as a whole.

[0034] For example, the first target model is trained using S3_1, S3_2, S3_3, and S3_4, and S3_5 is input into the first target model to obtain the predicted label and corresponding predicted probability. The second target model is trained using S3_1, S3_2, S3_3, and S3_5, and S3_4 is input into the second target model to obtain the predicted label and corresponding predicted probability. The third target model is trained using S3_1, S3_2, S3_4, and S3_5, and S3_3 is input into the third target model to obtain the predicted label and corresponding predicted probability. The fourth target model is trained using S3_1, S3_3, S3_4, and S3_5, and S3_2 is input into the fourth target model to obtain the predicted label and corresponding predicted probability. The fifth target model is trained using S3_2, S3_3, S3_4, and S3_5, and S3_1 is input into the fifth target model to obtain the predicted label and corresponding predicted probability.

[0035] S150. Select data belonging to the first dataset from another subset and calculate the mean predicted probability of the correct label as the baseline value.

[0036] S160. Select data belonging to the second dataset from another subset and compare the predicted probability of the correct label with the baseline value.

[0037] For the first target model, identify the portion of S3_5 belonging to the first dataset, filter for correct labels, and calculate the average predicted probability of the correct labels, denoted as the baseline value X. For example, if there are 20 images in S3_5 belonging to the first dataset, and 18 of them have correctly predicted labels, then the average predicted probability of these 18 images is 78%, which is the baseline value. Next, identify the portion of S3_5 belonging to the second dataset, filter for correct labels and their predicted probabilities, and only retain data with a predicted probability greater than or equal to 78%.

[0038] Similarly, for the second, third, fourth, and fifth objective models, the same method is used to determine the baseline value, and then another subset of data whose predicted probability is greater than or equal to the corresponding baseline value is selected.

[0039] S170. Retain data whose predicted probability is greater than or equal to the baseline value as incremental training samples for the second stage of training of the supervised learning model.

[0040] Optionally, for data with incorrect labels or data whose predicted probability is less than the baseline value, the labels are manually corrected; the corrected data is then used as incremental training samples to train the supervised learning model in the second stage.

[0041] For example, in image classification, suppose the model predicts a probability of 0.9 for the label "cat," while predicting a probability of only 0.5 for "dog" in another image. Through a confidence learning mechanism, the system will identify the low-confidence label (0.5) and mark it as unreliable, requiring manual correction or removal. During incremental model learning, a small number of problematic samples can be manually labeled for incremental correction. Problem samples are those that the model predicts incorrectly or have low confidence. Data annotators only need to label these samples, thus reducing the workload of manual labeling. In this way, the model can be continuously updated, requiring only a few incorrect labels, significantly reducing the need for manual labeling.

[0042] For example, in an image classification task, suppose the model makes predictions on a batch of 1000 images, generates labels, and performs confidence level filtering. The user only needs to label the 20 images with lower confidence levels, correct these labels, and then add them to the training set to continue optimizing the model.

[0043] The labels cleaned through confidence learning are then used for incremental learning to further optimize the model. Incremental learning refers to training the model using newly labeled samples (especially high-confidence labels) on top of the existing model, without retraining the entire model. This method not only keeps the model continuously updated but also effectively avoids redundant computation, improving the model's learning efficiency. For example, all data with a predicted probability greater than or equal to 78% can be used as incremental training samples to retrain the supervised learning model, improving the model's classification accuracy for new image samples.

[0044] This application also provides a data annotation system based on incremental learning and confidence learning, including: The user terminal is used to manually label the first dataset. A processing device is configured to input an unlabeled second dataset into a supervised learning model trained in the first stage, obtain the predicted labels output by the supervised learning model, and label the predicted labels onto the second dataset; mix the first and second datasets and divide the mixed dataset into multiple data subsets; train a target model using a subset of the multiple data subsets, and obtain the labels and predicted probabilities of another subset using the trained target model; select data belonging to the first dataset from the other subset and calculate the mean predicted probability of the correct label as a baseline value; select data belonging to the second dataset from the other subset and compare the predicted probability of the correct label with the baseline value; retain data with a predicted probability greater than or equal to the baseline value as incremental training samples for the second stage training of the supervised learning model; Storage device for storing the first dataset, the second dataset, the supervised learning model, and the target model.

[0045] Optionally, the user terminal has a human-computer interaction interface; the human-computer interaction interface displays the training progress (e.g., the first and second stages, the display of predicted labels and probabilities), and supports users to manually divide the data subsets.

[0046] In summary, this application has the following technical effects: 1. By automatically filtering labels through confidence learning, the process of manually filtering low-quality labels is avoided. At the same time, the model is continuously optimized through incremental learning, which improves the efficiency of data labeling and processing.

[0047] 2. Utilize incremental learning to continuously update the model, enabling it to adapt to new data and maintain high performance over the long term, thus possessing self-iterative capabilities.

[0048] 3. By combining confidence learning and incremental learning, the system can automatically identify and prioritize the use of high-confidence labels for model training, thereby improving the training efficiency and accuracy of the model.

[0049] 4. Incremental corrections are made using a small number of manually labeled problem samples, which greatly reduces the reliance on a large number of manual labels and lowers the labeling cost.

[0050] This embodiment provides an electronic device, see [link / reference] Figure 2 It includes at least one processor 301 and a memory 302 communicatively connected to at least one of the processors 301; The memory 302 stores instructions that can be executed by at least one of the processors 301, which enable at least one of the processors 301 to perform the above-described data labeling method based on incremental learning and confidence learning, and thus have at least the same advantages as the above-described method.

[0051] Optionally, the electronic device also includes interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The components are interconnected using different buses and can be mounted on a common motherboard or otherwise installed as needed. The processor can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI (Graphical User Interface) on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors can be used with multiple memories, and / or multiple buses can be used with multiple memories, if desired. Similarly, multiple electronic devices (e.g., as a server array, a group of blade servers, or a multiprocessor system) can be connected, each providing some of the necessary operations.

[0052] The memory 302, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the data annotation method based on incremental learning and confidence learning in this embodiment. The processor 301 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 302, thereby implementing the aforementioned data annotation method based on incremental learning and confidence learning.

[0053] The memory 302 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 302 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 302 may further include memory remotely configured relative to the processor, which can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0054] The electronic device may also include an input device 303 and an output device 304. The processor 301, memory 302, input device 303, and output device 304 may be connected via a bus or other means.

[0055] Input device 303 can receive input digital or character information, and output device 304 may include a display device, an auxiliary lighting device (e.g., an LED), and a haptic feedback device (e.g., a vibration motor). The display device may include, but is not limited to, a liquid crystal display (LCD), a light-emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touchscreen.

[0056] This embodiment provides a medium storing computer instructions for instructing a computer to perform the methods described above. The computer instructions on this medium, used to instruct the computer to perform the methods described above, thus possess at least the same advantages as the methods described above.

[0057] The medium in this application may be any combination of one or more computer-readable media. The medium may be a computer-readable signal medium or a computer-readable storage medium. The medium may be, for example,, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of the medium (a non-exhaustive list) include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, the medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0058] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0059] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF (Radio Frequency), or any suitable combination thereof.

[0060] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0061] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0062] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data annotation method based on incremental learning and confidence learning, characterized in that, include: The first stage of training for the supervised learning model was conducted using a first dataset with manually labeled data. The unlabeled second dataset is input into the supervised learning model trained in the first stage to obtain the predicted labels output by the supervised learning model, and the predicted labels are then labeled on the second dataset. The first and second datasets are combined, and the combined dataset is then divided into multiple subsets. The target model is trained using a subset of the multiple data subsets, and the labels and predicted probabilities of another subset are obtained through the trained target model. In another subset, select data belonging to the first dataset and calculate the mean of the predicted probabilities of the correct labels as a baseline. In the other subset, select data belonging to the second dataset and compare the predicted probability of the correct label with the baseline value; Data with predicted probabilities greater than or equal to the baseline value are retained as incremental training samples for the second stage of training of the supervised learning model. The first and second datasets consist of images, text, or audio; both the supervised learning model and the target model are classification models. The target model is a pre-trained model.

2. The method according to claim 1, characterized in that, After comparing the predicted probability of the correct label with the baseline value, the method further includes: For data with incorrect labels and data whose predicted probability is less than the baseline value, the labels are manually corrected. The corrected data is used as incremental training samples in the first dataset to train the supervised learning model in the second stage.

3. The method according to claim 1, characterized in that, The target model is trained using a subset of the multiple data subsets, and the labels and predicted probabilities of another subset are obtained from the trained target model, including: Select N-1 subsets from N data subsets that have not been selected as a whole; The target model is trained using the N-1 subsets mentioned above; The labels and predicted probabilities of the remaining subset are obtained through the trained target model; Returns the selection operation of N-1 subsets until all N-1 subsets have been selected as a whole.

4. The method according to claim 1, characterized in that, Supervised learning models include support vector machines, random forests, or convolutional neural networks.

5. The method according to any one of claims 1-4, characterized in that, The first stage of training for the supervised learning model is performed using a manually labeled first dataset, including: The first dataset is preprocessed, and the preprocessing includes at least outlier removal and encoding. The first dataset is input into the supervised learning model to obtain the predicted labels output by the supervised learning model; The parameters of the supervised learning model are iterated to narrow the gap between the predicted labels and the manually labeled labels.

6. A data annotation system based on incremental learning and confidence learning, characterized in that, include: The user terminal is used to manually label the first dataset. A processing device is configured to input an unlabeled second dataset into a supervised learning model trained in the first stage, obtain the predicted labels output by the supervised learning model, and label the predicted labels onto the second dataset; mix the first and second datasets and divide the mixed dataset into multiple data subsets; train a target model using a subset of the multiple data subsets, and obtain the labels and predicted probabilities of another subset using the trained target model; select data belonging to the first dataset from the other subset and calculate the mean predicted probability of the correct label as a baseline value; select data belonging to the second dataset from the other subset and compare the predicted probability of the correct label with the baseline value; retain data with a predicted probability greater than or equal to the baseline value as incremental training samples for the second stage training of the supervised learning model; A storage device is used to store a first dataset, a second dataset, a supervised learning model, and a target model; wherein the first dataset and the second dataset are images, text, or audio; the supervised learning model and the target model are both classification models; and the target model is a pre-trained model.

7. The system according to claim 6, characterized in that, The user terminal has a human-computer interaction interface; The human-computer interaction interface displays the training progress and supports users to manually divide the data subset.

8. An electronic device, characterized in that, include: At least one processor, and a memory communicatively connected to at least one of the processors; The memory stores instructions executable by at least one of the processors, which are executed to enable the at least one processor to perform the data labeling method based on incremental learning and confidence learning as described in any one of claims 1-5.

Images