Multi-agent cooperative labeling method
By employing a multi-agent collaborative annotation method, utilizing clustering and metric learning models, and combining expert and amateur strategies, the problem of expert scarcity and high cost in data annotation is solved, achieving high-precision and low-cost annotation results.
Patent Information
- Application Number
- CN202310675828.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-30
- Publication Date
- 2026-02-17
- Estimated Expiration
- 2041-06-30
AI Technical Summary
Existing technologies suffer from several drawbacks in data annotation, including a scarcity of domain experts, the inability of ordinary annotators to annotate professional data, high costs, and a lack of effective multi-agent collaboration mechanisms, resulting in low annotation accuracy and efficiency.
Initial relationships between sample points are discovered through clustering, cluster centers are labeled by experts, similarity is compared by amateurs, distance metrics are updated using a metric learning model, and labeled data is selected by combining expert and amateur strategies, thus achieving multi-agent collaborative labeling.
Maximizing annotation accuracy within a given cost enables efficient use of domain experts and amateurs, reducing annotation costs and improving annotation accuracy.
Smart Images
Figure CN116821860B_ABST
Abstract
Description
[0001] This application is a divisional application of Chinese invention patent application No. 202110739898.4 (application date: June 30, 2021, invention title: annotation method for multi-agent cooperation). Technical Field
[0002] This invention relates to the fields of machine learning model inference, human-computer interaction, and active learning, and particularly to a labeling method for multi-agent collaboration. Background Technology
[0003] Providing high-quality labeled data services for training machine learning algorithms has become a crucial condition for the advancement of artificial intelligence applications. However, due to the low barriers to entry and inconsistent service quality in the data labeling industry, clients often encounter pain points when selecting data services, such as data quality, service efficiency, data security, management capabilities, and service capacity. These pain points have become core issues hindering the industry's development. With the diversification of AI application scenarios, the barriers to entry for labeling tasks will gradually increase, showing a trend towards specialization. That is, in addition to relatively simple labeling tasks that can be mastered through training, there will be labeling tasks requiring professional backgrounds, such as bird labeling and plant labeling. At this point, domain experts are scarce, and labeling the entire dataset by experts would be very costly. While ordinary labelers are inexpensive, they lack professional backgrounds and cannot directly perform labeling. Therefore, combining a small number of experts with a large number of amateurs for professional data labeling will be the challenge facing the data labeling industry in the future.
[0004] Existing methods focus on automatic annotation, few-shot learning, active learning, and crowdsourced learning. However, these methods still have the following limitations:
[0005] 1. Automatic annotation methods cannot be started without training data, their effectiveness depends on training data, and they have poor portability and versatility;
[0006] 2. The few-shot learning method assumes that there is a small amount of labeled data at the beginning. When faced with a large number of unlabeled data samples, it is difficult to determine the initial data that needs to be labeled.
[0007] 3. Conventional active learning scenarios rely on consulting experts without considering human resource costs;
[0008] 4. Existing crowdsourcing work mainly adopts a single annotation method, focusing on task allocation and annotation result integration, ignoring the differences in ability and cost among experts, amateurs and machine learning models, and lacking a way to enable effective and low-cost collaboration among the three parties. Summary of the Invention
[0009] In view of this, the main objective of the present invention is to provide a labeling method for multi-agent collaboration, in order to partially solve at least one of the above-mentioned technical problems.
[0010] To achieve the above objectives, as one aspect of the present invention, a multi-agent collaborative annotation method is provided. This method selects data points suitable for annotation by both experts and amateurs through an active learning process of similarity between sample points, thereby maximizing annotation accuracy under a given cost. The annotation method includes:
[0011] Initial relationships between sample points were discovered through clustering.
[0012] Experts labeled the centroids of each cluster;
[0013] Amateurs compare whether the point closest to the center point in each cluster belongs to the same cluster as the center point;
[0014] Train a metric learning model using labeled samples and update the distance measurement method between each image sample point;
[0015] Use appropriate expert data selection strategies and amateur data selection strategies to select labeled data.
[0016] The initial relationship of the sample points refers to the similarity between the sample points, that is, the clustering method divides similar sample points into the same cluster and dissimilar sample points into different clusters.
[0017] The clustering method used is K-means.
[0018] The step of having experts label the centroids of each cluster also includes recording the discovered categories and the costs incurred.
[0019] The amateur's step of comparing whether the point closest to the center point in each cluster belongs to the same class as the center point also includes: assigning the same label to samples determined to belong to the same class, and recording the excluded classes for samples determined to belong to different classes.
[0020] The metric learning model learns a distance measurement method, and the distance obtained makes the similarity between images of different categories in the training set small, while the similarity between images of the same category is large.
[0021] The metric learning model employs the LMNN metric learning algorithm, which is an adaptive clustering algorithm.
[0022] The data selection strategy in the step of selecting labeled data using the corresponding expert data selection strategy and amateur selection strategy is related to the labeling process and the identity of the labeler; the labeling is divided into two stages according to whether the category is completely discovered. The data selection strategy for experts is different in these two stages, while the data selection strategy for amateurs is the same in these two stages.
[0023] If the categories are not fully discovered, for unlabeled sample points, the points with the highest confidence level below the threshold are filtered out, and points with the highest confidence level are randomly selected from the remaining points for expert labeling. If the categories are fully discovered, on non-uniform datasets, sample points are randomly selected for expert labeling, and on uniform datasets, points with the lowest confidence level in the category with the fewest labeled samples and the largest intra-class variance are preferentially selected for expert labeling.
[0024] For amateurs, regardless of whether all categories have been discovered, the amateurs are labeled with the points that have the highest confidence in the category with the fewest labeled samples and the smallest intra-class variance.
[0025] Based on the above technical solution, it can be seen that the multi-agent collaborative annotation method of the present invention has at least some of the following beneficial effects compared with the prior art:
[0026] For data labeling scenarios where there are few domain experts and many amateurs, requiring specialized knowledge for single-classification labeling, we design a data labeling mechanism based on multi-agent interaction (including domain experts, amateurs, and machine learning models) to perform single-classification labeling of image datasets, aiming to maximize labeling accuracy under given costs. Attached Figure Description
[0027] Figure 1 This is a schematic diagram of the annotation method for multi-agent collaboration provided in an embodiment of the present invention.
[0028] Figure 2 This is a flowchart illustrating the operation of the present invention. Detailed Implementation
[0029] The innovations of this invention are as follows: (1) It is the first crowdsourced annotation system that combines experts and amateurs; (2) Our system annotates the dataset from scratch without any pre-annotated data; (3) Due to the characteristics of metric learning, our system can annotate different target datasets while maintaining cost-effectiveness.
[0030] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.
[0031] like Figure 1As shown, this invention provides a multi-agent collaborative annotation method. By learning the similarity between sample points, an active learning process selects data points suitable for both experts and amateurs to annotate, thereby maximizing annotation accuracy under a given cost. The method includes:
[0032] Initial relationships between sample points were discovered through clustering.
[0033] Experts labeled the centroids of each cluster;
[0034] Amateurs compare whether the point closest to the center point in each cluster belongs to the same cluster as the center point;
[0035] Train a metric learning model using labeled samples and update the distance measurement method between each image sample point;
[0036] Use appropriate expert data selection strategies and amateur data selection strategies to select labeled data.
[0037] In the above method, the initial relationship of sample points refers to the similarity between sample points. That is, the clustering method divides similar sample points into the same cluster and dissimilar sample points into different clusters.
[0038] The second and third steps of the above method are the annotation methods of experts and amateurs. The annotation method of experts is to directly give labels, while the annotation method of amateurs is to perform point-to-point comparison to determine whether two sample points belong to the same category.
[0039] In the above method, the metric learning model learns a distance measure that makes images of different categories have low similarity while images of the same category have high similarity in the training set.
[0040] The data selection strategy in the above method is related to the annotation process and the annotator's identity (expert or amateur). The annotation process is divided into two stages based on whether a complete discovery is found in each category. The data selection strategy for experts differs between these two stages, while the data selection strategy for amateurs is the same in both stages.
[0041] The method of this invention constructs a mapping relationship in the output space of previously isolated heterogeneous models and integrates knowledge from multiple sources and domains to achieve adaptive collaboration among heterogeneous models, thereby improving the accuracy and range of labels obtained under limited resources. Due to the utilization of heterogeneous model collaboration, this invention has the advantage of obtaining high-precision, wide-range output labels under limited computing resources, and is suitable for scenarios such as large-scale data analysis in the cloud, real-time data analysis at the edge of the Internet of Things, and low-power data analysis on edge devices.
[0042] The method of this invention combines high-cost and low-cost annotation methods, utilizes the correlation between the two methods, and leverages a machine learning model to select data for annotation by different types of annotators, thereby achieving the goal of saving annotation costs and improving annotation accuracy.
[0043] like Figure 2 The diagram shown is a flowchart of the operation of this invention.
[0044] The embodiments of the present invention will be described in further detail below.
[0045] The multi-intelligent collaborative annotation method provided in this embodiment of the invention mainly includes the following steps:
[0046] Step 1: Discover the initial relationships between sample points through clustering;
[0047] Step 2.1: Experts label the centroids of each cluster;
[0048] Step 2.2: The amateur compares whether the point closest to the center point in each cluster belongs to the same cluster as the center point;
[0049] Step 3: Train the metric learning model using the labeled samples and update the distance measurement method between each image sample point;
[0050] Step 4.1: Calculate the accuracy rate of amateurs in the previous round and the number of annotations that experts and amateurs should mark in this round;
[0051] Step 4.2: Select labeled data using the appropriate expert data selection strategy and amateur data selection strategy;
[0052] Step 5, repeat steps 3, 4.1, and 4.2 until the cost is 0.
[0053] The specific steps are as follows:
[0054] Step 1: First, use a network trained on ImageNet to extract image features. Then, cluster all features using the K-means method. Assuming there are k categories, k cluster centers are obtained. Then, use Euclidean distance to determine the sample point closest to each cluster center.
[0055] Step 2.1: Experts label the k cluster centers obtained in the previous step, assigning labels to these k sample points. During this process, the discovered categories and the costs incurred are recorded.
[0056] Step 2.2: The amateur compares the k centroids from Step 1 with their nearest sample points to determine if they belong to the same category. Based on the amateur's judgment, samples classified as belonging to the same category are labeled with the same name, while samples classified as belonging to different categories are recorded as having been excluded from the category list.
[0057] Step 3: Since K-means is chosen as the clustering method, a suitable metric learning method, Large Margin Nearest Neighbor (LMNN), is considered. The basic idea is to learn a transformation matrix to transform existing features, such that the Euclidean distance of the transformed features brings sample points of the same class closer together and pushes sample points of different classes further apart. The LMNN is trained using labeled sample points to update the distance metric. The confidence score of each sample point in each class is the inverse ratio of the distance between this sample point and the center of the known sample points in each class.
[0058] In step 4.2, if the categories are not fully discovered, for unlabeled sample points, points with the highest confidence level below the threshold are removed, and points from the remaining points are randomly selected based on the number of experts to be labeled. If the categories are fully discovered, on non-uniform datasets, sample points are randomly selected for expert labeling; on uniform datasets, points with the lowest confidence level in the category with the fewest labeled samples and the largest intra-class variance are preferentially selected for expert labeling. For amateurs, regardless of whether the categories are fully discovered, points with the highest confidence level in the category with the fewest labeled samples and the smallest intra-class variance are selected for amateur labeling.
[0059] Because it utilizes multi-agent collaboration, this invention has the advantage of saving annotation costs and is suitable for single-classification image scenarios where expert annotation costs are much higher than amateur annotation costs or where amateurs cannot provide specific labels.
[0060] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for multi-agent collaborative annotation, characterized in that, The multiple agents include domain experts, amateurs, and machine learning models, and the labeling method includes: The initial relationship of the sample points is found by clustering, including: extracting picture features of the picture sample points by using a network trained on ImageNet, and clustering all the picture features by using a Kmeans method, the initial relationship of the sample points refers to the similarity between the picture sample points; The clustering of all the picture features by using the Kmeans method includes: dividing similar picture sample points into the same cluster, and dividing dissimilar picture sample points into different clusters, assuming that there are k categories, k cluster centers are obtained, and the picture sample points closest to each cluster center are determined by using the Euclidean distance; Receiving labels labeled by experts on the center points of each cluster picture; Receiving amateur determination results, wherein the amateur determination results are obtained by comparing whether the points closest to the picture center points in each cluster and the picture center points are of the same category; The receiving of the amateur determination results includes: labeling the picture samples determined as the same category with the same label, and recording the categories excluded by each sample for the picture samples determined as different categories; A distance measurement method between the picture sample points is updated by using the labeled picture samples to train a metric learning model, wherein the distance makes the similarity of the picture sample points of different categories small and the similarity of the picture sample points of the same category large; Corresponding expert data selection strategies and amateur selection strategies are used to select the labeling data; The selection of the labeling data by using the corresponding expert data selection strategies and amateur selection strategies includes: if the category is not completely discovered, for the unlabeled picture sample points, points with a confidence less than a threshold value are filtered out, and a number of expert points are randomly selected from the remaining points for labeling by the experts; if the category is completely discovered, on a non-uniform data set, picture sample points are selected by using a random selection method for labeling by the experts; if on a uniform data set, points with the smallest confidence in the category with the smallest number of labeled picture samples and the largest intra-class variance are preferentially selected for labeling by the experts; for the amateurs, whether the category is completely discovered or not, points with the largest confidence in the category with the smallest number of labeled picture samples and the smallest intra-class variance are selected for labeling by the amateurs.
2. The labeling method of claim 1, wherein Through the active learning paradigm, the process of learning the similarity between the picture sample points is used to select data points suitable for labeling by the experts and the amateurs, so as to maximize the labeling accuracy under a given cost.
3. The labeling method of claim 1, wherein The step of labeling the center points of each cluster by the experts further includes: recording the discovered categories and the cost.
4. The labeling method of claim 1, wherein The metric learning model learns a distance measurement method.
5. The labeling method of claim 1, wherein The metric learning model uses a metric learning algorithm LMNN adapted to the clustering algorithm.
6. The labeling method of claim 5, wherein The basic idea of the LMNN is to learn a transformation matrix to transform the existing features, so that the Euclidean distance of the transformed features is closer to the sample points of the same category and farther away from the sample points of different categories.
7. The labeling method according to claim 5 or 6, characterized in that, The LMNN is trained by using the labeled sample points, and the confidence of each sample point on each category is inversely proportional to the distance between the sample point and the center of the known sample point of each category.
8. The labeling method of claim 1, wherein By combining the two labeling methods of high cost and low cost, using the correlation between the two labeling methods, using the assistance of a machine learning model, and selecting the data labeled by different types of labelers, the cost of labeling is saved, and the accuracy of labeling is improved. The purpose is suitable for the picture single classification scene that the expert labeling cost is much higher than the amateur labeling cost or the amateur cannot give specific label.