Data labeling method, device and equipment, readable storage medium and program product
By clustering and spreading the data sets, the problems of slow data labeling speed and low accuracy in the prior art are solved, efficient and accurate data labeling are achieved, and cost and time investment are reduced.
Patent Information
- Application Number
- CN202411191554.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-17
AI Technical Summary
The existing data annotation methods have problems such as slow labeling speed and low labeling accuracy, which leads to poor data annotation effect, especially when processing large-scale data sets, which are difficult to ensure high cost, low efficiency, and label consistency.
By clustering the data in the dataset, the target class cluster is generated, and representative data is selected from each class cluster for annotation. The descriptive sub-features of the pseudo-label are used to propagate the clustering and the annotation results, improving the labeling efficiency and accuracy .
It effectively reduces the workload of labeling personnel, improves the consistency and accuracy of labeling, reduces the labeling cost and time cost, and adapts to the processing needs of large-scale data sets.
Smart Images
Figure CN119150167A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data annotation method, device, equipment, readable storage medium and program product. Background Art
[0002] Data annotation typically involves associating raw data with predefined labels or categories in order to train machine learning models or perform other data analysis tasks.
[0003] In the prior art, data are usually labeled manually one by one to obtain corresponding labeled data for model training and learning.
[0004] However, the existing data labeling method has problems such as slow labeling speed and low labeling accuracy, resulting in poor final data labeling effect. Summary of the invention
[0005] The present application provides a data labeling method, apparatus, device, readable storage medium and program product to solve the problem of poor existing data labeling effect.
[0006] In a first aspect, an embodiment of the present application provides a data labeling method, including:
[0007] Obtain a descriptor of a pseudo label of each first data in the data set;
[0008] Clustering the first data in the data set according to the descriptor to form at least one target cluster;
[0009] Selecting second data from the first data included in each target cluster, and obtaining a labeling result corresponding to the second data;
[0010] Based on the labeling result of the second data, other data in the target cluster where the second data is located except the second data is labeled.
[0011] In a possible design of the first aspect, before clustering the first data in the data set to form at least one target cluster, the method further includes:
[0012] Determine third data in the data set that has been annotated with pseudo labels;
[0013] The third data whose pseudo-label satisfies the first set condition is determined as the first data.
[0014] In another possible design of the first aspect, the first set condition includes one or more of: the confidence of the pseudo label is less than or equal to a confidence threshold, and the size of the pseudo label does not meet a set size requirement.
[0015] In another possible design of the first aspect, the method further includes:
[0016] If the confidence of the pseudo label of the third data is greater than the confidence threshold, a labeling result of the third data is generated based on the pseudo label.
[0017] In another possible design of the first aspect, the pseudo-label is generated by predicting the unlabeled data through a supervised training model or a semi-supervised training model; and the descriptor of the pseudo-label is determined based on feature information of the area identified by the pseudo-label in the corresponding first data.
[0018] In another possible design of the first aspect, clustering the first data in the data set according to the descriptor to form at least one target cluster includes:
[0019] According to the descriptor of the pseudo label of each first data, the first data in the data set is clustered at least once until the clustering end condition is met, and each second cluster obtained by the last clustering operation is determined as the target cluster; wherein the i+1th clustering operation includes:
[0020] Determine the first cluster to be clustered;
[0021] Based on the descriptor of the pseudo label of each first data in the first cluster, clustering the first data in each first cluster to obtain at least one second cluster;
[0022] If the at least one second type cluster does not satisfy the clustering end condition, then enter the (i+2)th clustering operation;
[0023] Among them, i is an integer not less than 0; when i is 0, the first cluster is a set consisting of the first data in the data set; when i is an integer greater than 0, the first cluster is the second cluster obtained by the i-th clustering operation.
[0024] In another possible design of the first aspect, the clustering end condition includes at least one of the following:
[0025] The total number of at least one second-class cluster obtained is less than a first threshold;
[0026] There is at least one third cluster in at least one second cluster, and the total amount of first data contained in the third cluster is less than a second threshold;
[0027] The total number of clustering operations is greater than or equal to a third threshold.
[0028] In another possible design of the first aspect, clustering the first data in the data set to form at least one target cluster includes:
[0029] Clustering the first data in the data set into N first-class clusters;
[0030] Continue clustering the first data in each first cluster to obtain M second clusters, and the number of first data contained in each second cluster is approximately equal to a set value;
[0031] The first data in each second cluster is clustered again to obtain K subdivided clusters as the at least one target cluster, and N, M and K are all set values.
[0032] In another possible design of the first aspect, clustering the first data in the data set to form at least one target cluster includes:
[0033] Obtaining the total amount of first data in the data set;
[0034] If the total amount of the first data in the data set is greater than a threshold amount, dividing the first data in the data set into at least two batches;
[0035] The first data of each batch in the data set is clustered in sequence to form the at least one target cluster.
[0036] In another possible design of the first aspect, the annotation result corresponding to the second data is obtained through manual annotation or automatic annotation using a set annotation model.
[0037] In another possible design of the first aspect, labeling other data except the second data in the target cluster where the second data is located based on the labeling result of the second data includes:
[0038] Using the labeling result of the second data as the labeling result of other data in the target cluster where the second data is located except the second data;
[0039] or;
[0040] In the target cluster where the second data is located, select target data whose distance from the cluster center of the target cluster meets a set distance condition;
[0041] The labeling result of the second data is used as the labeling result of the target data.
[0042] In another possible design of the first aspect, the first data includes at least one of an image, audio, text, multimedia information, and a point cloud.
[0043] In a second aspect, an embodiment of the present application provides a data labeling device, including:
[0044] A descriptor acquisition module, used to acquire a descriptor of a pseudo label of each first data in the data set;
[0045] A data clustering module, used for clustering the first data in the data set according to the descriptor to form at least one target cluster;
[0046] A result acquisition module, used to select second data from the first data included in each target cluster, and obtain a labeling result corresponding to the second data;
[0047] The labeling propagation module is used to label other data in the target cluster where the second data is located, except the second data, based on the labeling result of the second data.
[0048] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method described above.
[0049] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to implement the method described above when executed by a processor.
[0050] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the above method when executed by a processor.
[0051] The data labeling method, apparatus, device, readable storage medium and program product provided in the embodiments of the present application cluster a large amount of first data in a data set into clusters, and then select second data from each cluster as representative data, give the representative data a labeling result, and then diffuse and propagate the labeling result to other data in the cluster. This is equivalent to labeling one second data, and can realize the labeling of other data in the entire cluster, thereby improving the labeling efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the description, are used to explain the principles of the present application;
[0053] Figure 1 A schematic diagram of an application scenario of data annotation provided in an embodiment of the present application;
[0054] Figure 2 A schematic diagram of the labeling process of the semi-automatic labeling system provided in an embodiment of the present application;
[0055] Figure 3A schematic diagram of a data annotation method flow chart provided in an embodiment of the present application;
[0056] Figure 4 A clustering diagram provided for an embodiment of the present application;
[0057] Figure 5 A schematic diagram of a clustering process provided in an embodiment of the present application;
[0058] Figure 6 A schematic diagram of the marking process provided for an embodiment of the present application;
[0059] Figure 7 A schematic diagram of the structure of a data annotation device provided in an embodiment of the present application;
[0060] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0061] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0062] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0063] Data labeling is a crucial step in modern data science and artificial intelligence applications. Data labeling usually involves associating raw data with predefined labels or categories in order to train machine learning models or perform other data analysis tasks. However, with the growth of data scale and the diversification of application scenarios, traditional manual labeling methods have shown a series of problems: (1) High cost, which is mainly reflected in the high cost of obtaining and labeling large-scale labeled data, including labor costs and time costs. (2) Low efficiency, which is mainly reflected in the slow speed of manual data labeling and cannot effectively cope with the processing requirements of large-scale data sets. (3) It is difficult to ensure labeling consistency, which is mainly reflected in the fact that manual labeling is easily affected by subjective factors, and the consistency and accuracy of labels are difficult to ensure, especially for complex or large-scale data sets.
[0064] In view of the above situation, the present application provides a data labeling solution, which is mainly used to solve the problems of high cost, low efficiency and low labeling quality currently faced when using limited labeled data for model training. Specifically, the model detects and identifies a large amount of unlabeled data, generates third data (the third data is the data after the unlabeled data is labeled with pseudo labels), and then selects the data whose pseudo labels meet the set conditions from the third data as the first data, and obtains the descriptor of the pseudo label of the first data; then based on the descriptor features of the pseudo labels, the massive first data is clustered into clusters, and the second data is selected from each cluster for labeling, and then the labeling results of the second data in each cluster are propagated to other data in the cluster, so as to realize the labeling of other data in the cluster. This can effectively reduce the workload of the labeling personnel and improve the consistency and accuracy of the labeling.
[0065] Figure 1 The following is a schematic diagram of an application scenario of data annotation provided in an embodiment of the present application. The application scenario may be in the field of object detection, such as Figure 1 As shown in the figure, the data set can contain several images, and the purpose of object detection is to identify the people in the image. If manual annotation is used, the annotator can simply draw a frame in the image and mark the people in the image. However, if artificial intelligence is used for annotation, then the artificial intelligence needs to perform object detection and identify the people in the image. Among them, some positive samples and negative samples can be given as training data to train artificial intelligence, which requires a large amount of data to be annotated in advance to determine which are positive samples and which are negative samples.
[0066] The technical solution of the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.
[0067] First, this solution can be a semi-automatic annotation system, for example, Figure 2 A schematic diagram of the labeling process of the semi-automatic labeling system provided in the embodiment of the present application is shown in FIG. Figure 2 As shown, the semi-automatic annotation system includes a semi-supervised training module, a pseudo-label generation module and a semi-automatic annotation system module.
[0068] Among them, the semi-supervised training module is mainly composed of a semi-supervised target detection model trainer and a target detection model. By using a semi-supervised framework to train and optimize the model for labeled data and unlabeled data, it is ensured that the semi-supervised model or supervised training model performs well under limited labeled data. The pseudo-label generation module uses the trained semi-supervised model or supervised training model to predict and generate pseudo-labels for unlabeled data. The semi-automatic labeling module extracts descriptors for pseudo-labels. After completing the descriptor extraction, the descriptors of the pseudo-labels are used to cluster the data carrying the pseudo-labels and meeting the set conditions. After clustering, the second data is selected from the cluster for labeling, and then the labeling results of the second data are propagated to other data in the cluster to achieve the labeling of other data in the cluster. During the labeling period, what needs to be done is to label the given second data. Through clustering and label propagation, each time a second data is labeled, the label of the second data is propagated to other data in the same cluster through label propagation, thereby reducing the workload of the labeling personnel.
[0069] The following is combined with the above Figure 2 , this data annotation scheme is explained in detail.
[0070] Figure 3 A schematic diagram of a data annotation method flow chart provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the method comprises the following steps:
[0071] Step S310: Obtain a descriptor of a pseudo label of each first data in the data set.
[0072] In this embodiment, different data are first distinguished and explained:
[0073] ① Unlabeled data refers to data that has not been annotated with pseudo labels;
[0074] ② The third data refers to the data with pseudo labels after the unlabeled data is labeled;
[0075] ③ The first data refers to the data whose pseudo-labels in the third data meet the first set condition;
[0076] ④The second data refers to the data selected from the cluster after clustering the first data.
[0077] In this embodiment, the pseudo-label is used to filter out the first data for clustering from the third data. The descriptor is used to cluster the first data, that is, cluster the first data identified by the pseudo-labels with similar or identical descriptor features into a cluster.
[0078] In this embodiment, the amount of unlabeled data to be labeled is usually large. After the unlabeled data is labeled, it can be distinguished and summarized in the form of data sets. For example, taking a certain time period [T1, T2] as an example, all unlabeled data collected in this time period are stored in the same data set after labeling, and other unlabeled data collected outside this time period are stored in other data sets after labeling, so that data from different periods can be distinguished.
[0079] Exemplarily, the first data may include at least one of an image, audio, text, multimedia information, and a point cloud. For example, taking the first data as an image, if a label box is marked at area A in the image, and there is indeed a corresponding target object (such as a pedestrian) in the label box, then the image can be regarded as a positive sample.
[0080] In this embodiment, the pseudo-label is generated by predicting the unlabeled data through the model. Exemplarily, the pseudo-label is generated by predicting the unlabeled data through a supervised training model or a semi-supervised training model; the descriptor of the pseudo-label is determined based on the feature information of the area identified by the pseudo-label in the corresponding first data.
[0081] Among them, the prediction results of the model may be unreliable. For example, in machine learning, the model performs object detection on an image, and the model selects area A in the image as a pseudo-label (the pseudo-label can carry confidence, and the higher the confidence, the more reliable the pseudo-label). Furthermore, the descriptor is used to describe the features in the pseudo-label. For example, if there is a portrait in area A, then the descriptor describes the features of the portrait.
[0082] Step S320: clustering the first data in the data set according to the descriptor to form at least one target cluster.
[0083] In this embodiment, two similar descriptors and their corresponding first data of pseudo labels will be clustered into the same cluster. For example, the pseudo-label descriptors of the first data W1 and the first data W2 are M1 and M2 respectively. If the descriptor M1 is similar to the descriptor M2, the first data W1 and the first data W2 will be clustered into the same cluster.
[0084] In this embodiment, the number of clustering is not limited, and multiple clustering can be performed until the target cluster is finally obtained. For example, assuming that all the first data in the data set are clustered to obtain three clusters L1, L2 and L3, then the first data in L1, L2 and L3 can be clustered again, so that there will be sub-clusters L11, L12, L13, ... L1n in cluster L1, and clusters L2 and L3 are similar, including several sub-clusters, which is equivalent to performing clustering twice.
[0085] Step S330: Select second data from the first data included in each target cluster, and obtain a labeling result corresponding to the second data.
[0086] In this embodiment, through the above clustering, there is at least one first data in each target cluster. At this time, one or more second data can be selected from the target cluster, and then the second data is labeled, for example, the second data is marked as a "positive sample" or "negative sample". In this way, the labeling result of the second data is obtained, so that the second data becomes labeled data.
[0087] For example, in some embodiments, the labeling result of the second data can be obtained by manual labeling or automatic labeling using a set labeling model. For example, when the second data is an image, a label frame can be manually drawn in the image, and a portrait in the image can be selected by the label frame.
[0088] If the number of selected second data is small, manual labeling can be used, and if the number of selected second data is large, a labeling model can be used to automatically label the data to improve the labeling efficiency of the second data.
[0089] The number of target clusters may be one or more, and at least one second data needs to be selected from each target cluster.
[0090] Step S340: Based on the labeling result of the second data, label other data in the target cluster where the second data is located except the second data.
[0091] In this embodiment, if there are other first data in the target cluster, since the data in the same target cluster are highly similar, the labeling results of the labeled second data in the target cluster (which has actually become labeled data) can be propagated and diffused to other data in the target cluster.
[0092] Exemplarily, assume that there are three first data S1, S2, and S3 in the target cluster L1, among which the first data S1 is selected as the second data, and the first data S1 is manually labeled, and the labeling result is a "positive sample". Then, the remaining two first data S2 and S3 in this target cluster L1 can be automatically labeled as "positive samples", thereby realizing the propagation of the labeling results.
[0093] In addition, in practical applications, the number of target clusters may be large, and the labeling result of the second data in each target cluster can be propagated to other data in the target cluster as the labeling result of other data.
[0094] The embodiment of the present application clusters the first data into clusters, and then selects a second data from the cluster as representative data, gives the representative data a labeling result, and then diffuses and propagates the labeling result to other data in the cluster. This is equivalent to only needing to label the selected second data to achieve labeling of other data in the entire cluster, thereby improving the labeling efficiency. At the same time, since the first data clustered into the same cluster have a high degree of similarity, the accuracy of the labeling can also be guaranteed.
[0095] In practical applications, the data in the data set is usually massive. In order to further improve the efficiency in the process of labeling large-scale data, in some embodiments, the data in the data set can be preliminarily screened to filter out the data that does not need to be clustered later, which can increase the subsequent clustering speed and further improve the labeling efficiency. This can be achieved through the following steps:
[0096] (1) determining third data in the data set that has been annotated with pseudo labels;
[0097] (2) If the confidence of the pseudo-label of the third data is greater than the confidence threshold, a labeling result of the third data is generated based on the pseudo-label.
[0098] In this embodiment, each pseudo-label of the third data has a confidence level, and the confidence levels of different pseudo-labels may be different. For example, some pseudo-labels have a higher confidence level (which means that the pseudo-label is more reliable and closer to the real label), while some pseudo-labels have a lower confidence level (which means that the pseudo-label is less reliable). Among them, when the confidence level of a pseudo-label is greater than the set confidence threshold, the pseudo-label is actually more reliable, and the pseudo-label can be directly used as the annotation result of the third data without the need for subsequent data clustering and label propagation processes. This is equivalent to reducing the amount of data for subsequent clustering, which can further improve clustering efficiency.
[0099] For example, continue to use images as unlabeled data. After predicting the image, the model generates third data with pseudo labels. Assuming that the pseudo label box selects the portrait in the image, and the confidence of the pseudo label is relatively high, it means that there is a high probability that there is a portrait in the image, that is, the third data can be directly regarded as a "positive sample". Similarly, if it is determined that there is a high probability that there is no portrait in the image, then the third data can be directly regarded as a "negative sample". In this way, compared with completing the labeling of these data first, clustering processing is avoided, and the efficiency of data labeling is improved.
[0100] The embodiment of the present application realizes pseudo-label generation through a semi-supervised training model, sets a confidence threshold based on the generated pseudo-label, and screens the third data carrying the pseudo-label with high confidence, directly using it as a positive sample to reduce the time and cost of subsequent labeling steps, thereby further improving the labeling efficiency of large-scale data. Moreover, through semi-supervised training and pseudo-label generation, the labeling cost and time cost can be significantly reduced, and the data processing efficiency is improved. Compared with the traditional manual labeling method, it can greatly save human resources and time investment while ensuring the quality of labeling.
[0101] In addition, in some other embodiments, if the pseudo-labels of the third data satisfy the first set condition, the third data are regarded as the first data, and the first data need to be subsequently clustered.
[0102] In this embodiment, taking the application of data annotation to the field of object detection as an example, in the field of object detection, the first data may refer to an image, and the pseudo label may refer to a certain annotation box in the image (for example, if the model recognizes that there is a person in the image, it is annotated by the pseudo label). The size of the pseudo label refers to the size of the annotation box.
[0103] Among them, since the portraits in the image are often diverse, the pseudo-labels of different first data may be different. For example, the size of the pseudo-label of an adult in the image may be larger, while the size of the pseudo-label of a child in the image is relatively small. In addition, when the pseudo-label of unlabeled data is generated by the model, the pseudo-label can be scored. For example, if the portrait is very clear, then the pseudo-label score is higher, and if the portrait clarity is not high, the score is relatively low. Among them, the score can be used to characterize the confidence of the pseudo-label, and the higher the score, the higher the confidence.
[0104] In this embodiment, the first set condition may be pre-set, and the first set condition is used to determine which third data in the data set have unreliable pseudo-labels, and use them as first data for subsequent data clustering and label propagation.
[0105] Exemplarily, in some embodiments, the first set condition includes one or more of the following: the confidence of the pseudo-label is less than or equal to the confidence threshold, and the size of the pseudo-label does not meet the set size requirement.
[0106] In this embodiment, the purpose of setting the first set condition for screening is mainly to find some third data with reliable pseudo-labels in the massive third data carrying pseudo-labels. These third data no longer need to be subsequently labeled. As for the third data with unreliable pseudo-labels, they will be regarded as first data and need to continue subsequent clustering processing to form clusters for manual or automatic labeling.
[0107] Among them, since pseudo labels are generated through model prediction, factors such as model performance and the quality of unlabeled data will lead to different confidence levels of the pseudo labels (i.e., the scoring described above). For example, if the entire image of some unlabeled data is relatively clear, the model prediction effect is good, and the final pseudo label confidence is high.
[0108] How to obtain the pseudo label of the first data is described in detail below through some embodiments.
[0109] In some embodiments, the unlabeled data can be predicted by a supervised training model or a semi-supervised training model to generate pseudo labels. The unlabeled data for which pseudo labels are generated is regarded as the third data. Figure 2 For example, some labeled data (e.g., X labeled data) and a large amount of unlabeled data (e.g., Y unlabeled data) can be obtained, and the model can be trained by semi-supervised training to obtain a semi-supervised target detection model. The trained semi-supervised target detection model is then used to predict the unlabeled data in the data set and generate a pseudo label for each unlabeled data. These unlabeled data with pseudo labels are regarded as the third data. Where Y is greater than X.
[0110] Among them, part of the labeled data (i.e., X labeled data) and a large amount of unlabeled data (i.e., Y unlabeled data) can be directly used to train the semi-supervised object detection model (SSOD), and finally a model for pseudo-label generation is obtained.
[0111] In this embodiment, the descriptor of the pseudo-label can be further determined by the feature information contained in the region identified by the pseudo-label. Since the feature information in the regions identified by different pseudo-labels is different, if the descriptor of each pseudo-label is different, it is convenient to perform clustering processing later.
[0112] Specifically, a descriptor extraction model can be used to extract features of each sample based on the image corresponding to its bounding box, so as to obtain a descriptor of the sample.
[0113] In addition, in other implementations, other target detection models (such as supervised training models) may also be selected to generate pseudo labels for unlabeled data.
[0114] The embodiment of the present application utilizes partially labeled data and a large amount of unlabeled data to train a model that is ultimately used for pseudo-label generation. This can overcome the reliance of traditional fully supervised methods on a large amount of labeled data, significantly reduce costs and improve data labeling efficiency. In addition, based on the introduction of a semi-automatic labeling system, combined with descriptor extraction and clustering optimization, the accuracy and consistency of labeling can be effectively improved. Compared with manual labeling, it can more accurately process large-scale and complex data sets, and improve the stability and performance of model training.
[0115] The following describes in detail how to cluster the first data through some embodiments, wherein the first data includes at least one of an image, an audio, a text, a multimedia information, and a point cloud.
[0116] Exemplarily, taking the first data as an image, the purpose of clustering is to assign similar images to a cluster, and then select the second data from each cluster for labeling, and then propagate the labels of the labeled data in each cluster to the remaining other data in each cluster, thereby improving the labeling efficiency by label propagation. In practical applications, the larger the amount of data, the worse the clustering effect. In order to improve the effect of the clusters obtained by the final clustering, multiple clustering can be performed. Specifically, clustering can be performed through the following steps: according to the descriptor of the pseudo-label of each first data, the first data in the data set is clustered at least once until the clustering end condition is met, and each second cluster obtained by the last clustering operation is determined as the target cluster.
[0117] The i+1th clustering operation includes:
[0118] Determine the first cluster to be clustered;
[0119] Based on the descriptor of the pseudo label of each first data in the first cluster, clustering the first data in each first cluster to obtain at least one second cluster;
[0120] If at least one of the second-class clusters does not meet the clustering end condition, the i+2th clustering operation is entered;
[0121] Among them, i is an integer not less than 0; when i is 0, the first cluster is a set consisting of the first data in the data set; when i is an integer greater than 0, the first cluster is the second cluster obtained by the i-th clustering operation.
[0122] In this embodiment, multiple clusters can be obtained after each clustering, and the number of first data contained in each cluster can also be determined. Since the more times of clustering, the better the final clustering effect, in order to ensure that the first data in each cluster is as similar as possible, a clustering end condition can be configured to control the clustering process.
[0123] For example, Figure 4 The clustering diagram provided in the embodiment of the present application is as follows: Figure 4 As shown, by clustering the first data in the data set once, cluster 1, cluster 2, and cluster 3 are obtained, and then the first data in cluster 1, cluster 2, and cluster 3 are further clustered to obtain cluster 11, cluster 12, cluster 13, cluster 21, cluster 22, cluster 31, and cluster 32.
[0124] Here, assuming that the clustering end condition is "the number of clusters obtained by the final clustering is greater than or equal to 5", refer to the above Figure 4 , clusters 11, cluster 12, cluster 13, cluster 21, cluster 22, cluster 31 and cluster 32 are obtained by clustering, totaling 7 clusters, which meet the clustering end condition. There is no need to continue subsequent clustering for the first data in clusters 11, cluster 12, cluster 13, cluster 21, cluster 22, cluster 31 and cluster 32.
[0125] Exemplarily, in some embodiments, the clustering termination condition includes at least one of the following: ① the total number of at least one second-class clusters obtained is less than a first threshold; ② there is at least one third-class cluster in at least one second-class cluster, and the total number of first data contained in the third-class cluster is less than the second threshold; ③ the total number of clustering operations is greater than or equal to the third threshold.
[0126] Continue to refer to the above Figure 4 After the first clustering, three clusters are obtained: Cluster 1, Cluster 2 and Cluster 3. If the first threshold value is 4, then the clustering end condition is met and no subsequent clustering is performed (equivalent to clustering once and then ending). In addition, assuming that after the second clustering, Cluster 1 includes Cluster 11, Cluster 12 and Cluster 13, and the total number of first data contained in Cluster 11 is 10, if the second threshold value is 12, no subsequent clustering is performed (equivalent to clustering twice and then ending). In addition, the total number of clustering times can also be set. For example, if the third threshold value is set to 2, clustering is performed twice and then ends.
[0127] Furthermore, in some other embodiments, other clustering constraints may be set to control the clustering process. Figure 5 The clustering process diagram provided in the embodiment of the present application configures the clustering constraint condition as "the number of clusters finally obtained is greater than or equal to K, and the number of first data in each cluster finally obtained is not greater than the set value", such as Figure 5 As shown, it includes the following steps:
[0128] Step S510: clustering the first data in the data set into N first clusters;
[0129] Step S520: Continue clustering the first data in each first cluster to obtain M second clusters, and the number of first data in each second cluster is approximately equal to a set value;
[0130] Step S530: clustering the first data in each second cluster again to obtain K subdivided clusters as at least one target cluster, wherein N, M and K are all set values.
[0131] In this embodiment, there may be a lot of first data in the data set (for example, 10,000). Taking N=10, hierarchical clustering can be used to obtain a total of 10 first-class clusters. Then, for each first-class cluster, the Kmeans clustering algorithm can be further used for clustering. By setting the minimum number of samples, the number of first data in each second-class cluster is ensured to be close to the minimum number of samples. Then, the Kmeans clustering algorithm is used again to cluster the first data in each second-class cluster, and finally K sub-clustering clusters are obtained.
[0132] Among them, at least one second data can be selected from each subdivision cluster as the representative data. At least K representative data can be selected from K subdivision clusters, and then manually labeled. After labeling, the labeling results of these K representative data are propagated to other data in each subdivision cluster, thereby improving the labeling efficiency.
[0133] Specifically, Figure 6 A schematic diagram of the marking process provided in the embodiment of the present application, such as Figure 6 As shown, it can be divided into the following four steps:
[0134] Step 1: For the first data in the data set (represented by samples), first use hierarchical clustering to roughly cluster into n_clusters (n_clusters = 10) clusters, so that all samples are roughly divided into n_clusters, where the samples in each cluster are roughly similar. However, the number of samples in each cluster of coarse clustering is uncertain, and it needs to be further subdivided to facilitate personnel annotation while ensuring the annotation effect. Therefore, for each cluster of coarse clustering, by setting the minimum number of samples min_cluster_size (min_cluster_size = 100), use kmeans clustering to subdivide the samples in the cluster into multiple clusters again, and the number of samples in each small cluster is around min_cluster_size.
[0135] Among them, the number of kmeans clustering clusters can be determined by the elbow rule to determine the optimal number of classification clusters.
[0136] Step 2: In the above clusters C, most of the samples are highly similar. However, in the actual clustering, there are a lot of noise data. These noise data are diverse and scattered in each cluster. Therefore, in order to filter out these noise data, it is necessary to further subdivide these clusters C and cooperate with the labeling personnel to complete the screening. Therefore, for each cluster C with a sample size of about min_cluster_size, it will be clustered into k pre-set sub-clusters Csub( Figure 6 Therefore, the number of samples in each sub-cluster Csub is small enough, and a sample can be randomly selected from each sub-cluster Csub as the representative sample of the sub-cluster Csub.
[0137] Step 3: Finally, k representative samples will be sampled for each small cluster C, and the labeler will label these k representative samples to determine whether they are positive samples (label = 1) or negative samples (label = 0). After labeling, each representative sample will propagate its label to all samples in the small cluster Csub where the representative sample is located, thereby improving labeling efficiency.
[0138] In addition, in some embodiments, for the first data in the data set, if the total amount of the first data is large (for example, exceeds a threshold amount), the first data in the data set may be divided into at least two batches; Figure 6 The steps shown cluster the first data of each batch in the data set in turn.
[0139] The embodiment of the present application can effectively process large-scale data sets and maintain an efficient annotation process through batch processing and multiple clustering strategies. When processing large-scale data, it can adapt to data set changes and growth more quickly, maintain the scalability and stability of the system, and through hierarchical clustering and k-means clustering methods, it can achieve fine division and optimized annotation of data sets, effectively reduce the impact of noise data, and improve clustering effects.
[0140] After determining the second data, some embodiments are used to describe in detail how to improve the data annotation efficiency.
[0141] In some embodiments, the labeling result of the second data may be used as the labeling result of other data in the target cluster where the second data is located except the second data.
[0142] In this embodiment, one second data can be selected from each cluster, and then the second data is labeled by manual labeling or automatic labeling by a labeling model. At this time, in addition to the second data, there are still a number of first data left in the cluster, and the label of the second data can be directly transferred to the remaining first data. Among them, the manual labeling method is relatively more accurate, while the automatic labeling method is more efficient.
[0143] The embodiment of the present application can quickly complete the labeling of other data in the cluster except the second data by propagating the labeling results in this way, thereby greatly improving the labeling efficiency.
[0144] In addition, in some embodiments, target data whose distance from the cluster center of the target cluster meets a set distance condition may be selected from the target cluster where the second data is located; and the labeling result of the second data is used as the labeling result of the target data.
[0145] In this embodiment, according to the clustering algorithm, each cluster obtained by clustering will have a corresponding cluster center, and the distance between each data in the cluster and the cluster center can be calculated by establishing coordinates. Therefore, a distance condition can be set, for example, the threshold distance is set to 10, when the distance between the data and the cluster center is less than 10, the data is regarded as the target data, and the annotation result of the second data can be propagated to the target data as the annotation result of the target data.
[0146] In the embodiment of the present application, target data close to the cluster center is selected from the cluster, and then the labeling result of the second data is used as the labeling result of the target data. This can further improve the labeling accuracy while ensuring the labeling efficiency.
[0147] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.
[0148] Figure 7 A schematic diagram of the structure of a data annotation device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the data annotation device 700 includes a descriptor acquisition module 710 , a data clustering module 720 , a result acquisition module 730 and an annotation propagation module 740 .
[0149] The descriptor acquisition module 710 is used to obtain the descriptor of the pseudo label of each first data in the data set. The data clustering module 720 is used to cluster the first data in the data set according to the descriptor to form at least one target cluster. The result acquisition module 730 is used to select the second data from the first data contained in each target cluster and obtain the labeling result corresponding to the second data. The labeling propagation module 740 is used to label other data in the target cluster where the second data is located except the second data based on the labeling result of the second data.
[0150] Optionally, it also includes a data determination module, which is used to determine the third data in the data set that has been labeled with a pseudo label; and determine the third data whose pseudo label meets the first set condition as the first data.
[0151] Optionally, the first set condition includes one or more of the following: the confidence of the pseudo label is less than or equal to a confidence threshold, and the size of the pseudo label does not meet a set size requirement.
[0152] Optionally, a screening module is further included, which is used to generate a labeling result of the third data based on the pseudo-label if the confidence of the pseudo-label of the third data is greater than a confidence threshold.
[0153] Optionally, the pseudo-label is generated by predicting the unlabeled data through a supervised training model or a semi-supervised training model; and the descriptor of the pseudo-label is determined based on feature information of a region identified by the pseudo-label in the corresponding first data.
[0154] Optionally, the data clustering module can be specifically used to: perform at least one clustering operation on the first data in the data set according to the descriptor of the pseudo label of each first data until the clustering end condition is met, and determine each second cluster obtained by the last clustering operation as the target cluster.
[0155] The i+1th clustering operation includes:
[0156] Determine the first cluster to be clustered;
[0157] Based on the descriptor of the pseudo label of each first data in the first cluster, clustering the first data in each first cluster to obtain at least one second cluster;
[0158] If at least one of the second-class clusters does not meet the clustering end condition, the i+2th clustering operation is entered;
[0159] Among them, i is an integer not less than 0; when i is 0, the first cluster is a set consisting of the first data in the data set; when i is an integer greater than 0, the first cluster is the second cluster obtained by the i-th clustering operation.
[0160] Optionally, the clustering termination condition includes at least one of the following: (1) the total number of at least one second-class clusters obtained is less than a first threshold; (2) there is at least one third-class cluster in at least one second-class cluster, and the total number of first data contained in the third-class cluster is less than the second threshold; (3) the total number of clustering operations is greater than or equal to the third threshold.
[0161] Optionally, the data clustering module can be specifically used to: cluster the first data in the data set into N first clusters; continue to cluster the first data in each first cluster to obtain M second clusters, and the number of first data contained in each second cluster is approximately equal to a set value; cluster the first data in each second cluster again to obtain K sub-clustering clusters as at least one target cluster. Wherein, N, M and K are all set values.
[0162] Optionally, the data clustering module can be specifically used to: obtain the total amount of first data in the data set; if the total amount of first data in the data set is greater than a threshold amount, divide the first data in the data set into at least two batches; and cluster the first data of each batch in the data set in turn to form at least one target cluster.
[0163] Optionally, the annotation result corresponding to the second data is obtained through manual annotation or automatic annotation using a set annotation model.
[0164] Optionally, the annotation propagation module may be specifically configured to: use the annotation result of the second data as the annotation result of other data in the target cluster where the second data is located except the second data.
[0165] Optionally, the annotation propagation module can be specifically used to: select target data whose distance from the cluster center of the target cluster meets a set distance condition from the target cluster where the second data is located; and use the annotation result of the second data as the annotation result of the target data.
[0166] Optionally, the first data includes at least one of an image, audio, text, multimedia information, and a point cloud.
[0167] The device provided in the embodiment of the present application can be used to execute the method in the above embodiment. Its implementation principle and technical effect are similar and will not be repeated here.
[0168] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the descriptor acquisition module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the above descriptor acquisition module. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or software instructions.
[0169] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 8 As shown, the electronic device 800 includes: at least one processor 801, a memory 802, a bus 803 and a communication interface 804. In the figure, the processor, the communication interface and the memory communicate with each other through the bus. The communication interface is used to communicate with other devices. The communication interface includes a communication interface for data transmission and a display interface or an operation interface for human-computer interaction. The processor is used to execute computer execution instructions, and specifically can execute the relevant steps in the method described in the above embodiment.
[0170] The processor may be a central processing unit, or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the electronic device may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs. The memory is used to store computer execution instructions. The memory may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.
[0171] This embodiment also provides a computer-readable storage medium, in which computer instructions are stored. When at least one processor of an electronic device executes the computer instructions, the electronic device executes the methods provided in the above-mentioned various implementation modes.
[0172] This embodiment also provides a computer program product, which includes computer instructions. The computer instructions are stored in a readable storage electronic device. At least one processor can read the computer instructions from the readable storage medium. At least one processor executes the computer instructions so that the electronic device implements the methods provided by the above-mentioned various embodiments.
[0173] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.
[0174] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.
[0175] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.
[0176] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0177] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0178] In the present application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship; in the formula, the character " / " indicates that the previous and next associated objects are in a "division" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.
[0179] It is to be understood that the various numerical numbers involved in the embodiments of the present application are only for the convenience of description and are not intended to limit the scope of the embodiments of the present application. In the embodiments of the present application, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0180] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
Claims
1. A data labeling method, characterized in that: include: Obtain a descriptor of a pseudo label of each first data in the data set; Clustering the first data in the data set according to the descriptor to form at least one target cluster; Selecting second data from the first data included in each target cluster, and obtaining a labeling result corresponding to the second data; Based on the labeling result of the second data, other data in the target cluster where the second data is located except the second data is labeled.
2. The method according to claim 1, characterized in that: Before clustering the first data in the data set to form at least one target cluster, the method further includes: Determine third data in the data set that has been annotated with pseudo labels; The third data whose pseudo-label satisfies the first set condition is determined as the first data.
3. The method according to claim 2, characterized in that The first set condition includes one or more of: the confidence of the pseudo label is less than or equal to a confidence threshold, and the size of the pseudo label does not meet a set size requirement.
4. The method according to claim 3, characterized in that The method further comprises: If the confidence of the pseudo label of the third data is greater than the confidence threshold, a labeling result of the third data is generated based on the pseudo label.
5. The method according to claim 1, characterized in that The pseudo-label is generated by predicting the unlabeled data through a supervised training model or a semi-supervised training model; the descriptor of the pseudo-label is determined based on feature information of the area identified by the pseudo-label in the corresponding first data.
6. The method according to claim 1, characterized in that The clustering of the first data in the data set according to the descriptor to form at least one target cluster includes: According to the descriptor of the pseudo label of each first data, the first data in the data set is clustered at least once until the clustering end condition is met, and each second cluster obtained by the last clustering operation is determined as the target cluster; wherein the i+1th clustering operation includes: Determine the first cluster to be clustered; Based on the descriptor of the pseudo label of each first data in the first cluster, clustering the first data in each first cluster to obtain at least one second cluster; If the at least one second type cluster does not satisfy the clustering end condition, then enter the (i+2)th clustering operation; Among them, i is an integer not less than 0; when i is 0, the first cluster is a set consisting of the first data in the data set; when i is an integer greater than 0, the first cluster is the second cluster obtained by the i-th clustering operation.
7. The method according to claim 6, characterized in that The clustering end condition includes at least one of the following: The total number of at least one second-class cluster obtained is less than a first threshold; There is at least one third cluster in at least one second cluster, and the total amount of first data contained in the third cluster is less than a second threshold; The total number of clustering operations is greater than or equal to a third threshold.
8. The method according to claim 1, characterized in that: The clustering of the first data in the data set to form at least one target cluster includes: Clustering the first data in the data set into N first-class clusters; Continue clustering the first data in each first cluster to obtain M second clusters, and the number of first data contained in each second cluster is approximately equal to a set value; The first data in each second cluster is clustered again to obtain K subdivided clusters as the at least one target cluster, and N, M and K are all set values.
9. The method according to claim 1, characterized in that: The clustering of the first data in the data set to form at least one target cluster includes: Obtaining the total amount of first data in the data set; If the total amount of the first data in the data set is greater than a threshold amount, dividing the first data in the data set into at least two batches; The first data of each batch in the data set is clustered in sequence to form the at least one target cluster.
10. The method according to claim 1, characterized in that The annotation result corresponding to the second data is obtained through manual annotation or automatic annotation using a set annotation model.
11. The method according to claim 1, characterized in that: The step of labeling other data in the target cluster where the second data is located, except for the second data, based on the labeling result of the second data, includes: Using the labeling result of the second data as the labeling result of other data in the target cluster where the second data is located except the second data; or; In the target cluster where the second data is located, select target data whose distance from the cluster center of the target cluster meets a set distance condition; The labeling result of the second data is used as the labeling result of the target data.
12. The method according to any one of claims 1 to 11, characterized in that: The first data includes at least one of an image, an audio, a text, a multimedia information, and a point cloud.
13. A data labeling device, characterized in that: include: A descriptor acquisition module, used to acquire a descriptor of a pseudo label of each first data in the data set; A data clustering module, used for clustering the first data in the data set according to the descriptor to form at least one target cluster; A result acquisition module, used to select second data from the first data included in each target cluster, and obtain a labeling result corresponding to the second data; The labeling propagation module is used to label other data in the target cluster where the second data is located, except the second data, based on the labeling result of the second data.
14. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 12.
15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 12 when executed by a processor.
16. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 12 when being executed by a processor.