Sample label labeling method and device and storage medium
By performing clustering and matching calculations on the sample sets and selecting appropriate labeled sample clusters for model training, the problems of high cost and low accuracy of sample labeling in the existing technology are solved, and more efficient and accurate labeling is achieved.
Patent Information
- Application Number
- CN202510244966.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-03
- Publication Date
- 2025-07-25
AI Technical Summary
In the prior art, the sample annotation method of machine learning models requires a lot of time and labor costs, and there are problems of mislabeling and missing labels, resulting in low accuracy of label labeling and affecting the prediction performance of the model.
By clustering the labeled sample set and the unlabeled sample set, the matching degree between the unlabeled sample cluster cluster and the labeled sample cluster cluster is calculated, and the labeled sample cluster cluster that meets the preset conditions is selected as the model training samples. These samples are used to train the initial sample annotation model to obtain the trained sample annotation model, which is used to label unlabeled samples.
The sample annotation model's processing capability of similar data is improved, the accuracy of label annotation is enhanced, and the time and cost of manual annotation is reduced.
Smart Images
Figure CN120372274A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical field of data processing, and particularly to a method, device, and storage medium for label annotation of samples. Background Art
[0002] Currently, many scenarios involve the training and testing of machine learning models, and the samples used for training or testing need to be labeled. Samples can be regarded as the input of the model for training and optimizing the model. The model optimizes its own parameters and structure by comparing the differences between the predicted results and the sample labels.
[0003] The existing manual label annotation method requires a large amount of time and labor costs. The method of soft label annotation by the model often has the problems of mislabeling and missing labeling, and the low accuracy of label annotation will seriously affect the prediction performance of the model. Summary of the Invention
[0004] To solve the above technical problems, this application provides at least a method, device, and storage medium for label annotation of samples.
[0005] In the first aspect of this application, a method for label annotation of samples is provided. The method includes: obtaining a set of labeled samples and a set of unlabeled samples, clustering the set of labeled samples and the set of unlabeled samples respectively to obtain a labeled sample clustering cluster and an unlabeled sample clustering cluster; calculating the matching degree between the unlabeled sample clustering cluster and the labeled sample clustering cluster to obtain a clustering cluster matching result corresponding to the unlabeled sample clustering cluster; taking any unlabeled sample clustering cluster as the current labeled sample set, and based on the clustering cluster matching result corresponding to the current labeled sample set, selecting a labeled sample clustering cluster with a matching degree meeting a preset condition from multiple labeled sample clustering clusters to obtain a model training sample set corresponding to the current labeled sample set; using the model training sample set to train an initial sample annotation model to obtain a trained sample annotation model, and inputting the current labeled sample set into the trained sample annotation model to obtain the sample label output by the trained sample annotation model.
[0006] In one embodiment, based on the clustering cluster matching results corresponding to the current labeled sample set, select the labeled sample clusters with matching degrees meeting the preset conditions from multiple labeled sample clusters to obtain the model training sample set corresponding to the current labeled sample set, including: based on the clustering cluster matching results corresponding to the current labeled sample set, count the labeled sample clusters that match the current labeled sample set successfully to obtain the labeled sample matching quantity; determine the cluster selection strategy based on the labeled sample matching quantity, and use the cluster selection strategy to select the labeled sample clusters with matching degrees meeting the preset conditions from multiple labeled sample clusters to obtain the model training sample set corresponding to the current labeled sample set.
[0007] In one embodiment, the cluster selection strategy includes: the first selection strategy: regard the labeled sample clusters that match successfully as the labeled sample clusters with matching degrees meeting the preset conditions; the second selection strategy: exclude the labeled sample clusters that match unsuccessfully from multiple labeled sample clusters, and regard the remaining labeled sample clusters as the labeled sample clusters with matching degrees meeting the preset conditions; determine the cluster selection strategy based on the labeled sample matching quantity, and use the cluster selection strategy to select the labeled sample clusters with matching degrees meeting the preset conditions from multiple labeled sample clusters to obtain the model training sample set corresponding to the current labeled sample set, including: if the labeled sample matching quantity is greater than or equal to the preset quantity threshold, use the first selection strategy to select the labeled sample clusters with matching degrees meeting the preset conditions from multiple labeled sample clusters to obtain the model training sample set corresponding to the current labeled sample set; if the labeled sample matching quantity is less than the preset quantity threshold, use the second selection strategy to select the labeled sample clusters with matching degrees meeting the preset conditions from multiple labeled sample clusters to obtain the model training sample set corresponding to the current labeled sample set.
[0008] In one embodiment, based on the clustering cluster matching results corresponding to the current labeled sample set, select the labeled sample clusters with matching degrees meeting the preset conditions from multiple labeled sample clusters to obtain the model training sample set corresponding to the current labeled sample set, including: based on the clustering cluster matching results corresponding to the current labeled sample set, sort each labeled sample cluster in descending order according to the matching degree to obtain the sorting result; select a preset number of the top-ranked labeled sample clusters from the sorting result to obtain the model training sample set corresponding to the current labeled sample set.
[0009] In one embodiment, the initial sample annotation model is trained by a labeled sample set, and the method further includes: obtain the unlabeled samples that fail to cluster in the unlabeled sample set; input the unlabeled samples that fail to cluster into the initial sample annotation model to obtain the sample labels output by the initial sample annotation model.
[0010] In one embodiment, the method further includes: inputting the labeled samples in the labeled sample set into a label verification model to obtain the verification sample labels of the labeled samples output by the label verification model; using the sample labels corresponding to the labeled samples as initial sample labels, comparing the differences between the initial sample labels and the verification sample labels to obtain the actual labeling differences, and comparing the differences between the expected sample labeling ranges corresponding to the label verification model and the expected sample labeling ranges corresponding to the labeled samples to obtain the expected labeling differences; if the actual labeling differences and the expected labeling differences are inconsistent, transferring the labeled samples corresponding to the initial sample labels to the unlabeled sample set.
[0011] In one embodiment, the expected labeling differences include that the expected sample labeling ranges corresponding to the label verification model and the expected sample labeling ranges corresponding to the labeled samples are different from each other, and / or the expected sample labeling ranges corresponding to the label verification model include the expected sample labeling ranges corresponding to the labeled samples, and / or the expected sample labeling ranges corresponding to the label verification model are included in the expected sample labeling ranges corresponding to the labeled samples.
[0012] In one embodiment, after inputting the current labeled sample set into the trained sample labeling model to obtain the sample labels output by the trained sample labeling model, the method further includes: using the sample labels output by the trained sample labeling model as the to-be-verified sample labels, inputting the unlabeled samples corresponding to the to-be-verified sample labels into the label verification model to obtain the verification sample labels output by the label verification model; comparing the differences between the to-be-verified sample labels and the verification sample labels to obtain the actual labeling differences, and comparing the differences between the expected sample labeling ranges corresponding to the label verification model and the expected sample labeling ranges corresponding to the sample labeling model to obtain the expected labeling differences; if the actual labeling differences and the expected labeling differences are inconsistent, transferring the unlabeled samples corresponding to the to-be-verified sample labels to the unlabeled sample set; if the actual labeling differences and the expected labeling differences are consistent, transferring the unlabeled samples corresponding to the to-be-verified sample labels to the labeled sample set.
[0013] The second aspect of the present application provides a label annotation device for samples. The device includes: a sample clustering module, configured to obtain a set of labeled samples and a set of unlabeled samples, cluster the set of labeled samples and the set of unlabeled samples respectively, and obtain a labeled sample clustering cluster and an unlabeled sample clustering cluster; a clustering cluster matching module, configured to calculate the matching degree between the unlabeled sample clustering cluster and the labeled sample clustering cluster, and obtain a clustering cluster matching result corresponding to the unlabeled sample clustering cluster; a sample selection module, configured to use any unlabeled sample clustering cluster as the current labeled sample set, and based on the clustering cluster matching result corresponding to the current labeled sample set, select a labeled sample clustering cluster with a matching degree meeting a preset condition from multiple labeled sample clustering clusters, and obtain a model training sample set corresponding to the current labeled sample set; a sample annotation module, configured to use the model training sample set to train an initial sample annotation model, obtain a trained sample annotation model, input the current labeled sample set into the trained sample annotation model, and obtain a sample label output by the trained sample annotation model.
[0014] The third aspect of the present application provides an electronic device, including a memory and a processor. The processor is configured to execute program instructions stored in the memory to implement the above-mentioned sample label annotation method.
[0015] The fourth aspect of the present application provides a computer-readable storage medium, on which program instructions are stored. When the program instructions are executed by a processor, the above-mentioned sample label annotation method is implemented.
[0016] In the above solution, by clustering the set of labeled samples and the set of unlabeled samples respectively, a labeled sample clustering cluster and an unlabeled sample clustering cluster are obtained; the matching degree between the unlabeled sample clustering cluster and the labeled sample clustering cluster is calculated to obtain a clustering cluster matching result corresponding to the unlabeled sample clustering cluster; any unlabeled sample clustering cluster is used as the current labeled sample set, and based on the clustering cluster matching result corresponding to the current labeled sample set, a labeled sample clustering cluster with a matching degree meeting a preset condition is selected from multiple labeled sample clustering clusters to obtain a model training sample set corresponding to the current labeled sample set; the initial sample annotation model is trained using the model training sample set to obtain a trained sample annotation model, and the current labeled sample set is input into the trained sample annotation model to obtain a sample label output by the trained sample annotation model, which improves the data processing ability of the sample annotation model for data similar to the current labeled sample set, and further improves the accuracy of label annotation.
[0017] It should be understood that the above general description and subsequent detailed description are only exemplary and explanatory, and do not limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The accompanying drawings here are incorporated into the specification and form a part of this specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to explain the technical solutions of the present application.
[0019] Figure 1 It is a schematic diagram of the solution implementation environment shown in an exemplary embodiment of the present application;
[0020] Figure 2 It is a flowchart of the label annotation method for samples shown in an exemplary embodiment of the present application;
[0021] Figure 3 It is a schematic diagram of sample label annotation shown in an exemplary embodiment of the present application;
[0022] Figure 4 It is a flowchart of sample label annotation shown in an exemplary embodiment of the present application;
[0023] Figure 5 It is a block diagram of the label annotation device for samples shown in an exemplary embodiment of the present application;
[0024] Figure 6 It is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of the present application;
[0025] Figure 7 It is a schematic diagram of the structure of a computer-readable storage medium shown in an exemplary embodiment of the present application. Detailed implementation manners
[0026] The following will describe the solutions of the embodiments of the present application in detail with reference to the accompanying drawings of the specification.
[0027] In the following description, specific details such as specific system architectures, interfaces, and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the present application.
[0028] The term "and / or" in this article is merely an association information describing associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally represents an "or" relationship between the associated objects before and after. In addition, "multiple" in this article means two or more than two. In addition, the term "at least one" in this article means any one of multiple or any combination of at least two of multiple. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set composed of A, B, and C.
[0029] The following will describe the label annotation method for samples provided by the embodiments of the present application.
[0030] Please refer to Figure 1 , Figure 1 which is a schematic diagram of the solution implementation environment shown in an exemplary embodiment of the present application. The solution implementation environment may include a terminal 110 and a server 120, and the terminal 110 and the server 120 are communicatively connected to each other.
[0031] The number of terminals 110 may be one or more. The terminal 110 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto.
[0032] The server 120 may be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms.
[0033] In one example, the server 120 may perform label annotation processing on the unlabeled samples obtained from the terminal 110 to obtain sample labels. Of course, the server 120 may store the sample labels locally, send them back to the terminal 110, or transmit them to other terminals.
[0034] In one example, a client of a target application is installed and run in the terminal 110. For example, the target application may be an application providing a label annotation function, and the unlabeled samples are processed by using the target application to obtain sample labels. The server 120 may be the background server of the target application, and is used to provide background services for the client of the target application.
[0035] For the sample label annotation method provided in the embodiments of the present application, the execution subject of each step may be the terminal 110, such as the client of the target application installed and run in the terminal 110, or the server 120, or the terminal 110 and the server 120 cooperate with each other to execute, that is, a part of the steps of the method are executed by the terminal 110 and another part of the steps are executed by the server 120.
[0036] It should be noted that the data types of the samples in the present application include but are not limited to images, videos, texts, audios, etc. The sample label annotation method provided in the present application can be applied to any scenario where sample label annotation is required.
[0037] Please refer to Figure 2 , Figure 2The flowchart of the sample label annotation method shown in an exemplary embodiment of the present application. The sample label annotation method can be applied to Figure 1 the implementation environment shown, and is specifically executed by the server in this implementation environment. It should be understood that this method can also be applicable to other exemplary implementation environments and be specifically executed by devices in other implementation environments. This embodiment does not limit the implementation environment applicable to this method.
[0038] As Figure 2 shown, the sample label annotation method at least includes steps S210 to S240, which are introduced in detail as follows:
[0039] Step S210: Obtain the labeled sample set and the unlabeled sample set, and perform clustering on the labeled sample set and the unlabeled sample set respectively to obtain the labeled sample clustering clusters and the unlabeled sample clustering clusters.
[0040] Among them, a labeled sample refers to a sample with a sample label, and an unlabeled sample refers to a sample without a sample label.
[0041] It should be noted that the sample label corresponding to the labeled sample can be manually labeled or labeled by a machine learning model. For example, the sample label corresponding to the labeled sample is labeled by a pre-trained sample annotation model. The present application does not limit this.
[0042] Perform clustering on the labeled samples in the labeled sample set to obtain one or more labeled sample clustering clusters; and perform clustering on the unlabeled samples in the unlabeled sample set to obtain one or more unlabeled sample clustering clusters.
[0043] Among them, the clustering methods used for clustering the labeled sample set and the unlabeled sample set include but are not limited to the k-means clustering algorithm (k-means Clustering Algorithm), the fuzzy C-means clustering algorithm (Fuzzy C-Means Clustering Algorithm, FCM), etc. The present application does not limit this.
[0044] Step S220: Calculate the matching degree between the unlabeled sample clustering clusters and the labeled sample clustering clusters to obtain the clustering cluster matching results corresponding to the unlabeled sample clustering clusters.
[0045] Calculate the matching degree between the unlabeled sample cluster and the labeled sample cluster. The higher the matching degree between the unlabeled sample cluster and the labeled sample cluster, the more similar the samples in the unlabeled sample cluster are to the samples in the labeled sample cluster; the lower the matching degree between the unlabeled sample cluster and the labeled sample cluster, the less similar the samples in the unlabeled sample cluster are to the samples in the labeled sample cluster.
[0046] For example, calculate the distance between the cluster center vectors of the unlabeled sample cluster and the labeled sample cluster, and obtain the matching degree between the unlabeled sample cluster and the labeled sample cluster based on the distance between the cluster center vectors. Among them, the distance between the cluster center vectors and the matching degree are inversely correlated.
[0047] Another example is to calculate the distance between the feature vectors of each sample in the unlabeled sample cluster and the feature vectors of each sample in the labeled sample cluster, and obtain the matching degree between the unlabeled sample cluster and the labeled sample cluster based on the distance between the feature vectors of each sample. Among them, the distance between the feature vectors of each sample and the matching degree are inversely correlated.
[0048] Of course, in addition to the above-exemplified embodiments, other algorithms can also be used to calculate the matching degree between two clusters. The specific matching degree calculation method can be flexibly selected according to the actual application situation, and the present application does not limit this.
[0049] Take the matching degree between the unlabeled sample cluster and each labeled sample cluster as the cluster matching result corresponding to the unlabeled sample cluster.
[0050] Step S230: Take any unlabeled sample cluster as the current labeled sample set, and based on the cluster matching result corresponding to the current labeled sample set, select the labeled sample clusters with matching degrees meeting the preset conditions from multiple labeled sample clusters to obtain the model training sample set corresponding to the current labeled sample set.
[0051] According to the cluster matching result corresponding to the current labeled sample set, select the labeled sample clusters with matching degrees meeting the preset conditions from multiple labeled sample clusters, and use each labeled sample in the selected labeled sample clusters as model training samples to obtain the model training sample set.
[0052] For example, select the labeled sample clusters with matching degrees greater than or equal to the first threshold to obtain the model training sample set corresponding to the current labeled sample set.
[0053] For another example, discard the labeled sample clusters with a matching degree less than or equal to the second threshold, and obtain the model training sample set corresponding to the current labeled sample set based on the remaining labeled sample clusters.
[0054] Among them, the first threshold and the second threshold can be preset empirical values, or can be values flexibly calculated according to the actual situation. For example, the first threshold and the second threshold are set according to the accuracy of the initial sample annotation model and / or the total number of unlabeled samples in the current labeled sample set. If the accuracy of the initial sample annotation model is lower and the total number of unlabeled samples in the current labeled sample set is larger, the first threshold and the second threshold are lower, so as to select more labeled samples and improve the model training effect.
[0055] For another example, select the labeled sample cluster with the largest matching degree to obtain the model training sample set corresponding to the current labeled sample set.
[0056] Of course, in addition to the above-exemplified embodiments, other methods can also be used to select the labeled sample clusters whose matching degrees meet the preset conditions. For example, the selection strategy of the labeled sample clusters can also be determined according to the cluster matching results corresponding to the current labeled sample set, so as to select the labeled sample clusters by using the selection strategy. The specific selection method can be flexibly set according to the actual situation, and the present application does not limit this.
[0057] Step S240: Use the model training sample set to train the initial sample annotation model to obtain a trained sample annotation model, and input the current labeled sample set into the trained sample annotation model to obtain the sample labels output by the trained sample annotation model.
[0058] Among them, the initial sample annotation model is a pre-trained machine learning model, or can also be an initialized machine learning model. In addition, the sample annotation model can be implemented based on network structures such as Convolutional Neural Networks (CNN), Recurrent Neural Network (RNN), Artificial Neural Network (ANN), and Long Short-Term Memory (LSTM), and can be flexibly selected according to the data type of the samples. The present application does not limit this.
[0059] When using the model training sample set to train the initial sample annotation model, it can be to adjust some parameters of the initial sample annotation model, or to adjust all parameters of the initial sample annotation model. The present application does not limit this.
[0060] For example, please refer to Figure 3 , Figure 3 which is a schematic diagram of sample label annotation shown in an exemplary embodiment of the present application. As Figure 3 shown, all the labeled samples in the labeled sample set are used to train the initialized machine learning model or the pre-trained machine learning model to obtain an initial sample annotation model. Then, the labeled sample set is clustered to obtain labeled sample clusters including A1 and A2, and the unlabeled sample set is clustered to obtain unlabeled sample clusters including B1 and B2.
[0061] Take the unlabeled sample cluster B1 as the current labeled sample set, calculate the matching degrees between B1 and A1, A2 respectively, and select the labeled sample clusters including A1 and A2 whose matching degrees meet the preset conditions. Then, use each labeled sample in A1 and A2 to train the initial sample annotation model to obtain a trained sample annotation model C1. Then, use the trained sample annotation model C1 to label each unlabeled sample in B1 to obtain the sample labels of B1.
[0062] Furthermore, take the unlabeled sample cluster B2 as the current labeled sample set, calculate the matching degrees between B2 and A1, A2 respectively, and select the labeled sample cluster including A1 whose matching degree meets the preset conditions. Then, use each labeled sample in A1 to train the initial sample annotation model to obtain a trained sample annotation model C2. Then, use the trained sample annotation model C2 to label each unlabeled sample in B2 to obtain the sample labels of B2.
[0063] The sample annotation model trained using the model training sample set has stronger processing ability for data similar to the model training sample set. Therefore, inputting the current labeled sample set into the trained sample annotation model to obtain the sample labels output by the trained sample annotation model can improve the accuracy of the output sample labels.
[0064] Next, some embodiments of the present application will be described in detail.
[0065] In some embodiments, the method further includes: inputting the labeled samples in the labeled sample set into a labeling verification model to obtain verification sample labels of the labeled samples output by the labeling verification model; using the sample labels corresponding to the labeled samples as initial sample labels, comparing the differences between the initial sample labels and the verification sample labels to obtain actual labeling differences, and comparing the differences between the expected sample labeling ranges corresponding to the labeling verification model and the expected sample labeling ranges corresponding to the labeled samples to obtain expected labeling differences; if the actual labeling differences and the expected labeling differences are inconsistent, transferring the labeled samples corresponding to the initial sample labels to the unlabeled sample set.
[0066] Exemplarily, the expected labeling differences include that the expected sample labeling ranges corresponding to the labeling verification model and the expected sample labeling ranges corresponding to the labeled samples are different from each other, and / or the expected sample labeling ranges corresponding to the labeling verification model include the expected sample labeling ranges corresponding to the labeled samples, and / or the expected sample labeling ranges corresponding to the labeling verification model are included in the expected sample labeling ranges corresponding to the labeled samples.
[0067] For example, if the expected sample labeling range corresponding to the labeling verification model includes vehicle target recognition, while the expected sample labeling range corresponding to the labeled sample includes building target recognition, then the expected sample labeling range corresponding to the labeling verification model and the expected sample labeling range corresponding to the labeled sample are different from each other; if the expected sample labeling range corresponding to the labeling verification model includes vehicle target recognition, while the expected sample labeling range corresponding to the labeled sample includes vehicle model recognition (vehicle model recognition requires vehicle target recognition first), then the expected sample labeling range corresponding to the labeling verification model is included in the expected sample labeling range corresponding to the labeled sample, that is, the expected sample labeling range corresponding to the labeled sample includes the expected sample labeling range corresponding to the labeling verification model.
[0068] By comparing the differences between the initial sample labels corresponding to the labeled samples and the verification sample labels output by the labeling verification model, actual labeling differences are obtained, and it is determined whether the actual labeling differences and the expected labeling differences are consistent. If they are consistent, it indicates that the credibility of the initial sample labels corresponding to the labeled samples is relatively high; if they are inconsistent, it indicates that the credibility of the initial sample labels corresponding to the labeled samples is relatively low.
[0069] Among them, the number of labeling verification models can be one or more.
[0070] Taking the case where the expected sample labeling ranges corresponding to the labeling verification model and the expected sample labeling ranges corresponding to the labeled samples are different from each other as an example, the judgment rules can be seen in Table 1 below:
[0071]
[0072] Table 1
[0073] Exemplarily, the initial sample label is obtained by the model recognition, and it corresponds to a model inference probability score. Additionally, the verification sample label also corresponds to a model inference probability score. Then, by calculating the absolute value of the difference between the model inference probability scores corresponding to the same annotation attribute (such as the same target object or the same target attribute) between the initial sample label and the verification sample label, the probability that the initial sample label is credible is obtained.
[0074] For example, the difference S1 between the model inference probability scores corresponding to the initial sample label and the verification sample label is S1 = |the model inference probability score corresponding to the initial sample label - the model inference probability score corresponding to the verification sample label|, and the credibility probability S of the initial sample label is S = S1.
[0075] Taking the example that the expected sample annotation range corresponding to the annotation verification model is included in or includes the expected sample annotation range corresponding to the annotated sample, the judgment rules are shown in Table 2 below:
[0076]
[0077] Table 2
[0078] Similarly, the difference S2 between the model inference probability scores corresponding to the initial sample label and the verification sample label is S2 = 1 - |the model inference probability score corresponding to the initial sample label - the model inference probability score corresponding to the verification sample label|, and the credibility probability S of the initial sample label is S = S2.
[0079] Taking the example that there are simultaneously differences between the expected sample annotation range corresponding to the annotation verification model and the expected sample annotation range corresponding to the annotated sample, and the expected sample annotation range corresponding to the annotation verification model is included in or includes the expected sample annotation range corresponding to the annotated sample, the judgment rules are shown in Table 3 below:
[0080]
[0081] Table 3
[0082] Similarly, referring to the above embodiments, S1 and S2 are calculated. Then, the credibility probability S of the final initial sample label is S = m×S1 + n×S2, where m and n are weighting weights.
[0083] Set an upper threshold Tt and a lower threshold Tb. When the credibility probability S of the initial sample label is greater than the upper threshold Tt, it is determined that the initial sample label is credible. When S is less than the lower threshold Tb, it is determined that the initial sample label is not credible. When S is less than the upper threshold Tt and greater than the lower threshold Tb, it is determined that the initial sample label is to be determined.
[0084] Among them, for the initial sample label to be determined, it can be input into a pre-trained sample annotation model for label annotation. If the obtained label is consistent with the initial sample label, it is determined that the initial sample label is credible; if not, it is determined that the initial sample label is not credible. It can also send the initial sample label to be determined to a human for further verification, which is not limited in this application.
[0085] For the initial sample labels that are not credible, perform correction and filtering to obtain the final set of labeled samples, so as to improve the training accuracy of the subsequent sample annotation model.
[0086] In some embodiments, in step S230, based on the clustering cluster matching result corresponding to the current labeled sample set, select the labeled sample clusters with a matching degree meeting the preset conditions from multiple labeled sample clusters to obtain the model training sample set corresponding to the current labeled sample set, including:
[0087] Step S2311: Based on the clustering cluster matching result corresponding to the current labeled sample set, count the labeled sample clusters that match the current labeled sample set successfully to obtain the labeled sample matching quantity.
[0088] Among them, the labeled sample clusters with a matching degree greater than or equal to the first threshold can be used as the labeled sample clusters that match successfully; it can also be to sort the labeled sample clusters in descending order according to the matching degree, and select the top N labeled sample clusters as the labeled sample clusters that match successfully. This application does not limit the judgment rule for successful matching.
[0089] Exemplarily, it can be to count the total number of labeled sample clusters that match the current labeled sample set successfully to obtain the labeled sample matching quantity. For example, if the number of labeled sample clusters that match the current labeled sample set successfully is 3, then the labeled sample matching quantity is 3.
[0090] Exemplarily, it can also be to count the total number of all labeled samples in the labeled sample clusters that match the current labeled sample set successfully to obtain the labeled sample matching quantity. For example, if the labeled sample clusters that match the current labeled sample set successfully include A1 and A2, and the sum of the number of labeled samples in A1 and A2 is 1000, then the labeled sample matching quantity is 1000.
[0091] Step S2312: Determine the clustering cluster selection strategy based on the labeled sample matching quantity, and use the clustering cluster selection strategy to select the labeled sample clusters with a matching degree meeting the preset conditions from multiple labeled sample clusters to obtain the model training sample set corresponding to the current labeled sample set.
[0092] For example, different clustering cluster selection strategies are preset for different quantity intervals, and the corresponding clustering cluster selection strategy is selected according to the quantity interval where the number of matched labeled samples is located.
[0093] Exemplarily, the clustering cluster selection strategy includes:
[0094] The first selection strategy: taking the clustering cluster of the successfully matched labeled samples as the clustering cluster of the labeled samples whose matching degree meets the preset conditions;
[0095] The second selection strategy: excluding the clustering clusters of the labeled samples with failed matches from multiple clustering clusters of the labeled samples, and taking the remaining clustering clusters of the labeled samples as the clustering clusters of the labeled samples whose matching degree meets the preset conditions.
[0096] Among them, the clustering clusters of the labeled samples with a matching degree less than or equal to the second threshold can be taken as the clustering clusters of the labeled samples with failed matches to be excluded.
[0097] Of course, in addition to the above first selection strategy and second selection strategy, more selection strategies can also be set, and the present application is not limited thereto.
[0098] Illustrating with an example, if the number of matched labeled samples is greater than or equal to the preset quantity threshold, the first selection strategy is adopted to select the clustering clusters of the labeled samples whose matching degree meets the preset conditions from multiple clustering clusters of the labeled samples, so as to obtain the model training sample set corresponding to the current labeled sample set; if the number of matched labeled samples is less than the preset quantity threshold, the second selection strategy is adopted to select the clustering clusters of the labeled samples whose matching degree meets the preset conditions from multiple clustering clusters of the labeled samples, so as to obtain the model training sample set corresponding to the current labeled sample set.
[0099] Among them, the preset quantity threshold can be a preset empirical value, or can be flexibly calculated according to specific situations. For example, the preset quantity threshold is calculated according to the total number of unlabeled samples in the current labeled sample set and / or the accuracy rate of the current initial sample labeling model, etc. For example, the larger the total number of unlabeled samples in the current labeled sample set and the lower the accuracy rate of the initial sample labeling model, the larger the preset quantity threshold.
[0100] For example, assuming that the value of the preset quantity threshold is 1 (the number of labeled sample clusters), it means that as long as there are labeled sample clusters that successfully match the current labeled sample set, the first selection strategy is used to select labeled sample clusters whose matching degree meets the preset conditions from multiple labeled sample clusters to obtain the model training sample set corresponding to the current labeled sample set; if there is no labeled sample cluster that successfully matches the current labeled sample set, the second selection strategy is used to select labeled sample clusters whose matching degree meets the preset conditions from multiple labeled sample clusters to obtain the model training sample set corresponding to the current labeled sample set.
[0101] In addition to the above-mentioned embodiments of selecting labeled sample clusters, other methods may be used to select labeled sample clusters:
[0102] In some implementations, in step S230, based on the cluster matching result corresponding to the current labeled sample set, a labeled sample cluster whose matching degree meets a preset condition is selected from multiple labeled sample clusters to obtain a model training sample set corresponding to the current labeled sample set, including:
[0103] Step S2321: based on the cluster matching result corresponding to the current labeled sample set, sort each labeled sample cluster in descending order according to the matching degree to obtain a sorting result;
[0104] Step S2322: Select a preset number of top-ranked labeled sample clusters from the sorting results to obtain a model training sample set corresponding to the current labeled sample set.
[0105] The preset number may be a pre-set empirical value or a value obtained by flexible calculation.
[0106] For example, based on the number of unlabeled samples in the current labeled sample set, the preset number is calculated. If the number of unlabeled samples in the current labeled sample set is larger, the unlabeled samples in the current labeled sample set are more diverse, and more labeled samples are needed for model training, then the preset number is larger. Otherwise, the preset number is smaller.
[0107] The initial sample annotation model is trained using the model training sample set corresponding to the selected current annotation sample set to obtain a trained sample annotation model, so that the current annotation sample set is input into the trained sample annotation model to obtain the sample labels output by the trained sample annotation model.
[0108] It should be noted that since the clustering cluster matching results of different unlabeled sample clustering clusters are different, the model training sample sets finally corresponding to different unlabeled sample clustering clusters are also different, so it is necessary to perform multiple model trainings on the initial sample annotation model respectively.
[0109] In some embodiments, in order to improve the efficiency of model training, the model training order between each model training sample set can be planned according to the data overlap degree between the model training sample sets finally corresponding to each unlabeled sample clustering cluster.
[0110] For example, sort each model training sample set in descending order according to the data overlap degree, train the initial sample annotation model in turn according to the sorting of each model training sample set, and input the unlabeled samples in the corresponding unlabeled sample clustering cluster into the currently trained sample annotation model for sample label annotation after each training is completed.
[0111] Illustrate with an example. The labeled sample clustering clusters include A1 and A2, and the unlabeled sample clustering clusters include B1 and B2. If the model training sample set corresponding to B1 (denoted as set 1) consists of the labeled samples in A1 and A2, and the model training sample set corresponding to B2 (denoted as set 2) consists of the labeled samples in A1, then the data overlap degree is obtained according to the number of identical labeled samples between the model training sample sets corresponding to B1 and B2. The sorting result obtained by sorting each model training sample set in descending order according to the data overlap degree is set 2, set 1. Then, first use set 2 to train the initial sample annotation model, and input the unlabeled samples in the corresponding unlabeled sample clustering cluster B2 into the currently trained sample annotation model for sample label annotation after the training is completed; then, use set 1 to train the sample annotation model obtained in the previous training cycle again. Only the different data A2 between set 1 and set 2 can be selected to train the sample annotation model obtained in the previous training cycle, and the unlabeled samples in the corresponding unlabeled sample clustering cluster B1 are input into the currently trained sample annotation model for sample label annotation after the training is completed.
[0112] In some embodiments, there will be some unlabeled samples that fail to cluster. For these unlabeled samples, all the labeled samples in the labeled sample set can be selected to train the initial sample annotation model, and the unlabeled samples that fail to cluster are input into the trained sample annotation model for sample label annotation.
[0113] In some embodiments, the initial sample annotation model is trained from a set of labeled samples, and the method further includes: obtaining the unlabeled samples that failed to cluster in the set of unlabeled samples; inputting the unlabeled samples that failed to cluster into the initial sample annotation model to obtain the sample labels output by the initial sample annotation model.
[0114] For example, please refer to Figure 4 , Figure 4 which is the flowchart of sample label annotation shown in an exemplary embodiment of the present application. As Figure 4 shown, it includes:
[0115] Step S401: Train an initial sample annotation model based on all the labeled samples in the set of labeled samples;
[0116] Step S402: Cluster the set of labeled samples and the set of unlabeled samples to obtain unlabeled sample clusters and labeled sample clusters;
[0117] Step S403: Determine whether the unlabeled samples are successfully clustered. If they are successfully clustered, execute Step S404. If they fail to cluster, execute Step S407;
[0118] Step S404: Determine whether there is a successfully matched labeled sample cluster in the unlabeled sample cluster. If there is, execute Step S405. If not, execute Step S406;
[0119] Step S405: Use the successfully matched labeled sample cluster as the model training sample set to train the initial sample annotation model, and use the trained sample annotation model to perform sample label annotation on the corresponding unlabeled sample cluster;
[0120] Step S406: Exclude the labeled sample clusters that failed to match from the multiple labeled sample clusters, use the remaining labeled sample clusters as the model training sample set to train the initial sample annotation model, and use the trained sample annotation model to perform sample label annotation on the corresponding unlabeled sample cluster;
[0121] Step S407: Use the initial sample annotation model to perform sample label annotation on the unlabeled samples that failed to cluster;
[0122] Step S408: Aggregate the sample labels corresponding to each unlabeled sample.
[0123] In some embodiments, after inputting the current labeled sample set into the trained sample labeling model and obtaining the sample labels output by the trained sample labeling model, the method further includes: using the sample labels output by the trained sample labeling model as the sample labels to be verified, inputting the unlabeled samples corresponding to the sample labels to be verified into the labeling verification model, and obtaining the verified sample labels output by the labeling verification model; comparing the differences between the sample labels to be verified and the verified sample labels to obtain the actual labeling differences, and comparing the differences between the expected sample labeling range corresponding to the labeling verification model and the expected sample labeling range corresponding to the sample labeling model to obtain the expected labeling differences; if the actual labeling differences and the expected labeling differences are inconsistent, transferring the unlabeled samples corresponding to the sample labels to be verified to the unlabeled sample set; if the actual labeling differences and the expected labeling differences are consistent, transferring the unlabeled samples corresponding to the sample labels to be verified to the labeled sample set.
[0124] The principle of verifying the sample labels output by the sample labeling model is similar to the principle of verifying the initial sample labels corresponding to the labeled samples, and will not be elaborated here.
[0125] By verifying the sample labels, the accuracy of the sample labels is improved.
[0126] Optionally, for the newly added samples in the labeled sample set (i.e., the unlabeled samples whose sample labels have passed the above verification), the initial sample labeling model is trained using the newly added samples, and the trained sample labeling model is used as the initial sample labeling model for the next cycle. The unlabeled samples in the unlabeled sample set are labeled with sample labels through multi-cycle iteration, ensuring the efficiency and accuracy of label labeling.
[0127] The sample label labeling method provided by the present application includes clustering the labeled sample set and the unlabeled sample set respectively to obtain the labeled sample clustering clusters and the unlabeled sample clustering clusters; calculating the matching degree between the unlabeled sample clustering clusters and the labeled sample clustering clusters to obtain the clustering cluster matching results corresponding to the unlabeled sample clustering clusters; using any unlabeled sample clustering cluster as the current labeled sample set, and based on the clustering cluster matching results corresponding to the current labeled sample set, selecting the labeled sample clustering clusters with matching degrees meeting the preset conditions from multiple labeled sample clustering clusters to obtain the model training sample set corresponding to the current labeled sample set; training the initial sample labeling model using the model training sample set to obtain the trained sample labeling model, and inputting the current labeled sample set into the trained sample labeling model to obtain the sample labels output by the trained sample labeling model, improving the data processing ability of the sample labeling model for data similar to the current labeled sample set, and further improving the accuracy of label labeling.
[0128] Figure 5 It is a block diagram of a sample labeling device shown in an exemplary embodiment of the present application. As Figure 5 shown, the exemplary sample labeling device 500 includes:
[0129] A sample clustering module 510, configured to obtain a labeled sample set and an unlabeled sample set, cluster the labeled sample set and the unlabeled sample set respectively, and obtain a labeled sample cluster and an unlabeled sample cluster;
[0130] A cluster matching module 520, configured to calculate the matching degree between the unlabeled sample cluster and the labeled sample cluster, and obtain a cluster matching result corresponding to the unlabeled sample cluster;
[0131] A sample selection module 530, configured to use any unlabeled sample cluster as the current labeled sample set, and based on the cluster matching result corresponding to the current labeled sample set, select a labeled sample cluster with a matching degree meeting a preset condition from multiple labeled sample clusters, and obtain a model training sample set corresponding to the current labeled sample set;
[0132] A sample labeling module 540, configured to use the model training sample set to train an initial sample labeling model, obtain a trained sample labeling model, input the current labeled sample set into the trained sample labeling model, and obtain a sample label output by the trained sample labeling model.
[0133] It should be noted that the sample labeling device provided in the above embodiment and the sample labeling method provided in the above embodiment belong to the same concept. The specific manners in which each module and unit perform operations have been described in detail in the method embodiment, and will not be elaborated here. In practical applications, the sample labeling device provided in the above embodiment can, according to needs, allocate the above functions to different functional modules, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above. This is not limited here.
[0134] Please refer to Figure 6 , Figure 6 It is a schematic structural diagram of an electronic device according to an embodiment of the present application. The electronic device 600 includes a memory 601 and a processor 602. The processor 602 is configured to execute program instructions stored in the memory 601 to implement the steps in any of the above sample labeling method embodiments. In a specific implementation scenario, the electronic device 600 may include, but is not limited to: a microcomputer, a server. In addition, the electronic device 600 may also include mobile devices such as a laptop computer, a tablet computer, etc., which are not limited here.
[0135] Specifically, the processor 602 is used to control itself and the memory 601 to implement the steps in the method embodiment of the label annotation method of any of the above samples. The processor 602 may also be referred to as a Central Processing Unit (CPU). The processor 602 may be an integrated circuit chip with signal processing capabilities. The processor 602 may also be a general-purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. Additionally, the processor 602 may be implemented jointly by integrated circuit chips.
[0136] Please refer to Figure 7 , Figure 7 which is a schematic structural diagram of an embodiment of the computer-readable storage medium of the present application. The computer-readable storage medium 700 stores program instructions 710 that can be run by a processor, and the program instructions 710 are used to implement the steps in the method embodiment of the label annotation method of any of the above samples.
[0137] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be elaborated here.
[0138] The above descriptions of the various embodiments tend to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to each other. For the sake of brevity, they will not be elaborated in this article.
[0139] In several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.
[0140] In addition, each functional unit in the various embodiments of the present application may be integrated into one processing unit, may exist separately as individual physical units, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units. If the integrated units are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods in the various embodiments of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.
Claims
1. A method for labeling a sample, characterized in that, The method includes: Obtaining a set of labeled samples and a set of unlabeled samples, clustering the set of labeled samples and the set of unlabeled samples respectively to obtain labeled sample clusters and unlabeled sample clusters; Calculating the matching degree between the unlabeled sample clusters and the labeled sample clusters to obtain the cluster matching results corresponding to the unlabeled sample clusters; Taking any unlabeled sample cluster as the current labeled sample set, and based on the cluster matching results corresponding to the current labeled sample set, selecting labeled sample clusters with matching degrees meeting the preset conditions from multiple labeled sample clusters to obtain the model training sample set corresponding to the current labeled sample set; Using the model training sample set to train an initial sample annotation model to obtain a trained sample annotation model, and inputting the current labeled sample set into the trained sample annotation model to obtain the sample labels output by the trained sample annotation model.
2. The method according to claim 1, wherein The step of, based on the cluster matching results corresponding to the current labeled sample set, selecting labeled sample clusters with matching degrees meeting the preset conditions from multiple labeled sample clusters to obtain the model training sample set corresponding to the current labeled sample set includes: Based on the cluster matching results corresponding to the current labeled sample set, counting the labeled sample clusters that match successfully with the current labeled sample set to obtain the number of labeled samples that match; Determining a cluster selection strategy based on the number of labeled samples that match, and using the cluster selection strategy to select labeled sample clusters with matching degrees meeting the preset conditions from multiple labeled sample clusters to obtain the model training sample set corresponding to the current labeled sample set.
3. The method according to claim 2, wherein The cluster selection strategy includes: The first selection strategy: taking the labeled sample clusters that match successfully as the labeled sample clusters with matching degrees meeting the preset conditions; The second selection strategy: excluding the labeled sample clusters that match unsuccessfully from multiple labeled sample clusters, and taking the remaining labeled sample clusters as the labeled sample clusters with matching degrees meeting the preset conditions; The step of determining a cluster selection strategy based on the number of labeled samples that match, and using the cluster selection strategy to select labeled sample clusters with matching degrees meeting the preset conditions from multiple labeled sample clusters to obtain the model training sample set corresponding to the current labeled sample set includes: If the number of labeled samples that match is greater than or equal to a preset quantity threshold, then using the first selection strategy to select labeled sample clusters with matching degrees meeting the preset conditions from multiple labeled sample clusters to obtain the model training sample set corresponding to the current labeled sample set; If the number of labeled samples that match is less than the preset quantity threshold, then using the second selection strategy to select labeled sample clusters with matching degrees meeting the preset conditions from multiple labeled sample clusters to obtain the model training sample set corresponding to the current labeled sample set.
4. The method according to claim 1, wherein Selecting, from multiple labeled sample clusters, the labeled sample clusters with matching degrees meeting a preset condition based on the cluster matching result corresponding to the current labeled sample set to obtain a model training sample set corresponding to the current labeled sample set, including: Based on the cluster matching result corresponding to the current labeled sample set, sorting each labeled sample cluster in descending order according to the matching degree to obtain a sorting result; Selecting a preset number of labeled sample clusters with higher rankings from the sorting result to obtain a model training sample set corresponding to the current labeled sample set.
5. The method according to claim 1, wherein The initial sample annotation model is trained by the labeled sample set, and the method further includes: Obtaining the unlabeled samples that failed to be clustered in the unlabeled sample set; Inputting the unlabeled samples that failed to be clustered into the initial sample annotation model to obtain the sample labels output by the initial sample annotation model.
6. The method according to claim 1, characterized in that, The method further includes: Inputting the labeled samples in the labeled sample set into a label verification model to obtain the verified sample labels of the labeled samples output by the label verification model; Taking the sample labels corresponding to the labeled samples as initial sample labels, comparing the differences between the initial sample labels and the verified sample labels to obtain actual annotation differences, and comparing the differences between the expected sample annotation range corresponding to the label verification model and the expected sample annotation range corresponding to the labeled samples to obtain expected annotation differences; If the actual annotation differences and the expected annotation differences are inconsistent, transferring the labeled samples corresponding to the initial sample labels to the unlabeled sample set.
7. The method according to claim 6, wherein The expected annotation differences include that the expected sample annotation range corresponding to the label verification model is different from the expected sample annotation range corresponding to the labeled samples, and / or the expected sample annotation range corresponding to the label verification model contains the expected sample annotation range corresponding to the labeled samples, and / or the expected sample annotation range corresponding to the label verification model is included in the expected sample annotation range corresponding to the labeled samples.
8. The method according to claim 1, wherein After obtaining the sample labels output by the trained sample annotation model by inputting the current labeled sample set into the trained sample annotation model, the method further includes: Taking the sample labels output by the trained sample annotation model as to-be-verified sample labels, inputting the unlabeled samples corresponding to the to-be-verified sample labels into a label verification model to obtain the verified sample labels output by the label verification model; Comparing the differences between the to-be-verified sample labels and the verified sample labels to obtain actual annotation differences, and comparing the differences between the expected sample annotation range corresponding to the label verification model and the expected sample annotation range corresponding to the sample annotation model to obtain expected annotation differences; If the actual annotation differences and the expected annotation differences are inconsistent, transferring the unlabeled samples corresponding to the to-be-verified sample labels to the unlabeled sample set; If the actual annotation difference is consistent with the expected annotation difference, transfer the unannotated sample corresponding to the sample label to be verified to the set of annotated samples.
9. An electronic device, characterized in that, The electronic device includes a memory and a processor, and the processor is configured to execute program instructions stored in the memory to implement the steps in the method according to any one of claims 1-8.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions, and the program instructions can be executed by a processor to implement the steps in the method according to any one of claims 1-8.