Confidence-based weakly supervised programming annotation method, device, equipment and medium
Through the confidence-based weakly supervised programming labeling method, the dual evaluation mechanism and distribution-guided pruning strategy are used to improve the accuracy and efficiency of data labeling, solve the problems of noisy labels and low-quality data in weakly supervised programming data labeling, and provide high-quality training datasets in high-value fields.
Patent Information
- Application Number
- CN202411554892.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-01
- Publication Date
- 2025-09-05
- Estimated Expiration
- 2044-11-01
AI Technical Summary
In existing programmatic weakly supervised data labeling methods, the labeling function is prone to outputting erroneous noisy labels, and some data points are of low quality, resulting in low data labeling accuracy and efficiency. This is especially costly in high-value fields such as finance, education, and healthcare.
A confidence-based programming weakly supervised labeling method is adopted. By obtaining unlabeled data and labeled datasets, weak labels are provided by weak supervision sources, reliable and unreliable model training is performed, and a dual evaluation mechanism is designed to evaluate the model and data quality. Label category distribution statistics are performed, the retention ratio is calculated, and pruning and fusion are performed based on the confidence score to obtain a high-quality target dataset.
It improves the accuracy and efficiency of weakly supervised data annotation for programming, improves the quality of data points, provides large-scale high-quality training data sets for model training, and reduces annotation costs.
Smart Images
Figure CN119475028B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data annotation, and in particular to a confidence-based weakly supervised programming annotation method, device, equipment and medium. Background Art
[0002] Programmatic Weak Supervision (PWS) data labeling is a particularly important component and research area in the field of deep learning and data labeling. In recent years, deep learning models have achieved remarkable results in various fields. However, training these deep learning models often requires a large amount of labeled data. In the era of big data and artificial intelligence, various internet platforms are flooded with redundant data. While data is not scarce, labeling this data is often very expensive. For example, in high-value fields such as finance, education, and healthcare that rely heavily on expert experience, data labeling is often time-consuming and labor-intensive, imposing a high cost on model training. Current PWS data labeling still has some challenges: first, the labeling function inevitably outputs erroneous noisy labels; second, some data points may be of low quality, such as blurry images or semantically ambiguous text.
[0003] As can be seen from the above, how to improve the accuracy and efficiency of weakly supervised data annotation in programming and improve the quality of data points is a problem to be solved in this field. Summary of the Invention
[0004] In view of this, the purpose of the present invention is to provide a confidence-based weakly supervised programming annotation method, device, equipment, and medium, which can improve the accuracy and efficiency of weakly supervised programming data annotation and enhance the quality of data points. The specific scheme is as follows:
[0005] In a first aspect, the present application discloses a confidence-based weakly supervised programming annotation method, comprising:
[0006] Obtain unlabeled data and a labeled dataset, and use the labeling function of the weak supervision source to programmatically provide weak labels for the unlabeled data to obtain a weak dataset;
[0007] Training the preset models based on the labeled dataset and the weak dataset respectively to obtain a reliable model and an unreliable model;
[0008] The unreliable model is evaluated using the labeled data set to obtain a model evaluation accuracy, and the weak data set is evaluated using the reliable model based on a dual evaluation mechanism to obtain a data set evaluation accuracy and a confidence score;
[0009] Performing label category distribution statistics on the labeled dataset and the weak dataset to obtain a label type distribution vector, and calculating a first retention ratio of the labeled dataset in the pruning strategy and a second retention ratio of the weak dataset in the pruning strategy according to the label type distribution vector, the model evaluation accuracy, and the dataset evaluation accuracy;
[0010] The weak data set is pruned and fused according to the second retention ratio and the confidence score to obtain a target data set, so as to complete the programmed weakly supervised data labeling of the weak data set.
[0011] Optionally, the training of preset models based on the labeled dataset and the weak dataset to obtain a reliable model and an unreliable model includes:
[0012] Training a preset model based on the labeled data set to obtain a reliable model;
[0013] The model is trained based on the weak data set, and data of the abstention category in the weak data set is used as a negative sample to obtain an unreliable model.
[0014] Optionally, the formula for performing label category distribution statistics on the annotated dataset is:
[0015] ;
[0016] ;
[0017] in, To label the dataset, is the data point, For labels, is the kth data point Tags, () is the indicator function, is the total number of data points of the i-th category in the labeled dataset, and d is the label category distribution vector of the labeled dataset;
[0018] Correspondingly, the formula for performing label category distribution statistics on the weak dataset is:
[0019] ;
[0020] Where m is the number of weak datasets, is the total number of data points of the i-th category in the weak dataset, For weak data sets, For unlabeled data, is the data point weakly labeled by the jth labeling function, for The number of data points contained in The i-th labeling function is A weak label assigned.
[0021] Optionally, the calculating, based on the label type distribution vector, the model evaluation accuracy, and the dataset evaluation accuracy, a first retention ratio of the labeled dataset in the pruning strategy and a second retention ratio of the weak dataset in the pruning strategy respectively include:
[0022] Calculating a first retention ratio of different label categories of the labeled dataset in the pruning strategy according to the label type distribution vector corresponding to the labeled dataset and the label type distribution vector corresponding to the weak dataset;
[0023] A second retention ratio of the weak dataset in the pruning strategy is calculated according to the model evaluation accuracy, the dataset evaluation accuracy, and the first retention ratio.
[0024] Optionally, the formula for calculating the first retention ratio of different label categories of the labeled data set in the pruning strategy is:
[0025] ;
[0026] in, is the first retention ratio, is the total number of data points of the i-th category in the weak dataset, is the total number of data points of the i-th category in the labeled dataset, is the proportion of the i-th label category that should be retained during the pruning process, is the total number of data belonging to the jth category in all weak data sets, is the number of data belonging to the jth category in the labeled dataset.
[0027] Optionally, the calculating a second retention ratio of the weak dataset in the pruning strategy according to the model evaluation accuracy, the dataset evaluation accuracy, and the first retention ratio includes:
[0028] Calculate the average of the model evaluation accuracy and the data set evaluation accuracy;
[0029] The product of the average value and the first retention ratio is used as the second retention ratio of the weak dataset in the pruning strategy.
[0030] Optionally, pruning and fusing the weak dataset according to the second retention ratio and the confidence score to obtain a target dataset includes:
[0031] Sorting the data in the weak data set according to the confidence scores to obtain the sorted weak data set;
[0032] The number of data points retained in the sorted weak dataset is calculated using the second retention ratio, the sorted weak dataset is pruned according to the number of data points retained to obtain pruned data, and the pruned data is fused to obtain a target dataset.
[0033] In a second aspect, the present application discloses a confidence-based weakly supervised programming annotation device, comprising:
[0034] A dataset acquisition module is used to acquire unlabeled data and labeled datasets, and to provide weak labels for the unlabeled data in a programmatic manner using the labeling function of the weak supervision source to obtain a weak dataset;
[0035] A model training module, configured to train a preset model based on the labeled dataset and the weak dataset to obtain a reliable model and an unreliable model;
[0036] An evaluation module is configured to evaluate the unreliable model using the labeled dataset to obtain a model evaluation accuracy, and to evaluate the weak dataset using the reliable model based on a dual evaluation mechanism to obtain a dataset evaluation accuracy and a confidence score;
[0037] a retention ratio calculation module, configured to perform label category distribution statistics on the labeled dataset and the weak dataset to obtain a label type distribution vector, and calculate a first retention ratio of the labeled dataset in the pruning strategy and a second retention ratio of the weak dataset in the pruning strategy according to the label type distribution vector, the model evaluation accuracy, and the dataset evaluation accuracy;
[0038] A pruning and fusion module is used to prune and fuse the weak data set according to the second retention ratio and the confidence score to obtain a target data set, so as to complete the programming weak supervision data labeling of the weak data set.
[0039] In a third aspect, the present application discloses an electronic device, comprising:
[0040] Memory, used to store computer programs;
[0041] A processor is used to execute the computer program to implement the aforementioned confidence-based programming weakly supervised labeling method.
[0042] In a fourth aspect, the present application discloses a computer storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the aforementioned disclosed confidence-based programming weakly supervised labeling method are implemented.
[0043] It can be seen that the present application provides a confidence-based programmable weak supervision labeling method, including obtaining unlabeled data and a labeled data set, using the labeling function of the weak supervision source and providing weak labels for the unlabeled data in a programmable manner to obtain a weak data set; training the preset models based on the labeled data set and the weak data set respectively to obtain a reliable model and an unreliable model; evaluating the unreliable model using the labeled data set to obtain a model evaluation accuracy, and evaluating the weak data set based on a dual evaluation mechanism and using the reliable model to obtain a data set evaluation accuracy and a confidence score; performing label category distribution statistics on the labeled data set and the weak data set to obtain a label type distribution vector, and calculating the first retention ratio of the labeled data set in the pruning strategy and the second retention ratio of the weak data set in the pruning strategy according to the label type distribution vector, the model evaluation accuracy, and the data set evaluation accuracy; pruning and fusing the weak data set according to the second retention ratio and the confidence score to obtain a target data set to complete the programmable weak supervision data labeling of the weak data set. This application utilizes the labeling function of the weak supervision source and provides weak labels for unlabeled data in a programmatic manner to obtain a weak dataset. The preset models are trained based on the labeled dataset and the weak dataset respectively to obtain a reliable model and an unreliable model. The unreliable model is then evaluated using the labeled dataset, and the weak dataset is evaluated using the reliable model. A dual correlation evaluation mechanism is designed to evaluate the quality of the model and data, which reflects the quality of the labeling function and the data points respectively. Label category distribution statistics are performed on the labeled dataset and the weak dataset to obtain a label type distribution vector. The first retention ratio of the labeled dataset in the pruning strategy and the second retention ratio of the weak dataset in the pruning strategy are calculated according to the label type distribution vector, the model evaluation accuracy, and the dataset evaluation accuracy. The weak dataset is pruned and fused according to the second retention ratio and the confidence score. A distribution-guided pruning strategy is proposed to aggregate weak labels and prune low-quality data, thereby improving the accuracy and efficiency of programmatic weak supervision data labeling, improving the quality of data points, and providing a large-scale, high-quality training dataset for model training. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0045] Figure 1This is a flow chart of a confidence-based weakly supervised programming annotation method disclosed in this application;
[0046] Figure 2 This is a specific flow chart of the confidence-based weakly supervised annotation of programming disclosed in this application;
[0047] Figure 3 This is a schematic diagram of the structure of a confidence-based weakly supervised programming annotation device disclosed in this application;
[0048] Figure 4 This is a structural diagram of an electronic device provided in this application. DETAILED DESCRIPTION
[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0050] PWS data labeling is a particularly important component and research area in the field of deep learning and data annotation. In recent years, deep learning models have achieved remarkable results in various fields. However, training these deep learning models often requires a large amount of labeled data. In the era of big data and artificial intelligence, various internet platforms are flooded with redundant data. While data is not scarce, labeling this data is often very expensive. For example, in high-value fields such as finance, education, and healthcare that rely heavily on expert experience, data labeling is often time-consuming and labor-intensive, imposing a high cost on model training. Current PWS data labeling still has several challenges: on the one hand, the labeling function inevitably outputs erroneous noisy labels; on the other hand, some data points may be of low quality, such as blurry images or semantically ambiguous text. As can be seen from the above, improving the accuracy and efficiency of weakly supervised data labeling and enhancing data point quality are unresolved issues in this field.
[0051] See also Figure 1 As shown, the embodiment of the present invention discloses a confidence-based weakly supervised programming annotation method, which may specifically include:
[0052] Step S11: Obtain unlabeled data and a labeled data set, use the labeling function of the weak supervision source and programmatically provide weak labels for the unlabeled data to obtain a weak data set.
[0053] Step S12: training the preset models based on the labeled dataset and the weak dataset respectively to obtain a reliable model and an unreliable model.
[0054] In this embodiment, the preset model is trained based on the labeled data set to obtain a reliable model; the model is trained based on the weak data set, and the data of the abstention category in the weak data set is used as a negative sample to obtain an unreliable model.
[0055] Specifically, we first obtain unlabeled data and a small-scale manually labeled dataset, abstract multiple weak supervision sources into the form of labeling functions, and use programming to provide weak labels for the unlabeled data to obtain a weak dataset. For example, given a large-scale unlabeled data set , a small number of small-scale manually annotated datasets are labeled datasets , and a set of m labeled functions , the task is in the label category space ( , containing n categories) Perform category label annotation to provide high-quality dataset [X, Y] for end model training.
[0056] This application proposes the methods of “reliable learning” and “unreliable learning” to train the model on different types of data sets (i.e., labeled data sets and weak data sets). Specifically, on the manually labeled data sets, The above training model method is called "reliable learning", and the trained model is recorded as , weak datasets generated by weak supervision sources or annotation functions The above training model method is called "unreliable learning", and the trained model is recorded as In traditional weakly supervised programming frameworks, many labeling functions are designed to only recognize specific label categories and abstain from other label categories. Therefore, the weak datasets generated by the labeling function often only contain data with a specific category label, while abstaining from other data that do not belong to the category or all uncertain cases. In this case, the weak dataset It is easy to lack negative samples in the dataset, which makes it difficult to effectively train the multi-classification model. Therefore, in the embodiment of the present invention, the data points recorded as abandoned are regarded as potential negative samples, which are formally expressed as follows:
[0057] ;
[0058] in, From the tag collection Randomly select a label in The training model can effectively solve the problem of missing negative samples and obtain the model .
[0059] Step S13: Evaluate the unreliable model using the labeled dataset to obtain a model evaluation accuracy, and evaluate the weak dataset based on a dual evaluation mechanism and using the reliable model to obtain a dataset evaluation accuracy and a confidence score.
[0060] This application proposes a dual evaluation mechanism method, through which the quality of the annotation function and the data points are evaluated. This method not only uses data to evaluate the model, but also uses the model to evaluate the data: using manually annotated data Models trained using the "unreliable learning" method For evaluation, since the embodiment of the present invention mainly performs data labeling for classification task data, the accuracy rate is used as the evaluation index, and the model evaluation accuracy rate is recorded as ; Use a model trained using Reliable Learning Weak dataset generated by annotation function Evaluate and get the overall accuracy , this accuracy index also reflects the quality of the corresponding annotation function. At the same time, since the quality of different data points in the dataset is different, a more fine-grained confidence score is further used to measure the quality of the data point, which is expressed as ,in, represents the confidence score of the jth data point in the weak dataset generated by the i-th labeling function, which is The probability value of the predicted category calculated for the data point is finally used to obtain the dataset evaluation accuracy and confidence score.
[0061] Step S14: Perform label category distribution statistics on the labeled dataset and the weak dataset to obtain a label type distribution vector, and calculate the first retention ratio of the labeled dataset in the pruning strategy and the second retention ratio of the weak dataset in the pruning strategy based on the label type distribution vector, the model evaluation accuracy, and the dataset evaluation accuracy.
[0062] In this embodiment, the formula for performing label category distribution statistics on the annotated dataset is:
[0063] ;
[0064] ;
[0065] in, To label the dataset, is the data point, For labels, is the kth data point Tags, () is the indicator function, is the total number of data points of the i-th category in the labeled dataset, and d is the label category distribution vector of the labeled dataset;
[0066] Correspondingly, the formula for performing label category distribution statistics on the weak dataset is:
[0067] ;
[0068] Where m is the number of weak datasets, is the total number of data points of the i-th category in the weak dataset, For weak data sets, For unlabeled data, is the data point weakly labeled by the jth labeling function, for The number of data points contained in The i-th labeling function is A weak label assigned.
[0069] Then, a first retention ratio of different label categories of the labeled dataset in the pruning strategy is calculated based on the label type distribution vector corresponding to the labeled dataset and the label type distribution vector corresponding to the weak dataset; a second retention ratio of the weak dataset in the pruning strategy is calculated based on the model evaluation accuracy, the dataset evaluation accuracy, and the first retention ratio;
[0070] The formula for calculating the first retention ratio of different label categories of the labeled data set in the pruning strategy is:
[0071] ;
[0072] in, is the first retention ratio, is the total number of data points of the i-th category in the weak dataset, is the total number of data points of the i-th category in the labeled dataset, is the proportion of the i-th label category that should be retained during the pruning process, is the total number of data belonging to the jth category in all weak data sets, is the number of data belonging to the jth category in the labeled dataset;
[0073] The process of calculating the second retention ratio of the weak dataset in the pruning strategy is as follows: calculating the average of the model evaluation accuracy and the dataset evaluation accuracy; and taking the product of the average and the first retention ratio as the second retention ratio of the weak dataset in the pruning strategy.
[0074] In this embodiment, in the distribution-guided pruning strategy, the data is first pre-pruned. Weakly supervised sources or labeling functions often output incorrect noise labels. For example, the true semantic label information of a data point x may not be the i-th labeling function. Output To do this, we first prune these data from the weak dataset, according to: .
[0075] The statistical process of label category distribution is as follows: In the distribution-guided pruning strategy, the pruning ratio of each label category data in each weak dataset is estimated in a fine-grained manner based on the quality evaluation results of the labeling function and data points and the distribution of the data label categories. In order to achieve this goal, a small number of small-scale manually labeled datasets are The distribution of label categories in is statistically analyzed and formally expressed mathematically as follows: ,in, express The label category distribution vector on , is the data point, For labels, is the kth data point The label of , II() represents the indicator function. Similarly, the category distribution on m weak datasets can be calculated as follows: ,in, , Represents the total number of data points of the i-th category in the weak data set, which is also a vector The value of the i-th dimension of .
[0076] The process of calculating the first retention ratio is as follows: Label distribution on , the calculation formula for the first retention ratio of different label categories in pruning is shown above.
[0077] The process of calculating the second retention ratio is as follows: considering the quality of different labeling functions and combining the evaluation results of the dual evaluation mechanism: , where avg(⋅,⋅) represents the average operation. Next, the fine-grained retention ratio matrix is calculated for different weak datasets and label categories. as follows: ,in, Therefore, if you set any or , the other elements in P will also be calculated according to the above formula.
[0078] Step S15: pruning and fusing the weak dataset according to the second retention ratio and the confidence score to obtain a target dataset, so as to complete the programmed weakly supervised data labeling of the weak dataset.
[0079] In this embodiment, the data in the weak data set is sorted according to the confidence score to obtain the sorted weak data set, the number of data points retained in the sorted weak data set is calculated using the second retention ratio, the sorted weak data set is pruned according to the number of data points retained to obtain pruned data, and the pruned data is fused to obtain the target data set to complete the programmed weak supervision data labeling of the weak data set.
[0080] Specifically, according to the fine-grained retention ratio matrix calculated for different weak datasets and label categories, the low-quality data points in the weak dataset are pruned. Specifically, according to the confidence scores of the data points calculated in the dual evaluation mechanism, the data points in the weak dataset are sorted in descending order according to the confidence scores. ,in, Then, the number of data points N that should be retained in each weak dataset is calculated based on the aforementioned matrix P as follows: ,in, Finally, from each weak dataset Before taking data points and their labels form a dataset After obtaining several data sets , these datasets are fused (i.e., duplicate data points in different weak datasets are deleted) to form the final high-quality target dataset [X, Y].
[0081] This application provides a programming weakly supervised data annotation method based on confidence evaluation. The specific process is as follows: Figure 2 As shown in the figure, first, multiple weak supervision sources are abstracted into the form of labeling functions, and weak labels are provided for unlabeled data in a programming way; then, "reliable learning" and "unreliable learning" methods are proposed to train the model on labeled datasets and weak datasets; next, the quality of the aforementioned model and data is evaluated based on a dual evaluation mechanism, which reflects the quality of the labeling function and the data points respectively. This dual evaluation mechanism not only uses data to evaluate the trained model, but also uses the trained model to evaluate the data; finally, based on the quality evaluation results and the label category distribution, a distribution-guided pruning strategy is used to aggregate weak labels and prune low-quality data, thereby providing a large-scale, high-quality target dataset for end model training and completing the programmatic weak supervision data labeling of the weak dataset.
[0082] This application targets large-scale unlabeled data and uses labeling functions abstracted from several weakly supervised sources to quickly provide weak labels for the data. It trains the model through unreliable learning and reliable learning, and evaluates the quality of the trained model and weak dataset through a dual evaluation mechanism, reflecting the quality of the labeling function and data points respectively. Finally, a distribution-guided pruning strategy is used to aggregate weak labels and prune low-quality data, thereby improving the accuracy and efficiency of programmatic weakly supervised data labeling and the quality of data points, thereby providing large-scale, high-quality training datasets for end-to-end model training.
[0083] In this embodiment, unlabeled data and labeled data sets are obtained, and weak labels are provided for the unlabeled data in a programmatic manner using a labeling function of a weak supervision source to obtain a weak data set; preset models are trained based on the labeled data set and the weak data set to obtain a reliable model and an unreliable model; the unreliable model is evaluated using the labeled data set to obtain a model evaluation accuracy, and the weak data set is evaluated based on a dual evaluation mechanism and using the reliable model to obtain a data set evaluation accuracy and a confidence score; label category distribution statistics are performed on the labeled data set and the weak data set to obtain a label type distribution vector, and a first retention ratio of the labeled data set in the pruning strategy and a second retention ratio of the weak data set in the pruning strategy are calculated according to the label type distribution vector, the model evaluation accuracy, and the data set evaluation accuracy; the weak data set is pruned and fused according to the second retention ratio and the confidence score to obtain a target data set to complete the programmatic weak supervision data labeling of the weak data set. This application utilizes the labeling function of the weak supervision source and provides weak labels for unlabeled data in a programmatic manner to obtain a weak dataset. The preset models are trained based on the labeled dataset and the weak dataset respectively to obtain a reliable model and an unreliable model. The unreliable model is then evaluated using the labeled dataset, and the weak dataset is evaluated using the reliable model. A dual correlation evaluation mechanism is designed to evaluate the quality of the model and data, which reflects the quality of the labeling function and the data points respectively. Label category distribution statistics are performed on the labeled dataset and the weak dataset to obtain a label type distribution vector. The first retention ratio of the labeled dataset in the pruning strategy and the second retention ratio of the weak dataset in the pruning strategy are calculated according to the label type distribution vector, the model evaluation accuracy, and the dataset evaluation accuracy. The weak dataset is pruned and fused according to the second retention ratio and the confidence score. A distribution-guided pruning strategy is proposed to aggregate weak labels and prune low-quality data, thereby improving the accuracy and efficiency of programmatic weak supervision data labeling, improving the quality of data points, and providing a large-scale, high-quality training dataset for model training.
[0084] See also Figure 3As shown, the embodiment of the present invention discloses a confidence-based weakly supervised programming annotation device, which may specifically include:
[0085] The data set acquisition module 11 is used to acquire unlabeled data and labeled data sets, and use the labeling function of the weak supervision source to programmatically provide weak labels for the unlabeled data to obtain a weak data set;
[0086] A model training module 12 is used to train the preset models based on the labeled data set and the weak data set to obtain a reliable model and an unreliable model;
[0087] An evaluation module 13 is configured to evaluate the unreliable model using the labeled dataset to obtain a model evaluation accuracy, and to evaluate the weak dataset using the reliable model based on a dual evaluation mechanism to obtain a dataset evaluation accuracy and a confidence score;
[0088] a retention ratio calculation module 14, configured to perform label category distribution statistics on the labeled dataset and the weak dataset to obtain a label type distribution vector, and calculate a first retention ratio of the labeled dataset in the pruning strategy and a second retention ratio of the weak dataset in the pruning strategy based on the label type distribution vector, the model evaluation accuracy, and the dataset evaluation accuracy;
[0089] The pruning and fusion module 15 is used to prune and fuse the weak data set according to the second retention ratio and the confidence score to obtain a target data set, so as to complete the programming weak supervision data labeling of the weak data set.
[0090] In this embodiment, unlabeled data and labeled data sets are obtained, and weak labels are provided for the unlabeled data in a programmatic manner using a labeling function of a weak supervision source to obtain a weak data set; preset models are trained based on the labeled data set and the weak data set to obtain a reliable model and an unreliable model; the unreliable model is evaluated using the labeled data set to obtain a model evaluation accuracy, and the weak data set is evaluated based on a dual evaluation mechanism and using the reliable model to obtain a data set evaluation accuracy and a confidence score; label category distribution statistics are performed on the labeled data set and the weak data set to obtain a label type distribution vector, and a first retention ratio of the labeled data set in the pruning strategy and a second retention ratio of the weak data set in the pruning strategy are calculated according to the label type distribution vector, the model evaluation accuracy, and the data set evaluation accuracy; the weak data set is pruned and fused according to the second retention ratio and the confidence score to obtain a target data set to complete the programmatic weak supervision data labeling of the weak data set. This application utilizes the labeling function of the weak supervision source and provides weak labels for unlabeled data in a programmatic manner to obtain a weak dataset. The preset models are trained based on the labeled dataset and the weak dataset respectively to obtain a reliable model and an unreliable model. The unreliable model is then evaluated using the labeled dataset, and the weak dataset is evaluated using the reliable model. A dual correlation evaluation mechanism is designed to evaluate the quality of the model and data, which reflects the quality of the labeling function and the data points respectively. Label category distribution statistics are performed on the labeled dataset and the weak dataset to obtain a label type distribution vector. The first retention ratio of the labeled dataset in the pruning strategy and the second retention ratio of the weak dataset in the pruning strategy are calculated according to the label type distribution vector, the model evaluation accuracy, and the dataset evaluation accuracy. The weak dataset is pruned and fused according to the second retention ratio and the confidence score. A distribution-guided pruning strategy is proposed to aggregate weak labels and prune low-quality data, thereby improving the accuracy and efficiency of programmatic weak supervision data labeling, improving the quality of data points, and providing a large-scale, high-quality training dataset for model training.
[0091] In some specific embodiments, the model training module 12 may specifically include:
[0092] A reliable model training module, used to train a preset model based on the labeled data set to obtain a reliable model;
[0093] The unreliable model training module is used to train the model based on the weak data set and use the data of the abstention category in the weak data set as negative samples to obtain an unreliable model.
[0094] In some specific embodiments, the formula for performing label category distribution statistics on the annotated dataset is:
[0095] ;
[0096] ;
[0097] in, To label the dataset, is the data point, For labels, is the kth data point Tags, () is the indicator function, is the total number of data points of the i-th category in the labeled dataset, and d is the label category distribution vector of the labeled dataset;
[0098] In some specific embodiments, the formula for performing label category distribution statistics on the weak dataset is:
[0099] ;
[0100] Where m is the number of weak datasets, is the total number of data points of the i-th category in the weak dataset, For weak data sets, For unlabeled data, is the data point weakly labeled by the jth labeling function, for The number of data points contained in The i-th labeling function is A weak label assigned.
[0101] In some specific embodiments, the retention ratio calculation module 14 may specifically include:
[0102] A first retention ratio calculation module is used to calculate a first retention ratio of different label categories of the labeled dataset in the pruning strategy according to the label type distribution vector corresponding to the labeled dataset and the label type distribution vector corresponding to the weak dataset;
[0103] A second retention ratio calculation module is used to calculate a second retention ratio of the weak dataset in the pruning strategy according to the model evaluation accuracy, the dataset evaluation accuracy and the first retention ratio.
[0104] In some specific embodiments, the formula for calculating the first retention ratio of different label categories of the labeled data set in the pruning strategy is:
[0105] ;
[0106] in, is the first retention ratio, is the total number of data points of the i-th category in the weak dataset, is the total number of data points of the i-th category in the labeled dataset, is the proportion of the i-th label category that should be retained during the pruning process, is the total number of data belonging to the jth category in all weak data sets, is the number of data belonging to the jth category in the labeled dataset.
[0107] In some specific embodiments, the retention ratio calculation module 14 may specifically include:
[0108] An average value calculation module, used to calculate the average value of the model evaluation accuracy and the data set evaluation accuracy;
[0109] A product calculation module is configured to use the product of the average value and the first retention ratio as the second retention ratio of the weak data set in the pruning strategy.
[0110] In some specific embodiments, the pruning and fusion module 15 may specifically include:
[0111] a sorting module, configured to sort the data in the weak data set according to the confidence scores to obtain the sorted weak data set;
[0112] a target data set determination module, configured to calculate the number of data points retained in the sorted weak data set using a second retention ratio, prune the sorted weak data set according to the number of data points retained to obtain pruned data, and fuse the pruned data to obtain a target data set.
[0113] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the confidence-based weakly supervised labeling method for programming performed by the electronic device as disclosed in any of the aforementioned embodiments.
[0114] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.
[0115] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon include an operating system 221, a computer program 222 and data 223, etc. The storage method can be temporary storage or permanent storage.
[0116] Among them, the operating system 221 is used to manage and control the various hardware devices and computer programs 222 on the electronic device 20 to realize the calculation and processing of the data 223 in the memory 22 by the processor 21, which can be Windows, Unix, Linux, etc. In addition to including a computer program that can be used to complete the confidence-based programming weak supervision labeling method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks. In addition to including data transmitted from an external device received by the confidence-based programming weak supervision labeling device, the data 223 can also include data collected by its own input and output interface 25, etc.
[0117] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0118] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the steps of the confidence-based programming weakly supervised labeling method disclosed in any of the aforementioned embodiments are implemented.
[0119] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.
[0120] The above is a detailed introduction to the confidence-based weakly supervised programming annotation method, device, equipment and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.
Claims
1. A confidence-based weakly supervised programming data labeling method, characterized by: include: Obtain unlabeled data and a labeled data set, and use a labeling function of a weak supervision source to programmatically provide weak labels for the unlabeled data to obtain a weak data set; wherein the unlabeled data and the labeled data set are image data or text; Training the preset models based on the labeled dataset and the weak dataset respectively to obtain a reliable model and an unreliable model; The unreliable model is evaluated using the labeled data set to obtain a model evaluation accuracy, and the weak data set is evaluated using the reliable model based on a dual evaluation mechanism to obtain a data set evaluation accuracy and a confidence score; Performing label category distribution statistics on the labeled dataset and the weak dataset to obtain a label type distribution vector, and calculating a first retention ratio of the labeled dataset in the pruning strategy and a second retention ratio of the weak dataset in the pruning strategy according to the label type distribution vector, the model evaluation accuracy, and the dataset evaluation accuracy; Pruning and fusing the weak dataset according to the second retention ratio and the confidence score to obtain a target dataset, so as to complete the programming weak supervision data labeling of the weak dataset; Wherein, calculating the first retention ratio of the labeled dataset in the pruning strategy and the second retention ratio of the weak dataset in the pruning strategy according to the label type distribution vector, the model evaluation accuracy, and the dataset evaluation accuracy respectively, including: calculating the first retention ratio of different label categories of the labeled dataset in the pruning strategy according to the label type distribution vector corresponding to the labeled dataset and the label type distribution vector corresponding to the weak dataset; calculating the second retention ratio of the weak dataset in the pruning strategy according to the model evaluation accuracy, the dataset evaluation accuracy, and the first retention ratio; The formula for calculating the first retention ratio of different label categories of the labeled data set in the pruning strategy is: ； in, is the first retention ratio, is the total number of data points of the i-th category in the weak dataset, is the total number of data points of the i-th category in the labeled dataset, is the proportion of the i-th label category that should be retained during the pruning process, is the total number of data belonging to the jth category in all weak data sets, is the number of data belonging to the jth category in the labeled dataset; Calculating a second retention ratio of the weak dataset in the pruning strategy based on the model evaluation accuracy, the dataset evaluation accuracy, and the first retention ratio, including: calculating an average of the model evaluation accuracy and the dataset evaluation accuracy; and taking the product of the average and the first retention ratio as the second retention ratio of the weak dataset in the pruning strategy.
2. The confidence-based weakly supervised programming data labeling method according to claim 1, characterized in that: The preset models are trained based on the labeled dataset and the weak dataset respectively to obtain a reliable model and an unreliable model, including: Training a preset model based on the labeled data set to obtain a reliable model; The model is trained based on the weak data set, and data of the abstention category in the weak data set is used as a negative sample to obtain an unreliable model.
3. The confidence-based weakly supervised programming data labeling method according to claim 1, characterized in that: The formula for performing label category distribution statistics on the annotated dataset is: ； ； in, To label the dataset, is the data point, For labels, is the kth data point Tags, () is the indicator function, is the total number of data points of the i-th category in the labeled dataset, and d is the label category distribution vector of the labeled dataset; Correspondingly, the formula for performing label category distribution statistics on the weak dataset is: ； Where m is the number of weak datasets, is the total number of data points of the i-th category in the weak dataset, For weak data sets, For unlabeled data, is the data point weakly labeled by the jth labeling function, for The number of data points contained in The i-th labeling function is A weak label assigned.
4. The confidence-based weakly supervised programming data labeling method according to any one of claims 1 to 3, characterized in that: The pruning and fusing of the weak dataset according to the second retention ratio and the confidence score to obtain a target dataset includes: Sorting the data in the weak data set according to the confidence scores to obtain the sorted weak data set; The number of data points retained in the sorted weak dataset is calculated using the second retention ratio, the sorted weak dataset is pruned according to the number of data points retained to obtain pruned data, and the pruned data is fused to obtain a target dataset.
5. A confidence-based weakly supervised data labeling device for programming, characterized in that: include: A dataset acquisition module is used to acquire unlabeled data and labeled datasets, and to programmatically provide weak labels for the unlabeled data using a labeling function of a weak supervision source to obtain a weak dataset; wherein the unlabeled data and the labeled dataset are image data or text; A model training module, configured to train a preset model based on the labeled dataset and the weak dataset to obtain a reliable model and an unreliable model; An evaluation module is configured to evaluate the unreliable model using the labeled dataset to obtain a model evaluation accuracy, and to evaluate the weak dataset using the reliable model based on a dual evaluation mechanism to obtain a dataset evaluation accuracy and a confidence score; a retention ratio calculation module, configured to perform label category distribution statistics on the labeled dataset and the weak dataset to obtain a label type distribution vector, and calculate a first retention ratio of the labeled dataset in the pruning strategy and a second retention ratio of the weak dataset in the pruning strategy according to the label type distribution vector, the model evaluation accuracy, and the dataset evaluation accuracy; A pruning and fusion module, configured to prune and fuse the weak data set according to the second retention ratio and the confidence score to obtain a target data set, so as to complete the programming weak supervision data labeling of the weak data set; Wherein, calculating the first retention ratio of the labeled dataset in the pruning strategy and the second retention ratio of the weak dataset in the pruning strategy according to the label type distribution vector, the model evaluation accuracy, and the dataset evaluation accuracy respectively, including: calculating the first retention ratio of different label categories of the labeled dataset in the pruning strategy according to the label type distribution vector corresponding to the labeled dataset and the label type distribution vector corresponding to the weak dataset; calculating the second retention ratio of the weak dataset in the pruning strategy according to the model evaluation accuracy, the dataset evaluation accuracy, and the first retention ratio; The formula for calculating the first retention ratio of different label categories of the labeled data set in the pruning strategy is: ； in, is the first retention ratio, is the total number of data points of the i-th category in the weak dataset, is the total number of data points of the i-th category in the labeled dataset, is the proportion of the i-th label category that should be retained during the pruning process, is the total number of data belonging to the jth category in all weak data sets, is the number of data belonging to the jth category in the labeled dataset; Calculating a second retention ratio of the weak dataset in the pruning strategy based on the model evaluation accuracy, the dataset evaluation accuracy, and the first retention ratio, including: calculating an average of the model evaluation accuracy and the dataset evaluation accuracy; and taking the product of the average and the first retention ratio as the second retention ratio of the weak dataset in the pruning strategy.
6. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the confidence-based programmable weakly supervised data labeling method according to any one of claims 1 to 4.
7. A computer-readable storage medium, characterized in that Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the confidence-based programming weakly supervised data labeling method according to any one of claims 1 to 4.