Programming weak supervision labeling method and device based on confidence coefficient, equipment and medium

Abstract

The invention discloses a confidence-based programming weak supervision labeling method and device, equipment and a medium, and relates to the technical field of data labeling, a labeling function is utilized to provide weak labels for unlabeled data, and a weak data set is obtained; training the model based on the labeled data set and the weak data set to obtain a reliable model and a non-reliable model; evaluating an unreliable model by using the labeled data set to obtain a model evaluation accuracy rate, and evaluating a weak data set by using a reliable model based on a dual evaluation mechanism to obtain a data set evaluation accuracy rate and a confidence score; performing label type distribution statistics on the labeled data set and the weak data set to obtain a label type distribution vector, and calculating a first retention ratio of the labeled data set and a second retention ratio of the weak data set in the pruning strategy; and pruning and fusing the weak data set according to the second retention proportion and the confidence score to obtain a target data set, thereby improving the accuracy and efficiency of programming weak supervision data labeling, and improving the quality of data points.

Description

Technical Field

[0001] The present invention relates to the technical field of data annotation, and in particular to a confidence-based programming weakly supervised annotation method, device, equipment and medium. Background Art

[0002] PWS (Programmatic Weak Supervision) data labeling is a particularly important component and research field in the field of deep learning and data labeling. In recent years, deep learning models have achieved remarkable results in various fields. However, training these deep learning models often requires a large amount of labeled data. In fact, in the era of big data and artificial intelligence, various Internet platforms are flooded with a large amount of redundant data, so data is not scarce, but the cost of labeling this data is often very expensive. For example, for high-value and highly expert-dependent fields such as finance, education, and medical care, data labeling is often time-consuming and labor-intensive, which brings expensive costs to model training. There are still some problems in the current PWS data labeling: on the one hand, the labeling function will inevitably output incorrect noise labels; on the other hand, some data points themselves may have low quality, such as blurred images or semantically ambiguous text.

[0003] From the above, it can be seen that how to improve the accuracy and efficiency of programming weakly supervised data labeling and improve the quality of data points is a problem to be solved in this field. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a confidence-based programming weak supervision labeling method, device, equipment and medium, which can improve the accuracy and efficiency of programming weak supervision data labeling and improve the quality of data points. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a confidence-based weakly supervised programming annotation method, comprising:

[0006] Obtain unlabeled data and a labeled data set, and use a labeling function of a weak supervision source to programmatically provide weak labels for the unlabeled data to obtain a weak data set;

[0007] Based on the labeled data set and the weak data set, the preset models are trained respectively to obtain a reliable model and an unreliable model;

[0008] The unreliable model is evaluated using the labeled data set to obtain a model evaluation accuracy, and the weak data set is evaluated based on a dual evaluation mechanism and using the reliable model to obtain a data set evaluation accuracy and a confidence score;

[0009] Performing label category distribution statistics on the labeled data set and the weak data set to obtain a label type distribution vector, and calculating a first retention ratio of the labeled data set in the pruning strategy and a second retention ratio of the weak data set in the pruning strategy according to the label type distribution vector, the model evaluation accuracy, and the data set evaluation accuracy;

[0010] The weak data set is pruned and fused according to the second retention ratio and the confidence score to obtain a target data set, so as to complete the programmed weak supervision data labeling of the weak data set.

[0011] Optionally, the training of the preset models based on the labeled data set and the weak data set respectively to obtain a reliable model and an unreliable model includes:

[0012] Training a preset model based on the labeled data set to obtain a reliable model;

[0013] The model is trained based on the weak data set, and the data of the abstention category in the weak data set is used as a negative sample to obtain an unreliable model.

[0014] Optionally, the formula for performing label category distribution statistics on the annotated data set is:

[0015] ;

[0016] ;

[0017] in, To label the dataset, is the data point, For labels, is the kth data point Tags, () is the indicator function, is the total number of data points of the i-th category in the labeled data set, and d is the label category distribution vector of the labeled data set;

[0018] Correspondingly, the formula for performing label category distribution statistics on the weak data set is:

[0019] ;

[0020] Where m is the number of weak data sets, is the total number of data points of the i-th category in the weak data set, For a weak data set, For unlabeled data, is the data point weakly labeled by the jth labeling function, for The number of data points contained in The i-th labeling function is Weak label assigned.

[0021] Optionally, the calculating, according to the label type distribution vector, the model evaluation accuracy, and the data set evaluation accuracy, respectively, a first retention ratio of the labeled data set in the pruning strategy and a second retention ratio of the weak data set in the pruning strategy includes:

[0022] Calculate a first retention ratio of different label categories of the labeled data set in the pruning strategy according to the label type distribution vector corresponding to the labeled data set and the label type distribution vector corresponding to the weak data set;

[0023] A second retention ratio of the weak data set in the pruning strategy is calculated according to the model evaluation accuracy, the data set evaluation accuracy, and the first retention ratio.

[0024] Optionally, the formula for calculating the first retention ratio of different label categories of the annotated data set in the pruning strategy is:

[0025] ;

[0026] in, is the first reserved ratio, is the total number of data points of the i-th category in the weak data set, is the total number of data points of the i-th category in the labeled dataset, is the proportion of the i-th label category that should be retained during the pruning process, is the total number of data belonging to the jth category in all weak data sets, is the number of data belonging to the jth category in the labeled dataset.

[0027] Optionally, the calculating a second retention ratio of the weak data set in the pruning strategy according to the model evaluation accuracy, the data set evaluation accuracy, and the first retention ratio includes:

[0028] Calculate the average of the model evaluation accuracy and the data set evaluation accuracy;

[0029] The product of the average value and the first retention ratio is used as the second retention ratio of the weak data set in the pruning strategy.

[0030] Optionally, pruning and fusing the weak data set according to the second retention ratio and the confidence score to obtain a target data set includes:

[0031] Sorting the data in the weak data set according to the confidence scores to obtain the sorted weak data set;

[0032] The number of data points retained in the sorted weak data set is calculated using the second retention ratio, the sorted weak data set is pruned according to the number of data points retained to obtain pruned data, and the pruned data is fused to obtain a target data set.

[0033] In a second aspect, the present application discloses a confidence-based programming weak supervision labeling device, comprising:

[0034] A data set acquisition module is used to acquire unlabeled data and labeled data sets, and use the labeling function of the weak supervision source to programmatically provide weak labels for the unlabeled data to obtain a weak data set;

[0035] A model training module, used to train the preset models based on the labeled data set and the weak data set to obtain a reliable model and an unreliable model;

[0036] An evaluation module, used to evaluate the unreliable model using the labeled data set to obtain a model evaluation accuracy, and to evaluate the weak data set based on a dual evaluation mechanism and using the reliable model to obtain a data set evaluation accuracy and a confidence score;

[0037] A retention ratio calculation module is used to perform label category distribution statistics on the labeled data set and the weak data set to obtain a label type distribution vector, and calculate a first retention ratio of the labeled data set in the pruning strategy and a second retention ratio of the weak data set in the pruning strategy according to the label type distribution vector, the model evaluation accuracy, and the data set evaluation accuracy;

[0038] A pruning and fusion module is used to prune and fuse the weak data set according to the second retention ratio and the confidence score to obtain a target data set, so as to complete the programmed weak supervision data labeling of the weak data set.

[0039] In a third aspect, the present application discloses an electronic device, comprising:

[0040] Memory, used to store computer programs;

[0041] A processor is used to execute the computer program to implement the aforementioned confidence-based programming weak supervision labeling method.

[0042] In a fourth aspect, the present application discloses a computer storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the aforementioned disclosed confidence-based programming weakly supervised labeling method are implemented.

[0043] It can be seen that the present application provides a confidence-based programmable weak supervision labeling method, including obtaining unlabeled data and a labeled data set, using the labeling function of the weak supervision source and providing weak labels for the unlabeled data in a programmable manner to obtain a weak data set; training the preset models based on the labeled data set and the weak data set respectively to obtain a reliable model and an unreliable model; evaluating the unreliable model using the labeled data set to obtain a model evaluation accuracy, evaluating the weak data set based on a dual evaluation mechanism and using the reliable model to obtain a data set evaluation accuracy and a confidence score; performing label category distribution statistics on the labeled data set and the weak data set to obtain a label type distribution vector, and calculating the first retention ratio of the labeled data set in the pruning strategy and the second retention ratio of the weak data set in the pruning strategy according to the label type distribution vector, the model evaluation accuracy, and the data set evaluation accuracy; pruning and merging the weak data set according to the second retention ratio and the confidence score to obtain a target data set to complete the programmable weak supervision data labeling of the weak data set. The present application utilizes the labeling function of the weak supervision source and provides weak labels for unlabeled data in a programmatic manner to obtain a weak data set. The preset models are trained based on the labeled data set and the weak data set respectively to obtain a reliable model and an unreliable model. The unreliable model is then evaluated using the labeled data set, and the weak data set is evaluated using the reliable model. A dual correlation evaluation mechanism is designed to evaluate the quality of the model and the data, which reflects the quality of the labeling function and the data point respectively. The label category distribution statistics of the labeled data set and the weak data set are performed to obtain a label type distribution vector. The first retention ratio of the labeled data set in the pruning strategy and the second retention ratio of the weak data set in the pruning strategy are calculated according to the label type distribution vector, the model evaluation accuracy, and the data set evaluation accuracy. The weak data set is pruned and fused according to the second retention ratio and the confidence score. A distribution-guided pruning strategy is proposed to aggregate weak labels and prune low-quality data, thereby improving the accuracy and efficiency of programmatic weak supervision data labeling, improving the quality of data points, and providing a large-scale high-quality training data set for model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0045] Figure 1A flowchart of a confidence-based weakly supervised annotation method for programming disclosed in this application;

[0046] Figure 2 A specific flow chart of confidence-based weakly supervised annotation for programming disclosed in this application;

[0047] Figure 3 A schematic diagram of the structure of a confidence-based programming weak supervision labeling device disclosed in this application;

[0048] Figure 4 A structural diagram of an electronic device provided for this application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] PWS data labeling is a particularly important component and research field in the field of deep learning and data labeling. In recent years, deep learning models have achieved remarkable results in various fields. However, training these deep learning models often requires a large amount of labeled data. In fact, in the era of big data and artificial intelligence, various Internet platforms are flooded with a large amount of redundant data, so data is not scarce, but the cost of labeling these data is often very expensive. For example, for high-value and highly expert-experience-dependent fields such as finance, education, and medical care, data labeling is often time-consuming and labor-intensive, which brings expensive costs to model training. There are still some problems in the current PWS data labeling: on the one hand, the labeling function will inevitably output incorrect noise labels; on the other hand, some data points may have low quality themselves, such as blurred images or semantically ambiguous text. As can be seen from the above, how to improve the accuracy and efficiency of programming weakly supervised data labeling and improve the quality of data points is a problem to be solved in this field.

[0051] See also Figure 1 As shown, the embodiment of the present invention discloses a programming weak supervision labeling method based on confidence, which may specifically include:

[0052] Step S11: Obtain unlabeled data and a labeled data set, and use the labeling function of the weak supervision source to programmatically provide weak labels for the unlabeled data to obtain a weak data set.

[0053] Step S12: training the preset models based on the labeled data set and the weak data set respectively to obtain a reliable model and an unreliable model.

[0054] In this embodiment, the preset model is trained based on the labeled data set to obtain a reliable model; the model is trained based on the weak data set, and the data of the abstention category in the weak data set is used as a negative sample to obtain an unreliable model.

[0055] Specifically, we first obtain unlabeled data and a small-scale manually labeled dataset, abstract multiple weak supervision sources into the form of labeling functions, and use programming to provide weak labels for the unlabeled data to obtain weak datasets. For example, given a large-scale unlabeled data set , a small number of small-scale manually annotated data sets are annotated data sets , and a set of m labeled functions , the task is to , containing n categories) Perform category label annotation to provide high-quality data sets [X, Y] for end model training.

[0056] This application proposes the methods of "reliable learning" and "unreliable learning" to train the model on different types of data sets (i.e., labeled data sets and weak data sets). The above training model is called "reliable learning", and the trained model is recorded as , in weak datasets generated by weak supervision sources or annotation functions The above training model is called "unreliable learning", and the trained model is recorded as In the traditional weakly supervised programming framework, many labeling functions are designed to only identify specific label categories and abstain from other label categories. Therefore, the weak datasets generated by the labeling function often only contain data of a specific category label, while abstaining from other cases that do not belong to this category or all uncertain cases. In this case, the weak dataset It is easy to lack negative samples in the model, which makes it difficult to effectively train the multi-classification model. Therefore, in the embodiment of the present invention, the data points recorded as abandoned are regarded as potential negative samples, which are formally expressed as follows:

[0057] ;

[0058] in, From the tag collection Randomly select a label in The training model can effectively solve the problem of missing negative samples and obtain the model .

[0059] Step S13: using the annotated data set to evaluate the unreliable model to obtain the model evaluation accuracy, and based on the dual evaluation mechanism and using the reliable model to evaluate the weak data set to obtain the data set evaluation accuracy and confidence score.

[0060] This application proposes a dual evaluation mechanism method, through which the quality of the annotation function and the data point is evaluated. This method not only uses data to evaluate the model, but also uses the model to evaluate the data: using manually annotated data Models trained using the “unreliable learning” method Since the embodiment of the present invention mainly performs data labeling on classification task data, the accuracy rate is used as the evaluation index, and the model evaluation accuracy rate is recorded as ; Use a model trained using the "Reliable Learning" method Weak datasets generated by annotation functions Evaluate and get the overall accuracy , this accuracy index also reflects the quality of the corresponding annotation function. At the same time, due to the different qualities of different data points in the dataset, a more fine-grained confidence score is further used to measure the quality of the data point, expressed as ,in, represents the confidence score of the jth data point in the weak dataset generated by the i-th annotation function, which is The probability value of the predicted category calculated for the data point finally obtains the evaluation accuracy and confidence score of the data set.

[0061] Step S14: Perform label category distribution statistics on the labeled data set and the weak data set to obtain a label type distribution vector, and calculate a first retention ratio of the labeled data set in the pruning strategy and a second retention ratio of the weak data set in the pruning strategy according to the label type distribution vector, the model evaluation accuracy, and the data set evaluation accuracy.

[0062] In this embodiment, the formula for performing label category distribution statistics on the annotated data set is:

[0063] ;

[0064] ;

[0065] in, To label the dataset, is the data point, For labels, is the kth data point Tags, () is the indicator function, is the total number of data points of the i-th category in the labeled data set, and d is the label category distribution vector of the labeled data set;

[0066] Correspondingly, the formula for performing label category distribution statistics on the weak data set is:

[0067] ;

[0068] Where m is the number of weak data sets, is the total number of data points of the i-th category in the weak data set, For a weak data set, For unlabeled data, is the data point weakly labeled by the jth labeling function, for The number of data points contained in The i-th labeling function is Weak label assigned.

[0069] Then, a first retention ratio of different label categories of the labeled data set in the pruning strategy is calculated according to the label type distribution vector corresponding to the labeled data set and the label type distribution vector corresponding to the weak data set; a second retention ratio of the weak data set in the pruning strategy is calculated according to the model evaluation accuracy, the data set evaluation accuracy and the first retention ratio;

[0070] The formula for calculating the first retention ratio of different label categories of the labeled data set in the pruning strategy is:

[0071] ;

[0072] in, is the first reserved ratio, is the total number of data points of the i-th category in the weak data set, is the total number of data points of the i-th category in the labeled dataset, is the proportion of the i-th label category that should be retained during the pruning process, is the total number of data belonging to the jth category in all weak data sets, is the number of data belonging to the jth category in the labeled data set;

[0073] The process of calculating the second retention ratio of the weak data set in the pruning strategy is: calculating the average of the model evaluation accuracy and the data set evaluation accuracy; and taking the product of the average and the first retention ratio as the second retention ratio of the weak data set in the pruning strategy.

[0074] In this embodiment, in the distribution-guided pruning strategy, the data is first pre-pruned. Weakly supervised sources or labeling functions often output incorrect noise labels. For example, the true semantic label information of a data point x may not be the i-th label function. Output To do this, we first prune these data from the weak dataset, based on: .

[0075] The statistical process of label category distribution is as follows: In the distribution-guided pruning strategy, the pruning ratio of each label category data in each weak dataset is estimated in a fine-grained manner based on the quality evaluation results of the labeling function and data points and the distribution of the data label categories. To achieve this goal, a small number of small-scale manually labeled datasets are The distribution of label categories in is statistically analyzed and formally expressed mathematically as follows: ,in, express The label category distribution vector on , is the data point, For labels, is the kth data point The label of , II() represents the indicator function. Similarly, the category distribution on m weak datasets can be calculated as follows: ,in, , Represents the total number of data points of the i-th category in the weak data set, which is also a vector The value of the i-th dimension of .

[0076] The process of calculating the first retention ratio is as follows: Label distribution on , the calculation formula for the first retention ratio of different label categories in pruning is shown above.

[0077] The process of calculating the second retention ratio is as follows: consider the quality of different annotation functions and integrate the evaluation results in the dual evaluation mechanism: , where avg(⋅,⋅) represents the average operation. Next, we calculate the fine-grained retention ratio matrix for different weak datasets and label categories. as follows: ,in, Therefore, if you set any or , the other elements in P will also be calculated according to the above formula.

[0078] Step S15: Pruning and merging the weak data set according to the second retention ratio and the confidence score to obtain a target data set to complete the programmed weakly supervised data labeling of the weak data set.

[0079] In this embodiment, the data in the weak data set is sorted according to the confidence score to obtain the sorted weak data set, the number of data points retained in the sorted weak data set is calculated using the second retention ratio, the sorted weak data set is pruned according to the number of data points retained to obtain pruned data, the pruned data is fused to obtain the target data set, so as to complete the programmatic weak supervision data labeling of the weak data set.

[0080] Specifically, according to the fine-grained retention ratio matrix calculated for different weak datasets and label categories, the low-quality data points in the weak dataset are pruned. Specifically, according to the confidence scores of the data points calculated in the dual evaluation mechanism, the data points in the weak dataset are sorted in descending order according to the confidence scores as follows: ,in, . Then, the number of data points N that should be retained in each weak data set is calculated based on the aforementioned matrix P as follows: ,in, Finally, from each weak dataset Before taking data points and their labels form a dataset After obtaining several data sets , these data sets are fused (i.e., duplicate data points in different weak data sets are deleted) to form the final high-quality target data set [X, Y].

[0081] This application provides a programming weakly supervised data labeling method based on confidence evaluation. The specific process is as follows: Figure 2 As shown in the figure, first, multiple weak supervision sources are abstracted into the form of labeled functions, and weak labels are provided for unlabeled data in a programming way; then, "reliable learning" and "unreliable learning" methods are proposed to train the model on labeled data sets and weak data sets; next, the quality of the aforementioned model and data is evaluated based on a dual evaluation mechanism, which reflects the quality of the labeled function and the data points respectively. The dual evaluation mechanism not only uses data to evaluate the trained model, but also uses the trained model to evaluate the data; finally, based on the quality evaluation results and the label category distribution, a distribution-guided pruning strategy is used to aggregate weak labels and prune low-quality data, thereby providing a large-scale, high-quality target data set for end model training and completing the programmatic weak supervision data labeling of the weak data set.

[0082] This application targets large-scale unlabeled data and uses labeling functions abstracted from several weakly supervised sources to quickly provide weak labels for the data; it trains the model through unreliable learning and reliable learning, and evaluates the quality of the trained model and weak data set through a dual evaluation mechanism, which reflects the quality of the labeling function and the data points respectively; finally, it aggregates weak labels and prunes low-quality data through a distribution-guided pruning strategy, thereby improving the accuracy and efficiency of programming weakly supervised data labeling and the quality of data points, thereby providing a large-scale, high-quality training data set for end model training.

[0083] In this embodiment, unlabeled data and labeled data sets are obtained, and weak labels are provided for the unlabeled data in a programmatic manner using a labeling function of a weak supervision source to obtain a weak data set; preset models are trained based on the labeled data set and the weak data set to obtain a reliable model and an unreliable model; the unreliable model is evaluated using the labeled data set to obtain a model evaluation accuracy, and the weak data set is evaluated based on a dual evaluation mechanism and using the reliable model to obtain a data set evaluation accuracy and a confidence score; label category distribution statistics are performed on the labeled data set and the weak data set to obtain a label type distribution vector, and a first retention ratio of the labeled data set in the pruning strategy and a second retention ratio of the weak data set in the pruning strategy are calculated according to the label type distribution vector, the model evaluation accuracy, and the data set evaluation accuracy; the weak data set is pruned and fused according to the second retention ratio and the confidence score to obtain a target data set to complete the programmatic weak supervision data labeling of the weak data set. The present application utilizes the labeling function of the weak supervision source and provides weak labels for unlabeled data in a programmatic manner to obtain a weak data set. The preset models are trained based on the labeled data set and the weak data set respectively to obtain a reliable model and an unreliable model. The unreliable model is then evaluated using the labeled data set, and the weak data set is evaluated using the reliable model. A dual correlation evaluation mechanism is designed to evaluate the quality of the model and the data, which reflects the quality of the labeling function and the data point respectively. The label category distribution statistics of the labeled data set and the weak data set are performed to obtain a label type distribution vector. The first retention ratio of the labeled data set in the pruning strategy and the second retention ratio of the weak data set in the pruning strategy are calculated according to the label type distribution vector, the model evaluation accuracy, and the data set evaluation accuracy. The weak data set is pruned and fused according to the second retention ratio and the confidence score. A distribution-guided pruning strategy is proposed to aggregate weak labels and prune low-quality data, thereby improving the accuracy and efficiency of programmatic weak supervision data labeling, improving the quality of data points, and providing a large-scale high-quality training data set for model training.

[0084] See also Figure 3As shown, the embodiment of the present invention discloses a programming weak supervision labeling device based on confidence, which may specifically include:

[0085] The data set acquisition module 11 is used to acquire unlabeled data and labeled data sets, and use the labeling function of the weak supervision source to programmatically provide weak labels for the unlabeled data to obtain a weak data set;

[0086] A model training module 12, used to train the preset models based on the labeled data set and the weak data set to obtain a reliable model and an unreliable model;

[0087] An evaluation module 13 is used to evaluate the unreliable model using the labeled data set to obtain a model evaluation accuracy, and to evaluate the weak data set based on a dual evaluation mechanism and using the reliable model to obtain a data set evaluation accuracy and a confidence score;

[0088] A retention ratio calculation module 14 is used to perform label category distribution statistics on the labeled data set and the weak data set to obtain a label type distribution vector, and calculate a first retention ratio of the labeled data set in the pruning strategy and a second retention ratio of the weak data set in the pruning strategy according to the label type distribution vector, the model evaluation accuracy, and the data set evaluation accuracy;

[0089] The pruning and fusion module 15 is used to prune and fuse the weak data set according to the second retention ratio and the confidence score to obtain a target data set to complete the programmed weak supervision data labeling of the weak data set.

[0090] In this embodiment, unlabeled data and labeled data sets are obtained, and weak labels are provided for the unlabeled data in a programmatic manner using a labeling function of a weak supervision source to obtain a weak data set; preset models are trained based on the labeled data set and the weak data set to obtain a reliable model and an unreliable model; the unreliable model is evaluated using the labeled data set to obtain a model evaluation accuracy, and the weak data set is evaluated based on a dual evaluation mechanism and using the reliable model to obtain a data set evaluation accuracy and a confidence score; label category distribution statistics are performed on the labeled data set and the weak data set to obtain a label type distribution vector, and a first retention ratio of the labeled data set in the pruning strategy and a second retention ratio of the weak data set in the pruning strategy are calculated according to the label type distribution vector, the model evaluation accuracy, and the data set evaluation accuracy; the weak data set is pruned and fused according to the second retention ratio and the confidence score to obtain a target data set to complete the programmatic weak supervision data labeling of the weak data set. The present application utilizes the labeling function of the weak supervision source and provides weak labels for unlabeled data in a programmatic manner to obtain a weak data set. The preset models are trained based on the labeled data set and the weak data set respectively to obtain a reliable model and an unreliable model. The unreliable model is then evaluated using the labeled data set, and the weak data set is evaluated using the reliable model. A dual correlation evaluation mechanism is designed to evaluate the quality of the model and the data, which reflects the quality of the labeling function and the data point respectively. The label category distribution statistics of the labeled data set and the weak data set are performed to obtain a label type distribution vector. The first retention ratio of the labeled data set in the pruning strategy and the second retention ratio of the weak data set in the pruning strategy are calculated according to the label type distribution vector, the model evaluation accuracy, and the data set evaluation accuracy. The weak data set is pruned and fused according to the second retention ratio and the confidence score. A distribution-guided pruning strategy is proposed to aggregate weak labels and prune low-quality data, thereby improving the accuracy and efficiency of programmatic weak supervision data labeling, improving the quality of data points, and providing a large-scale high-quality training data set for model training.

[0091] In some specific embodiments, the model training module 12 may specifically include:

[0092] A reliable model training module, used to train a preset model based on the labeled data set to obtain a reliable model;

[0093] The unreliable model training module is used to train the model based on the weak data set and use the data of the abstention category in the weak data set as negative samples to obtain an unreliable model.

[0094] In some specific embodiments, the formula for performing label category distribution statistics on the annotated data set is:

[0095] ;

[0096] ;

[0097] in, To label the dataset, is the data point, For labels, is the kth data point Tags, () is the indicator function, is the total number of data points of the i-th category in the labeled data set, and d is the label category distribution vector of the labeled data set;

[0098] In some specific embodiments, the formula for performing label category distribution statistics on the weak data set is:

[0099] ;

[0100] Where m is the number of weak data sets, is the total number of data points of the i-th category in the weak data set, For a weak data set, For unlabeled data, is the data point weakly labeled by the jth labeling function, for The number of data points contained in The i-th labeling function is Weak label assigned.

[0101] In some specific embodiments, the retention ratio calculation module 14 may specifically include:

[0102] A first retention ratio calculation module, configured to calculate a first retention ratio of different label categories of the labeled data set in a pruning strategy according to the label type distribution vector corresponding to the labeled data set and the label type distribution vector corresponding to the weak data set;

[0103] A second retention ratio calculation module is used to calculate a second retention ratio of the weak data set in the pruning strategy according to the model evaluation accuracy, the data set evaluation accuracy and the first retention ratio.

[0104] In some specific embodiments, the formula for calculating the first retention ratio of different label categories of the labeled data set in the pruning strategy is:

[0105] ;

[0106] in, is the first reserved ratio, is the total number of data points of the i-th category in the weak data set, is the total number of data points of the i-th category in the labeled dataset, is the proportion of the i-th label category that should be retained during the pruning process, is the total number of data belonging to the jth category in all weak data sets, is the number of data belonging to the jth category in the labeled dataset.

[0107] In some specific embodiments, the retention ratio calculation module 14 may specifically include:

[0108] An average value calculation module, used to calculate the average value of the model evaluation accuracy and the data set evaluation accuracy;

[0109] A product calculation module is used to use the product of the average value and the first retention ratio as the second retention ratio of the weak data set in the pruning strategy.

[0110] In some specific embodiments, the pruning and fusion module 15 may specifically include:

[0111] A sorting module, used for sorting the data in the weak data set according to the confidence score to obtain the sorted weak data set;

[0112] The target data set determination module is used to calculate the number of data points retained in the sorted weak data set using the second retention ratio, prune the sorted weak data set according to the number of data points retained to obtain pruned data, and fuse the pruned data to obtain the target data set.

[0113] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the confidence-based programming weakly supervised labeling method performed by the electronic device disclosed in any of the aforementioned embodiments.

[0114] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0115] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 221, a computer program 222 and data 223, etc. The storage method can be temporary storage or permanent storage.

[0116] Among them, the operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20 to realize the calculation and processing of the data 223 in the memory 22 by the processor 21, which can be Windows, Unix, Linux, etc. In addition to including a computer program that can be used to complete the confidence-based programming weak supervision labeling method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks. In addition to including data transmitted from an external device received by the confidence-based programming weak supervision labeling device, the data 223 can also include data collected by its own input and output interface 25, etc.

[0117] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0118] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the steps of the confidence-based programming weakly supervised labeling method disclosed in any of the aforementioned embodiments are implemented.

[0119] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0120] The above is a detailed introduction to a confidence-based programming weak supervision labeling method, device, equipment and storage medium provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A confidence-based weakly supervised programming annotation method, characterized in that: include: Obtain unlabeled data and a labeled data set, and use a labeling function of a weak supervision source to programmatically provide weak labels for the unlabeled data to obtain a weak data set; Based on the labeled data set and the weak data set, the preset models are trained respectively to obtain a reliable model and an unreliable model; The unreliable model is evaluated using the labeled data set to obtain a model evaluation accuracy, and the weak data set is evaluated based on a dual evaluation mechanism and using the reliable model to obtain a data set evaluation accuracy and a confidence score; Performing label category distribution statistics on the labeled data set and the weak data set to obtain a label type distribution vector, and calculating a first retention ratio of the labeled data set in the pruning strategy and a second retention ratio of the weak data set in the pruning strategy according to the label type distribution vector, the model evaluation accuracy, and the data set evaluation accuracy; The weak data set is pruned and fused according to the second retention ratio and the confidence score to obtain a target data set, so as to complete the programmed weak supervision data labeling of the weak data set.

2. The confidence-based weakly supervised programming annotation method according to claim 1, characterized in that: The step of training the preset models based on the labeled data set and the weak data set to obtain a reliable model and an unreliable model includes: Training a preset model based on the labeled data set to obtain a reliable model; The model is trained based on the weak data set, and the data of the abstention category in the weak data set is used as a negative sample to obtain an unreliable model.

3. The confidence-based weakly supervised programming annotation method according to claim 1, characterized in that: The formula for performing label category distribution statistics on the annotated data set is: ； ； in, To label the dataset, is the data point, For labels, is the kth data point Tags, () is the indicator function, is the total number of data points of the i-th category in the labeled data set, and d is the label category distribution vector of the labeled data set; Correspondingly, the formula for performing label category distribution statistics on the weak data set is: ； Where m is the number of weak data sets, is the total number of data points of the i-th category in the weak data set, For a weak data set, For unlabeled data, is the data point weakly labeled by the jth labeling function, for The number of data points contained in The i-th labeling function is Weak label assigned.

4. The confidence-based weakly supervised programming annotation method according to claim 1, characterized in that: The calculating, according to the label type distribution vector, the model evaluation accuracy, and the data set evaluation accuracy, respectively, a first retention ratio of the labeled data set in the pruning strategy and a second retention ratio of the weak data set in the pruning strategy, comprises: Calculate a first retention ratio of different label categories of the labeled data set in the pruning strategy according to the label type distribution vector corresponding to the labeled data set and the label type distribution vector corresponding to the weak data set; A second retention ratio of the weak data set in the pruning strategy is calculated according to the model evaluation accuracy, the data set evaluation accuracy, and the first retention ratio.

5. The confidence-based weakly supervised programming annotation method according to claim 4, characterized in that: The formula for calculating the first retention ratio of different label categories of the labeled data set in the pruning strategy is: ； in, is the first reserved ratio, is the total number of data points of the i-th category in the weak data set, is the total number of data points of the i-th category in the labeled dataset, is the proportion of the i-th label category that should be retained during the pruning process, is the total number of data belonging to the jth category in all weak data sets, is the number of data belonging to the jth category in the labeled dataset.

6. The confidence-based weakly supervised programming annotation method according to claim 4, characterized in that: The calculating the second retention ratio of the weak data set in the pruning strategy according to the model evaluation accuracy, the data set evaluation accuracy and the first retention ratio includes: Calculate the average of the model evaluation accuracy and the data set evaluation accuracy; The product of the average value and the first retention ratio is used as the second retention ratio of the weak data set in the pruning strategy.

7. The confidence-based weakly supervised programming annotation method according to any one of claims 1 to 6, characterized in that: The pruning and fusing the weak data set according to the second retention ratio and the confidence score to obtain a target data set includes: Sorting the data in the weak data set according to the confidence scores to obtain the sorted weak data set; The number of data points retained in the sorted weak data set is calculated using the second retention ratio, the sorted weak data set is pruned according to the number of data points retained to obtain pruned data, and the pruned data is fused to obtain a target data set.

8. A confidence-based weakly supervised programming annotation device, characterized in that: include: A data set acquisition module is used to acquire unlabeled data and labeled data sets, and use the labeling function of the weak supervision source to programmatically provide weak labels for the unlabeled data to obtain a weak data set; A model training module, used to train the preset models based on the labeled data set and the weak data set to obtain a reliable model and an unreliable model; An evaluation module, used to evaluate the unreliable model using the labeled data set to obtain a model evaluation accuracy, and to evaluate the weak data set based on a dual evaluation mechanism and using the reliable model to obtain a data set evaluation accuracy and a confidence score; A retention ratio calculation module is used to perform label category distribution statistics on the labeled data set and the weak data set to obtain a label type distribution vector, and calculate a first retention ratio of the labeled data set in the pruning strategy and a second retention ratio of the weak data set in the pruning strategy according to the label type distribution vector, the model evaluation accuracy, and the data set evaluation accuracy; A pruning and fusion module is used to prune and fuse the weak data set according to the second retention ratio and the confidence score to obtain a target data set, so as to complete the programmed weak supervision data labeling of the weak data set.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the confidence-based programming weakly supervised labeling method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the confidence-based programming weakly supervised labeling method as described in any one of claims 1 to 7.

Images