Method and device for converting labeling result in classification task into soft label

Abstract

The invention discloses a method and device for converting annotation results in a classification task into soft labels, and the method comprises the steps: expanding the annotation results of a target category on a sample by an annotator to all categories, and obtaining a conditional confidence coefficient set which is a set of annotation results with discrete confidence coefficients in all categories; gaussian distribution obeyed by the conditional confidence coefficient set is constructed, logic values of confidence coefficients of all categories corresponding to all sub-intervals on the Gaussian distribution are determined, and the logic values of the confidence coefficients of all categories are obtained through sampling in all the sub-intervals; normalizing the obtained logic values of the confidence of each category on the sample to obtain a probability label labeled on the sample by a labeling person; and obtaining a soft label of the sample by aggregating the probability labels of the plurality of labelers on the sample.

Description

Technical Field

[0001] The present application relates to the field of machine learning, and more specifically, to a method and device for converting annotation results in a classification task into soft labels. Background Art

[0002] For data mining and machine learning tasks, especially for classification tasks, labeled data for training models is crucial. It is a common practice to ask human annotators to label samples. Humans express their uncertainty in a variety of ways. Taking classification tasks as an example, annotators are often asked to give the most likely category and the confidence level. When humans express their confidence in the form of discrete grades (e.g., low, medium, high) rather than probabilistic confidence scores (e.g., 85%), it will be difficult to convert the annotator's annotation information into a probability distribution of the annotator over all categories, and then form soft labels on the samples and provide them to the model for training. If human annotation results with discrete confidence can be converted into soft labels, it will help to deal with the uncertainty and ambiguity of the labels and help the model learn more information.

[0003] When the annotation task involves subjective judgment, discrete confidence levels can better capture the annotator's subjective feeling of uncertainty. Compared with probabilistic confidence scores, discrete confidence levels are easy to interpret and use, although they are more difficult to calculate than probabilistic confidence scores. However, this method makes it difficult to convert the annotator's annotation information into a probability distribution of the annotator over all categories to form a soft label, which can then be provided to the model for training.

[0004] Therefore, providing a method to convert the annotation results with discrete confidence of the annotator in the classification task into soft labels has become an urgent problem to be solved. Summary of the invention

[0005] In response to at least one defect or improvement need in the prior art, the present invention provides a method and device for converting the labeling results in a classification task into soft labels, thereby realizing the conversion of the labeling results in the classification task into soft labels. The constructed soft labels have richer information and can more accurately reflect the inter-class similarity of samples.

[0006] To achieve the above-mentioned purpose, according to a first aspect of the present invention, a method for converting the labeling results in a classification task into soft labels is provided, the method comprising: expanding the labeling results of the target category on the sample by the labeler to all categories to obtain a conditional confidence set, wherein the conditional confidence set is a collection of labeling results with discrete confidence in all categories; constructing a Gaussian distribution obeyed by the conditional confidence set, determining the logical values ​​of the confidence of each category corresponding to each subinterval on the Gaussian distribution, wherein the logical values ​​of the confidence of each category are obtained by sampling from each subinterval; normalizing the logical values ​​of the confidence of each category on the sample to obtain a probability label marked by the labeler on the sample; and obtaining a soft label of the sample by aggregating the probability labels of multiple labelers on the sample.

[0007] In an exemplary embodiment, the labeling results of a single category on a sample by the annotator are extended to all categories to obtain a conditional confidence set, including: determining the categories other than the target category in the sample by the annotator as categories to be extended, wherein all categories include the categories to be extended and the target category; defining the labeling results of the categories to be extended as extended labeling results based on the labeling results of the target category; and determining the union of the labeling results of the target category and the extended labeling results as the conditional confidence set.

[0008] In an exemplary embodiment, constructing the Gaussian distribution obeyed by the conditional confidence set includes: obtaining the confidence of each category in the conditional confidence set; and fitting the Gaussian distribution obeyed by the conditional confidence set based on the probability distribution corresponding to the confidence of each category in the conditional confidence set.

[0009] In an exemplary embodiment, a Gaussian distribution obeyed by a conditional confidence set is constructed, and the logical numerical value of the confidence of each category corresponding to each sub-interval on the Gaussian distribution is determined, including: dividing the Gaussian distribution into multiple sub-intervals, wherein the multiple sub-intervals are divided according to the confidence of each category in the conditional confidence set; sampling the multiple sub-intervals of the Gaussian distribution to obtain the logical numerical value of the confidence corresponding to each sub-interval.

[0010] In an exemplary embodiment, after sampling multiple subintervals of the Gaussian distribution and obtaining the logical values ​​of the confidence corresponding to each subinterval, the method further includes: obtaining the logical values ​​of all categories of the annotator in the sample under given data set parameters; determining the optimal data set parameters by calculating the consistency error; and determining the optimal logical values ​​based on the optimal data set parameters.

[0011] In an exemplary embodiment, the logical values ​​of the confidence of each category on the sample are normalized to obtain the probability label annotated by the annotator on the sample, including: normalizing the logical values ​​of the confidence of the annotator on each category of the sample through an activation function to convert the logical values ​​into a probability distribution; and determining the probability label annotated by the annotator on the sample based on the probability distribution.

[0012] In an exemplary embodiment, obtaining a soft label of a sample by aggregating the probability labels of multiple annotators on the sample includes: aggregating the probability labels of multiple annotators on the sample using an average weighted method to obtain an aggregated probability label; and determining the aggregated probability label as the soft label of the multiple annotators on the sample.

[0013] According to the second aspect of the present invention, there is also provided a device for converting the labeling results in the classification task into soft labels, which includes: an expansion unit, used to expand the labeling results of the target category on the sample by the labeler to all categories, and obtain a conditional confidence set, wherein the conditional confidence set is a collection of labeling results with discrete confidence in all categories; a construction unit, used to construct a Gaussian distribution obeyed by the conditional confidence set, and determine the logical values ​​of the confidence of each category corresponding to each sub-interval on the Gaussian distribution, wherein the logical values ​​of the confidence of each category are obtained by sampling from each sub-interval; a normalization unit, used to normalize the logical values ​​of the confidence of each category on the sample to obtain the probability label marked by the labeler on the sample; an aggregation unit, used to obtain the soft label of the sample by aggregating the probability labels of multiple labelers on the sample.

[0014] According to a third aspect of the present invention, a computer-readable storage medium is also provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned method of converting the annotation results in the classification task into soft labels when running.

[0015] According to a fourth aspect of the present invention, there is also provided an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the method of converting the annotation results in the classification task into soft labels through the computer program.

[0016] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0017] (1) The present invention provides a method for converting the labeling results in a classification task into soft labels. The soft labels constructed by this method have richer information and can more accurately reflect the inter-class similarity of samples. The concept of supplementary confidence solves the problem of the lack of confidence labeling results of the annotator in all categories except the labeled category, thereby enabling analysis of richer data. In addition, constructing a Gaussian distribution by category can more accurately model the actual situation of the annotator group. By sampling from the restored Gaussian distribution, a more accurate estimated value corresponding to the discrete confidence can be obtained, thereby improving the accuracy of the formed soft labels.

[0018] (2) The present invention provides a method for converting the labeling results in a classification task into soft labels. Converting the human labeling results with discrete confidence levels into soft labels will help deal with the uncertainty and ambiguity of the labels and help the model learn more information. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A flowchart of an optional method for converting a labeling result in a classification task into a soft label provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of an optional confidence expansion provided in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of an optional distribution of logical values ​​provided in an embodiment of the present application;

[0023] FIG4( a) is a schematic diagram of an optional image sample provided in an embodiment of the present application;

[0024] FIG4( b ) is a schematic diagram of an optional method for converting a labeling result in a classification task into a soft label provided in an embodiment of the present application;

[0025] FIG4( c ) is a schematic diagram of a soft tag constructed using the comparison method;

[0026] Figure 5 A schematic diagram of the structure of an optional device for converting a labeling result in a classification task into a soft label provided in an embodiment of the present application;

[0027] Figure 6A schematic diagram of the structure of an optional electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0029] The terms "first", "second", "third", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0030] According to one aspect of the embodiments of the present application, a method for converting the labeling results in a classification task into soft labels is provided. Figure 1 The present invention describes a method for converting the annotation results in a classification task into soft labels provided in an embodiment of the present application.

[0031] Figure 1 FIG. 1 is a flow chart of an optional method for converting the labeling results in a classification task into soft labels provided in an embodiment of the present application. Figure 1 As shown, the process of the method may include the following steps:

[0032] S102, expanding the annotation results of the target category on the sample by the annotator to all categories to obtain a conditional confidence set, wherein the conditional confidence set is a set of annotation results with discrete confidence in all categories;

[0033] S104, constructing a Gaussian distribution obeyed by the conditional confidence set, and determining the logical value of each category of confidence corresponding to each sub-interval on the Gaussian distribution, wherein the logical value of each category of confidence is obtained by sampling from each sub-interval;

[0034] S106, normalizing the obtained logical values ​​of the confidence of each category on the sample to obtain a probability label annotated on the sample by the annotator;

[0035] S108, obtaining a soft label for the sample by aggregating the probability labels of the multiple annotators on the sample.

[0036] The method provided in the present application for converting the labeling results in a classification task into soft labels can be applied to the scenario where the annotator expresses his confidence in the classification task in the form of discrete confidence. The labeling information of the annotator is converted into a probability distribution of the annotator over all categories to form a soft label, which is then provided to the model for training.

[0037] In this embodiment, after expanding the annotation results of the target category on the sample by the annotator, the Gaussian distribution of the expanded conditional confidence set can be constructed category by category, and then the Gaussian distribution is divided into sub-intervals, and the optimal parameters and corresponding logical values ​​are obtained, and the logical values ​​are normalized and aggregated to finally obtain the soft label.

[0038] Optionally, when converting the annotation results with discrete confidence levels into soft labels, the following sets may be predefined: Define sample set The category set is where x i is the sample feature vector, y i Belongs to One of the possible labels is the true label. In addition, the annotator set is defined as It should be noted that |·| is represented as the number of elements in a set, such as Representation Collection The number of elements in .

[0039] Defining discrete confidence level sets The confidence level reflects the annotator's confidence in his or her annotation results. Optionally, "<" can be used to indicate a potential order relationship between two different confidence levels, for example, confidence "low" < confidence "high". In the regulations

[0040] Define the category labeling result matrix and the discrete confidence level labeling result matrix as and where z i,j It is annotated by u j The category labeling result provided for the i-th sample, similarly, h i,j It is annotated by u j The discrete form confidence provided for the i-th sample. Among them, For example, for some i and j, (z i,j ,h i,j )=("cat","low") indicates the annotator u jLabel the i-th sample as "cat" with a confidence level of "low".

[0041] In the embodiment of the present application, in order to construct the soft label of the sample, the annotator u j The discrete confidence level provided on the i-th sample will label the result (z i,j ,h i,j ) is converted to in It is annotated by u j For the i-th sample belongs to the category The probability confidence score of is the probability label we need, then the soft label p of the i-th sample i for

[0042]

[0043] Through the above steps S102 to S108, the labeling results of the target category on the sample by the labeler are extended to all categories to obtain a conditional confidence set, wherein the conditional confidence set is a set of labeling results with discrete confidence in all categories; a Gaussian distribution obeyed by the conditional confidence set is constructed, and the logical values ​​of the confidence of each category corresponding to each sub-interval on the Gaussian distribution are determined, wherein the logical values ​​of the confidence of each category are obtained by sampling from each sub-interval; the logical values ​​of the confidence of each category on the sample are normalized to obtain the probability label labeled by the labeler on the sample; the soft label of the sample is obtained by aggregating the probability labels of multiple labelers on the sample, and the soft label constructed by this method has richer information and can more accurately reflect the inter-class similarity of the sample. The proposed concept of supplementary confidence (i.e., expansion based on the labeling results of the labeler on a single category) solves the lack of the labeler's confidence labeling results on all categories except the labeled category, thereby realizing the analysis of richer data. In addition, by constructing a Gaussian distribution based on the labeling results of all annotators on all categories of samples, we can more accurately model the actual situation of the annotator group. By sampling from the restored Gaussian distribution, we can get a more accurate estimate of the discrete confidence. By using the correlation between the labeling results of multiple annotators and the real categories in the classification task samples, we can convert the labeling results of the annotators into a probability distribution of the annotators on all categories, and finally form a soft label, which can achieve the classification task more accurately.

[0044] In an exemplary embodiment, the annotation results of a single category on the sample by the annotator are extended to all categories, and the conditional confidence set obtained includes:

[0045] S11, determining the categories other than the target category in the sample as the categories to be expanded, wherein all categories include the categories to be expanded and the target category;

[0046] S12, defining the annotation result of the to-be-expanded category as the expanded annotation result based on the annotation result of the target category;

[0047] S13, determining the union of the target category annotation result and the expanded annotation result as a conditional confidence set.

[0048] In this embodiment, the confidence of other categories in the sample can be expanded based on the annotation result of a single category on the sample by the annotator. For a certain sample, suppose the annotation result of a certain annotator is (c, l k ), where c and l k are the confidence levels of a category and a discrete form respectively.

[0049] Faced with the lack of confidence annotation results of the annotator on all categories except c, based on "supplementary confidence", that is, discrete confidence l k The complementary confidence is l ′ k , which means: when the human annotator has confidence l for a certain category c k When, for the sample set All categories except c (i.e., categories to be expanded), that is For any category in , the human annotator has a discrete confidence level l ′ k , where \ represents set subtraction.

[0050] According to the definition of complementary confidence, the expanded discrete confidence level set (i.e., conditional confidence set) can be obtained:

[0051] Figure 2 A schematic diagram of an optional confidence expansion provided in an embodiment of the present application, such as Figure 2 As shown, illustratively, taking a three-classification task as an example, the labeling situation of some users is shown.

[0052] In the given information, the annotator u j The category labeling result and confidence labeling result of the i-th sample are (z i,j ,h i,j ). According to the definition of supplementary confidence, the extended human confidence annotation result is represented as vector h i,j , from h i,j It can be expanded. Indicates the annotator u j The confidence of the category c of the i-th sample, here and It should be noted that bold letters represent vectors, such as h; non-bold letters represent real numbers, such as h.

[0053] Through this embodiment, the annotation results of the annotator in a single category are expanded to all categories, which solves the problem of the annotator's lack of confidence annotation results in all categories except the annotated category, and can achieve analysis of richer data and more comprehensive data analysis.

[0054] In an exemplary embodiment, constructing a Gaussian distribution obeyed by a conditional confidence set includes:

[0055] S21, obtaining the confidence of each category in the conditional confidence set;

[0056] S22, based on the probability distribution corresponding to the confidence of each category in the conditional confidence set, fit the Gaussian distribution obeyed by the conditional confidence set.

[0057] In this embodiment, a Gaussian distribution that the conditional confidence set obeys can be constructed by category. In order to model the Gaussian distribution in different situations, a statistical analysis can be performed by category. Specifically, after the confidence expansion, all annotators will have a confidence level for each category of all samples, and then the potential confidence level set of the annotator on category c2 can be obtained under the sample whose true category is c1. Here c1 and c2 are both category sets An element in The specific calculation method is as follows:

[0058]

[0059] Specifically, ∪ represents the union operation of sets, is a set that contains both confidence and complementary confidence. In addition, The confidence levels in have the following size relationship: for all All have l ′ k <l k Because l k It measures the annotator's confidence level in the most likely category, while l ′ k represents the potential confidence level of other categories. Therefore, there must be l ′ k <l k Based on l ′ k and l k The corresponding relationship between them, that is, the sum of the confidence levels of all categories on the same sample is a fixed value, and there exists l′ k ∝-l k Therefore, given the condition When

[0060] Before the annotator makes the final prediction, the logical value of each category can be obtained, and the relative order of the logical values ​​of each category ultimately determines the final annotation result and confidence level. According to the central limit theorem, a large number of independent logical values ​​will converge to a Gaussian distribution. Based on the probability distribution corresponding to the confidence of each category in the conditional confidence set, the Gaussian distribution obeyed by the conditional confidence set can be fitted.

[0061] Through this embodiment, the Gaussian distribution is constructed category by category, which can accurately model the real situation of the annotator group.

[0062] In an exemplary embodiment, constructing a Gaussian distribution that the conditional confidence set obeys, and determining the logical value of each category of confidence corresponding to each sub-interval on the Gaussian distribution includes:

[0063] S31, dividing the Gaussian distribution into a plurality of sub-intervals, wherein the plurality of sub-intervals are divided according to the confidence of each category in the conditional confidence set;

[0064] S32, sampling multiple sub-intervals of the Gaussian distribution to obtain a logical value of the confidence level corresponding to each sub-interval.

[0065] In this embodiment, the Gaussian distribution can be segmented. Will correspond to a set of logical values For example, suppose There are four confidence levels, and low′, low, high′, high correspond to logical values ​​4, 6, 3, and 7 respectively. Then the logical value set It is represented as {4,6,3,7}. Therefore, It obeys Gaussian distribution, that is here, and Control The mean and standard deviation of .

[0066] Alternatively, it can be calculated by as well as

[0067] (1) Approximately use the logical value corresponding to the frequency value (range [0,1]) of the annotator labeling the sample with the true category c1 as category c2 to represent Right now in is the inverse function of the sigmoid function, and Count is the counting function. Here, logit(p) converts the frequency value (range [0,1]) to a logical value (range (-∞,∞)). Simulates the confusion level of a human annotator population. It means that the annotator is more inclined to annotate c1 as c2. It is expressed as:

[0068]

[0069] Here, α is an adjustable parameter related to the original distribution of the dataset.

[0070] (2) The value of can be approximated using an indicator that can calculate the degree of uncertainty or confusion of the data in the collection, which can be used The information entropy is Since information entropy can simulate the degree of confusion in the data to a certain extent, it can be reasonably used to replace the variance value.

[0071]

[0072] here, express The number of occurrences of l in .

[0073] according to The number of confidence levels in the Gaussian distribution This ensures that each confidence level corresponds to an interval, and then samples logical values ​​corresponding to different discrete confidence levels from the obtained interval. In l, assume that its corresponding logical value is X l , the numerical interval in the Gaussian distribution is expressed as It can be expressed as:

[0074]

[0075] Further, it can be simplified to:

[0076]

[0077] Here, Φ represents the cumulative distribution function (CDF) of the standard Gaussian distribution, Φ -1 represents its inverse function.

[0078] Interval The logical value range of the confidence level l is determined, and random sampling can be performed within this interval to obtain the estimated logical value corresponding to l. Specifically, the logical value matrix can be defined as in represents the human annotator u j For the potential logical value of the i-th sample, here Indicates u j The logical value of a specific category c in the i-th sample. The detailed sampling process is: For In The value obtained by random sampling within the sampling range.

[0079] In an exemplary embodiment, after sampling multiple subintervals of the Gaussian distribution and obtaining logical values ​​of confidences corresponding to each subinterval, the method further includes:

[0080] S41, given the data set parameters, obtain the logical values ​​of all categories of the annotator in the sample;

[0081] S42, determining the optimal data set parameters by calculating the consistency error;

[0082] S43, determining an optimal logical value based on the optimal data set parameters.

[0083] In this embodiment, a grid search strategy can be used to find the parameter α in the search space By constructing Gaussian distribution and splitting Gaussian distribution by category, all logical values ​​under the condition of given parameter α can be obtained.

[0084] Figure 3 A schematic diagram of an optional distribution of logical values ​​provided in an embodiment of the present application, such as Figure 3 As shown in the figure, the evaluation was performed on the ImageNet-16H image classification dataset, which contains 16 categories, such as chair, oven, knife, etc., with 75 images in each category. In order to change the difficulty of annotation, four different phase noises (ω∈{80,95,110,125}) were applied to each image. The annotator was asked to provide the image category and its confidence level (low, medium and high). The true category in the figure refers to the correct annotation by the annotator, and the non-true category refers to the incorrect annotation by the annotator.

[0085] Furthermore, in order to obtain more accurate logical values, the error between the restored logical values ​​and the original annotation information can be measured. Specifically, the consistency error can be calculated, that is, the inconsistency ratio between the category corresponding to the maximum logical value predicted by the human annotator on the sample and the original predicted category (i.e., consistency error). Here, the goal is to ensure that no additional information is introduced and to be as close as possible to the original manual annotation result z i,j (z i,j Represents the annotator u jThe labeling category results in the i-th sample) remain consistent. The optimal α value α * Corresponding to the minimum consistency error, that is, α * =argmin α E(α). E(α) is the above consistency error, and its calculation method is as follows:

[0086]

[0087] in It is an indicator function that outputs 1 if the condition is met, otherwise it outputs 0.

[0088] In an exemplary embodiment, the obtained logical values ​​of confidence levels of each category on the sample are normalized to obtain the probability labels annotated by the annotator on the sample, including:

[0089] S51, normalizing the logical values ​​of the annotator's confidence in each category of the sample through an activation function, and converting the logical values ​​into probability distribution;

[0090] S52, determining the probability label annotated on the sample by the annotator based on the probability distribution.

[0091] In this embodiment, in order to aggregate and obtain the soft labels of the samples, the logical values ​​of the confidences of the categories may be normalized.

[0092] For example, an activation function such as a sigmoid function can be used to convert all logical values (range (-∞,∞)) converted to probability confidence score (range [0,1]):

[0093]

[0094] At this time, all logical values ​​are converted to the range of [0,1], that is, normalization is completed.

[0095] In order to obtain the sample soft label, since it is a probability distribution result, for example, the softmax rule can be used to convert the probability confidence score Convert to human probability labels

[0096]

[0097] It should be noted that the temperature parameter τ controls the influence of the highest probability score on the classification and simulates the noise in the decision-making environment of human standard makers. It can be selected according to actual conditions. Commonly used values ​​include {0.01, 0.05, 0.1, 0.2, 0.3}. In the embodiment of the present application, the value of τ can be 0.05, or other optional values, which are not limited here.

[0098] In an exemplary embodiment, obtaining a soft label of a sample by aggregating probabilistic labels of multiple annotators on the sample includes:

[0099] S61, using an average weighted method to aggregate the probability labels of multiple annotators on the sample to obtain an aggregated probability label;

[0100] S62, determining the aggregated probability label as the soft label of multiple annotators on the sample.

[0101] In this embodiment, after the logical values ​​of the confidence of each category on the sample are normalized to obtain the probability label annotated by the annotator on the sample, the probability labels of multiple annotators on the sample can be aggregated using the average weighted method to obtain an aggregated probability label, and the aggregated probability label can be determined as a soft label of multiple annotators on the sample.

[0102] Optionally, aggregate the human probability labels of all annotators on the same sample to obtain the sample soft label p i :

[0103]

[0104] The soft label constructed by the embodiment of the present application will reflect the similarity between categories. Figure 4(a) is a schematic diagram of an optional image sample provided by the embodiment of the present application. As shown in Figure 4(a), for this sample, since a wheel is displayed, it is very likely to be a truck or a car; Figure 4(b) is a schematic diagram of an optional method of converting the annotation result in the classification task into a soft label provided by the embodiment of the present application. As shown in Figure 4(b), it is a soft label constructed by the method of converting the annotation result in the classification task into a soft label provided by the present application. It can be seen that the probabilities of each category are inconsistent. It is worth noting that the label shows a relatively high probability in the two categories of truck and car; Figure 4(c) is a schematic diagram of the soft label constructed by the comparison method. As shown in Figure 4(c), it treats each category other than human annotations equally and cannot reflect the similarity between categories at all.

[0105] Table 1 shows the cross entropy loss of the model when using different soft labels. It shows the cross entropy loss value on the test set after the VGG-19 model is trained with different soft labels in the training set under five different random seeds. The results are reported in the form of μ±σ, where μ and σ represent the mean and standard deviation of the five experimental results, respectively. The lower the cross entropy, the better the effect. It can be seen that the model trained using this method has a lower cross entropy loss in all cases.

[0106] Table 1

[0107]

[0108] Through this embodiment, converting human annotation results with discrete confidence levels into soft labels will help deal with the uncertainty and ambiguity of the labels and help the model learn more information.

[0109] According to another aspect of an embodiment of the present application, a conversion device for implementing the above-mentioned method of converting the annotation results in the classification task into soft labels is also provided. Figure 5 is a schematic diagram of a structure of an optional device for converting a labeling result in a classification task into a soft label according to an embodiment of the present application, such as Figure 5 As shown, the device may include:

[0110] An expansion unit 502 is used to expand the annotation results of the target category on the sample by the annotator to all categories to obtain a conditional confidence set, wherein the conditional confidence set is a set of annotation results with discrete confidence in all categories;

[0111] A construction unit 504 is used to construct a Gaussian distribution that the conditional confidence set obeys, and determine the logical value of the confidence of each category corresponding to each sub-interval on the Gaussian distribution, wherein the logical value of the confidence of each category is obtained by sampling from each sub-interval;

[0112] A normalization unit 506 is used to normalize the obtained logical values ​​of the confidence of each category on the sample to obtain the probability label annotated by the annotator on the sample;

[0113] The aggregation unit 508 is used to obtain a soft label of the sample by aggregating the probabilistic labels of multiple annotators on the sample.

[0114] It should be noted that the expansion unit 502 in this embodiment can be used to execute the above step S102, the construction unit 504 in this embodiment can be used to execute the above step S104, the normalization unit 506 in this embodiment can be used to execute the above step S106, and the aggregation unit 508 in this embodiment can be used to execute the above step S108.

[0115] Through the above modules, the labeling results of the target category on the sample by the labeler are extended to all categories to obtain the conditional confidence set, where the conditional confidence set is the set of labeling results with discrete confidence in all categories; the Gaussian distribution obeyed by the conditional confidence set is constructed, and the logical values ​​of the confidence of each category corresponding to each sub-interval on the Gaussian distribution are determined, where the logical values ​​of the confidence of each category are sampled from each sub-interval; the logical values ​​of the confidence of each category on the sample are normalized to obtain the probability label marked by the labeler on the sample; by aggregating the probability labels of multiple labelers on the sample, the soft label of the sample is obtained. The soft label constructed by this method has richer information and can more accurately reflect the inter-class similarity of the sample. The concept of supplementary confidence proposed solves the lack of confidence labeling results of the labeler on all categories except the labeled category, and thus can realize the analysis of richer data. In addition, constructing Gaussian distribution by category can more accurately model the actual situation of the annotator group. By sampling from the restored Gaussian distribution, more accurate estimated values ​​corresponding to discrete confidence levels can be obtained, thereby improving the accuracy of the formed soft labels.

[0116] In an exemplary embodiment, the expansion unit includes:

[0117] A first determination module is used to determine the categories other than the target category in the sample as the categories to be expanded, wherein all categories include the categories to be expanded and the target category;

[0118] A definition module, used to define the annotation results of the to-be-expanded category as the expanded annotation results based on the annotation results of the target category;

[0119] The union of the target category annotation results and the expanded annotation results is determined as the conditional confidence set.

[0120] In one exemplary embodiment, the building block includes:

[0121] An acquisition module is used to obtain the confidence of each category in the conditional confidence set;

[0122] The fitting module is used to fit the Gaussian distribution obeyed by the conditional confidence set based on the probability distribution corresponding to the confidence of each category in the conditional confidence set.

[0123] In one exemplary embodiment, the building block includes:

[0124] A partitioning module, used for partitioning the Gaussian distribution into a plurality of sub-intervals, wherein the plurality of sub-intervals are partitioned according to the confidence of each category in the conditional confidence set;

[0125] The sampling module is used to sample multiple sub-intervals of the Gaussian distribution and obtain the logical value of the confidence level corresponding to each sub-interval.

[0126] In an exemplary embodiment, the apparatus further comprises:

[0127] The acquisition unit is used to obtain the logical values ​​of all categories of the sample by the annotator under the given data set parameters;

[0128] A first determining unit, configured to determine an optimal data set parameter by calculating a consistency error;

[0129] The second determining unit is used to determine an optimal logical value based on the optimal data set parameters.

[0130] In an exemplary embodiment, the normalization unit comprises:

[0131] The normalization module is used to normalize the logical values ​​of the annotator's confidence in each category of the sample through an activation function, and convert the logical values ​​into probability distribution;

[0132] The second determination module is used to determine the probability label annotated on the sample by the annotator based on the probability distribution.

[0133] In one exemplary embodiment, the polymeric unit comprises:

[0134] Aggregation module, used to aggregate the probability labels of multiple annotators on the sample using the average weighted method to obtain the aggregated probability label;

[0135] The third determination module is used to determine the aggregated probability label as the soft label of multiple annotators on the sample.

[0136] It should be noted here that the examples and scenarios implemented by the above-mentioned modules and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned embodiments. It should be noted that the above-mentioned modules as part of the device can run in a hardware environment and can be implemented by software or hardware, wherein the hardware environment includes a network environment.

[0137] According to another aspect of the embodiments of the present application, a storage medium is further provided. Optionally, in this embodiment, the storage medium can be used to execute the program code of any of the above methods for converting the annotation results in the classification task into soft labels in the embodiments of the present application.

[0138] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps:

[0139] S1, expand the annotator's annotation results of the target category on the sample to all categories to obtain a conditional confidence set, where the conditional confidence set is a set of annotation results with discrete confidence in all categories.

[0140] S2, constructing a Gaussian distribution obeyed by the conditional confidence set, and determining the logical value of the confidence of each category corresponding to each sub-interval on the Gaussian distribution, wherein the logical value of the confidence of each category is obtained by sampling from each sub-interval.

[0141] S3, normalize the logical values ​​of the confidence of each category on the sample to obtain the probability label annotated by the annotator on the sample.

[0142] S4, obtains the soft label of the sample by aggregating the probabilistic labels of multiple annotators on the sample.

[0143] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, which will not be described in detail in this embodiment.

[0144] Among them, computer-readable storage media may include, but are not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0145] According to another aspect of an embodiment of the present application, an electronic device for implementing the above-mentioned method of converting the annotation results in the classification task into soft labels is also provided. The electronic device can be a server, a terminal, or a combination thereof.

[0146] Figure 6 is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application, such as Figure 6 As shown, it includes a processor 602, a communication interface 604, a memory 606 and a communication bus 608, wherein the processor 602, the communication interface 604, and the memory 606 communicate with each other through the communication bus 608, wherein,

[0147] Memory 606, used to store computer programs;

[0148] The processor 602 is used to implement the following steps when executing the computer program stored in the memory 606:

[0149] S1, expand the annotator's annotation results of the target category on the sample to all categories to obtain a conditional confidence set, where the conditional confidence set is a set of annotation results with discrete confidence in all categories.

[0150] S2, constructing a Gaussian distribution obeyed by the conditional confidence set, and determining the logical value of the confidence of each category corresponding to each sub-interval on the Gaussian distribution, wherein the logical value of the confidence of each category is obtained by sampling from each sub-interval.

[0151] S3, normalize the logical values ​​of the confidence of each category on the sample to obtain the probability label annotated by the annotator on the sample.

[0152] S4, obtains the soft label of the sample by aggregating the probabilistic labels of multiple annotators on the sample.

[0153] Optionally, the communication bus may be a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The communication interface is used for communication between the electronic device and other devices.

[0154] The memory may include RAM, or may include non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0155] As an example, the memory 606 may include, but is not limited to, the expansion unit 502, the construction unit 504, the normalization unit 506, and the aggregation unit 508 in the device for converting the labeling result in the classification task into a soft label. In addition, other module units in the device for converting the labeling result in the classification task into a soft label may also be included but are not limited to, which will not be repeated in this example.

[0156] The above-mentioned processor can be a general-purpose processor, which can include but not be limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processing), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0157] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0158] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0159] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0160] In the several embodiments provided in the present application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0161] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0163] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program code.

[0164] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, which can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0165] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure here, those skilled in the art will easily think of the implementation scheme of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not recorded in the present disclosure. The description and examples are regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

[0166] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0167] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for converting the annotation results in the classification task into soft labels, It is characterized in that include: Expand the annotator's annotation results of the target category on the sample to all categories to obtain a conditional confidence set, where the conditional confidence set is a set of annotation results with discrete confidence in all categories; Constructing a Gaussian distribution obeyed by the conditional confidence set, and determining the logical values ​​of the confidences of each category corresponding to each subinterval on the Gaussian distribution, wherein the logical values ​​of the confidences of each category are obtained by sampling from each subinterval; Normalizing the obtained logical values ​​of the confidence of each category on the sample to obtain the probability label annotated by the annotator on the sample; A soft label of the sample is obtained by aggregating the probability labels of multiple annotators on the sample.

2. The method for converting the labeling result in the classification task into a soft label according to claim 1, It is characterized in that The labeling result of a single category on the sample is extended to all categories, and the conditional confidence set obtained includes: Determine the categories other than the target category in the sample as categories to be expanded, wherein all categories include the categories to be expanded and the target category; Defining the annotation result of the to-be-expanded category as an expanded annotation result based on the annotation result of the target category; The union of the annotation result of the target category and the expanded annotation result is determined as the conditional confidence set.

3. The method for converting the labeling result in the classification task into a soft label according to claim 1, It is characterized in that The constructing of the Gaussian distribution obeyed by the conditional confidence set includes: Obtaining the confidence of each category in the conditional confidence set; Based on the probability distribution corresponding to the confidence of each category in the conditional confidence set, the Gaussian distribution obeyed by the conditional confidence set is fitted.

4. The method for converting the labeling result in the classification task into a soft label according to claim 1, It is characterized in that The step of constructing the Gaussian distribution to which the conditional confidence set obeys and determining the logical values ​​of the confidences of various categories corresponding to each sub-interval on the Gaussian distribution comprises: Dividing the Gaussian distribution into a plurality of sub-intervals, wherein the plurality of sub-intervals are divided according to the confidence of each category in the conditional confidence set; Sampling multiple subintervals of the Gaussian distribution to obtain the logical values ​​of the confidences corresponding to the subintervals.

5. The method for converting the labeling result in the classification task into a soft label as claimed in claim 4, It is characterized in that After sampling the multiple subintervals of the Gaussian distribution and obtaining the logical values ​​of the confidences corresponding to the subintervals, the method further includes: Under the given data set parameters, obtaining the logical values ​​of all categories of the sample by the annotator; Determine the optimal data set parameters by calculating the consistency error; The optimal logical value is determined based on the optimal data set parameters.

6. The method for converting the labeling result in the classification task into a soft label according to claim 1, It is characterized in that The step of normalizing the obtained logical values ​​of the confidences of the categories on the sample to obtain the probability labels annotated by the annotator on the sample includes: Normalizing the logical values ​​of the confidence of the annotator in each category of the sample by an activation function, and converting the logical values ​​into probability distribution; A probability label annotated by the annotator on the sample is determined based on the probability distribution.

7. The method for converting the labeling result in the classification task into a soft label according to claim 1, It is characterized in that The step of obtaining a soft label of the sample by aggregating the probability labels of the multiple annotators on the sample comprises: Aggregating the probability labels of the multiple annotators on the sample using an average weighted method to obtain an aggregated probability label; The aggregated probability label is determined as the soft label of the multiple annotators on the sample.

8. A device for converting the labeling results in the classification task into soft labels, It is characterized in that include: An expansion unit, used to expand the annotation results of the target category on the sample by the annotator to all categories to obtain a conditional confidence set, wherein the conditional confidence set is a set of annotation results with discrete confidence in all categories; A construction unit, used to construct a Gaussian distribution obeyed by the conditional confidence set, and determine the logical values ​​of the confidences of each category corresponding to each sub-interval on the Gaussian distribution, wherein the logical values ​​of the confidences of each category are obtained by sampling from each sub-interval; A normalization unit, used to normalize the logical values ​​of the confidences of the categories obtained on the sample to obtain the probability labels annotated on the sample by the annotator; The aggregation unit is used to obtain a soft label of the sample by aggregating the probability labels of multiple annotators on the sample.

9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 7 when executed.

10. An electronic device comprising a memory and a processor, It is characterized in that A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 through the computer program.

Images