A method and device for converting a labeling result in a classification task into a soft label

Abstract

The application discloses a method and device for converting annotation results in a classification task into soft labels, the method comprising: expanding annotation results of a target class of a sample by an annotator to all classes to obtain a conditional confidence set, wherein the conditional confidence set is a set of annotation results with discrete confidence in all classes; constructing a Gaussian distribution to which the conditional confidence set is subjected, and determining logical values of confidence of each class corresponding to each sub-interval on the Gaussian distribution, wherein the logical values of confidence of each class are obtained by sampling in each sub-interval; normalizing the logical values of confidence of each class obtained on the sample to obtain a probability label annotated by the annotator on the sample; and obtaining a soft label of the sample by aggregating probability labels of the sample annotated by multiple annotators.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of machine learning, and more particularly, to a method and device for converting annotation results in a classification task into soft labels. BACKGROUND

[0002] For data mining and machine learning tasks, especially for classification tasks, label data used for training models is crucial. It is a common practice to require human annotators to label samples. Humans express their uncertainty in various ways. Taking classification tasks as an example, annotators are often required to give the most likely class and to give a confidence level. When humans express their confidence in the form of discrete ratings (e.g., low, medium, high) rather than in the form of a probability confidence score (e.g., 85%), it will be difficult to convert the annotation information of the annotators into a probability distribution of the annotators over all classes, and then form a soft label on the sample to provide the model for training. If the human annotation results with discrete confidence can be converted into soft labels, it will be helpful to handle the uncertainty and ambiguity of the labels, and it will be beneficial for the model to learn more information.

[0003] When the annotation task involves subjective judgment, discrete confidence levels can better capture the subjective feelings of the annotators about uncertainty. Compared with probability confidence scores, discrete confidence levels are easy to interpret and use, although they are more difficult to calculate than probability confidence scores. However, it is difficult to convert the annotation information of the annotators into a probability distribution of the annotators over all classes to form a soft label, and then provide the model for training.

[0004] Therefore, it becomes an urgent problem to provide a method for converting annotation results with discrete form confidence of annotators in a classification task into soft labels. SUMMARY

[0005] In view of at least one defect or improvement demand of the prior art, the present application provides a method and device for converting annotation results in a classification task into soft labels, which realizes the conversion of annotation results in a classification task into soft labels, and the constructed soft labels have more abundant information and can more accurately reflect the inter-class similarity of samples.

[0006] To achieve the above object, according to a first aspect of the present application, a method for converting annotation results in a classification task into soft labels is provided, the method comprising: expanding annotation results of a target class of a sample by an annotator to all classes to obtain a conditional confidence set, wherein the conditional confidence set is a set of annotation results of all classes having discrete confidences; constructing a Gaussian distribution to which the conditional confidence set is subjected, and determining logical values of confidences of the classes corresponding to each sub-interval on the Gaussian distribution, wherein the logical values of the confidences of the classes are obtained by sampling from each sub-interval; normalizing the logical values of the confidences of the classes obtained from the sample to obtain a probability label annotated by the annotator on the sample; and obtaining a soft label of the sample by aggregating the probability labels of the sample annotated by multiple annotators.

[0007] In an example embodiment, expanding the annotation results of a single class of the sample by the annotator to all classes to obtain the conditional confidence set comprises: determining classes of the sample other than the target class as to-be-expanded classes, wherein all classes include the to-be-expanded classes and the target class; defining annotation results of the to-be-expanded classes as expanded annotation results based on the annotation results of the target class; and determining a union of the annotation results of the target class and the expanded annotation results as the conditional confidence set.

[0008] In an example embodiment, constructing the Gaussian distribution to which the conditional confidence set is subjected comprises: obtaining confidences of the classes in the conditional confidence set; and fitting the Gaussian distribution to which the conditional confidence set is subjected based on probability distributions corresponding to the confidences of the classes in the conditional confidence set.

[0009] In an example embodiment, constructing the Gaussian distribution to which the conditional confidence set is subjected and determining the logical values of the confidences of the classes corresponding to each sub-interval on the Gaussian distribution comprises: dividing the Gaussian distribution into a plurality of sub-intervals, wherein the plurality of sub-intervals are divided according to the confidences of the classes in the conditional confidence set; and sampling the plurality of sub-intervals of the Gaussian distribution to obtain the logical values of the confidences corresponding to each sub-interval.

[0010] In an example embodiment, after sampling the plurality of sub-intervals of the Gaussian distribution to obtain the logical values of the confidences corresponding to each sub-interval, the method further comprises: obtaining the logical values of the classes of the sample by the annotator given a dataset parameter; determining an optimal dataset parameter by calculating a consistency error; and determining an optimal logical value based on the optimal dataset parameter.

[0011] In an example embodiment, the method for converting the labeling result in the classification task into the soft label comprises: extending the labeling result of the target class of the sample by the labeler to all classes to obtain a conditional confidence set, wherein the conditional confidence set is a set of labeling results of all classes having discrete confidence; constructing a Gaussian distribution to which the conditional confidence set is subjected, and determining a logical value of the confidence of each class corresponding to each sub-interval on the Gaussian distribution, wherein the logical value of the confidence of each class is obtained by sampling from each sub-interval; normalizing the logical value of the confidence of each class obtained on the sample to obtain a probability label labeled by the labeler on the sample; and aggregating the probability labels of the sample by multiple labelers to obtain a soft label of the sample.

[0012] In an example embodiment, the method for converting the labeling result in the classification task into the soft label comprises: extending the labeling result of the target class of the sample by the labeler to all classes to obtain a conditional confidence set, wherein the conditional confidence set is a set of labeling results of all classes having discrete confidence; constructing a Gaussian distribution to which the conditional confidence set is subjected, and determining a logical value of the confidence of each class corresponding to each sub-interval on the Gaussian distribution, wherein the logical value of the confidence of each class is obtained by sampling from each sub-interval; normalizing the logical value of the confidence of each class obtained on the sample to obtain a probability label labeled by the labeler on the sample; and aggregating the probability labels of the sample by multiple labelers to obtain a soft label of the sample.

[0013] According to a second aspect of the present application, there is also provided an apparatus for converting a labeling result in a classification task into a soft label, comprising: an extension unit configured to extend the labeling result of a target class of a sample by a labeler to all classes to obtain a conditional confidence set, wherein the conditional confidence set is a set of labeling results of all classes having discrete confidence; a construction unit configured to construct a Gaussian distribution to which the conditional confidence set is subjected, and determine a logical value of the confidence of each class corresponding to each sub-interval on the Gaussian distribution, wherein the logical value of the confidence of each class is obtained by sampling from each sub-interval; a normalization unit configured to normalize the logical value of the confidence of each class obtained on the sample to obtain a probability label labeled by the labeler on the sample; and an aggregation unit configured to aggregate the probability labels of the sample by multiple labelers to obtain a soft label of the sample.

[0014] According to a third aspect of the present application, there is also provided a computer readable storage medium having a computer program stored therein, wherein the computer program is configured to execute the method for converting a labeling result in a classification task into a soft label when running.

[0015] According to a fourth aspect of the present application, there is also provided an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the method for converting a labeling result in a classification task into a soft label through the computer program.

[0016] Overall, the above technical solutions conceived by the present application can achieve the following beneficial effects compared with the prior art:

[0017] (1) The application provides a method for converting the labeling results in a classification task into soft labels, the soft labels obtained by the method have more abundant information, and the inter-class similarity of samples can be more accurately reflected. The concept of supplementing confidence is proposed, which solves the lack of confidence labeling results of the labeler on all categories except the labeled category, and thus can realize analysis on more abundant data. In addition, the Gaussian distribution is constructed category by category, which can accurately model the real situation of the labeler group, and through sampling from the restored Gaussian distribution, more accurate estimated values corresponding to the discrete confidence can be obtained, thereby improving the accuracy of the formed soft labels.

[0018] (2) The application provides a method for converting the labeling results in a classification task into soft labels, which converts the human labeling results with discrete confidence into soft labels, which is helpful for processing the uncertainty and ambiguity of labels and is beneficial to the model learning more information. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0020] Figure 1 An optional flowchart of a method for converting the labeling results in a classification task into soft labels provided by the embodiments of the present application is shown in the figure.

[0021] Figure 2 An optional schematic diagram of confidence expansion provided by the embodiments of the present application is shown in the figure.

[0022] Figure 3 An optional distribution diagram of logical values provided by the embodiments of the present application is shown in the figure.

[0023] Fig. 4(a) is an optional schematic diagram of an image sample provided by the embodiments of the present application.

[0024] Fig. 4(b) is an optional schematic diagram of a method for converting the labeling results in a classification task into soft labels provided by the embodiments of the present application.

[0025] Fig. 4(c) is a schematic diagram of soft labels obtained by using a comparative method.

[0026] Figure 5 An optional structural schematic diagram of a device for converting the labeling results in a classification task into soft labels provided by the embodiments of the present application is shown in the figure.

[0027] Figure 6An optional structural schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0029] The terms "first", "second", "third", etc. in the specification and claims of the present application and the above-described drawings are used to distinguish different objects, and are not used to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed or can optionally include other steps or units inherent to the process, method, product or device.

[0030] According to an aspect of the embodiments of the present application, a method for converting annotation results in a classification task into soft labels is provided. The following will be described in combination with Figure 1 The method for converting annotation results in a classification task into soft labels provided by the embodiments of the present application is described.

[0031] Figure 1 An optional flowchart of the method for converting annotation results in a classification task into soft labels provided by the embodiments of the present application is shown in FIG. 1, which can include the following steps: Figure 1

[0032] S102, the annotation results of the target class of the annotator on the sample are expanded to all classes to obtain a conditional confidence set, wherein the conditional confidence set is a set of annotation results with discrete confidence in all classes;

[0033] S104, a Gaussian distribution to which the conditional confidence set is subjected is constructed, and logical values of confidences of the classes corresponding to each sub-interval on the Gaussian distribution are determined, wherein the logical values of the confidences of the classes are obtained by sampling in the sub-intervals;

[0034] S106, the logical values of the confidences of the classes on the sample obtained are normalized to obtain a probability label annotated by the annotator on the sample;

[0035] ​S108, obtaining a soft label of the sample by aggregating the probability labels of the sample provided by the multiple annotators.

[0036] The method for converting the annotation result in a classification task into a soft label provided in the application can be applied to a scenario in which an annotator expresses his / her confidence in the form of discrete confidence in a classification task. The annotation information of the annotator is converted into a probability distribution of the annotator on all categories to form a soft label, thereby providing the model for training.

[0037] In the embodiment, after the annotation result of the annotator on the target category of the sample is expanded, a Gaussian distribution of the expanded conditional confidence set of each category is constructed, then the Gaussian distribution is sub-interval segmented, and the optimal parameters and corresponding logical values are obtained. The soft label is finally obtained by normalizing the logical values and aggregating.

[0038] Optionally, when converting the annotation result with discrete confidence into a soft label, the following set can be defined in advance: defining a sample set The category set is where x i is a sample feature vector, y i is a true label belonging to one of the possible labels. In addition, the annotator set is defined as It should be noted that |·| represents the number of elements in a set, such as expressing the number of elements in the set .

[0039] The discrete confidence level set is defined as where the confidence reflects the confidence of the annotator on the annotation result. Optionally, the potential order relationship between two different confidence levels can be represented by "<", for example, confidence "low" < confidence "high". In , it is stipulated that

[0040] The category annotation result matrix and the discrete confidence level annotation result matrix are defined as and where z i,j is the category annotation result provided by the annotator u j for the i-th sample, and h i,j is the discrete form of confidence provided by the annotator u j for the i-th sample. Wherein, For example, for some i and j, (z i,j , h i,j ) = ("cat", "low") indicates that the annotator u jThe i-th sample is labeled as "cat" with a confidence level of "low".

[0041] In this embodiment of the application, in order to construct soft labels for samples, the annotator u can be used. j The discrete confidence level provided on the i-th sample will be used to label the results (z). i,j ,h i,j ) converted to in It is the annotator u j The i-th sample belongs to category Probability confidence score If it is the probability label we need, then the soft label p of the i-th sample... i for

[0042]

[0043] Through steps S102 to S108, by extending the annotation results of the annotator on the target category of the sample to all categories, a conditional confidence set is obtained. This conditional confidence set is the collection of annotation results with discrete confidence in all categories. A Gaussian distribution is constructed to follow the conditional confidence set, and the logical values ​​of the confidence scores for each category corresponding to each sub-interval on the Gaussian distribution are determined. The logical values ​​of the confidence scores for each category are sampled from each sub-interval. The logical values ​​of the confidence scores for each category on the sample are normalized to obtain the probability labels annotated by the annotator on the sample. By aggregating the probability labels of multiple annotators on the sample, soft labels for the sample are obtained. The soft labels constructed by this method have richer information and can more accurately reflect the inter-class similarity of the samples. The proposed concept of supplementary confidence (i.e., extension based on the annotator's annotation results on a single category) solves the problem of missing confidence annotation results for annotators in all categories other than the labeled category, thereby enabling analysis of richer data. Furthermore, by constructing a Gaussian distribution for each category based on the annotation results of all annotators on the samples, we can accurately model the true situation of the annotator group. By sampling from the reconstructed Gaussian distribution, we can obtain more accurate estimates of the discrete confidence scores. Utilizing the inherent correlations between the annotation results of multiple annotators and the true categories in the classification task samples, we transform the annotators' results into a probability distribution for each annotator across all categories, ultimately forming soft labels. These soft labels can achieve more precise classification tasks.

[0044] In one exemplary embodiment, the annotation results of the annotator on a single category on the sample are extended to all categories to obtain a conditional confidence set including:

[0045] S11, determining the class other than the target class in the sample as a to-be-expanded class by the annotator, wherein all the classes include the to-be-expanded class and the target class;

[0046] S12, defining the annotation result of the to-be-expanded class as an expanded annotation result based on the annotation result of the target class;

[0047] S13, determining the union of the annotation result of the target class and the expanded annotation result as a conditional confidence set.

[0048] In the embodiment, the confidence of other classes in the sample can be expanded based on the annotation result of a single class of the sample by the annotator. For a certain sample, it is assumed that the annotation result of a certain annotator is (c, l k ), wherein c and l k are the confidence of a certain class and a certain discrete form, respectively.

[0049] In the face of the lack of confidence annotation results of the annotator on all classes other than c. Based on the "complementary confidence", that is, the complementary confidence of the discrete confidence l k is l ′ k , which means that when a human annotator has a confidence l k on a certain class c, the human annotator has a discrete confidence l ′ on any class in the sample set other than c (i.e., to-be-expanded class), that is, , wherein \ represents set subtraction. k

[0050] According to the definition of the complementary confidence, the expanded discrete confidence level set (i.e., the conditional confidence set)

[0051] Figure 2 An optional confidence expansion diagram provided by the embodiment of the application is shown in FIG. 1. As shown in FIG. 1, an example is shown with a three-classification task to show the annotation of some users. Figure 2

[0052] In the given information, the class annotation result and the confidence annotation result of the i-th sample by the annotator u j are (z i,j , h i,j ). According to the definition of the complementary confidence, the expanded human confidence annotation result is represented as a vector h i,j , which is expanded from h i,j . The confidence of the annotator u j on the class c of the i-th sample can be represented as h , wherein​ and It should be noted that the bold letter represents a vector, such as h; the non-bold letter represents a real number, such as h.

[0053] Through this embodiment, the annotation results of the annotators in a single category are expanded to all categories, the lack of confidence annotation results of the annotators in all categories except the annotation category is solved, and more abundant data can be analyzed, and the comprehensiveness of data analysis is higher.

[0054] In an example embodiment, constructing the Gaussian distribution to which the conditional confidence set conforms includes:

[0055] S21, obtaining the confidence of each category in the conditional confidence set;

[0056] S22, fitting the Gaussian distribution to which the conditional confidence set conforms based on the probability distribution corresponding to the confidence of each category in the conditional confidence set.

[0057] In this embodiment, the Gaussian distribution to which the conditional confidence set conforms can be constructed category by category. In order to model the Gaussian distribution in different cases, statistical analysis can be performed category by category. Specifically, after the confidence expansion, all annotators will have a confidence level for each category of all samples, and then the set of potential confidence levels of the annotators in category c2 under the condition that the real category is c1 can be obtained. Here, c1 and c2 are both elements in the category set . The specific calculation method of is as follows:

[0058]

[0059] Specifically, ∪ represents the set operation of the union set, is a set containing both the confidence and the complementary confidence. In addition, The confidence levels in have the following size relationship: for all , there is ′ k . k Because l k measures the confidence level of the annotator on the category that he / she considers most likely, and l ′ k represents the potential confidence level of other categories. Therefore, there must be ′ k . k Based on the corresponding relationship between l ′ k and l k , that is, the sum of the confidence levels of all categories on the same sample is a fixed value, there is l′ k a-l k Therefore, when given conditions , the following can be obtained

[0060] Before the annotator makes the final prediction, the logical values of each category can be obtained, and the relative order of the logical values of each category ultimately determines the final annotation result and the confidence level. According to the central limit theorem, a large number of independent logical values will converge to a Gaussian distribution. Based on the probability distribution corresponding to the confidence of each category in the conditional confidence set, a Gaussian distribution to which the conditional confidence set conforms can be fitted.

[0061] Through this embodiment, the Gaussian distribution is constructed category by category, which can accurately model the real situation of the annotator group.

[0062] In an example embodiment, constructing a Gaussian distribution to which the conditional confidence set conforms, and determining the logical values of the confidence of each category corresponding to each sub-interval on the Gaussian distribution includes:

[0063] S31, dividing the Gaussian distribution into a plurality of sub-intervals, wherein the plurality of sub-intervals are divided according to the confidence of each category in the conditional confidence set;

[0064] S32, sampling the plurality of sub-intervals of the Gaussian distribution to obtain the logical values of the confidence corresponding to each sub-interval.

[0065] In this embodiment, the Gaussian distribution can be cut into a plurality of sub-intervals, corresponding to a set of logical values For example, it is assumed that four confidence levels, and low', low, high', high correspond to logical values 4, 6, 3, 7 respectively, then the set of logical values is represented as {4, 6, 3, 7}. Therefore, conforms to a Gaussian distribution, that is Here, and control the mean and standard deviation of .

[0066] Optionally, the following methods can be used to calculate and

[0067] (1) approximately use the frequency value (range [0, 1]) of the annotator to label the sample with the real category c1 as the category c2 to represent , that is where is the inverse function of sigmoid function, and Count is the counting function. Here, logit(p) converts the frequency value (range [0, 1]) to the logic value (range (-∞, ∞)), The confusion level of the human annotator group is simulated. The larger indicates that the annotator is more inclined to label c1 as c2, and can be expressed as:

[0068]

[0069] Here, α is an adjustable parameter related to the original distribution of the data set.

[0070] (2) The value of can be approximated using an index that can calculate the uncertainty or confusion degree of the data in the set, and the information entropy of can be used as the approximation value of, since the information entropy can simulate the confusion degree in the data to some extent, it can be reasonably used instead of the variance value.

[0071] Here, represents the number of occurrences of l in

[0072] According to the number of each confidence level in, the Gaussian distribution can be divided, which can ensure that each confidence level corresponds to an interval, and then the logic value corresponding to different discrete confidence levels is sampled from the obtained interval. For l in, suppose its corresponding logic value is X l The value interval in the Gaussian distribution is represented as

[0073] can be expressed as: Further, it can be simplified as:

[0074]

[0075] Here, Φ represents the cumulative distribution function (CDF) of the standard Gaussian distribution, and Φ -1 represents its inverse function.

[0076] The interval determines the logic value range of the confidence level l, and the estimated logic value corresponding to l can be obtained by randomly sampling in this interval. Specifically, the logic value matrix

[0077] can be defined as:

[0078] The interval determines the logic value range of the confidence level l, and the estimated logic value corresponding to l can be obtained by randomly sampling in this interval. Specifically, the logic value matrix ​​in Indicates human annotator u j For the latent logical value of the i-th sample, here Indicate u j For the logical value of a specific category c in the i-th sample. The detailed sampling process is as follows: Let From In Values ​​obtained by random sampling within the sampling range.

[0079] In an exemplary embodiment, after sampling multiple sub-intervals of the Gaussian distribution and obtaining the logical values ​​of the confidence levels corresponding to each sub-interval, the method further includes:

[0080] S41, Given the dataset parameters, obtain the logical values ​​of the annotators for all categories of the sample;

[0081] S42, determine the optimal dataset parameters by calculating the consistency error;

[0082] S43, determine the optimal logical value based on the optimal dataset parameters.

[0083] In this embodiment, a grid search strategy can be used to find the parameter α in the search space. The optimal value is obtained by performing a class-by-class construction of a Gaussian distribution and Gaussian distribution splitting, which yields all logical values ​​given a parameter α.

[0084] Figure 3 This is a schematic diagram illustrating an optional distribution of logical values ​​provided in an embodiment of this application, such as... Figure 3 As shown, the evaluation was performed on the ImageNet-16H image classification dataset, which contains 16 categories, such as chair, oven, knife, etc., with 75 images per category. To vary the annotation difficulty, four different phase noises (ω∈{80,95,110,125}) were applied to each image. Annotators were asked to provide the image category and their confidence level (low, medium, and high). In the figure, the true category refers to the correctly labeled category, and the non-true category refers to the incorrectly labeled category.

[0085] Furthermore, to obtain more accurate logistic values, the error between the recovered logistic values ​​and the original annotation information can be measured. Specifically, the consistency error can be calculated, which is the proportion of inconsistency between the class corresponding to the maximum logistic value predicted by the human annotator on the sample and the original predicted class (i.e., the consistency error). Here, the goal is to ensure that no additional information is introduced and that the results are as consistent as possible with the original human annotation results z. i,j (z i,j Indicates the commenter u jthe label class result in the i-th sample) is consistent. The optimal α value α * corresponding to the minimum consistency error, i.e. α * = argmin α E(α). E(α) is the consistency error described above, which is calculated as follows:

[0086]

[0087] where is an indicator function that outputs 1 if the condition is met, and 0 otherwise.

[0088] In an example embodiment, the logical values of the confidence of each class obtained on the sample are normalized to obtain the probability label annotated by the annotator on the sample, including:

[0089] S51, the logical values of the confidence of each class of the sample annotated by the annotator are normalized by an activation function to convert the logical values into a probability distribution;

[0090] S52, determining the probability label annotated by the annotator on the sample based on the probability distribution.

[0091] In this embodiment, in order to aggregate the soft label of the sample, the logical values of the confidence of each class can be normalized.

[0092] For example, all logical values (range (-∞, ∞)) can be converted into probability confidence scores (range [0, 1]) using an activation function such as a sigmoid function:

[0093]

[0094] At this time, the logical values are all converted to the range [0, 1], i.e. the normalization is completed.

[0095] In order to obtain the soft label of the sample, since it is a probability distribution result, the probability confidence scores can be converted into human probability labels using the softmax rule:

[0096]

[0097] It should be noted that the temperature parameter τ controls the degree of influence of the highest probability score on classification, and simulates the noise in the decision environment of human annotators. The value can be selected according to the actual situation, and common values include {0.01, 0.05, 0.1, 0.2, 0.3}. In the embodiment of the present application, the value of τ can be 0.05, and can also be other optional values, which are not limited herein.

[0098] In an example embodiment, obtaining the soft label of the sample by aggregating the probability labels of the sample by multiple annotators comprises:

[0099] S61, obtaining the aggregated probability label by aggregating the probability labels of the sample by multiple annotators using the average weighting method;

[0100] S62, determining the aggregated probability label as the soft label of the sample by multiple annotators.

[0101] In the embodiment, after the confidence values of each category on the sample are normalized to obtain the probability label of the sample annotated by the annotator, the aggregated probability label is obtained by aggregating the probability labels of the sample by multiple annotators using the average weighting method, and the aggregated probability label is determined as the soft label of the sample by multiple annotators.

[0102] Optionally, the soft label of the sample is obtained by aggregating the human probability labels of all annotators on the same sample. i :

[0103]

[0104] The soft label constructed by the embodiment of the present application reflects the similarity between categories. FIG. 4(a) is a schematic diagram of an optional image sample provided by the embodiment of the present application. As shown in FIG. 4(a), for the sample, since a wheel is displayed, it is likely to be a truck or a car. FIG. 4(b) is a schematic diagram of an optional method for converting the annotation result in a classification task into a soft label provided by the embodiment of the present application. As shown in FIG. 4(b), the soft label obtained by using the method for converting the annotation result in a classification task into a soft label provided by the present application can be seen. The probabilities of each category are inconsistent. It is worth noting that the label exhibits relatively high probabilities on the truck and car categories. FIG. 4(c) is a schematic diagram of a soft label obtained by using a comparison method. As shown in FIG. 4(c), the comparison method treats each category equally except for human annotation, and cannot reflect the similarity between categories.

[0105] Table 1 is the cross-entropy loss of the model when using different soft labels. It shows the cross-entropy loss values of the VGG-19 model on the test set after training the model using different soft labels on the training set under five different random seeds. The results are reported in the form of μ±σ, where μ and σ represent the mean and standard deviation of the five experimental results, respectively. The lower the cross-entropy, the better the effect. It can be seen that the model trained using the method has lower cross-entropy loss in various situations.

[0106] Table 1

[0107]

[0108] By this embodiment, converting the human annotation result with discrete confidence into soft label will help to handle the uncertainty and ambiguity of the label and facilitate the model to learn more information.

[0109] According to another aspect of the embodiments of the present application, there is also provided a conversion device for implementing the above method of converting the annotation result in a classification task into a soft label. Figure 5 is a structural schematic diagram of an optional device for converting the annotation result in a classification task into a soft label according to an embodiment of the present application, as shown in the figure, the device can include: Figure 5

[0110] an expanding unit 502 configured to expand the annotation result of the target class of the sample by the annotator to all classes to obtain a conditional confidence set, wherein the conditional confidence set is a set of annotation results with discrete confidence in all classes;

[0111] a constructing unit 504 configured to construct a Gaussian distribution to which the conditional confidence set is subjected, and determine a logical value of the confidence of each class corresponding to each sub-interval on the Gaussian distribution, wherein the logical value of the confidence of each class is obtained by sampling in each sub-interval;

[0112] a normalizing unit 506 configured to normalize the logical value of the confidence of each class obtained on the sample to obtain a probability label annotated by the annotator on the sample;

[0113] an aggregating unit 508 configured to obtain a soft label of the sample by aggregating the probability labels of the sample by multiple annotators.

[0114] It should be noted that the expanding unit 502 in this embodiment can be configured to perform the above step S102, the constructing unit 504 in this embodiment can be configured to perform the above step S104, the normalizing unit 506 in this embodiment can be configured to perform the above step S106, and the aggregating unit 508 in this embodiment can be configured to perform the above step S108.

[0115] ​By the above module, the annotation result of the target class of the annotator on the sample is expanded to all classes to obtain a conditional confidence set, wherein the conditional confidence set is a set of annotation results with discrete confidence in all classes; a Gaussian distribution to which the conditional confidence set is subjected is constructed, and a logical value of the confidence of each class corresponding to each sub-interval on the Gaussian distribution is determined, wherein the logical value of the confidence of each class is obtained by sampling from each sub-interval; the logical value of the confidence of each class obtained on the sample is normalized to obtain a probability label annotated by the annotator on the sample; and a soft label of the sample is obtained by aggregating the probability labels of the sample annotated by multiple annotators. The soft label constructed by the method has more abundant information and can more accurately reflect the inter-class similarity of the sample. The concept of supplementing confidence solves the lack of annotation results of the annotator on all classes except the annotation class, and thus can realize analysis on more abundant data. In addition, the Gaussian distribution is constructed class by class, which can more accurately model the real situation of the annotator group. By sampling from the restored Gaussian distribution, more accurate estimated values of the discrete confidence can be obtained, and the accuracy of the formed soft label is improved.

[0116] In an example embodiment, the expanding unit comprises:

[0117] The first determining module is configured to determine the classes of the sample except the target class as the to-be-expanded classes, wherein all the classes include the to-be-expanded classes and the target class.

[0118] The defining module is configured to define the annotation result of the to-be-expanded class as an expanded annotation result based on the annotation result of the target class.

[0119] The union of the annotation result of the target class and the expanded annotation result is determined as the conditional confidence set.

[0120] In an example embodiment, the constructing unit comprises:

[0121] The obtaining module is configured to obtain the confidence of each class in the conditional confidence set.

[0122] The fitting module is configured to fit the Gaussian distribution to which the conditional confidence set is subjected based on the probability distribution corresponding to the confidence of each class in the conditional confidence set.

[0123] In an example embodiment, the constructing unit comprises:

[0124] The dividing module is configured to divide the Gaussian distribution into a plurality of sub-intervals, wherein the plurality of sub-intervals are divided according to the confidence of each class in the conditional confidence set.

[0125] The sampling module is configured to sample a plurality of sub-intervals of the Gaussian distribution to obtain logical values of the confidence corresponding to each sub-interval.

[0126] In an example embodiment, the apparatus further includes:

[0127] The obtaining unit is configured to obtain the logical values of the confidence of the annotators on all categories of the sample under the given dataset parameters.

[0128] The first determining unit is configured to determine the optimal dataset parameters by calculating the consistency error.

[0129] The second determining unit is configured to determine the optimal logical values based on the optimal dataset parameters.

[0130] In an example embodiment, the normalization unit includes:

[0131] The normalization module is configured to normalize the logical values of the confidence of the annotators on each category of the sample by using the activation function to convert the logical values into a probability distribution.

[0132] The second determining module is configured to determine the probability label of the sample annotated by the annotators based on the probability distribution.

[0133] In an example embodiment, the aggregation unit includes:

[0134] The aggregation module is configured to aggregate the probability labels of the sample annotated by the plurality of annotators by using the average weighting method to obtain an aggregated probability label.

[0135] The third determining module is configured to determine the aggregated probability label as the soft label of the sample annotated by the plurality of annotators.

[0136] It should be noted that the above modules and the examples and scenarios realized by the corresponding steps are the same as those disclosed in the above embodiments, but are not limited to the above disclosed content. It should be noted that the above modules as part of the apparatus can run in a hardware environment, can be implemented by software, or can be implemented by hardware, wherein the hardware environment includes a network environment.

[0137] According to another aspect of the embodiments of the present application, a storage medium is provided. Optionally, in the present embodiment, the storage medium can be used to execute the program code of any one of the above-mentioned methods for converting the annotation results in the classification task into soft labels in the embodiments of the present application.

[0138] Optionally, in the present embodiment, the storage medium is configured to store program code for executing the following steps:

[0139] S1, expanding the labeling result of the target class of the annotator on the sample to all classes to obtain a conditional confidence set, wherein the conditional confidence set is a set of labeling results with discrete confidence in all classes.

[0140] S2, constructing a Gaussian distribution to which the conditional confidence set is subjected, and determining a logical value of the confidence of each class corresponding to each sub-interval on the Gaussian distribution, wherein the logical value of the confidence of each class is obtained by sampling in each sub-interval.

[0141] S3, normalizing the logical value of the confidence of each class obtained on the sample to obtain a probability label labeled by the annotator on the sample.

[0142] S4, obtaining a soft label of the sample by aggregating the probability labels of the sample labeled by multiple annotators.

[0143] Optionally, specific examples in the embodiment can refer to examples described in the above embodiments, and details are not described herein.

[0144] The computer readable storage medium can include, but is not limited to, any type of disk including floppy disks, optical disks, DVD, CD-ROM, micro drive, and magneto-optical disk, ROM, RAM, EPROM, EEPROM, DRAM, VRAM, flash memory device, magnetic card or optical card, nano system (including molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0145] According to another aspect of the embodiment of the present application, an electronic device for implementing the above method of converting the labeling result in the classification task into a soft label is also provided, which can be a server, a terminal, or a combination thereof.

[0146] Figure 6 is a structural schematic diagram of an optional electronic device according to the embodiment of the present application, as shown in Figure 6 including a processor 602, a communication interface 604, a memory 606 and a communication bus 608, wherein the processor 602, the communication interface 604 and the memory 606 complete mutual communication through the communication bus 608, wherein,

[0147] The memory 606 is configured to store a computer program.

[0148] The processor 602 is configured to execute the computer program stored in the memory 606 to implement the following steps:

[0149] S1, expanding the labeling result of the target class of the annotator on the sample to all classes to obtain a conditional confidence set, wherein the conditional confidence set is a set of labeling results with discrete confidence in all classes.

[0150] S2, the logic value of the confidence of each category corresponding to each sub-interval on the Gaussian distribution is determined, wherein the logic value of the confidence of each category is obtained by sampling in each sub-interval.

[0151] S3, the logic value of the confidence of each category on the sample is normalized to obtain the probability label of the sample labeled by the labeler.

[0152] S4, the soft label of the sample is obtained by aggregating the probability labels of the sample labeled by multiple labelers.

[0153] Optionally, the communication bus can be a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 6 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used for communication between the electronic device and other devices.

[0154] The memory can include a RAM and can also include a non-volatile memory, for example, at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0155] As an example, the memory 606 can include, but is not limited to, the extension unit 502, the construction unit 504, the normalization unit 506, and the aggregation unit 508 in the device for converting the labeling result in the classification task into a soft label. In addition, other module units in the device for converting the labeling result in the classification task into a soft label can also be included, but not limited to, which will not be described in detail in this example.

[0156] The processor can be a general processor, which can include but is not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; can also be DSP (Digital Signal Processing), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component.

[0157] Optionally, the specific examples in the embodiment can refer to the examples described in the above embodiments, and the embodiment will not be described here.

[0158] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action sequence described, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application.

[0159] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can refer to the related description of other embodiments.

[0160] In several embodiments provided by the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the units is only a logical function division. There can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different parts can be indirect couplings or communication connections through some services, devices or units, and can be electrical or other forms.

[0161] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0162] In addition, each of the function units in the embodiments of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0163] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable memory. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned memory includes: a U disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0164] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by a program instructing relevant hardware, and the program can be stored in a computer readable memory, which can include a flash disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, etc.

[0165] The above is only exemplary embodiments of the present disclosure, and cannot limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. Those skilled in the art will easily think of embodiments of the present disclosure after considering the specification and practicing the disclosure herein. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field not recorded in the present disclosure. The specification and examples are only considered as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

[0166] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope recorded in the present disclosure.

[0167] Those skilled in the art can easily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for converting label results in a classification task into soft labels, applied to the field of image classification tasks, the classification task being an image classification task, using image samples as samples, characterized in that, The method comprises the following steps: Extending the labeling result of a target class of a sample by an annotator to all classes to obtain a conditional confidence set, wherein the conditional confidence set is a set of labeling results of all classes with discrete confidence; Constructing a Gaussian distribution to which the conditional confidence set conforms, and determining a logical value of confidence of each class corresponding to each sub-interval on the Gaussian distribution, wherein the logical value of confidence of each class is obtained by sampling from the sub-interval; Normalizing the logical value of confidence of each class obtained on the sample to obtain a probability label labeled by the annotator on the sample; Obtaining a soft label of the sample by aggregating the probability labels of the sample labeled by multiple annotators; The method comprises the following steps: Obtaining the confidence of each class in the conditional confidence set; Based on the probability distribution corresponding to the confidence of each class in the conditional confidence set, fitting the Gaussian distribution to which the conditional confidence set conforms. 2.The method of claim 1, wherein, The method comprises the following steps: Determining the classes other than the target class in the sample as to-be-extended classes, wherein the all classes include the to-be-extended classes and the target class; Defining the labeling result of the to-be-extended classes as an extended labeling result based on the labeling result of the target class; Determining the union of the labeling result of the target class and the extended labeling result as the conditional confidence set. 3.The method of claim 1, wherein, The method comprises the following steps: Dividing the Gaussian distribution into multiple sub-intervals, wherein the multiple sub-intervals are divided according to the confidence of each class in the conditional confidence set; Sampling the multiple sub-intervals of the Gaussian distribution to obtain the logical value of confidence corresponding to each sub-interval. 4.The method of claim 3, wherein, After the sampling of the multiple sub-intervals of the Gaussian distribution to obtain the logical value of confidence corresponding to each sub-interval, the method further comprises the following steps: Given the dataset parameters, obtaining the logical value of each class of the sample by the annotator; Determining the optimal dataset parameters by calculating the consistency error; Determining the optimal logical value based on the optimal dataset parameters. 5.The method of claim 1, wherein, The method comprises the following steps: Normalizing the logical value of confidence of each class of the sample by the annotator through an activation function to convert the logical value into a probability distribution; Determining the probability label labeled by the annotator on the sample based on the probability distribution. 6.The method of claim 1, wherein, The method comprises the following steps: Aggregating the probability labels of the sample by multiple annotators to obtain an aggregated probability label by using the average weighting method. The probability label is determined as the soft label of the sample by aggregating the probability labels of the multiple annotators on the sample.

7. An apparatus for converting annotation results in a classification task into soft labels, comprising performing the method for converting annotation results in a classification task into soft labels as described in any one of claims 1-6, applied in the field of image classification tasks, wherein the classification task is an image classification task, and the samples used are image samples, characterized in that, The method comprises the following steps: The expanding unit is configured to expand the annotation result of the target class of the annotator on the sample to all classes to obtain a conditional confidence set, wherein the conditional confidence set is a set of annotation results of all classes with discrete confidence. The constructing unit is configured to construct a Gaussian distribution to which the conditional confidence set is subjected, and determine a logical value of the confidence of each class corresponding to each sub-interval on the Gaussian distribution, wherein the logical value of the confidence of each class is obtained by sampling in the sub-interval. The normalizing unit is configured to normalize the logical value of the confidence of each class obtained on the sample to obtain a probability label of the annotation of the sample by the annotator. The aggregating unit is configured to obtain a soft label of the sample by aggregating the probability labels of multiple annotators on the sample.

8. A computer readable storage medium, characterized in that, The computer readable storage medium comprises a stored program, wherein the program executes the method of any one of claims 1 to 6 when running. 9.An electronic device comprising a memory and a processor, the electronic device characterized by, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 6 by using the computer program.

Images