A crowdsourcing labeling truth value reasoning method, device and storage device

Abstract

The present invention discloses a crowdsourcing labeling truth value reasoning method, device and storage device. The method is specifically as follows: construct a data set corresponding to each crowdsourcing worker; for a worker, if the worker has annotated a target instance, calculate the distance between each instance in the data set corresponding to the worker and the target instance, otherwise directly set the worker's labeling quality on the target instance to 0; sort the distance between each instance and the target instance, and find the K neighbor instances with the closest distance to the instance in the data set; use the K neighbor instances to measure the deviation and variance of the worker on the target instance; linearly combine the deviation and variance as the worker's labeling quality w on the target instance ir ; Repeat the above steps until the marking quality of all crowdsourcing workers on the target instance is estimated; select the marking of the worker with the highest marking quality from all workers as the inferred marking of the target instance. The beneficial effects of the present invention are: simple and easy to implement, high marking quality, and solve the problem of low performance of the truth value reasoning method.

Description

Technical Field

[0001] The present invention relates to the field of big data processing, and in particular to a crowdsourcing labeling truth value reasoning method, device and storage device. Background Art

[0002] Due to the rapid development of artificial intelligence, the demand for large amounts of labeled data has also increased accordingly.

[0003] The emergence of crowdsourcing platforms provides an effective and low-cost way for instances to collect multiple labels from crowdsourcing workers. Therefore, on crowdsourcing platforms, each instance can obtain a set of multiple noisy labels. Truth-value reasoning methods are used to infer the true label of an instance from its multiple noisy label set.

[0004] Given a crowdsourcing dataset N is the number of instances contained in D, x i is the i-th instance, L i is x i Multiple noise marker sets. Contains M attributes, x i The mth attribute of is denoted as x im . represents the set of all crowdsourcing workers, and R represents the number of all crowdsourcing workers. Represents R workers for instance x i The rth marker in the provided marker set By the rth worker u r supply, From the class label set {0,c1,c2,L,c Q}, where 0 means that the rth worker has no mark x i The truth value reasoning method is x i The inferred inference markers are represented as

[0005] The simplest truth-value inference method is MV, which simply assumes that the quality of the markings of each crowdworker is the same, and infers the markings with the most occurrences in the multi-noise marking set as the true markings. However, in real scenarios, the marking quality of each crowdworker is different, which leads to the low quality of the inferred markings in MV inference.

[0006] In order to improve the performance of truth-valued reasoning methods, a large number of truth-valued reasoning methods have been proposed, which use various machine learning techniques to optimize the labeling inference process. However, these methods still have the following two defects:

[0007] (1) Most of them rely on the estimation of worker quality, but they only consider the deviation between the worker’s labeling and the true labeling, while ignoring the variance of the worker’s own labeling, that is, the volatility of the worker’s labeling.

[0008] (2) Most of them attempt to aggregate the marks of all workers in the multi-noise mark set through some strategy (such as weighted majority voting). However, mark aggregation is not always effective. In a few cases, the inferred marks obtained by mark aggregation are consistent with the marks of low-quality workers, but inconsistent with the marks of high-quality workers. In this case, the inferred marks are not credible. Summary of the invention

[0009] In order to further improve the performance of the truth value reasoning method, the present application provides a crowdsourcing labeled truth value reasoning method, device and storage device, and the method specifically includes the following steps:

[0010] S1. Build each crowdsourcing worker The corresponding data set Where D r Indicates that only crowdsourcing worker u is included r A dataset of all labeled instances;

[0011] S2. For worker u r , if u r For the target instance x i It is marked, that is Then calculate the data set D r Each instance in is associated with instance x i Otherwise, the worker u r On the target instance x i The mark quality on is set to 0;

[0012] S3. Dataset D r Each instance in is associated with instance x i Sort the distances and find D r In and Example x i The K nearest neighbor instances If K is less than D r The number of instances contained in, set the K value to D r The number of instances contained in x ikr In the dataset D r The kth instance found in i Recent instance; x ikr The multi-noise label set is in Denotes crowdsourcing worker u r' In x ikr Marks on

[0013] S4, from worker u r with other workers on target instance x i The label consistency on the target instance and the set of neighbor instances The labeling consistency on worker u is calculated in two aspects r In instance x i Deviation b ir ;

[0014] S5. Using Worker u r On the target instance x i The labels on the target instance and the set of neighbor instances The consistency of the markings on worker u is measured r In instance x i The variance of the markers on ir ;

[0015] S6, the deviation b ir and variance v ir Linear combination, as worker u r On the target instance x i Marking quality on ir ;

[0016] S7, repeat steps S2 to S6 until all crowdsourcing workers On the target instance x i Marking quality on All have been estimated;

[0017] S8. Select the label of the worker with the highest label quality from all workers as the target instance x. i Inferred tag.

[0018] A storage device stores instructions and data for implementing a crowdsourcing labeling truth value reasoning method.

[0019] A crowd-sourced labeling truth-value reasoning device comprises: a processor and a storage device; the processor loads and executes instructions and data in the storage device to implement a crowd-sourced labeling truth-value reasoning method.

[0020] The beneficial effects provided by the present invention are: not only can a comprehensive evaluation be made on the marking quality of crowdsourcing workers, but also a new truth-value reasoning strategy that is different from mark aggregation is explored, which can ensure that the inferred inferred marks are always consistent with the marks of high-quality workers, thereby improving the marking quality of the truth-value reasoning method, being simple and easy to implement, and having high marking quality, thereby solving the problem of poor performance of the truth-value reasoning method. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of the method of the present invention;

[0022] Figure 2 It is a working diagram of the hardware device of the present invention. DETAILED DESCRIPTION

[0023] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0024] Please refer to Figure 1 , Figure 1 is a flow chart of the method of the present invention.

[0025] The present invention provides a crowdsourcing labeling truth value reasoning method, device and storage device, wherein the method specifically comprises the following steps:

[0026] S1. Build each crowdsourcing worker The corresponding data set Where D r Indicates that only crowdsourcing worker u is included r A dataset of all labeled instances;

[0027] S2. For worker u r , if u r For the target instance x i It is marked, that is Then calculate the data set D r Each instance in is associated with instance x i Otherwise, the worker u r On the target instance x i The mark quality on is set to 0;

[0028] In step S2, data set D r Each instance in is associated with instance x i The calculation formula of the distance is shown in formula (1):

[0029]

[0030] In formula (1), x j Indicates D r The jth instance in d m (x i ,x j ) represents x i With x j The distance on the mth attribute is calculated as follows (2):

[0031]

[0032] In formula (2), overlap(x im ,x jm ) represents xi With x j The calculation method for noun attributes is as follows:

[0033]

[0034] In formula (3), rn_diff(x im ,x jm ) represents x i With x j The calculation method for numerical attributes is as follows:

[0035]

[0036] In formula (4), max m and min m Indicates D r The maximum and minimum values ​​of the mth feature in .

[0037] S3. Dataset D r Each instance in is associated with instance x i Sort the distances and find D r In and Example x i The K nearest neighbor instances If K is less than D r The number of instances contained in, set the K value to D r The number of instances contained in x ikr In the dataset D r The kth instance found in i Recent instance; x ikr The multi-noise label set is in Denotes crowdsourcing worker u r' In x ikr Marks on

[0038] S4, from worker u r with other workers on target instance x i The label consistency on the target instance and the set of neighbor instances The labeling consistency on worker u is calculated in two aspects r In instance x i Deviation b ir ;

[0039] It should be noted that b ir The larger the value, the greater the worker u r The higher the quality of the marking, the better the deviation b ir The calculation formula is shown in (5):

[0040]

[0041] In formula (5), Represents worker u r with other workers on target instance x i The label consistency on is calculated as follows (6):

[0042]

[0043] In formula (6), Represents a binary function whose value is 1 when its two parameters are the same, otherwise it is 0;

[0044] In formula (5), Represents worker u r The set of neighbor instances of the target instance with other workers The label consistency on is calculated as follows (7):

[0045]

[0046] S5. Using Worker u r On the target instance x i The labels on the target instance and the set of neighbor instances The consistency of the markings on worker u is measured r In instance x i The variance of the markers on ir ;

[0047] It should be noted that v ir The larger the value, the greater the worker u r The higher the quality of the marker, the higher the variance v ir The calculation formula is as follows (8):

[0048]

[0049] S6, the deviation b ir and variance v ir Linear combination, as worker u r On the target instance x i Marking quality on ir ;

[0050] Marking quality ir The calculation formula is as follows (9):

[0051]

[0052] S7, repeat steps S2 to S6 until all crowdsourcing workers On the target instance x i Marking quality on All have been estimated;

[0053] S8. Select the label of the worker with the highest label quality from all workers as the target instance x. i The inferred tag is expressed as (10):

[0054]

[0055] See also Figure 2 , Figure 2 4 is a schematic diagram of the working of the hardware device of an embodiment of the present invention, wherein the hardware device specifically comprises: a crowdsourcing labeling truth value reasoning device 401, a processor 402 and a storage device 403.

[0056] A crowd-sourced labeling truth-value reasoning device 401: The crowd-sourced labeling truth-value reasoning device 401 implements the crowd-sourced labeling truth-value reasoning method.

[0057] Processor 402: The processor 402 loads and executes instructions and data in the storage device 403 to implement the crowdsourcing labeling truth value reasoning method.

[0058] Storage device 403: The storage device 403 stores instructions and data; the storage device 403 is used to implement the crowdsourcing labeling truth value reasoning method.

[0059] As an embodiment, the present invention proposes a crowd-sourced labeling truth reasoning method based on labeling consistency, and the generated model is called crowd-sourced labeling truth reasoning based on labeling consistency (abbreviated as LCGTI).

[0060] In the experimental part, the crowdsourced label truth inference (abbreviated as LCGTI) based on label consistency proposed in the present invention is compared with majority voting (abbreviated as MV), DS, KOS, GTIC, multi-noise label distribution propagation (abbreviated as MNLDP), crowdsourced label amplification and weighted majority voting (abbreviated as LAWMV).

[0061] MV selects the label with the highest frequency in the multi-noise label set as the inferred label of the instance.

[0062] DS uses the EM algorithm to jointly estimate the confusion matrix of each crowdworker and the true label of the instance.

[0063] KOS uses the singular value decomposition technique of low-rank matrices and a belief propagation-like technique to complete truth-value reasoning by considering the reliability of workers.

[0064] GTIC clusters all instances into different clusters according to the multiple noisy label sets of each instance, and then maps each cluster to a specific class respectively.

[0065] MNLDP estimates a multi-noise label distribution from the multi-noise label set of each instance and propagates it to its nearest neighbors. Finally, each instance absorbs a part of the multi-noise label distribution from its nearest neighbors while keeping a part of the original multi-noise label distribution.

[0066] LAWMV uses the K nearest neighbor algorithm to find the K neighbors of an instance, and merges the multi-noise tag set of the K neighbors with the original multi-noise tag set to obtain an expanded multi-noise tag set. Finally, the tags in the expanded multi-noise tag set are weighted majority voted by tag similarity and instance distance to obtain the inferred tags.

[0067] In order to verify the effectiveness of the crowdsourced labeling truth reasoning method based on labeling consistency proposed in this invention, the classification performance of LCGTI, MV, DS, KOS, GTIC, MNLDP and LAWMV was experimentally compared.

[0068] In the experiment, the MNLDP and LAWMV methods were reproduced, and the existing implementations of the MV, DS, KOS and GTIC methods on the CEKA platform were used. The hyperparameter K of LCGTI was set to 35.

[0069] This application tests these methods on two widely used crowdsourced real-world datasets. Table 1 describes the main features of these two datasets in detail. The specific data can be downloaded from the CEKA platform website.

[0070] Table 1 Datasets used in the experiments

[0071] Dataset Number of instances Number of features Total number of crowdworkers Number of categories Total number of markers Income 600 10 67 2 6000 Music_genre 700 31 44 10 2946

[0072] Table 2 shows the labeling quality obtained by each method on each dataset. Each row is marked in bold to indicate the truth-value reasoning method with the best labeling quality on the dataset corresponding to that row.

[0073] Table 2 Comparison of labeling quality of MV, DS, KOS, GTIC, MNLDP, LAWMV and LCGTI

[0074] Dataset MV DS KOS GTIC MNLDP LAWMV LCGTI Income 71.76 72.67 71.33 71.83 71.17 74 74.33 Music_genre 71.00 52.57 50.56 71.43 78 76.71 78.57

[0075] From the experimental results in Table 2, we can see that:

[0076] On the real dataset Income, the labeling quality (74.33%) of the LCGTI method proposed in the present invention is significantly better than that of all the comparison objects, among which the labeling quality of MV is 71.76%, the labeling quality of DS is 72.67%, the labeling quality of KOS is 71.33%, the labeling quality of GTIC is 71.83%, the labeling quality of MNLDP is 71.17%, and the labeling quality of LAWMV is 74.00%.

[0077] On the real dataset Music_genre, the labeling quality (78.57%) of the LCGTI method proposed in the present invention is significantly better than all the comparison objects, among which the labeling quality of MV is 71.00%, the labeling quality of DS is 52.57%, the labeling quality of KOS is 50.56%, the labeling quality of GTIC is 71.43%, the labeling quality of MNLDP is 78%, and the labeling quality of LAWMV is 76.71%.

[0078] The labeling quality of the LCGTI method proposed in this invention is higher than that of all the comparison objects, which proves that comprehensively considering the deviation and variance of the workers' labels can mine more information, and the strategy of selecting the labels of the workers with the highest labeling quality as the inferred labels is effective.

[0079] The beneficial effects of the present invention are: not only can it comprehensively evaluate the marking quality of crowdsourcing workers, but it also explores a new truth-value reasoning strategy that is different from mark aggregation, which can ensure that the inferred inferred marks are always consistent with the marks of high-quality workers, thereby improving the marking quality of the truth-value reasoning method, being simple and easy to implement, and having high marking quality, thereby solving the problem of poor performance of the truth-value reasoning method.

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A crowd-sourced labeled truth-value reasoning method, characterized by: The following steps are involved: S1. Build each crowdsourcing worker The corresponding data set Where D r Indicates that only crowdsourcing workers u are included r A data set of all labeled instances; the data set includes a real data set Income; S2. For worker u r , if u r For the target instance x i It is marked, that is Then calculate the data set D r Each instance in is associated with instance x i Otherwise, the worker u r On the target instance x i The mark quality on is set to 0; S3. Dataset D r Each instance in is associated with instance x i Sort the distances and find D r In and Example x i The K nearest neighbor instances If the K value is less than the number of instances contained in Dr, set the K value to the number of instances contained in Dr; ikr In the dataset D r The kth instance found in i Recent instance; x ikr The multi-noise label set is in Denotes crowdsourcing worker u r' In x ikr Marks on S4, from worker u r with other workers on target instance x i The label consistency on the target instance and the set of neighbor instances The labeling consistency on worker u is calculated in two aspects r In instance x i Deviation b ir ; S5. Using Worker u r On the target instance x i The labels on the target instance and the set of neighbor instances The consistency of the markings on worker u is measured r In instance x i The variance of the markers on ir ; S6, the deviation b ir and variance v ir Linear combination, as worker u r On the target instance x i Marking quality on ir ; S7, repeat steps S2 to S6 until all crowdsourcing workers On the target instance x i Marking quality on All have been estimated; S8. Select the label of the worker with the highest label quality from all workers as the target instance x. i Inferred tag.

2. A crowd-sourced labeling truth-value reasoning method as claimed in claim 1, characterized in that: In step S2, data set D r Each instance in is associated with instance x i The calculation formula of the distance is shown in formula (1): In formula (1), x j Indicates D r The jth instance in d m (x i ,x j ) represents x i With x j The distance on the mth attribute is calculated as follows (2): In formula (2), overlap(x im ,x jm ) represents x i With x j The calculation method for noun attributes is as follows: In formula (3), rn_diff(x im ,x jm ) represents x i With x j The calculation method for numerical attributes is as follows: In formula (4), max m and min m Indicates D r The maximum and minimum values ​​of the mth feature in .

3. A crowd-sourced labeling truth-value reasoning method as claimed in claim 1, characterized in that: Deviation b in step S4 ir The calculation formula is shown in (5): In formula (5), Represents worker u r with other workers on target instance x i The label consistency on is calculated as follows (6): In formula (6), Represents a binary function whose value is 1 when its two parameters are the same, otherwise it is 0; In formula (5), Represents worker u r The set of neighbor instances of the target instance with other workers The label consistency on is calculated as follows (7):

4. A crowd-sourced labeling truth-value reasoning method as claimed in claim 3, characterized in that: In step S5, the variance v ir The calculation formula is as follows (8):

5. A crowd-sourced labeling truth-value reasoning method as claimed in claim 4, characterized in that: The marking quality w in step S6 ir The calculation formula is as follows (9):

6. A crowd-sourced labeling truth-value reasoning method as claimed in claim 1, characterized in that: In step S8, the expression of the inferred mark is as follows (10):

7. A storage device, characterized in that: The storage device stores instructions and data for implementing a crowdsourcing labeling truth value reasoning method as described in any one of claims 1 to 6.

8. A crowd-sourced labeling truth-value reasoning device, characterized in that: include: A processor and a storage device; the processor loads and executes instructions and data in the storage device to implement a crowdsourcing labeling truth value reasoning method as described in any one of claims 1 to 6.

Images