Label histogram creation device, label histogram creation method, and label histogram creation program

Abstract

A label histogram creation unit (14) of a label histogram creation device (1) sets, to a data set (X) composed of N data items (x), the number of samplings (β) for each of the data items (x), performs a first sampling process using a crowdsourcing (2), and creates a set (L) of label histograms. A pickup unit (16) performs, on the basis of uncertainty of information included in the label histograms, a process of picking up, from the data set (X), data items (x) to be subjected to a second-round sampling process. The label histogram creation unit (14) performs the second-round sampling process on the data items (x) picked up by the pickup unit (16), while increasing the number of samplings (β) as compared with that in the first sampling process.

Description

[Technical Field]

[0001] The present invention relates to a label histogram creating device, a label histogram creating method, and a label histogram creating program. [Background technology]

[0002] A label histogram shows the probability distribution of labels that classify a given data set. Label histograms are created by sampling data in which multiple people independently assign labels to the data. Such label histograms are typically created using crowdsourcing.

[0003] In the field of machine learning, there are many sets of label histograms created by sampling multiple different data sets with labels of the same classification, known as benchmark datasets (see, for example, Non-Patent Document 1). Benchmark datasets are used to evaluate the performance of data classifiers constructed by machine learning. In many of these benchmark datasets, a label is assigned to each piece of data only once, i.e., the number of samples is 1. On the other hand, it has been proposed to improve the accuracy of performance evaluation of data classifiers by increasing the number of samples to increase the diversity of the label histogram (see, for example, Non-Patent Document 2). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Yann Lecun, et.al., “THE MNIST DATABASE,” [online], [Retrieved May 6, 2022], Internet<URL:http: / / yann.lecun.com / exdb / mnist / > [Non-patent document 2] Mimori, T., Sasada, K., Matsui, H., and Sato, I. (2021). “Diagnostic uncertainty calibration: Towards reliable machine predictions in medical domain”, In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 3664-3672. PMLR. Summary of the Invention [Problem to be solved by the invention]

[0005] However, when using crowdsourcing, the more samples you use, the higher the financial cost. Furthermore, depending on the data that makes up the dataset, even if you increase the number of samples, the number of votes may be concentrated on a specific label for the majority of the data.

[0006] In such cases, it is possible to increase diversity by discarding label histograms of data where the number of votes is concentrated in a specific label. However, the cost required to create the discarded label histograms would be wasted. In other words, the number of samples included in the final label histogram set may be small compared to the cost required to create the label histogram set, and the resulting product may not be worth the cost.

[0007] There is a demand for a label histogram creation device that can create a highly diverse set of label histograms while reducing costs. [Means for solving the problem]

[0008] The label histogram creation device according to the present invention comprises: A label histogram showing a probability distribution of the labels in the data is created by performing a sampling process of assigning labels to classify the data using crowdsourcing, a label histogram creation unit that sets the number of samples for each data set consisting of multiple data, performs an initial sampling process using the crowdsourcing, and creates a set of label histograms; a pickup unit that performs a pickup process of data to be subjected to a second sampling process from the data set based on the uncertainty of information contained in the label histogram, The label histogram creation unit is characterized in that it increases the number of samples from the data picked up by the picking unit to a number greater than that of the first sampling process, and performs a second sampling process using the crowdsourcing. [Effects of the Invention]

[0009] According to the present invention, it is possible to create a highly diverse set of label histograms while reducing costs. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a conceptual diagram of a label histogram creation system to which a label histogram creation device according to an embodiment of the present invention is applied. [Figure 2] FIG. 10 is a diagram illustrating a label histogram. [Figure 3] FIG. 2 is a block diagram showing a functional configuration of the label histogram creating device. [Figure 4] FIG. 10 is a diagram illustrating an example of a set of label histograms. [Figure 5] FIG. 10 is a diagram illustrating another example of a set of label histograms. [Figure 6] 10 is a flowchart illustrating a processing flow of the label histogram creating device. [Figure 7]10 is a flowchart illustrating the flow of the second and subsequent sampling processes. [Figure 8] 10 is a flowchart illustrating the flow of a pickup process. [Figure 9] FIG. 10 is a diagram illustrating an example of image data included in a data set. [Figure 10] FIG. 10 is a diagram illustrating an example of a pin arrangement. [Figure 11] FIG. 10 is a diagram illustrating an example of picked up data. [Figure 12] FIG. 10 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the label histogram creation device according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Next, an embodiment of the present invention (hereinafter referred to as "the present embodiment") will be described with reference to the drawings. FIG. 1 is a conceptual diagram of a label histogram creation system to which a label histogram creation device according to this embodiment is applied. FIG. 2 is a diagram illustrating a label histogram. FIG. 3 is a block diagram showing the functional configuration of the label histogram creating device. As shown in FIG. 1, the label histogram creation system 100 includes a label histogram creation device 1 and a crowdsourcing device 2. A data set X, which is a target for creating a label histogram, is input from the outside to the label histogram creation device 1. The data set X is composed of multiple data x. The data x is, for example, data such as images, audio, and video, and is data used in machine learning. The data x belonging to the data set X can be classified using the same label set Y. The label set Y is composed of K types of labels y, as shown below.

[0012]

number

[0013] A label histogram shows the probability distribution of label y in data x. In other words, x and y are random variables sampled from a probability distribution P(x,y). A label histogram is created by multiple people independently assigning labels y to data x. The act of assigning labels y to data x is called sampling. Figure 2 shows an example where 100 people sampled a single piece of data x, with 70 people assigning label 1, 20 people assigning label 2, and 10 people assigning label 3. In this case, the label histogram for data x is expressed as [70, 20, 10]. Here, the number of times label y is assigned to construct a label histogram is called the number of samples. The number of times each label y is assigned is called the number of votes. In the example of Figure 2, the number of samples is 100, with label 1 receiving 70 votes, label 2 receiving 20 votes, and label 3 receiving 10 votes.

[0014] As shown in Fig. 1, a label histogram creation device 1 creates a label histogram for each data x in a dataset X using crowdsourcing 2. Crowdsourcing 2 is a system that recruits an unspecified number of workers Op over the Internet and requests them to perform tasks. The label histogram creation device 1 outputs a set T of data x to be sampled to a crowdsourcing 2 via a network. A worker Op who receives a request via crowdsourcing 2 samples each data x in set T. In crowdsourcing 2, the number of votes from the workers Op is tallied and a label histogram is created for each data x. A label histogram set L, which is a compilation of label histograms created for each data x in set T, is input from crowdsourcing 2 to the label histogram creation device 1.

[0015] The label histogram creation device 1 normalizes the label histogram created using crowdsourcing 2 in this way, and outputs the set P of normalized label histograms for the dataset X to the outside as a final product.

[0016] As shown in FIG. 3, the label histogram creating device 1 includes an input unit 11, an output unit 12, a storage unit 13, a label histogram creating unit 14, an information entropy calculating unit 15, and a pickup unit 16.

[0017] The input unit 11 and the output unit 12 are composed of a communication interface, an input / output interface, etc. The communication interface transmits and receives information to and from the crowdsourcing 2, etc., via a communication network. The input / output interface inputs and outputs information between an input device such as a keyboard, not shown, and an output device such as a display.

[0018] The storage unit 13 stores a program (label histogram creation program) for executing each functional unit of the label histogram creation device 1, and information required for the processing of each functional unit. As an example, the storage unit 13 stores a data set X input from the outside. The storage unit 13 also stores a set L of label histograms input from crowdsourcing 2. Furthermore, the storage unit 13 stores parameters and the like used in the processing described below.

[0019] The label histogram creation unit 14 creates a set L of label histograms by performing sampling processing using crowdsourcing 2 as described above. In this embodiment, the label histogram creating unit 14 performs sampling processing using crowdsourcing 2 multiple times. The label histogram creation unit 14 sets a set T and a sampling count β for each sampling process. As described above, the set T is made up of data x to be sampled. The sampling count β is the number of samples of each data x that constitutes the set T. That is, for each sampling process, the label histogram creation unit 14 outputs the set T of data x to be sampled, specifying the sampling count β, to the crowdsourcing 2. When the label histogram creation unit 14 receives a set L of label histograms, which are the sampling results of the set T, from the crowdsourcing 2, it stores the set L in the storage unit 13.

[0020] The label histogram creating unit 14 changes the number of data α constituting the set T and the number of samples β of each data x for each sampling process. Specifically, in the first sampling process, the label histogram creating unit 14 performs sampling process on all of the N pieces of data x included in the data set X. That is, in the first sampling process, the number of pieces of data α=N. In the second sampling process, the label histogram creating unit 14 reduces the number α of data included in the set T from N, while increasing the number β of samples of each data x from that in the first sampling process. When performing the second or subsequent sampling process multiple times, the label histogram creation unit 14 reduces the number of data α in the set T compared to the previous sampling process, while increasing the number of samples β of each data x compared to the previous sampling process. That is, the label histogram creating unit 14 narrows down the data x to be sampled, and then performs sampling intensively on the narrowed down data x. In each sampling process, the number α of data to be sampled and the number β of samples of each data x are set, for example, by an operator of the label histogram creation device 1 and stored as parameters in the storage unit 13. The number M of times the sampling process is performed from the second time onwards is also set by the operator in the same way and stored as a parameter in the storage unit 13.

[0021] When the sampling process is completed, the label histogram creating unit 14 creates a label histogram set P for the data set X by normalizing the label histogram set L stored in the storage unit 13 .

[0022] The information entropy calculation unit 15 and the pickup unit 16 perform processing to narrow down the data x to be sampled. The information entropy calculation unit 15 calculates the information entropy H of the label histogram for the data x for which the label histogram has been created by the sampling process. Information entropy H indicates the uncertainty of the information contained in the label histogram. The more uncertain the information contained in the label histogram, the greater the amount of information contained in the label histogram. In other words, if the information entropy H is large, the label histogram contains a lot of information, and if the information entropy H is small, the label histogram contains less information.

[0023] In the first sampling process, the information entropy calculation unit 15 calculates the information entropy H for each of the label histograms of the N pieces of data x that make up the data set X. In the second and subsequent sampling processes, the information entropy calculation unit 15 calculates the information entropy H only for the data x that is the sampling target and for which a label histogram has been created.

[0024] The pickup unit 16 performs a pickup process of the data x to be subjected to the next sampling process based on the uncertainty of the information contained in the label histogram of each data x. Specifically, the picking unit 16 refers to the information entropy H of the label histogram of each data x calculated by the information entropy calculation unit 15, and picks out data x whose information entropy H is dispersed from one another. The pickup unit 16 collects the picked up data x and sets a set T for the next sampling process. The number of data x picked up by the pickup unit 16 is equal to the number of data α to be sampled, which is set as a parameter for each sampling process. That is, the pickup unit 16 reduces the number of data x to be picked up each time it performs sampling processing. The pickup process of the pickup unit 16 will be described in detail later.

[0025] In this way, in this embodiment, the data x to be sampled is narrowed down based on the information entropy H, and sampling is performed selectively on the narrowed down data x, thereby reducing the overall number of samples and increasing the diversity of the label histograms in the set P of label histograms that are finally created.

[0026] The set of label histograms created by the label histogram creation device 1 is used, for example, to evaluate the performance of a data classifier constructed by machine learning. Here, the more diverse the label histograms constituting the set, the more accurately the performance of the data classifier can be evaluated. For example, when the label histograms constituting the set contain a balanced mix of large, small, and medium amounts of information and there is little bias in the amount of information, the label histograms can be said to be diverse. In order to increase the diversity of the label histogram, it is possible to increase the number of samples of each data x. However, simply increasing the number of samples of all data x included in dataset X may not increase the diversity of the label histogram.

[0027] FIG. 4 is a diagram showing an example of a set of label histograms. FIG. 5 is a diagram showing another example of a set of label histograms. Figure 4 shows an example in which a label histogram set L1 is created by sampling a data set consisting of six pieces of data x1 to x6. Figure 5 shows an example in which a label histogram set L2 is created by sampling a data set consisting of six pieces of data x7 to x12. Note that the number of samples is set to be the same in the examples of Figures 4 and 5.

[0028] For example, in the label histogram for data x1 in Figure 4, the number of votes is concentrated on label 1. Such a label histogram can be said to have a low amount of information, i.e., a low information entropy H. On the other hand, in the label histogram for data x4 in Figure 4, the number of votes is distributed almost equally among labels 1, 2, and 3. Such a label histogram can be said to have a high amount of information, i.e., a high information entropy H. In the label histogram for data x5 in Figure 4, the number of votes for label 3 is relatively high, but the number of votes is not extremely concentrated on one label like in data x1, so it can be said to have a medium amount of information. As described above, in the set L1 in FIG. 4, the label histograms of the data x1 to x6 have little bias in the amount of information, are dispersed among each other, and the diversity of the label histograms can be said to be relatively high.

[0029] On the other hand, in Figure 5, the label histogram for data x12 has votes that are somewhat distributed across labels 1 to 3, but the label histograms for data x7 to x11 have votes that are concentrated on one label. In other words, in set L2, the majority of data has low information content in the label histogram, and the amount of information contained in the label histogram is highly biased, which means that the diversity of the label histogram is relatively low.

[0030] As such, set L2 in Figure 5 has a lower diversity of label histograms compared to set L1 in Figure 4, but because it was created using the same number of samples as set L1, the cost of using crowdsourcing 2 is the same as set L1. To increase the diversity of set L2 in Figure 5, it is possible to discard from set L2 some of the label histograms of data x7 to x11, where the number of votes is concentrated in one label. However, in this case, the cost required for sampling the discarded label histograms would be wasted. In other words, the number of samples of data included in the final output set L2 would be small compared to the total cost required to create set L2, and the resulting product may not be worth the cost.

[0031] On the other hand, the label histogram device of this embodiment narrows down the data x to be sampled based on the information entropy H, as described above, and increases the number of samples for the narrowed down data x to perform sampling intensively. That is, in the first sampling process, the label histogram creating unit 14 samples all of the N pieces of data x included in the data set X, but the number of samples β is set to be small. Then, the picking unit 16 picks out data x with highly diverse combinations based on the information entropy H from the label histogram obtained in the first sampling process, and performs a second sampling process on the picked-up data x by increasing the sampling number β. This minimizes the number of samples taken for data x that has a biased amount of information and for which it is difficult to increase diversity even if the number of samples is increased, thereby reducing costs. Furthermore, in this embodiment, when the second and subsequent sampling processes are performed multiple times, the number of data items α to be subjected to the next sampling process is reduced and the number of samples β is increased each time a sampling process is performed, thereby enabling the present embodiment to further narrow down the highly diverse combinations of data x and perform sampling in a focused manner.

[0032] The processing of the label histogram creating device 1 according to this embodiment will be described with reference to a flowchart. FIG. 6 is a flowchart illustrating the flow of processing performed by the label histogram creating device. FIG. 7 is a flowchart illustrating the flow of the sampling process from the second time onwards. FIG. 8 is a flowchart illustrating the flow of the pickup process. Fig. 9 is a diagram showing an example of data included in a dataset. In Fig. 9, as an example, multiple image data included in the dataset are arranged according to the information entropy of each label histogram. FIG. 10 is a diagram showing an example of pin arrangement. FIG. 11 is a diagram showing an example of the picked up data.

[0033] 6, the label histogram creation device 1 starts processing when a dataset X for creating a label histogram is input (step S01: Yes). The label histogram creation unit 14 stores the dataset X input via the input unit 11 in the storage unit 13. If the dataset X is not input (step S01: No), the label histogram creation device 1 waits until it is input.

[0034] According to the input data set X, the operator of the label histogram creating device 1 sets the following parameters:

[0035]

number

[0036] By setting the parameters as described above, the number of data α to be sampled decreases each time sampling processing is performed, and the number of samples β for each data x increases.

[0037] The label histogram creating unit 14 sets a set T of data x to be subjected to sampling processing (step S02). In the first sampling process, the set T includes all of the N pieces of data x that make up the data set X, as shown below.

[0038]

number

[0039] The label histogram creating unit 14 performs an initial sampling process on the set T using crowdsourcing 2, and creates a set L of label histograms (step S03). Specifically, the label histogram creating unit 14 sets the number of samples β based on the parameters, and outputs the set T of data x to the crowdsourcing 2. In crowdsourcing 2, each piece of data x included in set T is sampled by workers Op, the number of which corresponds to the sampling number β. A worker Op assigns one of K types of labels y included in a label set Y to data x.

[0040] FIG. 9 shows an example in which the dataset X is composed of image data x of handwritten digits provided in MNIST of Non-Patent Document 1. Each of these data x is assigned one of 10 labels from 0 to 9. For example, if the initial sampling number β is set to 10, 10 workers Op assign a label y to each of the data x.

[0041] In crowdsourcing 2, for each piece of data x, the number of votes for the label y assigned by each worker Op is tallied, and a label histogram is created. The label histogram li of the data xi included in the set T is expressed by the following equation (1).

[0042]

number

[0043] Here, using crowdsourcing 2, the operation of multiple workers Op sampling one piece of data xi is regarded as a function and is denoted as Sampling. In this case, the label histogram li created for the data xi with the sampling number β and the label histogram set L which is a collection of the label histograms li can be expressed by the following equations (2) and (3).

[0044]

number

[0045] For example, when sampling is performed on data xa in Figure 9 with a sampling number of 10, if 7 people vote for 1 and 3 people vote for 7, the label histogram of image data xa is expressed as [0, 7, 0, 0, 0, 0, 3, 0, 0].

[0046] The crowdsourcing 2 inputs a set L of label histograms created for a set T to the input unit 11 of the label histogram creation device 1. The label histogram creating unit 14 stores the set L of label histograms of the set T input from the crowdsourcing 2 in the storage unit 13, thereby completing the first sampling process.

[0047] 6, the label histogram creation unit 14 sets the number k of sampling processes from the second onwards to 1 (step S04), where k is a natural number from 1 to M. That is, the label histogram creation unit 14 repeats the sampling process until the number k of second sampling processes becomes M. The label histogram creating unit 14 performs the second and subsequent sampling processes (step S05). In the second and subsequent sampling processes, the label histogram creating unit 14 performs sampling processes on the data x narrowed down by the processes of the information entropy calculating unit 15 and the picking unit 16 .

[0048] As shown in FIG. 7, the information entropy calculation unit 15 calculates the information entropy H of the label histogram li of each data xi using the set L of label histograms created in the first sampling process (step S51).

[0049] Specifically, the information entropy calculation unit 15 can calculate the information entropy H from the label histogram li, for example, by the following method.

[0050]

number

[0051] The function H relating to the uncertainty of the information contained in the label histogram is not limited to the information entropy, and can be calculated, for example, by the following formula (9) or (10).

[0052]

number

[0053] The pickup unit 16 sets a set T' of data x to be subjected to the next sampling process (step S52). The pickup unit 16 sets the minimum value A and the maximum value B of the set T′ as shown below. Specifically, the pickup unit 16 extracts the minimum value A and the maximum value B from the information entropy H of each piece of data x calculated by the information entropy calculation unit 15.

[0054]

number

[0055] In the example of FIG. 9, the image data x included in the data set X are arranged according to the magnitude of the information entropy H of each label histogram. The information entropy H of data Xb is the minimum value A, and the information entropy H of data Xc is the maximum value B. All data x are located within the interval (BA) between the minimum value A and the maximum value B. As is clear from FIG. 9, image data x of characters that are easy to distinguish, i.e., characters that are easy to classify, has a small information entropy H. Image data x of characters that are difficult to distinguish, i.e., characters that are difficult to classify, has a large information entropy H.

[0056] As shown in FIG. 7, the pickup unit 16 performs a pickup process to pick up data x to be subjected to the next sampling process from the set T' (step S53). 9, the data x is not uniformly distributed within the interval (BA), but is biased toward the minimum value A, the center, and the maximum value B. The picking unit 16 picks out a combination of data from these data x whose information entropy H is distributed among each other.

[0057] 10, the pickup unit 16 divides the section (BA) into equal intervals and places pins u at the boundary positions of each section. The positions where pins u are placed include minimum value A and maximum value B. The number of pins u to be placed matches the number α of data to be subjected to the next sampling process. For example, if α=9, the pickup unit 16 divides the section (BA) into eight equal parts and places nine pins u. The pickup unit 16 further sequentially picks up data x having the closest information entropy H to each pin u and adds it to the set T'. As shown in Figure 11, the picking process picks up nine pieces of data x, including the data xb and xc with the minimum value A and maximum value B, into the set T'. The picked up data x have a relatively small bias in information entropy H, and are a well-balanced combination of data with low, medium, and high classification difficulty. That is, even for a data set X having a bias in information entropy H as shown in FIG. 9, the picking process of this embodiment can obtain a combination of data x with highly diverse label histograms in which the information entropy H is dispersed, as shown in FIG.

[0058] 8, the pickup unit 16 sets the number of data x to be picked up into the set T' to r=1 (step S531). Here, r is a natural number between 1 and αk. αk is the number of data to be subjected to the next sampling process, which is set by a parameter. The pickup unit 16 determines the position of the pin u using the following equation (11) (step S532).

[0059]

number

[0060] According to the above formula (11), the position of the pin u is determined in order from the side of the minimum value A, that is, from the side with the least information entropy H. That is, in the first pickup, the pin u is determined to have the minimum value A, and in the αkth pickup, the pin u is determined to have the maximum value B. The pickup unit 16 picks up data t having information entropy H closest to the determined pin u into a set T' (step S533). The pickup unit 16 uses, for example, the following equation (12) to identify the data t having the information entropy H closest to the pin u.

[0061]

number

[0062] The pickup unit 16 further includes the data t in the new set T' and excludes it from the original set T using the following equation (13).

[0063]

number

[0064] If the pickup number r is not αk (step S534: No), the pickup unit 16 sets r to r+1 (step S535), returns to steps S532 to S533, determines the next pin u, and sequentially picks up data t closest to the pin u into the set T'. If the pickup number r=αk (step S534: Yes), the pickup unit 16 ends the pickup process.

[0065] 7, the pickup unit 16 overwrites the original set T with a set T' consisting of the picked-up data t (step S54). As a result, the set T is updated to one consisting only of the data x that will be the target of the next sampling process.

[0066] The label histogram creating unit 14 performs sampling processing on the newly set set T using crowdsourcing 2 (step S55). Specifically, the label histogram creation unit 14 sets the sampling number βk and outputs the data x included in the set T to the crowdsourcing 2. Here, the label histogram creation unit 14 increases the sampling number βk from the sampling number β in the initial sampling process based on the set parameters.

[0067] As described above, the set T updated by the picking process includes combinations of data x with highly diverse label histograms. By increasing the sampling number β for these combinations, the diversity of the label histograms of these data x can be further increased.

[0068] In crowdsourcing 2, the data xi that is the subject of the second sampling process is sampled in the same manner as in the first sampling process, to create a label histogram, which is input to the input unit 11 of the label histogram creation device 1. The storage unit 13 stores a label histogram li for the data xi created in the first sampling process. The label histogram creating unit 14 adds and stores the label histogram created in the second sampling process to the label histogram li of the data xi created in the first sampling process, as shown in the following equation (14).

[0069]

number

[0070] As shown in FIG. 6, if the number k of sampling processes from the second onward is not equal to M (step S06: No), the label histogram creating unit 14 sets k to k+1 (step S07), returns to step S05, and performs further sampling processes. Each time a sampling process is performed, the number of data αk to be subjected to the next sampling process is narrowed down from the previous sampling process by the information entropy calculation unit 15 and the pickup process of the pickup unit 16. The label histogram creation unit 14 performs sampling on the narrowed down data x by increasing the number of samples βk from the previous sampling process.

[0071] As an example, the label histogram creating device 1 can perform the following process. For example, when a data set X consisting of 10,000 pieces of data x is input, the label histogram creation unit 14 performs sampling on the 10,000 pieces of data x in the first sampling process by setting the number of samples β for each piece of data x to 10. The pickup unit 16 performs a pickup process using the label histogram of the 10,000 pieces of data x, and narrows down the number α of data to be sampled next to 1,000 pieces. In the second picking process, the label histogram creating unit 14 increases the number of samples β for each data x to 90 and performs sampling on 1000 data x. The pickup unit 16 performs a pickup process using the label histogram of the 1000 pieces of data x, and narrows down the number α of data to be sampled next to 200 pieces. The label histogram creating unit 14 increases the number of samples β for each data x to 9900 and performs sampling on the 200 data x.

[0072] In this way, by repeatedly narrowing down the data x and increasing the number of samples β, sampling is performed intensively on combinations of data x with high diversity in label histograms. This makes it possible to create a set L of highly diverse label histograms from dataset X while keeping the number of samples in the entire dataset X low.

[0073] 6, when the number k of sampling processes from the second time onwards reaches M (step S06: Yes), the label histogram creation unit 14 ends the sampling process. The label histogram creation unit 14 normalizes the label histogram li of the data x included in the data set X stored in the storage unit 13 by the following equation (15), and creates a set P of label histograms for the data set X (step S07).

[0074]

number

[0075] The label histogram creating unit 14 outputs the set P of label histograms for the normalized data set X to the outside via the output unit 12 (step S08), and the process ends.

[0076] <Hardware configuration> The label histogram creating device 1 according to this embodiment is realized by, for example, a computer 900 as shown in FIG. FIG. 12 is a hardware configuration diagram showing an example of a computer 900 that realizes the functions of the label histogram creating device 1 according to this embodiment. The computer 900 has a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a HDD (Hard Disk Drive) 904, an input / output I / F (Interface) 905, a communication I / F 906, and a media I / F 907.

[0077] The CPU 901 operates based on a program (label histogram creation program) stored in the ROM 902 or the HDD 904, and performs processing of each functional unit of the label histogram creation device 1 shown in Fig. 2. The ROM 902 stores a boot program executed by the CPU 901 when the computer 900 is started up, programs related to the hardware of the computer 900, and the like.

[0078] The CPU 901 controls an input device 910 such as a mouse or keyboard, and an output device 911 such as a display, via an input / output I / F 905. The CPU 901 acquires data from the input device 910 via the input / output I / F 905, and outputs generated data to the output device 911. Note that a GPU (Graphics Processing Unit) or the like may be used as a processor together with the CPU 901.

[0079] The HDD 904 stores programs executed by the CPU 901 and data used by the programs. The communication I / F 906 receives data from Crowdsourcing 2 (see FIG. 1) or other devices via a communication network (e.g., NW (Network) 920) and outputs the data to the CPU 901, and also transmits data generated by the CPU 901.

[0080] The media I / F 907 reads a program or data stored in the recording medium 912 and outputs it to the CPU 901 via the RAM 903. The CPU 901 loads a program related to a target process from the recording medium 912 onto the RAM 903 via the media I / F 907, and executes the loaded program. The recording medium 912 is an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), a magneto-optical recording medium such as an MO (Magneto Optical disc), a magnetic recording medium, a conductive memory tape medium, a semiconductor memory, or the like.

[0081] For example, when the computer 900 functions as the label histogram creation device 1 according to this embodiment, the CPU 901 of the computer 900 executes a program loaded onto the RAM 903 to realize the functions of the label histogram creation device 1. The HDD 904 stores data in the RAM 903. The CPU 901 reads and executes a program related to a target process from a recording medium 912. Alternatively, the CPU 901 may read a program related to a target process from another device via a communication network (NW 920).

[0082] <Configuration and Effects of the Above-described Embodiment> (1) The label histogram creation device 1 creates a label histogram showing the probability distribution of label y in data x by performing sampling processing to assign label y that classifies data x using crowdsourcing 2. Crowdsourcing 2 is a system in which sampling processing requested by the label histogram creation device 1 is performed by an unspecified number of workers Op recruited over the Internet. The label histogram creating device 1 includes a label histogram creating unit 14 and a pickup unit 16 . The label histogram creation unit 14 sets the number of samples β for each data x for a data set X consisting of N (plural) data x, performs an initial sampling process using crowdsourcing 2, and creates a set L of label histograms. The picking unit 16 picks out data x from the data set X, which is to be subjected to the second sampling process, based on the uncertainty of the information contained in the label histogram. The label histogram creation unit 14 increases the number of samples β for the data x picked up by the pickup unit 16 from the number of samples in the first sampling process, and performs a second sampling process using crowdsourcing 2.

[0083] With this configuration, the label histogram creating device 1 can create a highly diverse set L of label histograms while reducing costs. Specifically, the pickup unit 16 performs the pickup process based on the uncertainty of the information contained in the label histogram (e.g., information entropy H), thereby narrowing down the targets for the next sampling process to combinations of highly diverse data x whose label histograms have little bias in the amount of information. By increasing the sampling number β for the narrowed-down data x and performing the sampling process, the diversity of the label histograms of the data x can be further increased. Furthermore, in the initial sampling process, which samples all data x in dataset X, the number of samples β can be set low, thereby reducing the overall number of samples and the cost of using crowdsourcing 2.

[0084] (2) After the first sampling process, the label histogram creating unit 14 performs the sampling process M times (multiple times) using crowdsourcing 2. The pickup unit 16 performs the pickup process of the data x to be subjected to the next sampling process, while decreasing the number of pickups from the previous sampling process each time the sampling process is performed. The label histogram creating unit 14 performs the next sampling process on the data x picked up by the picking unit 16 by increasing the number of samples β from the previous sampling process.

[0085] As the sampling number β increases, the diversity of the label histogram increases, but the cost of using crowdsourcing 2 also increases. Therefore, the label histogram creation device 1 of this embodiment further narrows down the number of data α to be sampled each time sampling processing is performed, while gradually increasing the sampling number β for each data x. This makes it possible to reduce the increase in cost while focusing on sampling combinations of highly diverse data x, thereby further increasing diversity.

[0086] (3) The label histogram creating device 1 includes an information entropy calculation unit 15 . The information entropy calculation unit 15 calculates the information entropy H of the label histogram for the data x for which the label histogram has been created by the sampling process.

[0087] By calculating the information entropy H, which indicates the uncertainty of the information contained in the label histogram, the pickup unit 16 can narrow down the data x based on the information entropy H. This allows the pickup unit 16 to select a combination of data x with high diversity in the label histogram.

[0088] (4) As a pickup process, the pickup unit 16 picks up data x whose information entropy H is dispersed.

[0089] The pickup unit 16 picks up data x whose information entropy H is dispersed from one another, and thus it is possible to select a combination of data x whose information entropy H is less biased and highly diverse as the next sampling target.

[0090] (5) As a pickup process, the pickup unit 16 divides the interval (BA) between the minimum value A and the maximum value B of the information entropy H according to the number of data x to be picked up, and picks up the data x having the information entropy H closest to the pin u indicating the boundary position of each division.

[0091] This allows the pickup unit 16 to pick up a well-balanced combination of data x in which the information entropy H is dispersed according to the number α of data to be subjected to the next sampling process. The pickup unit 16 can pick up an even better-balanced combination of data x by, for example, dividing the section (BA) at equal intervals and arranging pins u.

[0092] The above-described effects can also be applied to the label histogram creation method performed by the label histogram creation device 1 and the label histogram creation program for causing the computer 900 to function as the label histogram creation device 1.

[0093] The present invention is not limited to the above-described embodiments, and many modifications can be made by a person skilled in the art within the technical concept of the present invention. [Explanation of symbols]

[0094] 1. Label histogram creation device 2. Crowdsourcing 11 Input section 12 Output section 13 Storage section 14 Label histogram creation section 15 Information entropy calculation unit 16 Pickup section 100 Label Histogram Creation System Op Worker

Claims

1. A label histogram creation device that creates a label histogram indicating a probability distribution of the labels in data by performing a sampling process of assigning labels that classify data using crowdsourcing, a label histogram creation unit that sets the number of samples for each data set consisting of multiple data, performs an initial sampling process using the crowdsourcing, and creates a set of label histograms; a pickup unit that performs a pickup process of data to be subjected to a second sampling process from the data set based on the uncertainty of information contained in the label histogram, The label histogram creation device is characterized in that the label histogram creation unit increases the number of samples from the first sampling process to the data picked up by the pickup unit, and performs a second sampling process using the crowdsourcing.

2. the label histogram creation unit performs the sampling process multiple times using the crowdsourcing after the initial sampling process; the pickup unit performs a pickup process of data to be subjected to the next sampling process while reducing the number of pickups compared to the previous sampling process each time the sampling process is performed; 2. The label histogram creation device according to claim 1, wherein the label histogram creation unit performs a next sampling process on the data picked up by the pickup unit by increasing the number of samples compared to the previous sampling process.

3. 3. The label histogram creation device according to claim 1, further comprising an information entropy calculation unit that calculates information entropy of the label histogram for the data for which the label histogram is created by the sampling process.

4. 4. The label histogram creation device according to claim 3, wherein the pickup unit performs the pickup process by picking up data whose information entropy is dispersed.

5. 5. The label histogram creation device according to claim 4, wherein the pickup unit, as the pickup process, divides an interval between the minimum value and the maximum value of the information entropy according to the number of data to be picked up, and picks up data having information entropy closest to a boundary position of each division.

6. A label histogram creation method for a label histogram creation device that creates a label histogram showing a probability distribution of the labels in the data by performing a sampling process of assigning labels that classify the data using crowdsourcing, comprising: The label histogram creation device includes: For a dataset consisting of a plurality of data, a sampling number of each data is set, and an initial sampling process is performed using the crowdsourcing; performing a process of picking out data to be subjected to a second sampling process from the data set based on the uncertainty of the information contained in the label histogram; A label histogram creation method, characterized in that the number of samples is increased from the first sampling process for the data picked up by the picking process, and a second sampling process is performed using the crowdsourcing.

7. A label histogram creating program for causing a computer to function as the label histogram creating device according to claim 1 or 2.

Images