System, method, and program for providing information about operator creating machine learning teacher data

Abstract

To specify an operator (i.e., an annotator) having sensibility similar to that of a user when creating teacher data.SOLUTION: A system 100 stores one or more data sets 140 and a plurality of labeling results 150 associated with the data sets. Each labeling result 150 is associated with each operator 130. The system 100 further creates and presents a sample data set 160 and accepts a labeling result 170 associated with the sample data set 160. The system 100 selects one or more operators 130 based on a result of calculating the similarity by comparing the labeling result 170 by a user 120 with a labeling result 180 of each of the plurality of operators 130 for the sample data set 160. The system 100 also outputs at least one of the selected one or more operators 130, teacher data created in the past by each operator 130, and information of a learned model learned from the past teacher data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to machine learning, and more particularly to a technology for providing information about workers who create training data for machine learning. [Background technology]

[0002] Artificial intelligence (AI) technology has seen remarkable development in recent years. Supervised learning, which uses a huge amount of training data to train an AI model, is well known. For example, an AI model trained through supervised learning can accurately identify various objects, such as people and animals. However, supervised learning requires a huge amount of training data. Training data is a set of example data input to an AI model and labels representing the answers to those examples. In many cases, training data is created by humans. Therefore, it takes a considerable amount of effort for users to obtain the necessary amount of training data. As a result, crowdsourcing the creation of training data is becoming more common.

[0003] Regarding crowdsourcing, for example, Japanese Patent Application Laid-Open Publication No. 2017-211689 (Patent Document 1) discloses a "technology for requesting the creation of training data for naming objects contained in images via crowdsourcing." This technology "requests the creation of training data for each task from multiple respondents who can answer. The answer results are confirmed using ensemble learning, and in parallel with machine recognition, ensemble learning is used to determine the answers of multiple respondents, and the reliability of the respondents is accumulated. The question creation unit detects the boundaries of objects in the received image and creates answer areas for each location of all objects contained in the image. The respondent input / output unit searches for respondents who can answer using the respondent status understanding unit and displays the questions on the respondent's terminal. To prevent spelling variations, the respondent input / output unit sends the input string from the respondent's terminal to a word / phrase supplementation unit, selects candidate words from name dictionary data, displays them on the respondent's terminal via the respondent input / output unit, obtains answers, and sends them to a response integration unit" (see [Abstract]). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2017-211689 Summary of the Invention [Problem to be solved by the invention]

[0005] For example, suppose there are multiple annotators who can undertake the creation of training data. An annotator is a worker who creates training data or labels each data set. The act of labeling data to create training data is also sometimes called annotation. In such a case, a user may wish to ask a worker with a similar sensibility to their own to create training data. However, the technology disclosed in Patent Document 1 is unable to identify a worker with a similar sensibility to the user's sensibility from among multiple workers. Therefore, there is a need for a technology that can identify a worker (i.e., an annotator) with a similar sensibility to the user's sensibility.

[0006] The present disclosure has been made in consideration of the above-described background, and in one aspect, its purpose is to provide a technology for identifying workers (i.e., annotators) who have sensibilities similar to those of a user when creating training data. [Means for solving the problem]

[0007] According to one embodiment, a system for providing information about workers who create training data for machine learning is provided. The system includes a storage unit that stores one or more datasets and a plurality of labeling results associated with each of the one or more datasets. Each of the plurality of labeling results is associated with a respective one of a plurality of workers. The system further includes a sample creation unit that creates a sample dataset from the one or more datasets, an output unit that presents the sample dataset to a user, an input unit that accepts the labeling results associated with the sample dataset from the user, an extraction unit that extracts the labeling results associated with the sample dataset for each of the plurality of workers from the storage unit, a comparison unit that calculates a similarity between the labeling results associated with the sample dataset by the user and the labeling results associated with the sample dataset for each of the plurality of workers, and a selection unit that selects one or more workers from the plurality of workers based on the comparison result. The output unit outputs at least one of information about the one or more workers, information about one or more past training data created by each of the one or more workers, and information about one or more trained models that have learned at least one of the one or more past training data.

[0008] In one aspect, creating a sample dataset includes selecting a candidate dataset from one or more datasets, and selecting one or more data from the candidate datasets to include in the sample dataset.

[0009] In one aspect, the input unit is configured to be able to accept input regarding desired teacher data, and selecting a candidate dataset includes selecting a dataset included in the desired teacher data.

[0010] In one aspect, selecting one or more pieces of data from the candidate dataset to be included in the sample dataset includes preferentially selecting pieces of data from the candidate dataset that have different labels assigned by multiple workers.

[0011] In one aspect, the comparison unit is configured to compare the user's label with each of the labels for the multiple workers for each piece of data included in the sample dataset, and calculate the similarity of the labeling results associated with the sample dataset between the user and each of the multiple workers based on the result of the comparison for each piece of data included in the sample dataset.

[0012] In one aspect, calculating the similarity of the labeling results associated with the comparison unit sample dataset includes counting the number of matches between the labels assigned by each of multiple workers and the labels assigned by the user for each piece of data included in the sample dataset.

[0013] In one aspect, calculating the similarity of labeling results associated with a sample dataset includes, when the labels assigned to the sample dataset include three or more candidate answers, evaluating the magnitude of the difference between the labels assigned by each of multiple workers and the labels assigned by the user for each piece of data included in the sample dataset.

[0014] In one aspect, when there are two or more identical labeling results associated with a certain dataset, the storage unit stores the two or more workers who created the two or more identical labeling results as a group, and the selection unit is configured to be able to select a group including two or more workers.

[0015] According to another embodiment, a method for providing information about workers who create training data for machine learning is provided. The method is executed by a computer. The method also includes storing one or more datasets and a plurality of labeling results associated with each of the one or more datasets. Each of the plurality of labeling results is associated with a respective one of a plurality of workers. The method further includes creating a sample dataset from the one or more datasets, presenting the sample dataset to a user, accepting from the user labeling results associated with the sample dataset, extracting from a storage unit the labeling results associated with the sample dataset for each of the plurality of workers, comparing the labeling results associated with the sample dataset by the user with each of the labeling results associated with the sample dataset for each of the plurality of workers to calculate a similarity, selecting one or more workers from among the plurality of workers based on the comparison result, and outputting at least one of information about the one or more workers, information about one or more past training data created by each of the one or more workers, and information about one or more trained models trained using at least one of the one or more past training data.

[0016] According to another embodiment, a program for providing information about workers who create training data for machine learning is provided. The program causes a computer to save one or more datasets and a plurality of labeling results associated with each of the one or more datasets. Each of the plurality of labeling results is associated with a respective one of a plurality of workers. The program further causes the computer to create a sample dataset from the one or more datasets, present the sample dataset to a user, accept from the user the labeling results associated with the sample dataset, extract from a storage unit the labeling results associated with the sample dataset for each of the plurality of workers, compare the labeling results associated with the sample dataset by the user with each of the labeling results associated with the sample dataset for each of the plurality of workers to calculate a similarity, select one or more workers from among the plurality of workers based on the comparison result, and output at least one of information about the one or more workers, information about one or more past training data created by each of the one or more workers, and information about one or more trained models that have learned at least one of the one or more past training data. [Effects of the Invention]

[0017] According to an embodiment, in creating training data, it is possible to identify workers (that is, annotators) who have sensibilities similar to those of the user.

[0018] The above and other objects, features, aspects and advantages of the present disclosure will become apparent from the following detailed description of the disclosure taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a diagram illustrating an example of a system 100 according to the present embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of functional blocks included in the system 100. [Figure 3]It is a diagram showing an example of the hardware included in system 100. [Figure 4] It is a diagram showing an example of the data stored in the storage unit 110. [Figure 5] It is a diagram showing a first example of the UI provided by system 100. [Figure 6] It is a diagram showing a second example of the UI provided by system 100. [Figure 7] It is a diagram showing a third example of the UI provided by system 100. [Figure 8] It is a diagram showing an example of the operation procedure of system 100.

Embodiments for Carrying Out the Invention

[0020] Hereinafter, embodiments of the technical idea according to the present disclosure will be described with reference to the drawings. In the following description, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated. Also, each embodiment, each modification example, each software configuration, each hardware configuration, each function, and each process, etc. may be selectively combined as appropriate.

[0021] <A. Outline of System Operation>

[0022] FIG. 1 is a diagram showing an example of a system 100 according to the present embodiment. System 100 is a system for a user to easily identify an operator (i.e., an annotator) having a sensibility similar to the user's own sensibility. By using system 100, the user can identify an operator having a sensibility similar to the user's own sensibility and request the creation of teacher data for use in machine learning from the operator. Referring to FIG. 1, the configuration of system 100 according to the present embodiment and a typical operation example of system 100 will be described. Also, various terms used in this specification will be described.

[0023] (a. System Configuration)

[0024] The system 100 is composed of one or more devices or virtual machines. When the system 100 is composed of one device, the system 100 may be read as a device. The system 100 is also configured to be able to refer to a memory unit 110. The memory unit 110 may be located inside the system 100 or may be external. The system 100 is also configured to be able to communicate with a terminal of a user 120. Furthermore, the system 100 may be configured to be able to communicate with the terminals of individual workers 130A, 130B, 130C, etc., and any other device.

[0025] The storage unit 110 stores information on one or more workers 130A, 130B, and 130C, one or more datasets 140, and labeling results 150A, 150B, and 150C that each worker 130A, 130B, and 130C associates with the dataset 140. In the example of FIG. 1, the storage unit 110 stores only one dataset, three workers, and labeling results, but this is merely an example. The storage unit 110 may store any number of datasets, worker information, and labeling results. Hereinafter, workers are collectively referred to as workers 130. Similarly, labels are collectively referred to as labeling results 150. Training data includes example data, such as images, and labels associated with the data. Therefore, in the example of FIG. 1, the combination of the dataset 140 and the labeling results 150 can be considered training data. In one aspect, the memory unit 110 may be configured to store some or all of information such as the profile and contact information of each worker 130, one or more pieces of training data previously created by each worker 130, and information on one or more trained models that have learned at least one of these pieces of training data. In the memory unit 110, each piece of training data and each trained model is linked to information on each worker 130.

[0026] In one aspect, system 100 may operate as a standalone system. In this case, user 120 may use the functions of system 100 via a UI (User Interface) or the like displayed on a display of system 100. In another aspect, system 100 may operate as a web application server. In this case, user 120 may use the functions of system 100 via a UI or the like displayed on a browser on a terminal.

[0027] (b. System Functions)

[0028] Next, the functions of the system 100 will be described. Conventionally, a user could use a crowdsourcing service or the like to request one or more workers to create training data. However, there was a problem in that the requested workers did not necessarily have sensibilities similar to those of the user. To solve this problem, one method is to hold a competition to select workers. However, holding a competition is time-consuming and costly, and therefore places a heavy burden on the user. To solve this problem, the system 100 has at least the following three functions.

[0029] The first function is a function for registering previously created teacher data. The system 100 pre-registers one or more teacher data sets and information about one or more workers who created each of the one or more teacher data sets in the storage unit 110. In some aspects, the system 100 may be configured to allow individual creators to register information about one or more previously created teacher data sets in the storage unit 110. The information registered in the system 100 includes at least the worker's own identification information and the teacher data. The system 100 may also be configured to acquire and register in the storage unit 110 the results of a past competition or information about previously created teacher data sets. The system 100 may receive this information from another device or may be input via a storage medium, etc. Here, assume that multiple workers 130 create teacher data sets from the same dataset 140 and store them directly in the storage unit 110. In this case, the dataset 140 is duplicated for each worker 130, resulting in unnecessary consumption of the storage unit 110's capacity. Therefore, the system 100 may decompose the registered training data into a data set 140 and labeling results 180 that each worker 130 assigns to the data set 140, and store the results in the storage unit 110.

[0030] The second function is to calculate the similarity of sensibilities between the user 120 and at least some of the one or more registered workers 130. As an example, the system 100 operates as a web application. In this case, the system 100 receives input of information regarding desired training data from the user 120. Next, the system 100 determines whether training data that matches or is similar to the desired training data is registered in the storage unit 110. That is, the system 100 determines whether a combination of a dataset 140 that matches or is similar to the desired training data and a labeling result 180 is registered in the storage unit 110. Here, it is assumed that training data that matches or is similar to the desired training data has been found. In this case, the system 100 creates a sample dataset 160 from the dataset 140, which is an example problem included in the found training data.

[0031] Next, the system 100 prompts the user 120 to label the sample dataset 160 (i.e., to create training data). The system 100 receives the labeling result 170 assigned to the sample dataset 160 from the user 120. Next, the system 100 extracts from the storage unit 110 one or more workers 130 who have created training data for the selected dataset 140. In the example of FIG. 1, workers 130A, 130B, and 130C are extracted.

[0032] Then, the system 100 extracts from the storage unit 110 labeling results 180A, 180B, and 180C previously assigned to the sample dataset 160 by each of the one or more extracted workers 130. These labels are collectively referred to as labeling results 180. The system 100 compares the labeling results 170 assigned to the sample dataset 160 by the user 120 with each of the extracted labeling results 180. Based on the comparison results, the system 100 calculates the degree of match or similarity between the training data of the user 120 (or the labeling results 170) and the training data of each worker 130 (or the labeling results 180). Finally, the system 100 ranks the workers 130 in descending order of the degree of match or similarity. The higher the ranking of a worker 130, the more similar the worker 130's sensibilities are to the user.

[0033] The third function is a function of presenting to the user 120 information about workers 130 who have sensibilities similar to those of the user 120. The system 100 may present to the user 120 information about workers 130 who have sensibilities similar to those of the user 120 (i.e., who are highly ranked) via a UI displayed on a browser or the like. In one aspect, the system 100 may present to the user 120 information about the highest-ranked worker 130. In this case, information about worker 130B is presented to the user 120. In another aspect, the system 100 may present to the user 120 information about the workers 130 in order of ranking. In another aspect, the system 100 may present to the user 120 a score of match or similarity. The information presented by the system 100 includes at least one of information about the worker 130 himself / herself, information about one or more training data previously created by the worker 130, and information about one or more trained models that have trained at least one of these training data.

[0034] By using the system 100 via a browser on the user's terminal or the like, the user 120 can identify a worker 130 who has a similar sensibility to the user's own. The user 120 can also make a new request to a worker 130 who has a similar sensibility to the user's own via the system 100. Furthermore, the user 120 can acquire and utilize, via the system 100, training data previously created by a worker 130 who has a similar sensibility to the user's own, or one or more trained models that have learned training data previously created by the worker 130.

[0035] In one aspect, the system 100 may be used as a crowdsourcing service, in which case the system 100 can match the user 120 with a worker 130 who has a similar sensibility to the user 120.

[0036] (c.Term)

[0037] Next, some terms used in this specification will be explained. In this specification, a "system" refers to a single device or a combination of multiple devices. The device may include a personal computer, a workstation, a server device, a tablet, or a smartphone. The device may also include a system-on-a-chip (SoC) or a system-on-module (SoM). The device may also include any peripheral devices such as a switch, a router, a display, a keyboard, and a mouse. The device may also include virtual machines and instances built in a cloud environment. In one aspect, system 100 may be connected to input / output devices such as a display and a keyboard and used as a standalone device. In another aspect, system 100 may provide various functions as a service or web application via a network. In this case, users may use the system's functions via a browser or client software installed on their terminals.

[0038] In this specification, the term "worker" refers to a person who creates training data. The worker 130 is also called an annotator. In this specification, the worker 130 may be read as an annotator.

[0039] In this specification, a "user" refers to a person who needs training data. A user 120 may request training data from a worker 130 (annotator) via a crowdsourcing service or the like. Therefore, the user 120 may be interpreted as a requester.

[0040] In this specification, a "model" refers to a mechanism for analyzing input data and outputting the evaluation results. A model may also refer to a mechanism by which AI performs machine learning. A model may also be called an AI model, a machine learning model, or a deep learning model. A model that has completed learning using training data may also be called a trained model. In this specification, a model includes a model before learning.

[0041] As used herein, "teacher data" refers to a set of example questions and answers used to train a model. As an example, suppose a model is trained to estimate the age of a person appearing in an image. In this case, the teacher data is a set of an image of a person (example question) and text data indicating the age (answer). As used herein, a set of data and the answers associated with that data may also be referred to as teacher data. A dataset and a set of labels associated with each data item in the dataset may also be referred to as teacher data. Teacher data may also be referred to as training data or learning data. Furthermore, the example questions included in the teacher data may include data in any format, such as image data, audio data, sensor data, and text data. Similarly, the answers included in the teacher data may also include data in any format.

[0042] In this specification, a "label" refers to an answer contained in training data. A label may be called an annotation or classification information. Typically, one label (answer) is associated with one piece of data (example question). A set of labels associated with each piece of data contained in a dataset is called a "labeling result" or a "label set." The task of assigning labels to data is sometimes called labeling or labeling. Furthermore, labeling may be called annotation or classification.

[0043] In this specification, the term "dataset" refers to a collection of multiple pieces of data. As an example, an image dataset is a collection of multiple pieces of image data. Typically, in supervised learning, a huge amount of training data is used. Therefore, training data is created by labeling individual examples included in a set of many examples. In other words, training data can be said to be created by labeling individual pieces of data included in some dataset.

[0044] As used herein, a "sub-dataset" refers to a part of dataset 140. As an example, assume that a set of 100 image data is taken out from a dataset 140 containing 1000 image data. In this case, the set of 100 image data taken out is a sub-dataset.

[0045] As used herein, a "sample dataset" is a set of one or more data extracted from dataset 140. The sample dataset 160 is used to have the user 120 perform labeling (creation of teacher data). Since the sample dataset 160 is a part of the dataset, it can also be said to be a sub-dataset.

[0046] As used herein, "sensibility" indicates how one feels when seeing an image or the like. Sensibility may be read as "subjectivity". In the creation of teacher data, sensibility may also be rephrased as the tendency of the labels given to the data. For example, assume that the first operator and the second operator perform labeling on a dataset A containing 100 data. At this time, if the labels given by each of the first operator and the second operator to the 100 data completely match, it can be said that the sensibilities of the two for dataset A are the same. Also, if the labels given by each of the first operator and the second operator to the 100 data match by 98%, it can be said that the sensibilities of the two for dataset A are similar. Further, if the labels given by each of the first operator and the second operator to the 100 data only match by 30%, it can be said that the sensibilities of the two for dataset A are not similar.

[0047] <B. Configuration of the System>

[0048] FIG. 2 is a diagram illustrating an example of functional blocks included in system 100. In one aspect, each functional block illustrated in FIG. 2 may be realized as a program. In this case, each functional block illustrated in FIG. 2 is realized by executing a program on hardware illustrated in FIG. 3. In another aspect, some or all of each functional block illustrated in FIG. 2 may be realized as hardware. In this case, system 100 further includes hardware corresponding to at least some of the functional blocks in addition to the configuration illustrated in FIG. 3.

[0049] The system 100 includes a storage unit 110, an input unit 210, a sample creation unit 220, an extraction unit 230, a comparison unit 240, a selection unit 250, and an output unit 260. In some aspects, the system 100 may further include functional blocks not shown in FIG.

[0050] The storage unit 110 stores one or more pieces of training data and information on a plurality of workers. More specifically, the storage unit 110 stores one or more data sets (i.e., example questions) and labeling results (i.e., answers) linked to each of the one or more data sets. Each of the labeling results is linked to each of one or more workers. An example of the structure of data stored in the storage unit 110 will be described later with reference to FIG. 4.

[0051] The input unit 210 is configured to acquire various information via a UI. For example, the input unit 210 may be configured to register worker information, etc. The input unit 210 may be configured to register previously created teacher data, etc. The input unit 210 may be configured to register user information, etc. The input unit 210 may be configured to accept a request to create teacher data, etc. The input unit 210 may be configured to accept a search input for teacher data or a dataset, etc. The input unit 210 may be configured to accept a label assigned to the sample dataset 160 by a user. The input unit 210 stores the input data in the storage unit 110 based on accepting the registration input for creator information, user information, and teacher data. The input unit 210 outputs the input data to the sample creation unit 220 based on accepting a request to create teacher data or a search input for teacher data or a dataset, etc. The input unit 210 outputs the input data to the comparison unit 240 based on accepting the label assigned to the sample dataset 160 by the user.

[0052] The sample creation unit 220 creates the sample dataset 160. As an example, the sample creation unit 220 creates the sample dataset 160 based on a user request acquired from the input unit 210. As an example, assume that the user inputs to the input unit 210 a request for or search for training data for person age estimation. In this case, the sample creation unit 220 creates a portion of example problems (i.e., a sample dataset) from the training data (i.e., a dataset) for person age estimation. The sample creation unit outputs the created sample dataset 160 to the output unit 260. The sample creation unit also outputs the created sample dataset 160 to the extraction unit 230. In a certain aspect, the sample creation unit 220 may preferentially include data with different labels (i.e., answers) for each worker in the sample data. This is because the system 100 aims to identify workers who have sensibilities similar to those of the user, and therefore should preferentially select data in which answers vary from worker to worker (i.e., data that reflects differences in sensibilities between workers 130).

[0053] The extraction unit 230 searches the storage unit 110 to search for workers who have assigned labels to the sample dataset 160 (hereinafter referred to as relevant individuals). If relevant individuals are found, the extraction unit 230 extracts the labels assigned to the sample dataset 160 by each relevant individual from the storage unit 110. The extraction unit 230 outputs the labels assigned to the sample dataset 160 by each relevant individual to the comparison unit 240.

[0054] The comparison unit 240 compares the label assigned by the user 120 to the sample dataset 160 with each of the labels assigned by each relevant person to the sample dataset 160. Through the comparison process, the comparison unit 240 calculates a score indicating the degree of similarity of sensibilities between the user and each relevant person (i.e., worker 130). The comparison unit 240 then determines a ranking of each worker 130 in order of score. More specifically, the comparison unit 240 compares the label of the user 120 with each of the labels assigned by each worker 130 for each piece of data included in the sample dataset 160, and calculates a similarity score.

[0055] The comparison unit 240 calculates the similarity of the labeling results associated with the sample dataset 160 between the user 120 and each worker 130 based on the score of each label. The similarity of the labeling results associated with the sample dataset 160 is the sum of the similarity scores for each label. As an example, if the sample dataset 160 includes 10 pieces of data, the comparison unit 240 calculates the sum of the similarities for each label assigned to these 10 pieces of data. The comparison unit 240 outputs the similarity of the labeling results to the selection unit 250.

[0056] In calculating the total similarity, the comparison unit 240 may calculate the similarity of the labeling results associated with the sample dataset 160 by counting the number of labels that are the same as those of the user 120 for each of the multiple workers 130.

[0057] Assume that the labels assigned to the sample dataset 160 include three or more candidate answers. In this case, the comparison unit 240 may calculate the magnitude of difference between the selected answer for each data item included in the sample dataset 160 between the label of the user 120 and each of the labels assigned by the multiple workers 130. For example, assume that an age label is assigned to a person appearing in a photograph. In this task, assume that the user 120 assigns the label "age: teens" to the data "image X." Also assume that worker 130A assigns the label "age: 30s" to "image X," and worker 130B assigns the label "age: 20s" to "image X." In this case, it can be said that the difference between the answer of worker 130A and the answer of the user is greater than the difference between the answer of worker 130B and the answer of the user. Therefore, with regard to the label of image X, worker 130B is assigned a higher score than worker 130A.

[0058] The selection unit 250 selects workers 130 to present to the user 120 based on the similarity. In one aspect, the selection unit 250 may select the worker 130 with the highest similarity. In another aspect, the selection unit 250 may select a predetermined number of workers 130 in order of decreasing similarity. In yet another aspect, the selection unit 250 may select all workers 130 compared with the user 120 and sort them in order of similarity. The selection unit 250 outputs information about the one or more selected workers 130 and the similarity score of each worker 130 to the output unit 260.

[0059] The output unit 260 outputs various UIs. In one aspect, the output unit 260 may output the various UIs to a display connected to the system 100. In another aspect, the output unit 260 may transmit the various UIs to the terminal of the user 120. In this case, the terminal of the user 120 may display the various UIs on a browser or the like. As an example, the output unit 260 may output a registration screen for the user 120, a registration screen for the worker 130, and a registration screen for training data. The output unit 260 may also output a training data request or search screen (see FIG. 5), a display screen for the sample dataset 160 (see FIG. 6), a display screen for the worker information (see FIG. 7), and the like. The display screen for the worker information displays information about the worker 130 who has sensibilities similar to those of the user 120. The output unit 260 may dynamically generate a display screen for the worker information based on information acquired from the selection unit 250. As an example, the worker information display screen displays at least one of information about one or more workers, information about training data previously created by each of the one or more workers, and information about one or more trained models that have learned at least one of these training data.

[0060] 3 is a diagram illustrating an example of hardware included in the system 100. The system 100 includes a processor 301, a primary storage device 302, a secondary storage device 303, an external device interface 304, an input interface 305, an output interface 306, and a communication interface 307.

[0061] The processor 301 may execute programs for implementing various functions of the system 100. The processor 301 may be configured, for example, with at least one integrated circuit. According to an embodiment, the integrated circuit may include at least one central processing unit (CPU), at least one graphics processing unit (GPU), at least one field programmable gate array (FPGA), at least one application specific integrated circuit (ASIC), at least one artificial intelligence (AI) chip, or a combination thereof.

[0062] The primary storage device 302 functions as a workspace for the processor 301. The primary storage device 302 stores programs executed by the processor 301 and data referenced by the processor 301. In one aspect, the primary storage device 302 can be realized by a dynamic random access memory (DRAM), a static random access memory (SRAM), or the like.

[0063] Secondary storage device 303 is a non-volatile memory that stores programs executed by processor 301 and data referenced by processor 301. Processor 301 executes programs read from secondary storage device 303 to primary storage device 302, and references data read from secondary storage device 303 to primary storage device 302. In one aspect, secondary storage device 303 can be realized by a hard disk drive (HDD), a solid state drive (SSD), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, or the like.

[0064] The external device interface 304 can be connected to any external device such as a printer, a scanner, an external HDD, etc. In one aspect, the external device interface 304 can be realized by a USB (Universal Serial Bus) terminal or the like.

[0065] The input interface 305 can be connected to any input device such as a keyboard, a mouse, a touchpad, a gamepad, etc. In one aspect, the input interface 305 can be realized by a USB terminal, a PS / 2 terminal, a Bluetooth (registered trademark) module, etc.

[0066] Output interface 306 can be connected to any output device such as a cathode ray tube display, a liquid crystal display, an organic electroluminescence (EL) display, etc. In one aspect, output interface 306 can be realized by a USB terminal, a D-sub terminal, a DVI (Digital Visual Interface) terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal, a DisplayPort terminal, etc.

[0067] Communication interface 307 is connected to other devices via a wired network or a wireless network. In one aspect, communication interface 307 may be implemented by a wired local area network (LAN) port, a Wi-Fi (registered trademark) (Wireless Fidelity) module, or the like. In another aspect, communication interface 307 may transmit and receive data using a communication protocol such as TCP / IP (Transmission Control Protocol / Internet Protocol), UDP (User Datagram Protocol), or the like.

[0068] 4 is a diagram showing an example of data stored in storage unit 110. In one aspect, the data stored in storage unit 110 may be represented as a table of a relational database, or may be represented in any other data format such as JSON (JavaScript (registered trademark) Object Notation).

[0069] The storage unit 110 includes one or more datasets 410. Each of the one or more datasets 410 is associated with a plurality of labeling results 420. Each of the plurality of labeling results 420 is associated with a plurality of pieces of worker information 430. In the example of FIG. 4, two datasets 410 are associated with three labeling results 420 for each 410, but this is merely an example. The storage unit 110 may store any number of datasets 410. Each dataset 410 may be associated with any number of labeling results 420.

[0070] The dataset 410 corresponds to a set of examples that are input to the model. Typically, the dataset 410 includes a large amount of data. The data included in the dataset 410 can be image, video, audio, text, or any other format.

[0071] The labeling result 420 corresponds to an answer input to the model. Typically, the labeling result 420 includes the same number of labels as the number of data included in the dataset 410. The label may be interpreted as an annotation, classification, tag, or the like. The labeling result may be interpreted as an annotation result, classification result, tagging result, or the like. The combination of the dataset 410 and the labeling result 420 becomes training data. Therefore, it can be said that the storage unit 110 stores training data.

[0072] The worker information 430 includes information about the worker 130. The information about the worker 130 may include the worker's 130 profile, contact information, and performance information. In one aspect, the worker information 430 may be linked to information about one or more pieces of past training data created by the worker 130. The worker information 430 may also be linked to information about one or more trained models that have trained at least one of the one or more pieces of past training data created by the worker 130.

[0073] In the example of FIG. 4, three labeling results, "first labeling result, second labeling result, and third labeling result," are linked to the "first data set." In other words, it can be said that the storage unit 110 stores three pieces of training data related to the "first data set." Each of the three labeling results, "first labeling result, second labeling result, and third labeling result," is linked to each of three workers, "first worker, second worker, and third worker." This allows the system 100 to identify the worker 130 who created each labeling result 420 from each labeling result 420.

[0074] The training data stored in the storage unit 110 may be collected by any means. Therefore, the workers 130 who labeled each data set 410 may be different. Conversely, the same worker 130 may label each data set 410. More specifically, three labeling results (fourth labeling result, fifth labeling result, and sixth labeling result) are linked to the "second data set." Each of the three labeling results (fourth labeling result, fifth labeling result, and sixth labeling result) is linked to three workers (first worker, fourth worker, and fifth worker), respectively. When the "second data set" is compared with the "first data set," all of them are linked to the "first worker." In other words, it can be said that the "first and second data sets" contain labeling results 420 from the same worker 130. On the other hand, the "first data set" is linked to the "second and third workers," and the "second data set" is linked to the "fourth and fifth workers." In other words, it can be said that the "first and second data sets" contain labeling results 420 by different workers 130. In this way, each data set 410 may contain labeling results 420 by the same worker 130 and labeling results 420 by different workers 130.

[0075] In one aspect, the system 100 may group workers 130 who are considered to have the same sensibility for a certain dataset 410. As an example, assume that the labels included in the "first labeling result" and the labels included in the "second labeling result" completely match. In this case, the sensibility of the "first worker" and the "second worker" for the "first dataset" can be considered to match. Therefore, the system 100 may group the "first worker and second worker" associated with the "first dataset" and manage them as a single group. The system 100 may group three or more workers 130. As described above, the extraction unit 230 and the selection unit 250 acquire the worker 130 or information related to the worker 130 from the storage unit 110. At this time, the extraction unit 230 and the selection unit 250 may select a group as the worker 130. Furthermore, when outputting information related to the worker 130, the output unit 260 may output information related to multiple workers included in the group. As an example, suppose that the selection unit 250 selects a group including "the first and second workers" as workers 130 who have similar sensibilities to those of the user 120. In this case, the output unit 260 outputs information related to "the first and second workers."

[0076] 1 to 4, the system 100 includes a storage unit 110 that stores one or more data sets 410 and a plurality of labeling results 420 associated with each of the one or more data sets 410. Each of the plurality of labeling results 420 is associated with a respective one of a plurality of workers 130. The system 100 further includes a sample creation unit 220 that creates a sample dataset 160 from one or more datasets 410, an output unit 260 that presents the sample dataset 160 to the user 120, an input unit 210 that accepts labeling results associated with the sample dataset 160 from the user 120, an extraction unit 230 that extracts labeling results associated with the sample dataset 160 for each of the multiple workers 130 from the memory unit 110, a comparison unit 240 that compares the labeling results associated with the sample dataset 160 by the user 120 with each of the labeling results associated with the sample dataset 160 for each of the multiple workers 130 to calculate a similarity, and a selection unit 250 that selects one or more workers 130 from the multiple workers 130 based on the comparison results. The output unit 260 outputs at least one of information on one or more workers 130, information on one or more past training data created by each of the one or more workers 130, and information on one or more trained models that have learned at least one of the one or more past training data.

[0077] In one aspect, the comparison unit 240 compares the label of the user 120 with each of the labels of the multiple workers 130 for each piece of data included in the sample dataset 160. The comparison unit 240 is also configured to calculate the similarity of the labeling results associated with the sample dataset 160 between the user 120 and each of the multiple workers 130 based on the result of the comparison for each piece of data included in the sample dataset 160.

[0078] As an example, calculating the similarity of the labeling results associated with the sample dataset 160 includes counting the number of matches between the labels assigned by each of the multiple workers 130 and the labels assigned by the user 120 for each piece of data included in the sample dataset 160.

[0079] As another example, calculating the similarity of the labeling results associated with the sample dataset 160 includes, when the label assigned to the sample dataset 160 includes three or more candidate answers, evaluating the magnitude of the difference between the labels assigned by each of the plurality of workers 130 and the label assigned by the user 120 for each data included in the sample dataset 160.

[0080] Furthermore, when there are two or more identical labeling results associated with a certain dataset 410, the storage unit 110 may group and store two or more workers 130 who created the two or more identical labeling results. Also, the selection unit 250 is configured to be able to select a group including two or more workers 130.

[0081] <C. Example of User Interface>

[0082] The system 100 may be provided as a web application that provides a crowdsourcing service. In this case, as an example, the system 100 may include a UI as shown in FIGS. 5 to 7.

[0083] FIG. 5 is a diagram showing a first example of the UI provided by the system 100. The screen 500 is a screen for the user 120 to select or search for desired teacher data. As an example, the user 120 can access the screen 500 by logging in to the system 100 via the browser of their terminal. The screen 500 may include any input items for the user 120 to search for desired teacher data. As an example, the screen 500 includes a first input item 510, a second input item 520, and a third input item 530.

[0084] The first input field 510 is an input field for searching for desired teacher data. The user 120 can search for desired teacher data by entering a free word in the first input field 510. The first input field 510 can accept any word, such as a dataset name, a dataset type, a label name, a label type, a teacher data name, and a teacher data type. In one aspect, the first input field 510 may be a text field.

[0085] The second input item 520 is an input item that allows the user to select desired teacher data from a list of teacher data registered in the storage unit 110. The user 120 can search for desired teacher data by selecting the teacher data from the second input item 520. In one aspect, the second input item 520 may be a pull-down menu.

[0086] The third input item 530 is an input item for searching for desired teacher data from tags registered in the system 100 or the storage unit 110. The user 120 can search for desired teacher data by selecting one or more tags using the first input item 510. Each tag is linked to at least one of one or more datasets 410, one or more labeling results 420, and one or more pieces of worker information 430. The system 100 may also have a separate UI for registering tags.

[0087] The system 100 searches for training data stored in the storage unit 110 based on operation input on the screen 500. When the system 100 finds the desired training data, it selects a dataset 410 (i.e., an example problem) included in the training data as a candidate dataset. The candidate dataset is a dataset 410 that meets the search criteria of the user 120 and is used to create the sample dataset 160.

[0088] The screen 500 shown in FIG. 5 is merely an example. The screen 500 may include any input fields other than those shown in FIG. 5. Furthermore, the user 120 may search for desired training data by combining multiple input fields on the screen 500. In some aspects, the system 100 may present the user 120 with training data that is similar to the user's 120's search keywords, even if it does not exactly match the user's 120's search keywords. In this case, the system 100 may use any existing search technology, such as semantic search. The user 120 may check whether the desired training data exists in the storage unit 110 via the screen 500. More specifically, the system 100 determines whether the training data desired by the user 120 exists in the storage unit 110 based on the data entered on the screen 500. The system 100 then selects the dataset 410 included in the training data desired by the user 120 as a candidate dataset.

[0089] FIG. 6 is a diagram illustrating a second example of a UI provided by system 100. Screen 600 displays sample dataset 160. Screen 600 also accepts input of a label to be associated with sample dataset 160 from a user. As an example, a screen displayed in a browser on a terminal of user 120 may transition to screen 600 based on the result of an operation input on screen 500. In some aspects, the browser screen may transition from screen 500 to screen 600 via one or more screens, such as a search result display screen. As an example, screen 600 includes a sample dataset display item 610, a label input item 620, and an OK button 630.

[0090] The sample dataset display item 610 displays the sample dataset 160. The sample dataset 160 is a sub-dataset extracted from within the candidate dataset. In the example of Figure 6, the sample dataset display item 610 displays two images. That is, the sample dataset 160 includes two images (i.e., two data).

[0091] The system 100 searches the storage unit 110 for desired training data based on information input via the screen 500. The system 100 then selects a dataset 410 included in the desired training data as a candidate dataset. The candidate dataset corresponds to an example of training data desired by the user 120. In a certain aspect, the system 100 may preferentially include data with different labels (i.e., answers) for each worker 130 in the sample dataset 160. This is because the system 100 aims to identify workers with sensibilities similar to those of the user 120, and should preferentially select data for which the answers differ for each worker 130 (i.e., data that reflects differences in sensibilities between workers 130).

[0092] The label input field 620 accepts input from the user 120 of a label to be associated with each data item included in the sample data set 160. In one aspect, the label input field 620 may be an input form using a selection method such as radio buttons. In another aspect, the label input field 620 may be an input form such as a text field into which any information can be input.

[0093] When the enter button 630 is pressed, the terminal of the user 120 transmits the answer entered in the label input field 620 to the system 100. That is, the terminal of the user 120 transmits the label that the user has associated with each data item in the sample dataset 160 to the system 100. In some cases, an error may be displayed on the screen 600 when the enter button 630 is pressed when there is an incomplete entry in the label input field 620.

[0094] The system 100 can acquire, via the screen 600, each label that the user 120 has associated with each piece of data included in the sample dataset 160. The information entered in the label input field 620 can be said to be data that indicates the user's 120 subjectivity or sensibility regarding the sample dataset 160. In other words, the system 100 can acquire, via the screen 600, data that indicates the user's 120 subjectivity or sensibility regarding the sample dataset 160.

[0095] As described with reference to Figures 5 and 6, creating the sample dataset 160 includes selecting a candidate dataset from one or more datasets 410 and selecting one or more data from the candidate dataset as data to be included in the sample dataset 160. More specifically, the system 100 searches for and acquires training data desired by the user 120 from the storage unit 110. The system 100 then selects a dataset 410 included in the acquired training data as a candidate dataset. Furthermore, the system 100 extracts some data from the candidate dataset and creates the sample dataset 160 including the extracted data.

[0096] To this end, the input unit 210 is configured to be able to accept input regarding the desired dataset 410 (or training data). Selecting a candidate dataset includes selecting a dataset 410 included in the desired training data. In one aspect, the system 100 may select a dataset 410 similar to the dataset 410 included in the desired training data as a candidate dataset. In another aspect, the system 100 may select one or more datasets 410 similar to the dataset 410 included in the desired training data and present them to the user. In this case, the system 100 selects the dataset 410 selected by the user as a candidate dataset.

[0097] In one aspect, selecting one or more data from the candidate dataset as data to be included in the sample dataset 160 may include preferentially selecting data from the candidate dataset that has different labels assigned by multiple workers 130. This allows the system 100 to create a sample dataset 160 that includes a large amount of data that results in different answers from different workers (i.e., that reflects differences in sensibilities between workers 130). In one aspect, the system 100 may calculate, for each piece of data included in the candidate dataset, a degree of inconsistency between the labels assigned by each worker 130, and sort each piece of data based on the degree of inconsistency. In this case, the system 100 may include a predetermined number of pieces of data in the sample dataset 160 in descending order of degree of inconsistency.

[0098] FIG. 7 is a diagram illustrating a third example of a UI provided by the system 100. The screen 700 displays information about one or more workers 130 who have similar sensibilities to the user 120. For each piece of data included in the sample dataset 160, the system 100 compares the label of the user 120 with each of the labels of the multiple workers 130. The system 100 then identifies one or more workers 130 who have similar sensibilities to the user 120 from the comparison results. Hereinafter, the one or more workers 130 who have similar sensibilities to the user 120 are referred to as “one or more selected workers 130.” The system 100 displays information related to the one or more selected workers 130 on the screen 700. As an example, the screen displayed in the browser of the terminal of the user 120 may transition to the screen 700 based on the result of an operation input on the screen 600.

[0099] As an example, screen 700 includes display items 710 related to one or more selected workers 130, display items 720 related to one or more past training data created by each of the one or more selected workers 130, and display items 730 related to one or more trained models that have learned at least one of the one or more past training data.

[0100] The display item 710 includes various information such as contact information, profile information, and past performance information of one or more selected workers 130. In one aspect, the display item 710 may include an operation item for requesting work from one or more selected workers 130.

[0101] The display item 720 includes various information such as an overview, details, and links to one or more pieces of teacher data previously created by each of the one or more selected workers 130. In one aspect, the display item 720 may include an operation item for selecting and acquiring one or more pieces of teacher data.

[0102] The display item 730 includes various information such as an overview, details, and links to the training data of one or more trained models that have learned at least one of one or more training data previously created by each of one or more selected workers 130. In one aspect, the display item 730 may include an operation item for selecting and acquiring one or more trained models.

[0103] In one aspect, screen 700 may display at least one of display items 710, 720, and 730. System 100 may also be configured to allow selection of display items to be displayed on screen 700 based on input of display settings for screen 700 from the user.

[0104] User 120 may request work from worker 130 who has similar sensibilities to his / her own via screen 700. Furthermore, if user 120 finds desired training data among the training data previously created by worker 130 who has similar sensibilities to his / her own via screen 700, he / she may use the training data. Furthermore, user 120 may refer to one or more trained models that have learned training data previously created by worker 130 who has similar sensibilities to his / her own. If user 120 finds a model that suits his / her purpose among the one or more trained models, he / she may use the model.

[0105] Suppose that teacher data created by a certain worker 130 in the past or a learned model that has learned the teacher data is downloaded from the system 100. In this case, an incentive such as money may be given to the certain worker 130. For this purpose, the system 100 can count the number of downloads of teacher data created by each worker 130 in the past and the number of downloads of each learned model. Further, the system 100 can calculate an incentive for each worker 130 based on the number of downloads of each data. In a certain situation, the system 100 may transfer money corresponding to the incentive to the account of each worker 130 via an approval system. By providing such a mechanism, the system 100 can encourage more workers 130 to register the teacher data created in the past with the system 100.

[0106] <D. Operating Procedure of the System>

[0107] FIG. 8 is a diagram showing an example of the operating procedure of the system 100. In a certain situation, the processor 301 may read a program for performing the processing of FIG. 8 from the secondary storage device 303 into the primary storage device 302 and execute the program. In other situations, a part or all of the processing may also be realized as a combination of circuit elements configured to execute the processing. Further, in other situations, the following steps may be executed in a swapped order.

[0108] In step S800, the system 100 determines whether there is a new data registration request. As an example, the system 100 may receive a registration request for past teacher data from an individual worker 130. As another example, the system 100 may receive a registration request for one or more past teacher data and information on one or more workers 130 who created these teacher data from an administrator or another system. If the system 100 determines that there is a new data registration request (YES in step S800), the control is transferred to step S810. Otherwise (NO in step S8, the system 100 transfers the control to step S820.

[0109] In step S810, the system 100 links the dataset 410, the labeling result 420, and the worker information 430 and registers them in the storage unit 110. More specifically, the system 100 divides the input training data into the dataset 410 (a set of example problems) and the labeling result 420 (a label set), and stores them in the storage unit 110. In this case, the system 100 links the dataset 410 and the labeling result 420. In a certain aspect, the system 100 may store the input training data directly in the storage unit 110.

[0110] In step S820, system 100 determines whether a search request has been received from user 120. The search request corresponds to an input on screen 500. The search request can be said to be a request to search for at least one of workers 130 who have sensibilities similar to those of user 120, training data previously created by the workers, and trained models that have learned the training data. If system 100 determines that a search request has been received from user 120 (YES in step S820), system 100 transfers control to step S830. If not (NO in step S820), system 100 transfers control to step S800.

[0111] In step S830, the system 100 generates and outputs a sample dataset 160. More specifically, the system 100 uses the search keywords, tags, etc. included in the search request to search the storage unit 110 and acquires a dataset 410 (i.e., a candidate dataset) that meets the conditions. The system 100 generates a sample dataset 160 that includes some of the data in the candidate dataset, and outputs it to the screen 600. When selecting some of the data from the candidate dataset, the system 100 may preferentially select data that has different labels assigned by multiple workers 130 (i.e., data that reflects differences in sensibilities between the workers 130).

[0112] In step S840, the system 100 acquires the labeling results of the sample dataset 160 by the user 120. The labeling results correspond to input into the label input field 620 on the screen 600.

[0113] In step S850, the system 100 compares the labeling results assigned to the sample data set 160 by the user 120 with each of the labeling results assigned to the sample data set 160 by each of the one or more workers 130. Based on the comparison, the system 100 calculates the similarity between the labeling results assigned by the user 120 and each of the labeling results assigned by the one or more workers 130.

[0114] In step S860, system 100 ranks one or more workers 130 in descending order of similarity. Furthermore, system 100 selects at least one worker 130 with a high similarity. In one aspect, system 100 may select the worker 130 with the highest similarity. In another aspect, system 100 may select a predetermined number of workers 130 in descending order of similarity. Furthermore, in another aspect, system 100 may select all of the ranked workers 130.

[0115] In step S870, the system 100 outputs at least one of information about the selected one or more workers 130, information about training data previously created by each of the one or more workers 130, and information about a trained model that has learned at least one of the training data previously created by the one or more workers 130. The data output in step S870 corresponds to the data displayed on the screen 700.

[0116] In step S880, the system 100 receives a user request and processes the request. As an example, the system 100 may receive a request to one or more workers 130 to create training data. In this case, the system 100 transmits a request notification to each worker 130. As another example, the system 100 may receive a request to acquire past training data. In this case, the system 100 transmits the requested past training data to the terminal of the user 120. As yet another example, the system 100 may receive a request to acquire a trained model that has learned past training data. In this case, the system 100 transmits the requested trained model to the terminal of the user 120.

[0117] In step S890, system 100 transitions the current screen to the initial screen based on a response to the request of user 120. In one aspect, system 100 may transition the current screen to a confirmation screen indicating a response to the request. In this case, system 100 may transition the current screen to the initial screen based on, for example, user 120 pressing a confirmation button on the confirmation screen.

[0118] 8, the system 100 may execute a program to perform instructions or methods defined in the program. The instructions or methods may include storing one or more datasets 410 and a plurality of labeling results 420 associated with each of the one or more datasets 410. Each of the plurality of labeling results 420 is associated with a respective one of the plurality of workers 130. The method further includes creating a sample dataset 160 from one or more datasets 410, presenting the sample dataset 160 to a user, accepting labeling results associated with the sample dataset 160 from the user 120, extracting from the memory unit 110 the labeling results associated with the sample dataset 160 for each of the multiple workers 130, comparing the labeling results associated with the sample dataset 160 by the user with each of the labeling results associated with the sample dataset 160 for each of the multiple workers 130 to calculate a similarity, selecting one or more workers 130 from the multiple workers 130 based on the comparison result, and outputting at least one of information about the one or more workers 130, information about one or more past training data created by each of the one or more workers 130, and information about one or more trained models that have learned at least one of the one or more past training data.

[0119] <E.まとめ>

[0120] As described above, the system 100 according to this embodiment can identify the worker 130 who has sensibilities similar to those of the user 120. The system 100 can also present to the user 120 information about the worker 130 and training data previously created by the worker 130. Furthermore, the system 100 can also present to the user 120 one or more trained models that have learned training data previously created by the worker 130.

[0121] By using the system 100, the user 120 can easily request the creation of training data from a worker 130 who has sensibilities similar to the user's own. Furthermore, the user 120 can use training data previously created by the worker 130 if it suits the user's purpose. Similarly, the user 120 can use a trained model that has learned training data previously created by the worker 130 if it suits the user's purpose.

[0122] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present disclosure is defined by the claims, not by the above description, and is intended to include all modifications within the meaning and scope equivalent to the claims. Furthermore, the disclosures described in the embodiments and each modification are intended to be implemented, as far as possible, either alone or in combination. [Explanation of symbols]

[0123] 100 System, 110 Memory Unit, 120 User, 130, 130A, 130B, 130C Worker, 140, 410, A Data Set, 150, 150A, 150B, 150C, 170, 180, 180A, 180B, 180C, 420 Labeling Result, 160 Sample Data Set, 210 Input Unit, 220 Sample Creation Unit, 230 Extraction Unit, 240 Comparison Unit, 250 Selection Unit, 260 Output Unit, 301 Processor, 302 Primary Storage Unit, 303 Secondary Storage Unit, 304 External Device Interface, 305 Input Interface, 306 Output Interface, 307 Communication Interface, 430 Worker Information, 500, 600, 700 Screen, 510 First Input Item, 520 Second Input Item, 530 Third input item, 610 sample dataset display item, 620 label input item, 630 decision button, 710, 720, 730 display items.

Claims

1. a storage unit configured to store one or more datasets and a plurality of labeling results associated with each of the one or more datasets, each of the plurality of labeling results being associated with a respective one of a plurality of workers; a sample creation unit that creates a sample data set from the one or more data sets; an output unit for presenting the sample data set to a user; an input unit that receives a labeling result associated with the sample dataset from the user; an extraction unit that extracts, from the storage unit, labeling results associated with the sample data set for each of the plurality of workers; a comparison unit that calculates a similarity by comparing a labeling result associated with the sample data set by the user with a labeling result associated with the sample data set by each of the plurality of workers; a selection unit that selects one or more workers from the plurality of workers based on a result of the comparison, The output unit outputs at least one of information about the one or more workers, information about one or more past training data created by each of the one or more workers, and information about one or more trained models that have learned at least one of the one or more past training data.

2. Creating the sample data set includes: selecting a candidate dataset from the one or more datasets; and selecting one or more data from the candidate data sets for inclusion in the sample data set.

3. the input unit is configured to be able to accept input regarding desired teacher data, The system of claim 2 , wherein selecting the candidate dataset includes selecting a dataset included in the desired training data.

4. 3. The system of claim 2, wherein selecting one or more pieces of data from the candidate dataset as data to be included in the sample dataset includes preferentially selecting, from the candidate dataset, data that has been assigned different labels by the plurality of workers.

5. The comparison unit comparing the user's label with each of the labels for the plurality of workers for each piece of data included in the sample dataset; The system according to claim 1 , configured to calculate a similarity between labeling results associated with the sample dataset between the user and each of the plurality of workers based on a result of the comparison for each piece of data included in the sample dataset.

6. 6. The system of claim 5, wherein calculating a similarity of the labeling results associated with the sample dataset includes counting, for each of the data included in the sample dataset, the number of matches between the labels assigned by each of the plurality of workers and the labels assigned by the user.

7. 6. The system of claim 5, wherein calculating a similarity of labeling results associated with the sample dataset includes, when the labels assigned to the sample dataset include three or more candidate answers, evaluating, for each of the data included in the sample dataset, a magnitude of difference between a label assigned by each of the plurality of workers and a label assigned by the user.

8. When there are two or more identical labeling results associated with a certain data set, the storage unit groups two or more workers who created the two or more identical labeling results and stores the group; The system according to claim 1 , wherein the selection unit is configured to be able to select a group including the two or more workers.

9. 1. A computer-implemented method comprising: storing one or more datasets and a plurality of labeling results associated with each of the one or more datasets, each of the plurality of labeling results being associated with a respective one of a plurality of workers; creating a sample data set from the one or more data sets; presenting the sample data set to a user; receiving, from the user, a labeling result associated with the sample dataset; extracting, from a storage unit, labeling results associated with the sample data sets for each of the plurality of workers; calculating a similarity between the labeling result associated with the sample data set by the user and each of the labeling results associated with the sample data set by each of the plurality of workers; selecting one or more workers from the plurality of workers based on a result of the comparison; and and outputting at least one of information about the one or more workers, information about one or more past training data created by each of the one or more workers, and information about one or more trained models that have learned at least one of the one or more past training data.

10. A program, The program causes a computer to store one or more datasets and a plurality of labeling results associated with each of the one or more datasets, each of the plurality of labeling results being associated with a respective one of a plurality of workers, and the program further comprises: creating a sample data set from the one or more data sets; presenting the sample data set to a user; receiving, from the user, a labeling result associated with the sample dataset; extracting, from a storage unit, labeling results associated with the sample data sets for each of the plurality of workers; calculating a similarity between the labeling result associated with the sample data set by the user and each of the labeling results associated with the sample data set by each of the plurality of workers; selecting one or more workers from the plurality of workers based on a result of the comparison; and A program that causes the computer to execute the following steps: output at least one of information about the one or more workers, information about one or more past teacher data created by each of the one or more workers, and information about one or more trained models that have learned at least one of the one or more past teacher data.

Images