Annotation device, annotation method, and annotation program

Abstract

To perform an annotation at a lower cost and with high accuracy in supervised learning in machine learning.SOLUTION: An annotation device 10 includes an acquisition unit 15a for acquiring first learning data to be used by machine learning, a first distribution unit 15b for distributing the first learning data acquired by the acquisition unit 15a to a plurality of annotators, a classification unit 15c for classifying the first learning data on the basis of the degree of confidence of a first correct answer label imparted to the first learning data by each annotator, and a second distribution unit 15e for distributing a classification result of the first learning data classified by the classification unit 15c.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an annotation device, an annotation method, and an annotation program. [Background technology]

[0002] Traditionally, supervised learning in machine learning requires training data and corresponding correct labels. In many studies, multiple people view the data and assign metadata (annotation).

[0003] For example, when annotating audio or video, a worker (sometimes called an "annotator") listens to a few to several tens of seconds of audio or video and adds metadata to meet the specifications. Specifically, if the annotation is for research and development of emotion recognition from audio, the worker selects the most appropriate emotion for the audio heard, and if the annotation is for object detection or recognition in an image, the worker selects the area of ​​the object in the image and adds a description of the object.

[0004] Conventional annotation methods can be divided into those that have a comparison target for the task and those that do not. When there is no comparison target, annotators view still images or a few seconds of audio or video and then add metadata. This method has a low time cost because the number of times data is viewed equals the total number of samples, N. It can also accurately annotate tasks that can be reliably understood by anyone even in a short amount of time (e.g., transcription, tagging objects, or clearly angry situations that anyone can see or hear).

[0005] On the other hand, when there is something to compare, annotators listen to audio or video for a long period of time (tens of seconds to several minutes) and assign metadata about continuous and relative changes in events (see, for example, Non-Patent Document 3), or they listen to multiple audio or video clips and assign relative rankings or scores (see, for example, Non-Patent Document 4). This method, because there is something to compare, reduces inconsistencies between annotators and allows for more accurate metadata to be assigned. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Mohammad Soleymani, and Martha Larson, “Crowdsourcing for Affective Annotation of Video: Development of a Viewer-reported Boredom Corpus”, 2010. [Non-patent document 2] Ryutaro Tanno, Ardavan Saeedi, Swami Sankaranarayanan, Daniel C. Alexander, and Nathan Silberman, “Learning From Noisy Labels By Regularized Estimation Of Annotator Confusion”, 2019. [Non-patent document 3] David Melhart, Antonios Liapis, and Georgios N. Yannakakis “PAGAN: Video Affect Annotation Made Easy”, 2019. [Non-patent document 4] Lifang Yang, and Rui Zhu, “Subjective Evaluation of Cooling Fan Sound based on Grade Scoring and Paired Comparison”, 2016. Summary of the Invention [Problem to be solved by the invention]

[0007] However, the above-mentioned conventional techniques cannot perform low-cost and highly accurate annotation in supervised learning in machine learning because annotation methods without a comparison target cannot perform accurate annotation for tasks that are difficult to understand in a short viewing period, and there are problems such as large variations in the labels assigned by annotators, which reduces the reliability of the annotation results.

[0008] To address this issue, measures have been taken, such as taking into consideration the quality of the annotator, answer trends, and reducing the impact of noise through annotation by multiple people, but these have not fundamentally resolved the issue of increasing the reliability of the annotation itself (see, for example, Non-Patent Documents 1 and 2). For example, when annotating the degree of concentration, if the person is very focused or not at all focused, in other words, if it is clear to anyone who sees or hears, then the votes of multiple annotators are likely to match, but if it is difficult to tell whether they are focused or not, accurate annotation is difficult, and as a result, subtle differences cannot be expressed.

[0009] On the other hand, annotation methods without a comparison object require viewing a long time or a large amount of data, which results in a huge cost for annotation. For example, when viewing several combinations of data simultaneously, selecting n items from the total N sample data can result in a maximum of N C n It may be possible to reduce the number of combinations while maintaining annotation quality by referring to psychological experiments, but careful consideration is required to determine which combinations to exclude. [Means for solving the problem]

[0010] In order to solve the above-mentioned problems and achieve the object, the annotation device of the present invention is characterized by comprising: an acquisition unit that acquires first training data to be used for machine learning; a first distribution unit that distributes the first training data acquired by the acquisition unit to a plurality of annotators; a classification unit that classifies the first training data based on the reliability of first correct labels assigned to the first training data by each annotator; and a second distribution unit that distributes the classification results of the first training data classified by the classification unit.

[0011] Furthermore, an annotation method according to the present invention is an annotation method executed by an annotation device, and is characterized by including: an acquisition step of acquiring first training data to be used for machine learning; a first distribution step of distributing the first training data acquired by the acquisition step to a plurality of annotators; a classification step of classifying the first training data based on the reliability of first correct labels assigned to the first training data by each annotator; and a second distribution step of distributing the classification results of the first training data classified by the classification step.

[0012] Furthermore, the annotation program of the present invention is characterized in that it causes a computer to execute the following steps: an acquisition step of acquiring first training data to be used in machine learning; a first distribution step of distributing the first training data acquired by the acquisition step to a plurality of annotators; a classification step of classifying the first training data based on the reliability of first correct labels assigned to the first training data by each annotator; and a second distribution step of distributing the classification results of the first training data classified by the classification step. [Effects of the Invention]

[0013] The present invention enables annotation to be performed at lower cost and with higher accuracy in supervised learning in machine learning. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an annotation system according to the first embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the configuration of the annotation device according to the first embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of learning data according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of first training data and first correct labels according to the first embodiment. [Figure 5] FIG. 5 is a diagram showing an example of second learning data and second correct labels according to the first embodiment. [Figure 6] FIG. 6 is a flowchart showing an example of the flow of the annotation process according to the first embodiment. [Figure 7] FIG. 7 is a flowchart showing an example of the flow of the first learning data classification process according to the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating a computer that executes a program. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, an annotation device, an annotation method, and an annotation program according to the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to the following embodiments.

[0016] [First embodiment] The configuration of the annotation system according to this embodiment, the configuration of the annotation device, a specific example of the annotation process, the flow of the annotation process, and the flow of the data classification process will be described below in this order, and finally the effects of this embodiment will be described.

[0017] [Annotation system configuration] The configuration of an annotation system 100 according to this embodiment (referred to as "this system" where appropriate) will be described in detail using Figure 1. Figure 1 is a diagram showing an example of an annotation system according to a first embodiment. The annotation system 100 has an annotation device 10 such as a server, annotators 20 (20A, 20B, 20C) such as various terminals, and various databases 30 (30A, 30B, 30C).

[0018] The annotation device 10, the annotator 20, and the database 30 are communicably connected via a predetermined communication network (not shown) by wire or wirelessly. The annotation system 100 shown in Fig. 1 may include multiple annotation devices 10.

[0019] First, the annotation device 10 acquires learning data necessary for research and development as first learning data from various databases 30 (step S1). Here, the acquired learning data is data such as audio, images, and videos, and is acquired in a medium and on a scale appropriate for the purpose of the research and development.

[0020] Next, the annotation device 10 distributes the acquired first training data to the annotator 20 (step S2). Here, the annotator 20 is a terminal and a user of the terminal that assigns correct labels to the distributed training data, respectively, but is not particularly limited thereto. The annotator 20 may also be a machine learning model that can assign specific correct labels that have been created separately.

[0021] Next, the annotator 20 assigns a correct label (first correct label) to the distributed first training data (step S3). The annotation device 10 also acquires the first training data to which the correct label has been assigned (step S4).

[0022] Thereafter, the annotation device 10 classifies the first training data based on the first correct label (step S5). At this time, the annotation device 10 selects training data to which reliable correct data has been assigned as a reference point (appropriately referred to as "reference data") S based on the answer acquired from the annotator 20. The annotation device 10 further classifies the training data other than the reference point S into data to which it is easy to assign an accurate correct label (appropriately referred to as "data D") and data to which it is difficult to assign an accurate correct label (appropriately referred to as "data E").

[0023] Furthermore, the annotation device 10 generates second learning data from the classified first learning data (step S6). At this time, the annotation device 10 generates a data group including the reference point S, data E, and data D, which have the same source. Note that the classification of the first learning data and the generation of the second learning data will be described later.

[0024] The annotation device 10 then distributes the generated second learning data to the annotator 20 (step S7). At this time, when distributing the data group of the second learning data, the annotation device 10 distributes each piece of data to the annotator 20 so that the user views the reference point S first, followed by data E and data D. The annotator 20 also assigns a correct label (second correct label) to the distributed second learning data (step S8). Finally, the annotation device 10 acquires the second learning data to which the correct label has been assigned (step S9).

[0025] In the annotation system 100 according to this embodiment, the annotation device 10 includes reliable data for events to which a correct label is to be assigned in a data group and indicates it explicitly, allowing the annotator 20 to use the data as a comparison target, thereby achieving more accurate annotation.

[0026] [Configuration of annotation device] The configuration of the annotation device 10 according to this embodiment will be described in detail with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the configuration of the annotation device according to this embodiment. The annotation device 10 includes an input unit 11, an output unit 12, a communication unit 13, a storage unit 14, and a control unit 15.

[0027] The input unit 11 controls the input of various information to the annotation device 10. The input unit 11 is, for example, a mouse or a keyboard, and accepts input of setting information and the like to the annotation device 10. The output unit 12 controls the output of various information from the annotation device 10. The output unit 12 is, for example, a display, and outputs setting information and the like stored in the annotation device 10.

[0028] The communication unit 13 controls data communication with other devices. For example, the communication unit 13 performs data communication with each communication device. The communication unit 13 can also perform data communication with an operator's terminal (not shown).

[0029] The storage unit 14 stores various pieces of information that the control unit 15 refers to when it operates and various pieces of information that the control unit 15 acquires when it operates. Here, the storage unit 14 is, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. Note that, in the example of Fig. 2, the storage unit 14 is installed inside the annotation device 10, but it may be installed outside the annotation device 10, or multiple storage units may be installed.

[0030] The memory unit 14 stores the first learning data acquired from the database 30 described later, the first learning data assigned with the first correct label acquired from the annotator 20, the classification results classified by the classification unit 15c of the control unit 15, the second learning data generated by the generation unit 15d, the second learning data assigned with the second correct label acquired from the annotator 20, etc., as well as information about the annotator 20, such as the user name and the identification number of the machine learning model.

[0031] The control unit 15 controls the entire annotation device 10. The control unit 15 has an acquisition unit 15a, a first distribution unit 15b, a classification unit 15c, a generation unit 15d, and a second distribution unit 15e. Here, the control unit 15 is, for example, an electronic circuit such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0032] The acquiring unit 15a acquires first training data used for machine learning. For example, the acquiring unit 15a acquires the first training data including audio, images, or videos. The acquiring unit 15a also acquires the first training data from the database 30. The acquiring unit 15a also acquires training data to which correct labels have been assigned from the annotator 20. The acquiring unit 15a then stores the first training data, the training data to which correct labels have been assigned, and the like in the storage unit 14.

[0033] The first distribution unit 15b distributes the first training data acquired by the acquisition unit 15a to a plurality of annotators 20. For example, the first distribution unit 15b distributes the first training data in a format in which a predetermined number is assigned as a first correct label. Furthermore, the first distribution unit 15b distributes the first training data to a machine learning model as an annotator 20. Note that detailed processing of the first training data and the first correct label will be described later.

[0034] The classification unit 15c classifies the first training data based on the reliability of the first correct labels assigned to the first training data by each annotator. For example, the classification unit 15c classifies the first training data into reference data, data that is easy to accurately assign a correct label to, or data that is difficult to accurately assign a correct label to, based on the variance of the first correct label as the reliability. The classification unit 15c also classifies the first training data based on the posterior probability of the first correct label as the reliability. Furthermore, the classification unit 15a stores the calculation result of the reliability of the first correct label and the classification result based on the reliability in the storage unit 14.

[0035] Here, when the annotator 20 is a human, the reliability refers to the variance of the numerical values ​​of the correct labels of each annotator for certain training data, but is not particularly limited to this. The index used for the reliability may be any index that represents the variance of the numerical values, and the smaller the variance of the numerical values, the higher the reliability of the correct labels. Furthermore, when the annotator 20 is a machine learning model, the reliability refers to the posterior probability of the numerical values ​​that are the estimation results of the machine learning model for certain training data, but is not particularly limited to this. The index used for the reliability may be any index that represents the accuracy of the estimation results of the machine learning model, and the higher the accuracy of the estimation results, the higher the reliability of the correct labels.

[0036] The generating unit 15d generates, as a classification result, second learning data that is a data group including a plurality of reference data that are reference data with different extreme values, data that are easy to accurately assign a correct label to, and data that are difficult to accurately assign a correct label to, and the data originates from the same source. Furthermore, the generating unit 15d stores the classification result of the second learning data, etc., in the storage unit 14.

[0037] Here, the extreme values ​​are, for example, the minimum number "1" and the maximum number "5" when the learning data has a format in which the degree of a specific state such as concentration is determined as a correct label using a number on a five-point scale {1, 2, 3, 4, 5}, but are not particularly limited to these. The extreme values ​​may be numbers that the annotator 20 can clearly determine as an extreme state, and are not limited to the minimum or maximum value of the range of numbers set in advance as the correct label.

[0038] The second distribution unit 15e distributes the classification results of the first learning data classified by the classification unit 15c. For example, the second distribution unit 15e first distributes multiple reference data with different extreme values. The second distribution unit 15e also distributes the classification results to multiple annotators who distributed the first learning data or to a predetermined annotator other than the multiple annotators who distributed the first learning data.

[0039] Here, the classification result refers to the first learning data classified by the classification unit 15c based on the reliability of the assigned correct labels, and is, for example, learning data labeled with three categories: reference point S (reference data), data E (data to which an accurate correct label is easily assigned), and data D (data to which an accurate correct label is difficult to assign), but is not particularly limited thereto. The classification result may be learning data labeled with the reliability of the correct label, or may be learning data selected by the generation unit 15d.

[0040] [Example of annotation processing] A specific example of the annotation process of the annotation device 10 according to this embodiment will be described with reference to Figs. 3 to 5. Fig. 3 is a diagram showing an example of training data according to the first embodiment. Fig. 4 is a diagram showing an example of first training data and a first correct label according to the first embodiment. Fig. 5 is a diagram showing an example of second training data and a second correct label according to the first embodiment.

[0041] (First annotation process) First, the first annotation process from the acquisition of first training data to the acquisition of first training data to which a first correct answer label has been assigned will be described. First, with reference to FIG. 3, the first training data acquired by the annotation device 10 from the database 30 or the like will be described. Here, the acquired first training data is data such as audio, images, and videos, and is data acquired in a medium and on a scale appropriate for the purpose of research and development. For example, when performing annotation to realize concentration level estimation from audio, the annotation device 10 acquires audio data from an audio database that stores audio data in the database 30.

[0042] Figure 3 shows a dataset X that holds voice data. The dataset X contains {x0, x1, x2, x3, x4, . . . x N 3 includes the following audio data. The audio data shown in Fig. 3 represents the audio waveform as a relationship between the passage of time and the audio signal strength.

[0043] In the following description, annotation processing using audio data as the first training data will be described, but the type of training data is not particularly limited. The first training data may be image data, video data, or a combination thereof in addition to audio data. Furthermore, the first training data may be data obtained by converting the audio data or the like into digital or text form.

[0044] Next, the first learning data that the annotation device 10 distributes to the annotator 20 and the first correct label assigned to the first learning data that the annotation device 10 acquires from the annotator 20 will be described with reference to Fig. 4. For example, when performing annotation to realize concentration level estimation from speech, the annotation device 10 sets five levels of concentration levels ("1": not concentrated, "2": slightly not concentrated, "3": neither concentrated nor unconcentrated (flat), "4": slightly concentrated, "5": concentrated) in advance, and distributes the first learning data to the annotator 20, which assigns a correct label indicating which concentration level is most appropriate for each piece of speech data held in dataset X.

[0045] When annotating to estimate the level of concentration from audio, the annotation device 10 distributes learning data for determining, on a five-point scale, whether the student to whom the question was asked was concentrating on the lesson, how concentrated, etc., based on audio data regarding questions and conversations between a teacher and a student during class. When assigning correct labels regarding the level of concentration from image data or video data, the annotation device 10 may have the annotator 20 read the facial expressions of the students from images or videos during class, and distribute learning data for determining the level of concentration.

[0046] Then, the annotation device 10 acquires the first training data to which the correct labels have been assigned by the annotator 20. In FIG. 3, the speech data “x0” to “x10” of the dataset X of “annotator 01” to “annotator 03” are N The correct labels for "ANNOT1(X)" are shown (see Figure 4 "ANNOT1(X)"). For example, the correct labels assigned by "Annotator 01" to "Annotator 03" for the audio data x0 are "2", "1", and "1", respectively.

[0047] (Second annotation process) Second, we will explain the second annotation process, from classifying the first training data assigned with the first correct label to acquiring the second training data assigned with the second correct label. First, we will explain a specific example of the classification process of the first training data based on the reliability of the correct label acquired from the annotator 20, using Figure 4. The annotation device 10 calculates the mean and variance values ​​of the correct label assigned to each speech data.

[0048] In Figure 4, x0 has a mean of 1.3 and a variance of 0.3 (small variance). Similarly, x1 has a mean of 5 and a variance of 0 (all annotators' answers match), x2 has a mean of 1 and a variance of 0 (all annotators' answers match), x3 has a mean of 3.3 and a variance of 0.3 (small variance), x4 has a mean of 4.0 and a variance of 1.0 (large variance), and xN The mean is 1.6 and the variance is 1.3 (large variance).

[0049] At this time, the annotation device 10 selects, as the reference point S, the learning data to which reliable correct answer data has been assigned from the answers acquired from the annotator 20. In the example of FIG. 4, the annotation device 10 selects x1 (extreme value "5") and x2 (extreme value "1") as the learning data for which the answers of all the annotators are consistent and the numerical values ​​of the assigned correct answer labels are extreme values.

[0050] Furthermore, the annotation device 10 further classifies the learning data other than the reference point S into data D, which is easy to accurately assign a correct label to, and data E, which is difficult to accurately assign a correct label to. For example, the annotation device 10 sets a reliability threshold, and classifies data D if the variance is 1.0 or more, and classifies data E if not. In the example of FIG. 4, the annotation device 10 classifies x0 and x3 into data E because the variance is less than 1.0, and x4 and x5 into data E. N Since the variance is greater than 1.0, it is classified as data D.

[0051] When the annotation device 10 uses the estimation results of a separately created machine learning model as the correct label, it classifies, for example, learning data whose posterior probability of the numerical value that becomes the estimation result is 80% or more as reference point S, learning data whose posterior probability is 50% or more but less than 80% as data E, and learning data whose posterior probability is less than 50% as data D. Furthermore, the annotation device 10 can statically or dynamically change the classification method, such as the number of classifications and thresholds of the learning data.

[0052] Next, a specific example of the generation and distribution process of the second training data based on the classification of the first training data will be described with reference to Fig. 5. The annotation device 10 generates a data group including three types of data, namely, a reference point S, data E, and data D, as the second training data. At this time, the reference point S is always made to include data of "1" and "5," which are its extreme values. In addition, the source of the data in each data group (speaker, person appearing in a video, object, etc.) is assumed to be the same.

[0053] For example, the annotation device 10 may generate a set of data {p0, p1, . . . p M} is generated. Here, the data set p0 contains {x0, x1, x2, x3, x4, x N} (see Figures 3 and 4) as elements, and the data group p M contains {x a ,x b ,x c ,x d ,x e ,x f} (not shown in FIGS. 3 and 4) are included as elements. In the example of FIG. 5, the annotation device 10 includes, as the data group p0, the reference point S{x1, x2}, the data E{x0, x3}, the data D{x4, x N} and select the data set p M As a reference point S{x a ,x b}, data E{x c ,x d}, data D{x e ,x f} has been selected.

[0054] The number of data in each data group and the selection method can be changed as desired, as long as they contain reference points S, which are different extreme values, and satisfy the condition of being from the same source, as described above. For example, the number of data may be a random number within a certain range. Also, the included data may be two each of data E and data D, with the average value of each annotation result being closer to "1" or "5."

[0055] Thereafter, the annotation device 10 distributes the data group selected as described above to the annotator 20 as second learning data. At this time, when distributing the data group of the second learning data, the annotation device 10 first distributes the reference point S for each data group, and then distributes data E and data D to the annotator 20. In the example of FIG. 5, when distributing the data group p0, the annotation device 10 distributes the reference point S{x1, x2}, data E{x0, x3}, and data D{x4, x N}, and the data set p M When distributing, the reference point S{x a ,x b}, data E{x c ,x d}, data D{x e ,x f} and are distributed in that order.

[0056] The annotation device 10 may instruct the annotator 20 to view the data in the order of data E and data D. The annotation device 10 may also distribute data E and data D so that they are viewed randomly. Furthermore, the annotation device 10 may present a correct label for the reference point S that is distributed first at the same time as distribution and instruct the annotator 20 not to assign a correct label, or may instruct the annotator 20 to assign a correct label to all training data regardless of the classification of the training data.

[0057] Finally, the annotation device 10 acquires the second learning data to which the correct labels have been assigned by the annotator 20. In the example of Fig. 5, the annotation device 10 acquires the correct labels of the data E and the data D excluding the reference point S for each data group for each annotator. For example, for "annotator 01", {x0, x 3, x4,x N} training data, in order {1,4 , 3,2}, and then use the data set p M {x c ,x d ,x e ,x f} training data, in order {2,4 , The correct label {3,3} is obtained (see Figure 5 "ANNOT2(X)").

[0058] The final processing of the correct labels assigned to the second training data is not particularly limited. The annotation device 10 may take a majority vote for each training data and determine the most common correct label as the final correct label, or may calculate the average score of the numerical values ​​and determine the numerical value as the final correct label.

[0059] [Annotation processing flow] The flow of annotation processing according to this embodiment will be described in detail with reference to Fig. 6. Fig. 6 is a flowchart showing an example of the flow of annotation processing according to the first embodiment.

[0060] First, the acquisition unit 15a of the annotation device 10 acquires first learning data including audio, images, video, etc. from the database 30 or the like (step S101). At this time, the acquisition unit 15a may acquire the first learning data from the storage unit 14. The acquisition unit 15a may also process the original data, such as audio data, acquired from the database 30 or the storage unit 14, and divide the data into appropriate sizes for learning data or classify them appropriately. Furthermore, the acquisition unit 15a may acquire audio data, etc. from the outside via the input unit 11.

[0061] Next, the first distribution unit 15b distributes the first training data to the annotator 20 (step S102). At this time, the first distribution unit 15b may select an annotator 20 to distribute the first training data according to the first training data. Furthermore, the acquisition unit 15a acquires the first training data to which the first correct label has been assigned by the annotator 20 (step S103).

[0062] The classification unit 15c then classifies the first training data based on the reliability of the first correct label (step S104). The generation unit 15d then generates second training data from the classified first training data (step S105). Subsequently, the second distribution unit 15e distributes the second training data to the annotator 20 (step S106).

[0063] The second distribution unit 15e can also distribute the second learning data to an annotator other than the annotator 20 that distributed the first learning data. For example, the second distribution unit 15e can distribute the first learning data to a human annotator and distribute the second learning data to an annotator that is a machine learning model.

[0064] Finally, the acquiring unit 15a acquires the second learning data to which the second correct label has been assigned by the annotator 20 (step S107), and the process ends. Note that if the accuracy of the acquired second correct label is not sufficient, the processes of steps S104 to S107 may be performed again.

[0065] [Flow of classification process for the first training data] The flow of the classification process for the first learning data according to this embodiment will be described in detail with reference to Fig. 7. Fig. 7 is a flowchart showing an example of the flow of the classification process for the first learning data according to the first embodiment. First, the acquisition unit 15a of the annotation device 10 acquires the first correct label assigned to the first learning data from the annotator 20 (step S201). Next, if the annotator 20 is a human (step S202: annotator is a human), the classification unit 15c performs classification processes of steps S208 to S210 based on the processes of steps S203 to S205.

[0066] If the answers of all the annotators match (step S203: YES) and the answers are extreme values ​​(step S204: YES), the classification unit 15c classifies the first learning data to which the correct label has been assigned as the reference point S (step S208). Furthermore, if the answers of the annotators 20 include an answer that does not match (step S203: NO) or if the answer of the annotator 20 is not an extreme value (step S204: NO), the classification unit 15c performs the process of step S205.

[0067] If the variance of the answer of the annotator 20 is 1.0 or more (step S205: Yes), the classification unit 15c classifies the first learning data to which the correct label has been assigned as data E (step S209). If the variance of the answer of the annotator 20 is less than 1.0 (step S205: No), the classification unit 15c classifies the first learning data to which the correct label has been assigned as data D (step S210). When the classification process of steps S208 to S210 is completed, the classification unit 15c ends the process.

[0068] On the other hand, when the annotator 20 is a machine learning model (step S202: annotator is a machine learning model), the classification unit 15c performs classification processing of steps S208 to S210 based on the processing of steps S206 to S207. When the posterior probability of the value that is the estimation result of the annotator 20 is 80% or more (step S206: Yes), the classification unit 15c classifies the first learning data to which the correct label has been assigned into the reference point S (step S208).

[0069] Furthermore, if the posterior probability of the value that is the estimation result of the annotator 20 is less than 80% (step S206: No) and the posterior probability is 50% or more (step S207: Yes), the classifying unit 15c classifies the first learning data to which the correct label has been assigned as data E (step S209). Furthermore, if the posterior probability of the value that is the estimation result of the annotator 20 is less than 50% (step S207: No), the classifying unit 15c classifies the first learning data to which the correct label has been assigned as data D (step S210). When the classification process of steps S208 to S210 is completed, the classifying unit 15c ends the process.

[0070] [Effects of the first embodiment] First, in the annotation process according to the present embodiment, first training data used in machine learning is acquired, the acquired first training data is distributed to multiple annotators, the first training data is classified based on the reliability of the first correct answer labels assigned to the first training data by each annotator, and the classification results of the classified first training data are distributed. Therefore, this process enables annotation to be performed at lower cost and with higher accuracy in supervised learning in machine learning.

[0071] Second, in the annotation process according to the present embodiment, first training data including audio, images, or video is acquired, the first training data is distributed in a format in which a predetermined number is assigned as a first correct label, and the first training data is classified into reference data, data that is easy to accurately assign a correct label, or data that is difficult to accurately assign a correct label based on the variance of the first correct label as a reliability measure. Therefore, in supervised learning in machine learning, this process enables the assignment of highly reliable correct labels even when there is no comparison target, enabling annotation to be performed at lower cost and with higher accuracy.

[0072] Third, in the annotation process according to the present embodiment, the annotator distributes first training data to the machine learning model and classifies the first training data based on the posterior probability of the first correct label as the reliability. Therefore, in supervised learning in machine learning, this process enables highly reliable correct labels to be assigned even if the annotator is not human, enabling annotation to be performed at lower cost and with higher accuracy.

[0073] Fourth, in the annotation process according to the present embodiment described above, as a classification result, second training data is generated as a group of data that includes multiple reference data with different extreme values, data that is easy to accurately label, and data that is difficult to accurately label, and the multiple reference data are distributed first. Therefore, in supervised learning in machine learning, this process enables reliable and efficient labeling of data even when there is no comparison target, enabling annotation to be performed at lower cost and with higher accuracy.

[0074] Fifth, in the annotation process according to the present embodiment described above, classification results are distributed to the multiple annotators who distributed the first training data, or to a predetermined annotator other than the multiple annotators who distributed the first training data. Therefore, in supervised learning in machine learning, this process enables reliable, efficient, and more flexible assignment of correct labels even when there is no comparison target, enabling annotation to be performed at lower cost and with higher accuracy.

[0075] [System configuration, etc.] The components of each device shown in the drawings according to the above embodiments are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of each device can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.

[0076] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0077] 〔program〕 It is also possible to create a program in which the processes executed by the annotation device 10 described in the above embodiment are written in a language executable by a computer. In this case, the same effects as those of the above embodiment can be obtained by having a computer execute the program. Furthermore, such a program may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read and executed by a computer to realize the same processes as those of the above embodiment.

[0078] Fig. 8 is a diagram showing a computer that executes a program. As shown in Fig. 8, a computer 1000 includes, for example, a memory 1010, a CPU (Central Processing Unit) 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070, and these components are connected by a bus 1080.

[0079] As shown in FIG. 8, the memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090 as shown in FIG. 8. The disk drive interface 1040 is connected to a disk drive 1100 as shown in FIG. 8. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120 as shown in FIG. 8. The video adapter 1060 is connected to a display 1130 as shown in FIG. 8.

[0080] 8, the hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the above programs are stored, for example, on the hard disk drive 1090 as program modules in which instructions to be executed by the computer 1000 are written.

[0081] The various data described in the above embodiment are stored as program data, for example, in the memory 1010 or the hard disk drive 1090. The CPU 1020 then reads the program module 1093 and the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as needed, and executes various processing procedures.

[0082] Note that the program module 1093 and program data 1094 related to the program are not limited to being stored in the hard disk drive 1090, and may be stored in, for example, a removable storage medium and read by the CPU 1020 via a disk drive or the like. Alternatively, the program module 1093 and program data 1094 related to the program may be stored in another computer connected via a network (such as a LAN (Local Area Network) or WAN (Wide Area Network)) and read by the CPU 1020 via the network interface 1070.

[0083] The above-described embodiments and their modifications are included in the technology disclosed in this application, as well as in the scope of the invention described in the claims and their equivalents. [Explanation of symbols]

[0084] 10 Annotation Device 11 Input section 12 Output section 13 Communications Department 14 Storage section 15 Control Unit 15a Acquisition part 15b 1st Distribution Section 15c Classification section 15d Generator 15e 2nd Distribution Department 20, 20A, 20B, 20C Annotator 30, 30A, 30B, 30C database 100 Annotation System

Claims

1. a classification unit that classifies first training data used in machine learning based on the reliability of first correct labels assigned by a plurality of annotators to the first training data; a generation unit that generates second training data including a classification result of the first training data; a distribution unit that distributes the generated second training data to the annotator; an acquisition unit that acquires the second training data to which the second correct answer label has been assigned by the annotator; An annotation device comprising:

2. The annotation device according to claim 1 , wherein the reliability represents a variance of the first correct label.

3. The annotation device according to claim 1 , wherein the classification result includes reference data selected based on the reliability.

4. 2. The annotation device according to claim 1, wherein the classification result includes classification of the first training data into data that is easy to accurately assign a correct label to, or data that is difficult to accurately assign a correct label to, based on the reliability.

5. The annotation device according to claim 3 , wherein the classification result includes a plurality of the reference data having different extreme values.

6. The annotation device described in Claim 1, characterized in that the generation unit generates the second learning data which includes the classification result and is a group of data in which the source of each piece of data included in the classification result is the same.

7. 6. The annotation device according to claim 3, wherein the distribution unit distributes the reference data first.

8. An annotation device described in any one of claims 1 to 7, characterized in that the distribution unit distributes the generated second learning data to the multiple annotators or a specified annotator other than the multiple annotators.

9. An annotation method performed by an annotation device, comprising: a classification step of classifying first training data used in machine learning based on the reliability of first correct labels assigned by a plurality of annotators to the first training data; a generating step of generating second training data including a classification result of the first training data; a distribution step of distributing the generated second training data to the annotator; an acquisition step of acquiring the second training data to which the second correct answer label has been assigned by the annotator; An annotation method comprising:

10. a classification step of classifying first training data used for machine learning based on the reliability of first correct labels assigned by a plurality of annotators to the first training data; a generating step of generating second training data including a classification result of the first training data; a distribution step of distributing the generated second training data to the annotator; an acquisition step of acquiring the second training data to which the second correct answer label has been assigned by the annotator; An annotation program characterized by causing a computer to execute the above.

Images