Annotation device, annotation method, and program

Abstract

To provide an annotation device, an annotation method, and a program that can shorten annotation work time.SOLUTION: An annotation device 8 causes workers to annotate a work object output to an output device 4, and creates teaching data for use in supervised machine learning. A calculation unit 17 causes multiple workers to perform preliminary annotation and calculates the accuracy of the annotation. An annotation processing unit 21 causes multiple workers to perform main annotation for one work object. An integration unit 22 integrates annotation results of the multiple workers based on their respective accuracy. An accumulation processing unit 23 stores and accumulates final annotation result Pt obtained by the integration unit 22 in a storage device 5 as teaching data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an annotation device, an annotation method, and a program for creating training data for supervised machine learning by adding annotation information to a work object. [Background technology]

[0002] A conventional type of machine learning is supervised machine learning, in which artificial intelligence learns based on training data provided by humans. Supervised machine learning requires an operator to create training data in advance, known as annotation. Annotation is the process in which an operator manually adds information (labels) to each piece of data that will be used as the training data.

[0003] Conventionally, an annotation device that adds annotation information to video data to create training data used in machine learning of a trained model has been well known, as disclosed in Patent Document 1. This annotation device identifies the positions of objects included in the video and adds, as annotation information, object labels that indicate the types of objects, as well as relationship labels that indicate the types of phenomena with which the objects are correlated. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-56744 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when supervised machine learning is performed using training data created through annotation, errors in the annotation results can affect the learning accuracy. For this reason, annotation results obtained through annotation must be checked by a reviewer to ensure there are no errors. However, because the annotation results must be checked manually by a reviewer, there was a concern that the checking process would take a long time. As a result, the annotation process tended to take a long time. [Means for solving the problem]

[0006] The annotation device that solves the above problem is a device that creates teacher data to be used in supervised machine learning by having a worker perform annotation for each work object that is output to an output device using an input device to add annotation information to the work object, and is equipped with: a calculation unit that has each of the multiple workers perform preliminary annotation and calculates the accuracy of the annotation; an annotation processing unit that has the multiple workers perform actual annotation for one of the work objects; an integration unit that integrates the annotation results of each of the multiple workers based on the accuracy of each of the multiple workers; and a storage processing unit that stores and accumulates the final annotation result obtained by the integration unit in a storage device as the teacher data.

[0007] The annotation method that solves the above problem is a method of creating teacher data to be used in supervised machine learning by having a worker perform annotation for each work object that is output to an output device using an input device to add annotation information to the work object, and includes the steps of having each of the multiple workers perform preliminary annotation to calculate the accuracy of the annotation, having the multiple workers perform actual annotation for one of the work objects, creating a final annotation result by integrating the annotation results of each of the multiple workers based on the accuracy of each of the multiple workers, and storing and accumulating the final annotation result in a storage device as the teacher data.

[0008] A program that solves the above problem has a computer used in an annotation device that has a worker perform annotation for each work object output to an output device using an input device to add annotation information to the work object, and creates teacher data to be used in supervised machine learning.The computer has the computer perform the following operations: have each of the multiple workers perform preliminary annotation to calculate the accuracy of the annotation; have the multiple workers perform actual annotation for one of the work objects; integrate the annotation results of each of the multiple workers based on the accuracy of each of the multiple workers; and store and accumulate the final annotation result obtained by the integration in a storage device as the teacher data. [Effects of the Invention]

[0009] The present invention can reduce the time required for annotation work. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a configuration diagram of an annotation device according to an embodiment. [Figure 2] FIG. 10 is a diagram showing an annotation screen. [Figure 3] FIG. 1 is an explanatory diagram showing the workflow of annotation. [Figure 4] 10 is a flowchart showing a process flow when calculating accuracy. [Figure 5] FIG. 1 is an explanatory diagram showing a conventional annotation workflow. [Figure 6] 10 is a flowchart showing the flow of actual annotation processing. DETAILED DESCRIPTION OF THE INVENTION

[0011] An embodiment of the present disclosure will be described below. (Work System 1) As shown in FIG. 1, the operation system 1 used in annotation includes a processing device 2, an input device 3, an output device 4, and a storage device 5. The operation system 1 is, for example, a personal computer. The processing device 2 has, for example, a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), and controls the overall operation of the operation system 1. The input device 3 is, for example, a keyboard, a mouse, a touchpad, etc. The output device 4 is, for example, a display device 6 such as a monitor. The storage device 5 is storage to which data can be written and deleted.

[0012] (Annotation device 8) As shown in Fig. 1, the annotation device 8 is configured using the resources of the work system 1. Specifically, the annotation device 8 is configured using a processing device 2, an input device 3, an output device 4, and a storage device 5. In this example, the annotation device 8 has a worker use the input device 3 to annotate each work object 9 (see Fig. 2) output from the output device 4, adding annotation information to the work object 9, thereby creating training data to be used in supervised machine learning.

[0013] As shown in FIG. 2, annotation by a worker is image annotation in which annotation information is added to an image displayed on a display device 6 using an input device 3, thereby creating training data to be used for image judgment in supervised machine learning. In this case, the work object 9 is the image displayed on the display device 6. The annotation information is, for example, a label added to the data of the work object 9 as information indicating the correct answer. The label includes, for example, a tag indicating meaning or attribute. The worker performs annotation on a large number of images required for supervised machine learning, for example.

[0014] Image annotation includes object detection, region extraction, polygonal region designation, landmark detection, and image classification. Object detection is, for example, the task of tagging each object displayed in an image or video by surrounding it with a rectangle. Region extraction is, for example, the task of extracting a specific object from an image or video. Polygonal region designation is, for example, the task of tagging an object displayed in an image or video by surrounding it with a polygon. Landmark detection is, for example, the task of identifying characteristic positions in an image or video using points. Image classification is, for example, the task of tagging a single image.

[0015] The annotation is not limited to an image annotation, but may be, for example, an audio annotation or a text annotation. Audio annotation is the process of adding tags and additional information to audio data. Text annotation is the process of adding tags and additional information to text data such as sentences and comments.

[0016] (Annotation work) 2, in the case of image annotation, the worker performs annotation by displaying annotation screen 12 on display device 6 and inputting annotation information into annotation screen 12 using input device 3. In the case of image annotation, annotation screen 12 has image display area 13 that displays the image to be annotated, and an input area 14 for annotation information.

[0017] In the case of object detection, the worker surrounds the image to be annotated with a square frame 15 and specifies the annotation class (pedestrian, bicycle, etc.). The worker performs this task for all objects displayed in the image, thereby tagging each object. The worker performs this task for each of multiple images as annotations.

[0018] (Calculation of annotation worker accuracy W) 1, the annotation device 8 includes a calculation unit 17 that has each of a plurality of workers perform preliminary annotation and calculates the accuracy W of the annotation. The annotation device 8 also includes a supervised data registration unit 19 that registers, in the storage device 5, annotation results without annotation errors that have been created in advance by an administrator for a specific work target 18. The calculation unit 17 and the supervised data registration unit 19 are constructed by, for example, the computer of the processing device 2.

[0019] When the annotation work object 9 is an image, the specific work object 18 is, for example, a predetermined number of images (specific images) extracted from a large number of images prepared as annotation objects. The correct answer data Dk is, for example, an annotation result with no annotation error or with little annotation error.

[0020] The calculation unit 17 has a plurality of workers perform pre-annotation for the specific work target 18 of the supervised data Dk registered by the supervised data registration unit 19, and calculates the accuracy W of each of the plurality of workers based on the annotation results of the pre-annotation. The calculation unit 17 writes and stores the calculated accuracy W of each worker in a predetermined area of ​​the storage device 5.

[0021] (Integration of annotation results from each worker) 1, the annotation device 8 includes an annotation processing unit 21, an integration unit 22, and an accumulation processing unit 23. The annotation processing unit 21, the integration unit 22, and the accumulation processing unit 23 are constructed, for example, by a computer of the processing device 2. The actual annotation is performed, for example, on the actual work object 24 (see FIG. 2), which is the remainder after subtracting the specific work object 18 from the work object 9.

[0022] The annotation processing unit 21 has multiple workers perform actual annotation on one image. In this example, the annotation processing unit 21 has each of the multiple workers perform annotation on the actual work object 24 prepared for the actual annotation. Note that since each worker has a different annotation skill (i.e., accuracy W), the annotation results of each worker on the actual work object 24 also differ.

[0023] The integration unit 22 integrates the annotation results of each of the multiple workers based on the accuracy W of each of the multiple workers. The integration unit 22 adds up the accuracy W of each worker each time an annotation by each worker is completed for the same image, and integrates the annotation results when the total value (score S) of the accuracy W exceeds a specified value Sth.

[0024] The accumulation processing unit 23 stores and accumulates the final annotation result Pt obtained by the integration unit 22 as training data in the storage device 5. Every time a final annotation result Pt is created, the accumulation processing unit 23 writes and holds the final annotation result Pt in the storage device 5.

[0025] (Additional function of the correct answer data Dk) 1, the supervised data registration unit 19 adds, to the storage device 5, the final annotation results Pt obtained by the integration unit 22 that satisfy conditions that make them reliable, as new supervised data Dk. In this way, the supervised data registration unit 19 automatically adds predetermined final annotation results Pt as supervised data Dk while performing the actual annotation. In this example, the conditions that make them reliable are, for example, conditions that make it possible to select supervised data Dk that is suitable for assessing the annotation skill of a worker.

[0026] (Update function of worker accuracy W) 1, the annotation device 8 includes an update unit 26 that updates the accuracy W of each worker calculated during preliminary annotation. In this example, the update unit 26 updates the accuracy W of each of the multiple workers using the annotation results of the actual annotation of each worker and a final annotation result Pt obtained by integrating the annotation results of the actual annotation. The update unit 26, for example, newly calculates the annotation error from the results of the preliminary and actual annotation, and recalculates the accuracy W from the error.

[0027] Next, the operation of the annotation device 8 (annotation method, program) of this embodiment will be described. (Annotation flow) As shown in FIG. 3, annotation tasks are broadly divided into five tasks: "Task 1," "Task 2," "Task 3," "Task 4," and "Task 5." "Task 1" is a task in which a person with high annotation skills, such as an administrator, creates correct answer data Dk without annotation errors. Specifically, the administrator extracts a predetermined number of task targets 9 from multiple task targets 9 as specific task targets 18, performs annotation on the data of these specific task targets 18 (hereinafter referred to as specific data Da), and creates correct answer data Dk without annotation errors. The correct answer data registration unit 19 registers the correct answer data Dk created by the administrator in the storage device 5.

[0028] "Task 2" is a task in which each worker performs annotation on specific data Da, and the accuracy W of each worker's annotation is calculated based on the annotation results. In this example, the calculation unit 17 has each worker perform annotation on the multiple specific data Da for which correct answer data Dk was created in "Task 1", and calculates the accuracy W of each worker's annotation based on the annotation results. The calculation unit 17 registers the calculated accuracy W of each worker's annotation in the storage device 5.

[0029] "Task 3" is a task in which each worker performs annotation for each of the multiple production work objects 24, and the final annotation results Pt are stored as training data in the storage device 5. The production work objects 24 are, for example, the remaining work objects 9 from the many work objects 9 originally prepared, excluding the specific work object 18 used to calculate the accuracy W. Each worker performs annotation for each piece of data in the production work objects 24 (hereinafter referred to as production data Db).

[0030] The integrating unit 22 integrates the annotation results of each worker to obtain a final annotation result Pt. In this example, the integrating unit 22 integrates the annotation results of each worker by calculating a weighted average of the annotation results of each worker. The integration of the annotation results of each worker is performed for each production work target 24. The accumulation processing unit 23 registers the final annotation result Pt obtained by the integrating unit 22 in the storage device 5 as training data.

[0031] "Task 4" is a task of adding, to the supervised answer data Dk, the final annotation results Pt obtained in "Task 3" that are suitable for assessing the annotation skills of the worker. In this example, each time the supervised answer data registration unit 19 finds a final annotation result Pt that is suitable for assessing the annotation skills of the worker, it adds that final annotation result Pt to the storage device 5 as the supervised answer data Dk.

[0032] "Task 5" is a task of updating the accuracy W of each worker's annotation based on the results of the annotation of the actual work objects 24 performed by each worker. The update unit 26 periodically updates the accuracy W, for example, every time annotation of a predetermined number of work objects 9 is completed in the actual annotation.

[0033] (Calculation of each worker's accuracy W) 4 is a flowchart showing the procedure for calculating the accuracy W of each worker's annotation. The calculation of the accuracy W of each worker is performed, for example, after annotation of the specific data Da for all workers is completed.

[0034] In step 101, the calculation unit 17 has each of the multiple workers perform annotation on the multiple prepared specific work objects 18 and acquires the annotation results. If the work objects 9 are images, the calculation unit 17 has each worker perform annotation on a predetermined number of images (e.g., 100 images). Then, the calculation unit 17 acquires the annotation results for the specific work objects 18 from each worker.

[0035] In step 102, the calculation unit 17 calculates the annotation error for each of the multiple workers. The annotation error is, for example, a value corresponding to the difference between the annotation results for the multiple specific work targets 18 and the correct answer data Dk, and is calculated for each worker.

[0036] In step 103, the calculation unit 17 calculates the accuracy W of the annotations of each worker based on the annotation errors calculated in step 102. The accuracy W may be, for example, a score expressed as a positive number. Then, the calculation unit 17 writes and registers the calculated accuracy W of each worker in the storage device 5.

[0037] (Production data DB annotation) 6 is a flowchart showing the procedure for annotating the production data Db. This flowchart is executed, for example, for each production work object 24. In other words, if the production work object 24 is an image, the processing of the flowchart is executed for each image.

[0038] In step 201, the integration unit 22 sets the score S, which is a variable that changes by adding up the accuracy W of each worker, to "0," and sets the variable i, which indicates the number of the worker, to "1." The score S increases by sequentially adding up the accuracy W of the workers who performed the annotation.

[0039] In step 202, the integration unit 22 acquires the annotation result for the annotation of the production data Db by the i-th worker. In step 203, the integrating unit 22 updates the score S. Specifically, the integrating unit 22 calculates the score S by adding the accuracy W of each worker up to the i-th worker.

[0040] In step 204, the integration unit 22 determines whether the score S exceeds a specified value Sth (whether S>Sth holds). The specified value Sth is set, for example, to a value that cannot be reached unless there are a specified number of workers with high annotation skills among the workers. If the score S exceeds the specified value Sth, it can be determined that the total value of the accuracy W has reached a value sufficient to integrate the annotation results. In step 204, if the score S does not exceed the specified value Sth, the process proceeds to step 205, and if the score S exceeds the specified value Sth, the process proceeds to step 206.

[0041] If the score S does not exceed the specified value Sth, the integrating unit 22 increments the variable i in step 205. That is, if the score S does not exceed the specified value Sth, the integrating unit 22 updates the variable i to obtain the annotation result of the next worker. After the variable i is incremented, the processing of steps 202 to 204 is executed again.

[0042] Then, the processes of steps 202 to 205 are repeated until the score S exceeds the specified value Sth. That is, the summation of the accuracy W is repeated until the total value of the accuracy W reaches a value that ensures the reliability of the integration.

[0043] In step 206, the integrating unit 22 calculates the final annotation result Pt by integrating the annotation results of multiple workers based on the accuracy W of each worker. In this example, the integrating unit 22 integrates the annotation results of each worker, placing emphasis on the annotation results of workers with high accuracy W. The integration is performed using a calculation method in which the weighting of the value increases as the worker's accuracy W increases, for example, using a weighted average.

[0044] In step 207, the supervised data registration unit 19 determines whether or not to add the final annotation result Pt calculated by integration to the supervised data Dk. In this example, the supervised data registration unit 19 adds the final annotation result Pt to the supervised data Dk when at least one of the following conditions (1) and (2) is met: (1) The annotation results of workers with high accuracy W are close to each other. (2) There is a large difference in the annotation results between workers with high accuracy W and those with low accuracy W. If the correct answer data Dk satisfies condition (1), it can be treated as data that can be correctly annotated by a worker with high accuracy W, data that is unlikely to cause confusion in annotation, or data whose annotation is not influenced by human intuition. Also, if the correct answer data Dk satisfies condition (2), it can be treated as data that is difficult to annotate for a worker with low accuracy W. This prevents data with simple annotations from being determined to be correct for everyone when calculating the worker's accuracy W.

[0045] If it is determined in step 207 that the final annotation result Pt should be added to the supervised data Dk, the process proceeds to step 208. If it is not determined in step 207 that the final annotation result Pt should be added to the supervised data Dk, the process ends.

[0046] If it is determined that the final annotation result Pt should be added to the supervised data Dk, then in step 208, the supervised data registration unit 19 registers the final annotation result Pt in the storage device 5. As a result, the final annotation result Pt created by the annotation of the production data Db is added as the supervised data Dk, and the supervised data Dk is automatically increased.

[0047] (Accuracy W update) As shown in "Task 5" in FIG. 3, the update unit 26 periodically updates the accuracy W of the worker. In this example, when a predetermined number of annotations for the production work target 24 are completed in the production annotation, the update unit 26 overwrites the accuracy W at that timing. For example, the update unit 26 adds the latest annotation results to past annotation results, calculates the error between these sets and each correct answer data Dk, and recalculates a new accuracy W. This changes the accuracy W of each worker to the latest value corresponding to their current annotation skill.

[0048] (Compared to traditional annotation) As shown in Figure 5, conventional annotation involves a reviewer sequentially checking the annotation results output by each worker. However, since each worker has different annotation skills, the accuracy of the annotation results output by each worker also varies. As a result, the reviewer must check the annotation results, which have varying levels of accuracy, which takes time to check the results. This has raised concerns that the annotation work time may become longer.

[0049] On the other hand, in this example, the annotation results of each worker for the annotation work object 9 are integrated based on the accuracy W of each worker's annotation, and the final annotation result Pt created by this integration is generated as training data. In this way, creating training data with sufficient accuracy does not require a reviewer to manually check the annotation results. This reduces the time required for annotation work. Furthermore, because training data with minimal annotation error is automatically created, it also contributes to simplifying the annotation work.

[0050] (Effects of the embodiment) According to the annotation device 8 (annotation method, program) of the above embodiment, the following effects can be obtained.

[0051] (1) The annotation device 8 has a worker annotate each work object 9 output to the output device 4 using the input device 3 to add annotation information to the work object 9, thereby creating teacher data to be used in supervised machine learning. The annotation device 8 is provided with a calculation unit 17 that has each of multiple workers perform preliminary annotation and calculates the accuracy W of the annotation. The annotation device 8 is provided with an annotation processing unit 21 that has multiple workers perform actual annotation for one work object 9. The annotation device 8 is provided with an integration unit 22 that integrates the annotation results of each of the multiple workers based on the accuracy W of each of the multiple workers. The annotation device 8 is provided with a storage processing unit 23 that stores and accumulates the final annotation result Pt obtained by the integration unit 22 in the storage device 5 as teacher data.

[0052] According to this configuration, the annotation results of each worker are integrated based on the annotation accuracy W of each worker, and the final annotation result Pt created by this integration is stored as training data in the storage device 5. As a result, even when workers with various annotation skills perform annotations, a final annotation result Pt close to the annotation result of a worker with a high accuracy W is automatically created and stored in the storage device 5. This makes it possible to automatically create highly accurate training data without requiring a reviewer to check the annotation results created by each worker one by one. This reduces the time required for annotation work.

[0053] (2) The annotation device 8 includes a supervised data registration unit 19 that registers, in the storage device 5, annotation results without annotation errors that have been created in advance by an administrator for the specific work object 18 as supervised data Dk. The calculation unit 17 has multiple workers perform pre-annotation for the specific work object 18 for which the supervised data Dk has been registered by the supervised data registration unit 19, and calculates the accuracy W of each of the multiple workers based on the annotation results of the pre-annotation. According to this configuration, the accuracy W of each worker's annotation is calculated based on the difference from the annotation result without annotation errors. Therefore, the accuracy W of each worker's annotation can be calculated with high accuracy.

[0054] (3) The supervised data registration unit 19 adds, to the storage device 5, those final annotation results Pt obtained by the integration unit 22 that satisfy the conditions for determining that the results are reliable (the above-mentioned conditions (1) and (2)), as new supervised data Dk. This configuration makes it possible to automatically increase the supervised data Dk while performing the actual annotation. Therefore, there is no need to manually add the supervised data Dk, which saves the effort of adding the supervised data Dk.

[0055] (4) The annotation device 8 is provided with an update unit 26 that updates the accuracy W of each of the multiple workers using the annotation results of each worker's actual annotation and the final annotation result Pt obtained by integrating the annotation results of each worker's actual annotation. This configuration ensures that the annotation accuracy W of each worker is a value that is in line with their current annotation skill, thereby contributing to improving annotation precision.

[0056] (5) The integration unit 22 adds up the accuracy W of each worker each time multiple workers complete annotation for the same work object, and integrates the annotation results when the total value (score S) of the accuracy W exceeds a specified value Sth. According to this configuration, the annotation results are integrated when the total value (score S) of the accuracy W reaches a sufficient value. Therefore, the final annotation result Pt has fewer annotation errors. This can improve the annotation accuracy.

[0057] (6) The work object 9 is an image displayed on the display device 6 as the output device 4. The annotation by the worker is an image annotation that creates training data to be used for image judgment in supervised machine learning by adding annotation information to the image displayed on the display device 6 using the input device 3. This configuration can reduce the work time for image annotation.

[0058] (Other embodiments) This embodiment can be modified as follows: This embodiment and the following modifications can be combined and implemented within the scope of technical compatibility.

[0059] The specific task object 18 may be a dedicated object for calculating the accuracy W. Various types of annotation information can be applied depending on the type of annotation.

[0060] Pre-annotation is not limited to having a worker actually annotate the work object 9, but other methods may be used, such as judging based on annotation experience or track record, or conducting a test.

[0061] The integration of annotation results is not limited to the method of calculating a weighted average, and for example, a geometric mean may be used. The image as the work object 9 is not limited to a still image, but may be a video.

[0062] When the display device 6 is, for example, a touch panel, the input device 3 may be a touch sensor provided on the touch panel. The work object 9 is not limited to an image, but may be audio data or text data.

[0063] The output device 4 is not limited to the display device 6, but may be an audio output device (speaker). The calculation unit 17, the correct answer data registration unit 19, the annotation processing unit 21, the integration unit 22, the accumulation processing unit 23, and the update unit 26 may be configured as [1] one or more processors operating according to a computer program (software), or [2] a combination of such a processor and one or more dedicated hardware circuits, such as application-specific integrated circuits (ASICs), that perform at least some of the various processes. The processor includes a CPU and memory, such as RAM and ROM, that stores program code or instructions configured to cause the CPU to perform the processes. The memory (computer-readable medium) includes any available medium accessible by a general-purpose or dedicated computer. Alternatively, instead of a computer including the above processor, a processing circuit configured by one or more dedicated hardware circuits that perform all of the various processes may be used.

[0064] The calculation unit 17, the supervised data registration unit 19, the annotation processing unit 21, the integration unit 22, the accumulation processing unit 23, and the update unit 26 may be configured from independent processors, or may be constructed from a processor that shares some of its functions. In this way, the calculation unit 17, the supervised data registration unit 19, the annotation processing unit 21, the integration unit 22, the accumulation processing unit 23, and the update unit 26 are not limited to being independent functional blocks, and may be configured from a single functional block, or may be configured from a functional block that shares some of its functions.

[0065] While the present disclosure has been described with reference to the embodiments, it is understood that the present disclosure is not limited to those embodiments or structures. The present disclosure also encompasses various modifications and equivalent modifications. In addition, various combinations and forms, including only one element, more than one element, or less than one element, are also within the scope and spirit of the present disclosure. [Explanation of symbols]

[0066] 3...input device, 4...output device, 5...storage device, 6...display device, 8...annotation device, 9...work object, 17...calculation unit, 18...specific work object, 19...correct data registration unit, 21...annotation processing unit, 22...integration unit, 23...accumulation processing unit, 26...update unit, W...accuracy, Dk...correct data, Pt...final annotation result, S...score as total value, Sth...specified value.

Claims

1. An annotation device that creates teacher data to be used in supervised machine learning by having a worker annotate, for each work object output to an output device, the work object by using an input device to add annotation information to the work object, a calculation unit that causes each of the plurality of workers to perform a preliminary annotation and calculates the accuracy of the annotation; an annotation processing unit that causes the plurality of workers to perform actual annotation on one of the work objects; an integration unit that integrates the annotation results of each of the plurality of workers based on the accuracy of each of the plurality of workers; An annotation device comprising: an accumulation processing unit that stores and accumulates the final annotation results obtained by the integration unit in a storage device as the teaching data.

2. a correct data registration unit that registers an annotation result without annotation errors, which has been created in advance by an administrator for a specific work target, in the storage device as correct data; 2. The annotation device according to claim 1, wherein the calculation unit has the plurality of workers perform the pre-annotation for the specific work target of the supervised data registered by the supervised data registration unit, and calculates the accuracy of each of the plurality of workers based on the annotation results of the pre-annotation.

3. The annotation device according to claim 2 , wherein the supervised data registration unit adds, to the storage device, as new supervised data, the final annotation results obtained by the integration unit that satisfy conditions that make the final annotation results reliable.

4. 2. The annotation device according to claim 1, further comprising an update unit that updates the accuracy of each of the plurality of workers using the annotation results of the actual annotations of each worker and the final annotation result obtained by integrating the annotation results of the actual annotations of each worker.

5. 2. The annotation device according to claim 1, wherein the integration unit adds up the accuracy of each of the plurality of workers each time the workers complete annotation for the same work object, and integrates the annotation results when the sum of the accuracy exceeds a specified value.

6. the work target is an image displayed on a display device serving as the output device, 2. The annotation device according to claim 1, wherein the annotation by the worker is an image annotation that creates the training data to be used for image judgment in supervised machine learning by adding annotation information to the image displayed on the display device using the input device.

7. An annotation method for creating teacher data to be used in supervised machine learning by having a worker annotate, for each work object output to an output device, the work object by using an input device to add annotation information, the method comprising: a step of having each of the plurality of workers perform preliminary annotation and calculating the accuracy of the annotation; a procedure of having the plurality of workers perform actual annotation on one of the work objects; a step of creating a final annotation result by integrating the annotation results of each of the plurality of workers based on the accuracy of each of the plurality of workers; and a step of storing and accumulating the final annotation result as the training data in a storage device.

8. A computer used in an annotation device that creates teacher data to be used in supervised machine learning by having a worker annotate a work object output to an output device by using an input device to add annotation information to the work object, having each of the plurality of workers perform a preliminary annotation and calculating the accuracy of the annotation; having the plurality of workers perform actual annotation on one of the work objects; aggregating the annotation results of each of the plurality of workers based on the accuracy of each of the plurality of workers; and storing and accumulating the final annotation results obtained by the integration in a storage device as the training data.

Images