Information processing device, control method, and program

Abstract

To provide an information processing apparatus, a control method, and a program that detect a difference in application standard between a plurality of operators in annotation application operations and allow a warning for the difference in application standard.SOLUTION: An information processing apparatus acquires a plurality of annotations based on input from a plurality of operators to the same application target, compares the plurality of acquired annotations, and has determination means (standard mismatch determination unit 205) that determines if there is a difference in annotation application standard between the plurality of operators on the basis of the comparison, and warning means (warning display unit 206) that performs warning when the difference in application standard between the plurality of operators is determined by the determination means.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to information processing technology. [Background technology]

[0002] Classifying objects by annotating or tagging them is a common practice. Recently, annotations and tags are not only used for classification, but are increasingly being used as training data for machine learning.

[0003] In particular, when classifying data as training data for machine learning, the number of items to be tagged can be large, so the work may be carried out by multiple workers after sharing annotation and tagging standards.

[0004] As a technique for multiple workers to add annotations, there is a method as shown in Patent Document 1. This method judges the reliability of each worker from the results of annotations added by multiple people, and determines which annotation targets to assign to each worker based on the reliability. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2015-166975 A Summary of the Invention [Problem to be solved by the invention]

[0006] When multiple workers annotate, if the annotation standards vary among workers, there is a risk that different annotations will be given to the same object. If different annotations are given to the same object, when these are used as training data for machine learning, it may not be possible to generate a highly accurate trained model.

[0007] The method described in Patent Document 1 cannot detect inconsistencies or discrepancies in the annotation standards between multiple workers.

[0008] The present invention has been made to solve the above-mentioned problems, and aims to detect discrepancies in annotation standards between multiple workers in annotation work, and to enable a warning against any discrepancies in the annotation standards. [Means for solving the problem]

[0009] As a means for achieving the above object, an information processing device according to the present invention has the following arrangement.

[0010] That is, An acquisition means for acquiring a plurality of annotations based on inputs from a plurality of workers to a same annotation target; A comparison means for comparing the plurality of annotations acquired by the acquisition means; a determination means for determining whether there is a discrepancy in annotation standards between the plurality of workers based on the comparison by the comparison means; a warning means for issuing a warning when the determining means determines that there is a difference in the application criteria between the plurality of workers; The determining means determines that there is a discrepancy in the assignment criteria between the plurality of workers when a comparison result by the comparing means indicates that the entropy of the plurality of annotations is equal to or greater than a predetermined value. It is characterized by: Effect of the Invention

[0011] According to the present invention, it is possible to detect a deviation in annotation standards between a plurality of workers in the annotation work and to issue a warning against the deviation in annotation standards. [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 is a diagram illustrating a hardware configuration of an information processing device. [Diagram 2] FIG. 2 is a diagram illustrating a functional configuration of an information processing device. [Diagram 3] FIG. 11 is a diagram showing an example of a screen output by an information processing device. [Figure 4]4 is a flowchart showing the overall process of the information processing device according to the first embodiment. [Diagram 5] FIG. 13 is a diagram illustrating an example of a table for managing grant targets. [Figure 6] 11 is a diagram showing an example of a work queue in which targets to be assigned that are used when a worker performs a task are stored. FIG. [Figure 7] FIG. 13 is a diagram illustrating an example of a table storing added data added by a worker. [Figure 8] 11 is a flowchart showing a process for determining the variability of assigned data assigned by a plurality of people. [Figure 9] FIG. 11 is a diagram illustrating a functional configuration of an information processing device according to a third embodiment. [Figure 10] FIG. 13 is a diagram showing an example of a table storing the results of estimating the assignment data to be assigned to the assignment target in the third embodiment. [Figure 11] 13 is a flowchart showing a process for determining a reference confirmation target in the third embodiment. [Figure 12] FIG. 13 is a diagram showing an example of a table for managing grant targets in the third embodiment. [Figure 13] 13 is a flowchart showing a process for determining the number of workers working on the same target to be assigned in the third embodiment. [Figure 14] FIG. 13 is a diagram showing an example of a work queue in which targets to be assigned that are used when a worker performs work in the third embodiment are stored. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the present invention will be described in detail based on preferred embodiments thereof with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the configurations shown in the drawings.

[0014] <Embodiment 1> FIG. 1 is a diagram showing a hardware configuration of an information processing device 100 in this embodiment. The information processing device 100 described in the following embodiment is a computer device having the configuration shown in the block diagram in FIG. 1. In addition, the information processing device 100 will be described as a single device hereinafter, but is not limited to this. That is, the methods in each embodiment may be realized by a single computer device, or each function may be distributed and realized in multiple computer devices as necessary. When configured by multiple computer devices, they are connected by a Local Area Network (LAN) or the like so as to be able to communicate with each other.

[0015] 1, reference numeral 101 denotes a Central Processing Unit (CPU) that controls the entire information processing device 100. Reference numeral 102 denotes a Read Only Memory (ROM) that stores programs and parameters that do not require modification. Reference numeral 103 denotes a Random Access Memory (RAM) that temporarily stores programs and data supplied from an external device or the like.

[0016] Reference numeral 104 denotes an external storage device such as a hard disk or a memory card installed in the information processing device 100. However, it may be fixed or removable. For example, it includes optical disks such as flexible disks (FD) and compact disks (CD), magnetic or optical cards, IC cards, and memory cards. Reference numeral 105 denotes an interface with input devices such as a pointing device and a keyboard 109 that receive user operations and input data. Reference numeral 106 denotes a display interface with a monitor 110 that displays data held by the information processing device 100 and data supplied thereto. Reference numeral 107 denotes a network interface for connecting to a network line such as the Internet 111. Reference numeral 108 denotes a system bus that connects each of the units 101 to 107 so that they can communicate with each other.

[0017] The information processing device 100 of this embodiment is a device that adds annotations such as the names and meanings of objects contained in an annotation target (for example, content such as an image or text). Furthermore, the information processing device 100 detects deviations in annotation standards between multiple workers who perform annotation work, and issues a warning or alert when a deviation in the annotation standards is detected.

[0018] The functional configuration of the information processing device 100 of this embodiment will be described with reference to Fig. 2. The functional configuration described below is realized by the CPU 101 reading and executing programs stored in the ROM 102 or the external storage device 104 to perform calculations and processing of information and control of each piece of hardware. Note that each functional configuration may be realized by hardware such as ASIC or FPGA.

[0019] The information processing device 100 in this embodiment has an assignment target management unit 201 and an assignment target presentation unit 202 as means for handling assignment targets to be presented to a worker. The information processing device 100 also has an assignment data input unit 203, a work result storage unit 204, and a criteria inconsistency determination unit 205 as means for processing the work of the worker assigning information to the assignment target and the work results. In addition, the information processing device 100 has a warning display unit 206 as means for warning that the assignment criteria differ between workers.

[0020] The target management unit 201 manages targets to which an operator can assign information. Specific examples of targets include images, videos, character strings constituting natural language such as sentences, files, etc. In the following, an image will be used as an example of the target to be assigned.

[0021] The grant target presentation unit 202 presents the grant targets managed by the grant target management unit 201 to the worker.

[0022] The attachment data input unit 203 accepts input of attachment data to be attached to the attachment target presented by the attachment target presentation unit 202. Fig. 3(a) shows an example of a user interface presented to a worker. The attachment target presentation window 300 has an attachment target display area 310, an attachment target change button 320, and an attachment data input area 330. The attachment target presentation unit 202 corresponds to the attachment target display area 310 and the attachment target change button 320, and the attachment data input unit 203 corresponds to the attachment data input area 330.

[0023] The task of assigning annotation data to one target involves the worker checking the target displayed in the target display area 310 and inputting the annotation information to be assigned to the target in the data input area 330. When the input is complete, the worker uses the target change button 320 to change the target displayed in the target display area 310, and repeats this process until there are no more targets in the work queue (described later).

[0024] FIG. 3(b) shows an example of presenting a user interface to a plurality of workers. In the example of FIG. 3(b), six workers are performing the assignment work. Each worker uses a client 351, a client 352, a client 353, a client 354, a client 355, and a client 356 to operate the above-mentioned assignment target presentation window 300. The six clients are connected to a server 340, and receive and display the assignment target from the server 340. The clients 351 to 354 and the server 340 have the same configuration as the information processing device 100.

[0025] The work result storage unit 204 stores the target presented by the target presentation unit 202 and the target data input by the target data input unit 203 in association with each other.

[0026] The criteria inconsistency determination unit 205 determines the variation in the assigned data assigned by a plurality of people, and judges whether or not there is an inconsistency in the criteria between the workers.

[0027] When the standard inconsistency determination unit 205 determines that there is an inconsistency in standards between workers, the warning display unit 206 displays a warning to the worker, an inspector who checks the worker's assignment results, a supervisor who supervises the worker, etc.

[0028] Next, the overall flow of the first embodiment will be described with reference to Fig. 4. Fig. 4 has two flow charts, Fig. 4(a) and Fig. 4(b). Fig. 4(a) shows the process from S401 to S404 in which an object to be assigned is presented to an operator and the result is received. Fig. 4(b) shows the process from S411 to S415 in which an event is triggered to determine whether or not a criterion is inconsistent when the assignment data entered by the operator is received.

[0029] 4 and the following flowcharts are implemented by CPU 101 reading and executing programs stored in ROM 102 or external storage device 104 to calculate and process information and control each piece of hardware. A configuration may be used in which some or all of the processing of each step in the flowcharts shown in FIG. 4 and the following flowcharts is implemented by hardware such as one or more ASICs or FPGAs.

[0030] First, the process shown in Fig. 4(a) will be described in detail. In the following description, the information processing device 100 will be described as the server 340 shown in Fig. 3(b).

[0031] In S401, the information processing device 100 creates a work queue for each worker for the grant targets managed by the grant target management unit 201. The grant targets managed by the grant target management unit 201 are managed in a grant target storage table 500 as shown in FIG. 5. The grant target storage table 500 is composed of two columns. One grant target is represented by one row in the grant target storage table 500. Column 501 stores an ID that identifies the grant target, and column 502 stores a criterion confirmation flag that indicates whether the grant target can be used for criterion confirmation. The criterion confirmation flag managed in column 502 may be determined randomly, or a designation may be received from a user such as an inspector, and the grant target identified by the designation may be determined.

[0032] FIG. 6 shows an example of a work queue when there are six workers. Queue 611 is for worker A, queue 612 is for worker B, queue 613 is for worker C, queue 614 is for worker D, queue 615 is for worker E, and queue 616 is for worker F. The six queues 611 to 616 are used in order from the top. Using the information in the target storage table 500 shown in FIG. 5, targets are assigned to each queue. Targets with target IDs 001 to 006 have a criterion confirmation flag of False, so they are set separately to the queues of the respective workers. Next, target with target ID 007 has a criterion confirmation flag of True, so it is set to the queues of all the workers. After that, targets with target IDs 008 to 013 also have a criterion confirmation flag of False, so they are set separately to the queues of the respective workers. This process is repeated to assign targets to work queues.

[0033] In S402, the information processing device 100 has the assignment target management unit 201 check whether all of the work queues generated in S401 are empty. If all are empty, the process ends. If there are still assignment targets in at least one of the work queues, the process proceeds to S403.

[0034] In S403, the information processing device 100 acquires the grant target from each work queue generated in S401, and displays the grant target on the grant target presentation unit 202 of each worker as shown in Fig. 3(b). That is, the server 340 displays the grant target on the clients 351 to 356.

[0035] In S404, the information processing device 100 receives the attached data input by the user for the attached target presented in S403 via the attached data input unit 203. After receiving the data, the attached data input unit 203 stores the received attached data in the work result storage unit 204. In the example shown in FIG. 3(b), the server 230 receives the attached data attached by the clients 351 to 356 from each of the clients. The attached data is data including annotation information indicating the name and meaning of an object included in the attached target. After that, the attached data input unit 203 starts an attached data reception event.

[0036] Next, the process shown in FIG. 4(b) will be described in detail.

[0037] In S411, in response to the attachment data reception event started in S404, the information processing device 100 causes the criterion inconsistency determination unit 205 to determine whether the criterion confirmation flag of the attachment target to which the attachment data has been attached is True. If True, the process proceeds to S412, and if False, the process ends. For example, in the case where the attachment targets managed by the attachment target management unit 201 are the example shown in FIG. 5, it can be confirmed that the criterion confirmation flag of the attachment target with ID 007 is True.

[0038] In S412, the information processing device 100 acquires all annotation data assigned to the annotation target for which the criterion confirmation flag is set to True by the criterion inconsistency determination unit 205 from the work result storage unit 204. That is, the information processing device 100 acquires annotations assigned by multiple workers to the same annotation target.

[0039] An example of the attached data managed by the work result storage unit 204 is shown in FIG. 7. The attached data is managed in an attached data storage table 700. The attached data storage table 700 is composed of four columns. One piece of attached data is represented by one row in the attached data storage table 700. Column 701 stores an ID that identifies the attached data. Column 702 stores the attached data input by the worker. Column 703 stores the ID of the target to which the attached data is attached. In the example of FIG. 7, the target is an image, so the ID is an image ID, but it is not limited to an image and may be a video, a character string constituting a natural language such as a sentence, a file, or the like. Column 704 stores the ID of the worker who inputted the attached data.

[0040] Fig. 7 shows the results of each worker's work up to the midpoint of the work queue shown in Fig. 6. For example, if the ID of the target to be assigned, for which the reference confirmation flag described in S411 is True, is 007, the assigned data with 007 in column 703 is acquired in S412 using the assigned data storage table 700 shown in Fig. 7. Specifically, in S412, the information processing device 100 acquires six pieces of assigned data with IDs of 007, 009, 010, 011, 014, and 015 stored in column 701.

[0041] In S413, the information processing device 100 judges whether the number of assigned data acquired in S412 by the criterion inconsistency judgment unit 205 is equal to the number of workers. If they are equal, the process proceeds to S414, and if they are not equal, the process ends.

[0042] In S414, the information processing device 100 checks the variation of the multiple attachment data acquired by the criterion inconsistency determination unit 205 in S412. If the criterion is correctly shared, the attachment data should match, so if multiple types of attachment data exist in the multiple attachment data, it is determined that there is variation. In other words, if the comparison result of annotations from multiple workers to the same attachment target is a result that is not consistent among the multiple workers, the information processing device 100 determines that there is a difference in the attachment criteria between the multiple workers. Note that the method of determining the variation is not limited to this, and it may be determined that there is variation if the entropy of the multiple attachment data is calculated and the entropy is equal to or greater than a predetermined value. If it is determined that there is variation, the process proceeds to S415, and if there is no variation, the process ends.

[0043] In S415, the information processing device 100 causes the warning display unit 206 to display a warning to the worker, inspector, and / or supervisor. The information processing device 100 may display a message on the monitor 110 to call attention or warn, or may output a sound to call attention or warn. Furthermore, the information processing device 100 may transmit information for outputting a warning to a device corresponding to the worker, inspector, and / or supervisor. Furthermore, the information processing device 100 may cause a warning to be displayed on a device corresponding to the worker, inspector, and / or supervisor.

[0044] The information processing device 100 may present information indicating that there may be inconsistencies or variations in the assignment criteria among multiple workers to the workers, inspectors, and / or supervisors as a call for attention or a warning. The information processing device 100 may also present information encouraging the workers, inspectors, and / or supervisors to take measures to standardize the assignment criteria as a call for attention or a warning. In this case, the information processing device 100 may be configured to send information indicating the warning to a corresponding address by e-mail or the like.

[0045] Furthermore, when only some of the workers have added annotations with different contents from those of the other workers, the information processing device 100 may present a warning only to these some of the workers. In this case, the information processing device 100 may issue a warning by presenting information that prompts the workers to check the annotation criteria, information indicating that the workers have added annotations with contents different from those of the other workers, and / or information indicating annotations that are thought to be correct and added by the other workers. Furthermore, the information processing device 100 may be configured to issue a warning to all devices constituting the system. Furthermore, the information processing device 100 may issue a combination of the above-mentioned exemplified calls for attention and warnings.

[0046] 3, when the information processing device 100 operates as the server 340, the information processing device 100 may transmit information for outputting a warning to the clients 351 to 356 corresponding to the worker, inspector, and / or supervisor to be warned. When the information processing device 100 operates as the server 340, the information processing device 100 may output a warning to the built-in or connected monitor 110. When the information processing device 100 operates as the clients 351 to 356, the information processing device 100 may be configured to output a warning in response to receiving information for outputting a warning from the server 340. When the information processing device 100 operates as the clients 351 to 356, the information processing device 100 may be configured to determine whether the assignment criteria of the worker corresponding to the information processing device 100 are different from those of other workers. In this case, the information processing device 100 receives annotations assigned by other workers from the server 340 or other clients. The information processing device 100 may then be configured to determine whether the assignment criteria of the worker corresponding to the device itself are different from those of other workers, and output a warning that the assignment criteria of the worker corresponding to the device itself are different from those of other workers.

[0047] This allows the worker, inspector, or supervisor who confirmed the warning to confirm that the annotation standards are not being followed among workers. If it is confirmed that the standards are not being followed, the worker can reconfirm his / her understanding, and the inspector or supervisor can directly instruct the worker who misunderstood the standards and gave the wrong annotation. As a result of the confirmation and instruction, the understanding of the standards is revised, making it possible to improve the accuracy of annotation throughout the entire workplace.

[0048] <Embodiment 2> As another embodiment, a second embodiment in which a discrepancy in the assignment criteria between workers is determined and a warning is issued will be described below. The hardware configuration of the information processing device 100 in the second embodiment is the same as that in the first embodiment. Also, the functional configuration of the information processing device 100 in the second embodiment is the same as that in the first embodiment shown in Fig. 2, but the process in which the criteria inconsistency determination unit 205 makes a determination is different from that in the first embodiment, and therefore only this point will be described.

[0049] In the second embodiment, the variation is determined by grouping a plurality of pieces of appending data. The variation determination process in the second embodiment will be described with reference to the flowchart shown in FIG.

[0050] In S801, the information processing device 100 has the criterion inconsistency determination unit 205 group a plurality of pieces of attached data. As an example, the case where the ID of the attached target is 007, which was also described in the first embodiment, will be described. In the example shown in FIG. 7, six workers have attached data to the attached target with ID 007. Specifically, the IDs of the attached data are 007, 009, 010, 011, 014, and 015. When checking the column 702 of these six pieces of data, the attached data for three pieces of data, 007, 009, and 011, is "SUV", and the attached data for three pieces of data, 010, 014, and 015, is "pickup truck". In this case, the data is divided into two groups, an SUV group and a pickup truck group. In this way, grouping in this embodiment means distinguishing according to the contents of the annotation.

[0051] In S802, the information processing device 100 determines whether or not the criteria mismatch determination unit 205 has created multiple groups as a result of the grouping in S801. If the determination result shows that multiple groups have been created, the process proceeds to S803, and if not, the process proceeds to S805. In the above example, two groups, an SUV group and a pickup truck group, have been created, so the process proceeds to S803.

[0052] In S803, the information processing device 100 determines whether or not there are a plurality of groups in which the number of pieces of attached data in the group is 2 or more. This is because, when the number of pieces of attached data constituting the group is less than 2, that is, when there is only one piece of attached data, it is possible that the worker simply made a mistake. If the result of the determination is that there is any attached data, the process proceeds to S804, and if there is no attached data, the process proceeds to S805. In the above example, since there are three pieces of attached data in both the SUV group and the pickup truck group, the process proceeds to S804. In this embodiment, the determination is made when the number of pieces of attached data present in the group is 2 or more, but is not limited to 2. If you want to increase the number at which you determine that the worker simply made a mistake, you can set a value of 2 or more.

[0053] In S804, the criterion inconsistency determination unit 205 of the information processing apparatus 100 determines that the added data varies.

[0054] In S805, the criterion inconsistency determination unit 205 of the information processing apparatus 100 determines that the added data is not varied.

[0055] This makes it possible to eliminate the possibility that the operator simply made a mistake, and therefore makes it possible to detect deviations from the standard with a higher degree of certainty.

[0056] <Embodiment 3> As another embodiment, a third embodiment in which a discrepancy in the criteria for assignment between workers is determined and a warning is issued will be described below. The hardware configuration of the information processing device 100 in the third embodiment is the same as that in the first embodiment. Also, the functional configuration of the information processing device 100 in the third embodiment is shown in Fig. 9. Note that the same numbers as those used in Fig. 2 are used for the same components as those in the first embodiment shown in Fig. 2, and the description thereof will be omitted.

[0057] The attachment data estimation unit 901 estimates in advance the attachment data to be assigned to the attachment targets managed by the attachment target management unit 201. To make the estimation, a pre-learning model is generated by machine learning the relationship between the attachment targets of the same type as the attachment targets managed by the attachment target management unit 201 and the attachment data to be assigned to the same type of attachment targets. Then, the pre-learning model is used to estimate the attachment data to be assigned to each of the attachment targets managed by the attachment target management unit 201. The estimated attachment data is stored in the work result storage unit 204 as estimated attachment data in association with the attachment targets.

[0058] The pre-learning model used by the attachment data estimation unit 901 may be a model that is machine-learned by the information processing device 100, or may be a machine-learned model acquired from outside. When the information processing device 100 executes machine learning, the information processing device 100 may further include a GPU (Graphical Processing Unit) and perform machine learning using the GPU in addition to the CPU 101. The GPU can perform efficient calculations by processing more data in parallel, and is therefore effective when learning is performed multiple times, such as deep learning.

[0059] Specific examples of machine learning algorithms include nearest neighbor methods, naive Bayes methods, decision trees, and support vector machines. In addition, deep learning, which uses a neural network to generate features and connection weighting coefficients for learning, can also be used. Any of the above algorithms that can be used can be used as appropriate and applied to this embodiment.

[0060] When creating a trained model, multiple combinations of input data and output data are prepared as training data (teacher data). Knowledge is acquired from them by machine learning, and a trained model is generated that outputs output data for the input data based on the acquired knowledge. The trained model may be updated after certain processing as necessary.

[0061] In addition, the same type of attachment target is, for example, an image in which a car is shown, different from the image managed by the attachment target management unit 201, when an operator sees a car in an image and assigns the type of the car as attachment data. In addition, the estimated attachment data is composed of the result of estimation using the pre-learning model and its probability. An example of the estimated attachment data will be described with reference to FIG. 10. In the example shown in FIG. 10, the attachment target is an image.

[0062] The estimated attachment data storage table 1000 shown in FIG. 10(a) is composed of five columns. One piece of estimated attachment data is represented by one row in the estimated attachment data storage table 1000. Column 1001 is a column that stores an ID that identifies the estimated attachment data that is the estimation result. Column 1002 is a column that stores the estimation result. Column 1003 is a column that stores accuracy information of the estimation result. Column 1004 is a column that stores the ID of the image that is the estimated target. Column 1005 is a column that stores the ID of the area included in the image that is the estimated target. The area is managed in an area table 1010 described later. Note that, since multiple attachment data may be attached to an attachment target, area information is stored. For example, when annotating the type of a car in an image, if there are multiple cars in the image, an area in the image is specified and annotation is assigned to that area. In the example shown in FIG. 10(a), two estimated attachment data with IDs of 008 and 009 both indicate that they were assigned to an attachment target with an image ID of 008 in column 1004.

[0063] Region table 1010 shown in FIG. 10(b) is composed of five columns. Furthermore, one region within the target is represented by one row in region table 1010. Column 1011 is a column that stores an ID that identifies the region. Column 1012 is a column that stores the X coordinate value of the starting point, starting from the top left point of the region. Column 1013 is a column that stores the Y coordinate value of the starting point, starting from the top left point of the region. Column 1014 is a column that stores the width value of the region. Column 1015 is a column that stores the height value of the region.

[0064] The reference confirmation target determining unit 902 determines a reference confirmation target suitable for determining whether the reference is blurred. The flow of the process of determining the reference confirmation target will be described with reference to the flowchart of FIG.

[0065] In S1101 , the reference-check-use grant target determination unit 902 of the information processing device 100 acquires one grant target from the grant target management unit 201 .

[0066] In S1102, the information processing device 100 causes the reference confirmation grant target determination unit 902 to acquire, from the work result storage unit 204, all of the estimated grant data estimated for the grant target acquired in S1101.

[0067] In S1103, the information processing device 100 judges the degree of difficulty when working on the assignment target from one or more estimated assignment data acquired in S1102. In this embodiment, it is judged whether the probability included in one or more estimated assignment data is 60% or less. If the result of the judgment is that there is estimated assignment data with a probability of being included in the estimated assignment data of 60% or less, the process proceeds to S1104, and if the probability included in all of the estimated assignment data is greater than 60%, the process ends. Note that 60% is an example and is not limited to this, and any predetermined threshold value may be set.

[0068] Furthermore, when using the accuracy of the estimated attachment data, different values ​​preset by the user may be used, or the trend in the probability of the group of estimated attachment data managed by the work result storage unit 204 may be used. The trend in probability may be, for example, the average probability. Note that other information may be used instead of using the accuracy information of the estimated attachment data. For example, the number of estimated attachment data included in one attachment target may be used, or in the case where the estimated attachment data has an area as shown in FIG. 10, the area of ​​overlapping areas may be used.

[0069] In S1104, the information processing device 100 has the reference confirmation target determination unit 902 set a concentration determination flag for the corresponding target. The concentration determination flag may be set as a column of a database managed by the target management unit 201, or may be assigned as metadata of the file if the target is a file. FIG. 12 shows an example of targets managed by the target management unit 201.

[0070] The grant target storage table 1200 shown in FIG. 12 is composed of three columns. Furthermore, one grant target is represented by one row in the grant target storage table 1200. Column 1201 is a column that stores an ID that identifies the grant target. Column 1202 is a column that stores information indicating whether the grant target is suitable as a grant target for reference confirmation. Column 1203 is a column that stores the number of workers to whom the grant target is presented if the grant target is a grant target for reference confirmation. Note that the value of column 1203 is 1 if the value of column 1202 is False, and a value other than 1 is stored if the value of column 1202 is True. A value greater than 1 in column 1203 means that the grant target is presented to that number of workers, and a negative value means that the grant target is presented to all workers.

[0071] In S1105, the information processing device 100 determines how many workers the reference confirmation target is to be presented to by the reference confirmation target determination unit 902. This process will be described later with reference to Fig. 13. Note that the reference confirmation target may be always presented to all workers without performing the process of S1105.

[0072] The processes from S1101 to S1105 are performed for one grant target, and the reference-confirmation grant target determination unit 902 performs this process for all grant targets managed by the grant target management unit 201.

[0073] The flow of the process for determining the number of workers to whom a reference confirmation target is to be presented will be described with reference to FIG.

[0074] In S1301, the information processing device 100 determines whether the probability of the lowest probability of the estimated assignment data among the estimated assignment data for the assignment target to be processed is less than 60% and 50% or more. If the result of the determination satisfies the condition, the process proceeds to S1302, and if not, the process proceeds to S1303.

[0075] In S1302, the information processing device 100 determines the number of workers to be presented with the reference confirmation target by the reference confirmation target determination unit 902 to be three, and sets this in the target management unit 201. Specifically, 3 is set in the column 1203 of the target storage table 1200 shown in FIG.

[0076] In S1303, the information processing device 100 determines whether the probability of the lowest probability of the estimated assignment data among the estimated assignment data for a certain assignment target is less than 50% and 40% or more. If the result of the determination satisfies the condition, the process proceeds to S1304, and if not, the process proceeds to S1305.

[0077] In S1304, the information processing device 100 determines the number of workers to be presented with the reference confirmation grant targets by the reference confirmation grant target determination unit 902 to be five, and sets this in the grant target management unit 201. Specifically, five is set in the column 1203 of the grant target storage table 1200 shown in FIG.

[0078] In S1305, the information processing device 100 causes the reference confirmation target determination unit 902 to determine the number of workers to whom the reference confirmation target is to be presented, and sets the number in the target management unit 201. Specifically, −1 is set in the column 1203 of the target storage table 1200 shown in FIG.

[0079] In addition, the probability and the number of branching judgments specified in the process shown in FIG. 13 are not limited to these. In this embodiment, the probability is divided into three: less than 60% and 50% or more, less than 50% and 40% or more, and less than 40%. However, this may be divided into four ranges of 5% increments from 60% to 40%, and a total of five ranges of less than 40%. Or, it may be divided into two ranges: less than 60% and 40% or more, and less than 40%. Also, as in the description of S1103, other information may be used instead of using the accuracy information of the estimated attachment data. For example, the range of the number of estimated attachment data included in one attachment target may be used, or, in the case where the estimated attachment data has an area as shown in FIG. 10, the range of the size of the area where the areas overlap or the range of the number may be used. In addition, the method of using a database when managing the number of workers who present the attachment target is one example, and is not limited to this. For example, if the attachment target is a file, it may be set as metadata of the file.

[0080] In addition, the number of workers to be checked for the annotation criteria is changed depending on the estimated accuracy of the annotation to be added, but this is not limited to the above, and the number may be fixed (for example, all workers) regardless of the estimated accuracy.

[0081] Next, Figure 14 shows an example of setting a work queue for each worker in the grant target management unit 201 when the estimated grant data is used to identify grant targets for standard judgment and the number of workers to which those grant targets for standard judgment are to be presented is determined.

[0082] The example shown in Fig. 14 is an example of a work queue for six workers, similar to that shown in Fig. 6. Queue 1401 is for worker A, queue 1402 is for worker B, queue 1403 is for worker C, queue 1404 is for worker D, queue 1405 is for worker E, and queue 1406 is for worker F. It is assumed that the six queues 1401 to 1406 are used in order from the top.

[0083] The assignment target management unit 201 sets assignment targets to queues based on the reference confirmation flag stored in column 1202 of the assignment target storage table 1200 in FIG. 12 and the number of workers stored in column 1203.

[0084] How the grant target management unit 201 allocates the grant targets shown in FIG. 12 to each queue will be described. Here, the explanation will be given in ascending order of ID in column 1201. The grant target with ID 001 is allocated to one worker because the reference confirmation flag is False. Specifically, the grant target with ID 001 is allocated to queue 1401. Next, the grant target with ID 002 is similarly allocated to queue 1402. The grant target with ID 003 has the reference confirmation flag True and the number of workers presented is three, so it is allocated to queues 1403, 1404, and 1405. The four grant targets with IDs 004 to 007 have the reference confirmation flag False, so they are allocated to queues 1406, 1401, 1402, and 1403, respectively. Next, the grant target with ID 008 has its reference confirmation flag set to True and the number of workers presented is 5, so it is distributed to queues 1404, 1405, 1406, 1401, and 1402. The four grant targets with IDs 009 to 012 have their reference confirmation flag set to False, so they are distributed to queues 1403, 1404, 1405, and 1406, respectively. Omitting the middle part, the grant target with ID N has its reference confirmation flag set to True and the number of workers is -1, so it is distributed to all workers. Specifically, it is distributed to all of queues 1401, 1402, 1403, 1404, 1405, and 1406.

[0085] This makes it possible to determine whether the annotation criteria are inconsistent for only those annotation targets that are likely to be inconsistent. As a result, it becomes possible to more effectively detect whether the criteria are inconsistent, and to efficiently improve the accuracy of annotation throughout the entire workplace.

[0086] According to the present embodiment having the above configuration, it is possible to issue a warning based on the variation in the results of annotations made by multiple people. This allows the worker, supervisor, or verifier who checks the warning, or the verifier who verifies the annotation results, to immediately determine which annotation targets are likely to cause deviations in the criteria and which worker's understanding is off. As a result, it becomes possible to take corrective measures against workers whose understanding of the annotation criteria is off, and it becomes possible to improve the accuracy of annotations overall.

[0087] <Other embodiments> The above-described multiple embodiments may be appropriately combined. For example, the process of determining the determination process of the grant criteria may be configured to switch between the methods of the respective embodiments every predetermined number of times. Alternatively, the method of each embodiment may be randomly switched.

[0088] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0089] 300 Grant target presentation window 310 Display area for granting 320 Change target button 330 Grant data input area 340 Server for managing annotation tasks by multiple people 351-356 Clients used by workers to annotate

Claims

1. An acquisition means for acquiring a plurality of annotations based on inputs from a plurality of workers to a same annotation target; A comparison means for comparing the plurality of annotations acquired by the acquisition means; a determination means for determining whether there is a discrepancy in annotation standards between the plurality of workers based on the comparison by the comparison means; and a warning means for issuing a warning when the determination means determines that there is a deviation in the application criteria between the plurality of workers, The information processing device is characterized in that the judgment means judges that there is a discrepancy in the assignment standards between the multiple workers when the comparison result by the comparison means indicates that the entropy of the multiple annotations is greater than or equal to a predetermined value.

2. An acquisition means for acquiring a plurality of annotations based on inputs from a plurality of workers to a same annotation target; A comparison means for comparing the plurality of annotations acquired by the acquisition means; a determination means for determining whether there is a discrepancy in annotation standards between the plurality of workers based on the comparison by the comparison means; and a warning means for issuing a warning when the determination means determines that there is a deviation in the application criteria between the plurality of workers, The information processing device is characterized in that the determination means determines that there is a discrepancy in the annotation standards between the multiple workers when the comparison result by the comparison means shows that there are multiple groups of annotations for the annotation target and that each of the multiple groups contains multiple data.

3. The method further includes a receiving means for receiving a designation of the target to be granted from a user, 3 . The information processing apparatus according to claim 1 , wherein the acquiring means acquires the plurality of annotations based on inputs from the plurality of workers to an object to be assigned that is specified by the designation received by the receiving means. 4 .

4. An acquisition means for acquiring a plurality of annotations based on inputs from a plurality of workers to a same annotation target; An estimation means for estimating an annotation to the annotation target; A comparison means for comparing the plurality of annotations acquired by the acquisition means; a determination means for determining whether there is a discrepancy in annotation standards between the plurality of workers based on the comparison by the comparison means; a warning means for issuing a warning when the determining means determines that there is a deviation in the standard for applying the standard between the plurality of workers; having The information processing device according to claim 1, wherein the comparison means changes the number of the plurality of annotations used for comparison depending on the estimation accuracy of the annotation estimated by the estimation means.

5. The information processing apparatus according to claim 4 , wherein the comparison means increases the number of the plurality of annotations used for comparison as the estimation accuracy of the annotation estimated by the estimation means decreases.

6. The information processing device according to claim 4 , wherein the estimation means estimates the annotation for the target by using a trained model that has been machine-learned.

7. 5. The information processing apparatus according to claim 1, wherein the warning means issues a warning by displaying information or outputting a sound.

8. 5. The information processing apparatus according to claim 1, wherein the warning means issues a warning by transmitting information for outputting a warning to another apparatus.

9. The information processing device according to any one of claims 1, 2 and 4, characterized in that the warning means displays information indicating that there may be a discrepancy in the annotation criteria or information indicating that the annotation added is different from annotations added by other workers.

10. 5. The information processing apparatus according to claim 1, wherein the warning means displays information encouraging a measure to be taken to standardize the criteria for granting the rating.

11. 5. The information processing apparatus according to claim 1, wherein the object to be tagged is an image or a text.

12. The information processing apparatus according to claim 1 , wherein the target to be tagged is a region including an object in an image.

13. The information processing apparatus according to claim 1 , wherein the annotation is a name of an object included in the target to which the annotation is to be added.

14. An acquisition step of acquiring a plurality of annotations based on inputs from a plurality of workers to a same annotation target; a determination step of determining whether there is a discrepancy in annotation standards between the plurality of workers based on a comparison of the plurality of annotations obtained; and a warning step of issuing a warning when a deviation in the application criteria is determined between the plurality of workers, A control method for an information processing device, characterized in that in the judgment process, if the comparison result indicates that the entropy of the multiple annotations is greater than or equal to a predetermined value, it is judged that there is a discrepancy in the assignment standards between the multiple workers.

15. An acquisition step of acquiring a plurality of annotations based on inputs from a plurality of workers to a same annotation target; a determination step of determining whether there is a discrepancy in annotation standards between the plurality of workers based on a comparison of the plurality of annotations obtained; and a warning step of issuing a warning when a deviation in the application criteria between the plurality of workers is determined by the determination step, A control method for an information processing device, characterized in that in the judgment process, if the comparison result shows that there are multiple groups of annotations for the annotation target and that each of the multiple groups contains multiple data, it is judged that there is a discrepancy in the annotation standards between the multiple workers.

16. An acquisition step of acquiring a plurality of annotations based on inputs from a plurality of workers to a same annotation target; An estimation step of estimating an annotation to the annotation target; a determination step of determining whether there is a discrepancy in annotation standards between the plurality of workers based on a comparison of the plurality of annotations obtained; and a warning step of issuing a warning when a deviation in the application criteria between the plurality of workers is determined by the determination step, A control method for an information processing device, characterized in that, in the comparison, the number of the multiple annotations used in the comparison is changed depending on the estimation accuracy of the annotation estimated in the estimation step.

17. A program for causing a computer to function as the information processing device according to any one of claims 1 to 4.

Images