Information processing device, control method, and program

The information processing apparatus simplifies the detection of worker concentration by comparing annotations and work time, addressing inaccuracies in existing methods and preventing errors through timely warnings.

JP7580947B2Active Publication Date: 2024-11-12CANON KK
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
JP2020103338
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-06-15
Publication Date
2024-11-12
Estimated Expiration
2040-06-15

AI Technical Summary

Technical Problem

Existing methods for determining worker concentration during annotation tasks require devices like electroencephalographs, leading to increased complexity and potential inaccuracies in recognition accuracy due to decreased worker focus.

Method used

An information processing apparatus that acquires and compares annotations, determines recognition accuracy based on work time and annotation quality, and issues warnings when concentration decreases, using a simplified approach without requiring additional devices.

Benefits of technology

Enables accurate determination of recognition accuracy and timely warnings to prevent errors by detecting concentration declines in annotation tasks, reducing costs and improving task efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007580947000001
    Figure 0007580947000001
  • Figure 0007580947000002
    Figure 0007580947000002
  • Figure 0007580947000003
    Figure 0007580947000003
Patent Text Reader

Abstract

To easily determine a reduction in an operator's recognition accuracy in annotation application operations and allow a warning for the reduction in recognition accuracy.SOLUTION: An information processing apparatus according to the present invention has: acquisition means that acquires a first annotation to an application target, the annotation based on input from an operator; comparison means that compares the first annotation acquired by the acquisition means with a second annotation to be compared; determination means that determines a reduction in the operator's recognition accuracy based on the comparison made by the comparison means; and warning means that performs warning when the reduction in the operator's recognition accuracy is determined by the determination means.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to information processing technology. [Background technology]

[0002] Classifying objects by annotating or tagging them is a common practice. Recently, annotations and tags are not only used for classification, but are increasingly being used as training data for machine learning.

[0003] However, the task of annotating is often monotonous, and there is a problem that performing monotonous tasks repeatedly reduces the worker's concentration and makes them more likely to make mistakes.

[0004] There is a method disclosed in Patent Document 1 that estimates whether or not the worker's concentration is declining by monitoring the worker himself / herself. In this method, the physiological parameters of the worker performing a certain task are measured to obtain the amount of brain activity and emotional state. In addition, the concentration level of the worker is determined, and a correspondence is identified by machine learning based on the data of the amount of brain activity, emotional state, and the concentration level. Then, based on the identified correspondence, it is estimated whether or not the worker is concentrating. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2019-159941 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, the method described in Patent Document 1 requires a device that is not directly related to the actual work, such as an electroencephalograph, to determine the concentration level. Therefore, there is room for simplification in determining the recognition accuracy caused by a decrease in the worker's concentration during the annotation work.

[0007] The present invention has been made to solve the above-mentioned problems, and aims to easily determine the recognition accuracy of a worker in annotation work and to enable a warning against a decline in recognition accuracy. [Means for solving the problem]

[0008] As a means for achieving the above object, an information processing device according to the present invention has the following arrangement.

[0009] That is, An acquisition means for acquiring a first annotation, which is an annotation for an annotation target, based on an input from a worker; a comparison means for comparing the first annotation acquired by the acquisition means with a second annotation to be compared; When a unit work time calculated by dividing the work time related to the annotation of the worker by the amount of work becomes equal to or greater than a threshold value, a determination means for determining a decrease in recognition accuracy of the worker based on the comparison by the comparison means; and a warning means for issuing a warning when the determining means determines that the worker has lost concentration. Effect of the Invention

[0010] According to the present invention, it is possible to easily judge the recognition accuracy of a worker in an annotation assignment task and to issue a warning against a decrease in recognition accuracy. [Brief description of the drawings]

[0011] [Figure 1] FIG. 2 is a diagram illustrating a hardware configuration of an information processing device. [Diagram 2] FIG. 2 is a diagram illustrating a functional configuration of an information processing device. [Diagram 3]FIG. 11 is a diagram showing an example of a screen output by an information processing device. [Figure 4] 13 is a flowchart showing a process of determining a target for concentration determination. [Diagram 5] 4 is a flowchart showing the overall processing of the information processing device. [Figure 6] 13 is a flowchart showing a process of receiving added data. [Figure 7] FIG. 11 is a diagram illustrating an example of data stored in a work result storage unit. [Figure 8] 11 is a flowchart showing a process for determining whether or not concentration is declining. [Figure 9] FIG. 11 is a diagram illustrating a functional configuration of an information processing device according to a second embodiment. [Figure 10] FIG. 11 is a diagram showing an example of a screen output by an information processing device in the second embodiment. [Figure 11] 13 is a flowchart showing a process for determining a target for concentration determination in the second embodiment. [Figure 12] 10 is a flowchart showing the overall processing of an information processing device according to a second embodiment. [Figure 13] 13 is a flowchart showing a process of receiving added data in the second embodiment. [Figure 14] FIG. 11 is a diagram showing an example of data stored in a work result storage unit in the second embodiment. [Figure 15] 13 is a flowchart showing a process for determining the timing to re-display an assignment target that has been presented once in the second embodiment. [Figure 16] 11 is a flowchart showing a process for determining whether or not concentration is declining in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the present invention will be described in detail based on preferred embodiments thereof with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the configurations shown in the drawings.

[0013] <Embodiment 1> FIG. 1 is a diagram showing a hardware configuration of an information processing device 100 in this embodiment. The information processing device 100 described in the following embodiment is a computer device having the configuration shown in the block diagram in FIG. 1. In addition, the information processing device 100 will be described as a single device hereinafter, but is not limited to this. That is, the methods in each embodiment may be realized by a single computer device, or each function may be distributed and realized in multiple computer devices as necessary. When configured by multiple computer devices, they are connected by a Local Area Network (LAN) or the like so as to be able to communicate with each other.

[0014] 1, reference numeral 101 denotes a Central Processing Unit (CPU) that controls the entire information processing device 100. Reference numeral 102 denotes a Read Only Memory (ROM) that stores programs and parameters that do not require modification. Reference numeral 103 denotes a Random Access Memory (RAM) that temporarily stores programs and data supplied from an external device or the like.

[0015] Reference numeral 104 denotes an external storage device such as a hard disk or a memory card installed in the information processing device 100. However, it may be fixed or removable. For example, it includes optical disks such as flexible disks (FD) and compact disks (CD), magnetic or optical cards, IC cards, and memory cards. Reference numeral 105 denotes an interface with input devices such as a pointing device and a keyboard 109 that receive user operations and input data. Reference numeral 106 denotes a display interface with a monitor 110 that displays data held by the information processing device 100 and data supplied thereto. Reference numeral 107 denotes a network interface for connecting to a network line such as the Internet 111. Reference numeral 108 denotes a system bus that connects each of the units 101 to 107 so that they can communicate with each other.

[0016] The information processing device 100 of this embodiment is a device that adds annotations such as the names and meanings of objects contained in an annotation target (for example, content such as an image or text). Furthermore, the information processing device 100 determines, based on the work performance, a decrease in recognition accuracy caused by a decrease in the concentration of a worker performing the annotation addition work, and if a decrease in recognition accuracy is detected based on the result, it issues a warning or a warning to the worker.

[0017] The functional configuration of the information processing device 100 of this embodiment will be described with reference to Fig. 2. The functional configuration described below is realized by the CPU 101 reading and executing programs stored in the ROM 102 or the external storage device 104 to perform calculations and processing of information and control of each piece of hardware. Note that each functional configuration may be realized by hardware such as ASIC or FPGA.

[0018] The information processing device 100 in this embodiment has an assignment target management unit 201 and an assignment target presentation unit 202 as means for processing the assignment target to be presented to the worker. The information processing device 100 also has an assignment data input unit 203 and a work result storage unit 204 as means for processing the work of the worker assigning information to the assignment target. In addition, the information processing device 100 has an assignment data estimation unit 205 as means for estimating in advance the assignment data to be assigned to the assignment target. Furthermore, the information processing device 100 has an assignment target determination unit 206 for concentration determination that identifies an assignment target suitable for determining the concentration from among the assignment targets. Finally, the information processing device 100 has a presentation timing determination unit 207, a concentration decline determination unit 208, and a warning display unit 209 as means for determining whether the worker is concentrating, and for calling attention and issuing a warning when it is determined that the worker is not concentrating.

[0019] The target management unit 201 manages targets to which a worker can assign information and targets for concentration determination, which will be described later. In addition to simply managing targets, the target management unit 201 has a target storage queue that stores targets to be worked on, and a target storage queue for concentration determination that manages targets for concentration determination. Specific examples of targets include images, videos, character strings that constitute natural language such as sentences, and files. In the following, an image will be used as an example of a target.

[0020] The annotation target presentation unit 202 presents to the worker the annotation targets managed by the annotation target management unit 201. The annotation data input unit 203 accepts input of annotation data including annotations that the worker will assign to the annotation targets presented by the annotation target presentation unit 202. Fig. 3 shows an example of a user interface presented to the worker.

[0021] 3, the grant target presentation window 300 has an grant target display area 310, an grant target change button 320, and an grant data input area 330. Note that the grant target presentation section 202 corresponds to the grant target display area 310 and the grant target change button 320, and the grant data input section 203 corresponds to the grant data input area 330.

[0022] The task of assigning annotation data to one target involves the worker checking the target displayed in the target display area 310 and inputting the annotation information to be assigned to the target in the data input area 330. When the input is complete, the worker uses the target change button 320 to change the target displayed in the target display area 310, and repeats this process until there are no more targets to assign.

[0023] The work result storage unit 204 stores the target presented by the target presentation unit 202 in association with the target data input by the target data input unit 203. Similarly, the work result storage unit 204 stores estimated target data, which is a result of estimation by the target data estimation unit 205 described later, in association with the target.

[0024] The attachment data estimation unit 205 estimates in advance the attachment data that is expected to be attached to the attachment targets managed by the attachment target management unit 201. In order to make the estimation, a pre-learning model is generated by machine learning the relationship between the attachment targets of the same type as the attachment targets managed by the attachment target management unit 201 and the attachment data to be attached to the attachment targets of the same type. Then, the information processing device 100 uses the pre-learning model to estimate the attachment data to be attached to each of the attachment targets managed by the attachment target management unit 201. The estimated attachment data is stored in the work result storage unit 204 as estimated attachment data in association with the attachment targets.

[0025] For example, when an operator sees a car in an image and assigns the type of the car as the assignment data, the same type of assignment target is an image that shows a car and is different from the images managed by the assignment target management unit 201. The estimated assignment data is composed of the result of estimation using the pre-learning model and its probability. An example of the estimated assignment data will be described later with reference to FIG. 7.

[0026] In addition, the pre-learning model used by the attachment data estimation unit 205 may be machine-learned by the information processing device 100, or may be acquired from an external device as a machine-learned model. When the information processing device 100 executes machine learning, the information processing device 100 may further include a GPU (Graphical Processing Unit) and perform machine learning using the GPU in addition to the CPU 101. The GPU can perform efficient calculations by processing more data in parallel, and is therefore effective when learning is performed multiple times, such as deep learning.

[0027] Specific examples of machine learning algorithms include nearest neighbor methods, naive Bayes methods, decision trees, and support vector machines. In addition, deep learning, which uses a neural network to generate features and connection weighting coefficients for learning, can also be used. Any of the above algorithms that can be used can be used as appropriate and applied to this embodiment.

[0028] When creating a trained model, multiple combinations of input data and output data are prepared as training data (teacher data). Knowledge is acquired from them by machine learning, and a trained model is generated that outputs output data for the input data based on the acquired knowledge. The trained model may be updated after certain processing as necessary.

[0029] The concentration-determining target determination unit 206 determines whether the target managed by the target management unit 201 can be used to determine whether the concentration is declining. The flow of the determination process executed by the concentration-determining target determination unit 206 will be described with reference to the flowchart shown in FIG.

[0030] 4 and the following flowcharts are implemented by CPU 101 reading and executing programs stored in ROM 102 or external storage device 104 to calculate and process information and control each piece of hardware. A configuration may be used in which some or all of the processing of each step in the flowcharts shown in FIG. 4 and the following flowcharts is implemented by hardware such as one or more ASICs or FPGAs.

[0031] In S401 , the target for concentration assessment determining unit 206 acquires one target for concentration assessment from the target management unit 201 .

[0032] In S402, the concentration ability assessment target determination unit 206 acquires, from the task result storage unit 204, estimated assignment data estimated for the target acquired in S401.

[0033] In S403, the concentration determination target determination unit 206 determines whether the accuracy information included in the estimated data acquired in S402 is equal to or greater than a first threshold. If the accuracy information is equal to or greater than the first threshold, the process proceeds to S404. If the accuracy information is less than the first threshold, the process ends.

[0034] The first threshold may be, for example, a value previously set by the user, or may use the tendency of the accuracy information included in the estimated grant data for the grant targets managed by the grant target management unit 201. The tendency of the accuracy information may be, for example, a value obtained by adding twice the standard deviation δ in the positive direction from the average probability as the first threshold. Also, without using the first threshold, it may be determined whether the accuracy information included in the estimated grant data for the grant target is included in the top 5% of all grant targets.

[0035] In S404, the concentration determination target determination unit 206 sets a concentration determination flag for the target. After that, the process ends. If the target management unit 201 uses a database to manage the target, the concentration determination flag may be set as a column in the database, or if the target is a file, it may be assigned as metadata for the file.

[0036] The processes from S401 to S404 are performed for one grant target, and the grant target determination unit 206 for concentration ability assessment performs this process for all grant targets managed by the grant target management unit 201.

[0037] In this way, the information processing device 100 can accurately determine the concentration level by using the target with high accuracy of the estimated attached data when the worker checks the concentration level.

[0038] The presentation timing determination unit 207 determines the timing for presenting to the worker the target for concentration determination, which is the result determined by the target for concentration determination determination unit 206 .

[0039] The concentration decline determination unit 208 determines whether or not the concentration declines by using the assigned data assigned by the worker to the target for concentration determination displayed by the target presentation unit 202. Details will be described later with reference to FIG.

[0040] The warning display unit 209 issues a warning or alert to the worker when the concentration decrease determination unit 208 determines that the worker's concentration is decreasing.

[0041] Next, the overall processing flow of the information processing device 100 in this embodiment will be described with reference to FIG.

[0042] In S501, the information processing device 100 stores in a grant target storage queue the grant target on which the worker is to work from among the grant targets managed by the grant target management unit 201. The grant target storage queue may be created individually for each worker, or one queue may be shared by multiple workers overall.

[0043] In S502, the information processing device 100 stores, from among the grant targets managed by the grant target management unit 201, grant targets having a flag for concentration determination in a grant target storage queue for concentration determination.

[0044] In S503, the information processing device 100 starts a concentration determining target presentation interval timer in order to obtain the elapsed time after the presentation of the target for concentration determining.

[0045] In S504, the information processing device 100 checks whether the grant target exists in the grant target storage queue in which the grant target was stored in S501. If the grant target exists, the process proceeds to S505, and if the grant target does not exist, the process ends.

[0046] In S505, the information processing device 100 judges whether it is time to present the concentration judgment target by the presentation timing judgment unit 207. Specifically, the judgment is made based on whether the time of the concentration judgment target presentation interval timer has elapsed for a certain period of time or more. That is, the information processing device 100 judges the decrease in concentration periodically at a predetermined period of time, and does not perform the judgment for a certain period of time from the previous judgment. Note that the certain period of time may be a fixed time such as 10 minutes, or a fixed time such as 10 minutes with a certain random fluctuation (-2 minutes to +2 minutes). If it is judged that it is not time to present the target yet, the process proceeds to S506, and if it is judged that it is time to present the target, the process proceeds to S507. In this way, the information processing device 100 periodically performs a process of judging whether the concentration has decreased, thereby making it possible to prevent the continuation of work in a state of decreased concentration and mistakes in the annotation work due to the decrease in concentration.

[0047] In S506, the information processing device 100 presents the target to be attached by the target presentation unit 202 and accepts the attachment data input by the worker. A detailed process flow will be described later with reference to Fig. 6. After that, the process proceeds to S504.

[0048] In S507, the information processing device 100 presents an assignment target for use in concentration determination on the assignment target presentation unit 202, and accepts the assignment data input by the worker. Note that the processing of S506 and the processing of S507 are basically the same, but the assignment target presented to the worker is different. Specifically, a normal assignment target is presented in S506, and an assignment target for use in concentration determination is presented in S507. Also, the normal assignment target and the assignment target for use in concentration determination can be treated as the same by the worker. The details of the processing flow will be described later with reference to FIG. 6 together with S506.

[0049] In S508, the information processing device 100 resets the concentration determining assignment object presentation interval timer started in S503.

[0050] In S509, the information processing device 100 uses the assignment data given by the worker to the assignment target presented in S507 and the estimated assignment data of the assignment target to determine whether the worker's concentration has decreased in the concentration decrease determination unit 208. Details of the determination process will be described later with reference to Fig. 8. If the determination result shows that the concentration has not decreased, the process proceeds to S504, and if it has decreased, the process proceeds to S510.

[0051] In S510, the information processing device 100 uses the annotation target presentation unit 202 to alert the worker that his / her concentration is declining. The information processing device 100 may display the alert or warning by a message on the monitor 110, or may output a sound to alert or warn the worker. The information processing device 100 may present, as the alert or warning, information indicating that his / her concentration may be declining to the worker. The information processing device 100 may also present, as the alert or warning, information indicating that the recognition accuracy of the worker may be declining. The information processing device 100 may also present, as the alert or warning, information warning the worker that the worker's eyes are tired. The information processing device 100 may also present, as the alert or warning, information encouraging the worker to take a break. The information processing device 100 may also present, as the alert or warning, information indicating that the annotation added may contain an error to the worker. The information processing device 100 may also perform a combination of the above-mentioned exemplified alerts or warnings.

[0052] Next, the process flow for presenting the grant target and accepting the grant data entered by the worker will be described with reference to FIG.

[0053] In S601, the information processing device 100 acquires the grant target from the grant target management unit 201, and presents the grant target to the worker by displaying it to the worker using the grant target presentation unit 202. When acquiring the grant target from the grant target management unit 201, the information processing device 100 acquires the grant target from the grant target storage queue in the case of acquiring a normal grant target, and acquires the grant target from the grant target storage queue for concentration determination in the case of acquiring a grant target for concentration determination.

[0054] In S602, the information processing device 100 judges whether the worker has completed inputting the attachment data. If the inputting is completed, the process proceeds to S603, and if the inputting is not completed, the process continues until the inputting is completed.

[0055] In S603, the information processing device 100 acquires the attachment data input by the worker using the attachment data input unit 203. Note that the attachment data is data including annotation information indicating the name and meaning of an object included in the attachment target.

[0056] In S604, the attachment target presented in S601 and the attachment data acquired in S603 are associated with each other and stored as a work result in the work result storage unit 204. An example of the work result data will be described with reference to FIG.

[0057] Examples of the estimated assignment data and the work result data will be described with reference to FIG.

[0058] 7(a), 7(b), and 7(c) show examples of tables in the database stored in the work result storage unit 204. FIG.

[0059] The attachment target storage table 710 shown in Fig. 7(a) is composed of three columns. Furthermore, one attachment target is represented by one row in the attachment target storage table 710. Column 711 is a column that stores an ID that identifies the image to which the attachment is to be applied. Column 712 is a column that stores the file path of the image. Column 713 is a column that stores a concentration determination flag that indicates whether the attachment target is for use in concentration determination.

[0060] The estimated attachment data storage table 720 shown in Fig. 7(b) is composed of four columns. Furthermore, one piece of estimated attachment data is represented by one row in the estimated attachment data storage table 720. Column 721 is a column that stores an ID that identifies the estimation result. Column 722 is a column that stores annotation information of the estimation result. Column 723 is a column that stores accuracy information of the estimation result. Column 724 is a column that stores the ID of the image that is the estimated target.

[0061] The attached data storage table 730 shown in Fig. 7(c) is composed of three columns. One piece of attached data is represented by one row in the attached data storage table 730. Column 731 is a column that stores an ID that identifies the attached data. Column 732 is a column that stores the attachment result. Column 733 is a column that stores the image ID of the image to which the data has been attached.

[0062] Next, the process of determining whether or not concentration is decreasing, which is performed by the concentration decrease determining unit 208, will be described with reference to FIG.

[0063] In S801, the information processing device 100 acquires estimated grant data for a grant target to be used for concentration determination in the concentration decrease determination unit 208.

[0064] In S802, the information processing device 100 acquires the annotation data for the annotation target, including the annotation input by the worker, in the concentration decrease determination unit 208. That is, the information processing device 100 acquires the annotation assigned to the annotation target by the input by the worker. In the concentration determination, the annotation indicated by the estimated annotation data is compared with the annotation input by the worker.

[0065] In S803, the information processing device 100 compares the estimated attachment data acquired in S801 with the attachment data acquired in S802, and determines whether they match with each other in the concentration decrease determination unit 208. If they match, the process ends. If they do not match, the process proceeds to S804.

[0066] In S804, the information processing device 100 determines that the concentration of the worker is declining by the concentration decline determination unit 208. In this embodiment, when determining the target for concentration determination from among the targets to be assigned, the accuracy information included in the estimated assignment data is set to be equal to or greater than a first threshold value. By setting this first threshold value to a very high value, the estimated assignment data can be considered to be equivalent to the correct answer. This is because a very high value of the accuracy information of the estimated assignment data means that the characteristics for identifying the assignment data are prominent, and it can be considered that the situation is easy for the worker to judge.

[0067] As a result, if a worker gives a different attribute data from the estimated attribute data, it means that the worker made a mistake in a situation where it is easy to judge, and this is judged as a decrease in concentration.In addition, instead of using the estimated attribute data as data considered to be correct, it may be judged whether the attribute data obtained in S802 matches the attribute data given in advance by another worker or a verifier who verifies the work result to the same target.

[0068] This allows the worker to continue working as usual, assigning assignment data to the assignment targets, while being warned if their concentration is declining. By taking a break in response to the warning, it is possible to prevent the worker from forcing themselves to work when their concentration is declining and making mistakes.

[0069] In S803, when the comparison result between the annotation indicated by the attached data of the estimated attached data and the annotation input by the worker does not match, it is determined that the concentration has decreased, but this is not limited to this. For example, when the comparison result between the annotation indicated by the attached data of the estimated attached data and the annotation input by the worker is a synonym or indicates a higher-level concept and a lower-level concept, it may be determined that the concentration has not decreased.

[0070] <Embodiment 2> The functional configuration of the information processing device 100 of the second embodiment is shown in Fig. 9. Note that the same components as those shown in Fig. 2 are denoted by the same numbers as those used in Fig. 2. Note that the hardware configuration of the information processing device 100 of the second embodiment is the same as that of the first embodiment.

[0071] In FIG. 2, the task time measurement means 901 measures the time from when the target presentation unit 202 presents the target to the worker until the input is received by the target data input unit 203 as the task time.

[0072] An example of a user interface presented to a worker in embodiment 2 is shown in Fig. 10. The grant target presentation window 300, the grant target display area 310, the grant target change button 320, and the grant data input area 330 in Fig. 10(a) are the same as those shown in Fig. 3.

[0073] FIG. 10 shows an example in which annotation data is added to a plurality of regions in an annotation target. In the second embodiment, an image is used as an annotation target as in the first embodiment, but the present invention is not limited to this. For example, the annotation target may be a character string that constitutes a sentence. In addition, when the annotation target is a character string that constitutes a sentence, a specific word in the sentence may be specified and annotated.

[0074] In FIG. 10, a cursor 1001 is used to specify an area within the target to be attached. The area is specified by clicking and dragging a pointing device such as a mouse to specify a rectangular position range. Note that the method of specifying an area by clicking and dragging using a pointing device is one example. As another example, a known technique for specifying an area on a UI, such as tapping or dragging on a touch panel with a finger or electronic pen, may be used.

[0075] The area 1011 shown in FIG. 10(a) represents an area estimated by the appended data estimation unit 205. In the example of FIG. 10, the area is indicated as an estimated area by displaying the boundary line of the area with a dotted line. Note that the area may be indicated as an estimated area by changing the color of the boundary line or by changing the color inside the area, other than by making the boundary line a dotted line. FIG. 10(b) shows a state in which the area 1011, which is an estimated area, is selected. In the example of FIG. 10(b), the area is indicated as selected by displaying the boundary line of the area with a thick line. Area 1031 is an estimated area and is in a selected state, so it is displayed with a thick dotted line. When the operator selects area 1031, the operator can freely change the size of the area. To change the size, a handle 1032 is used. In the example of FIG. 10, the handles are placed at the four corners of the boundary line of the area and at the midpoints of each side, but the placement of the handles is only an example and the handles may be placed at other positions.

[0076] Area 1012 shown in Fig. 10(a) is an area newly generated by the operator using cursor 1001. Fig. 10(c) shows the state when area 1012 is selected. As with the inferred area described above, in Fig. 10(c) too, the border is displayed in thick lines to indicate that the area has been selected. As with Fig. 10(b), it is also possible to freely change the size of the selected area in Fig. 10(c) too. To change the size, use handle 1042. The handle is positioned in the same way as in the inferred area described above.

[0077] In addition, regardless of whether the area is an estimated area or a newly created area, the operator can add the added data to the area by inputting the added data into the added data input area 330 while the area is selected. In the example shown in Fig. 10(b), inputting data into the added data input area 330 adds the estimated area to the selected area 1031. In the example shown in Fig. 10(c), inputting data into the added data input area 330 adds the newly created area to the selected area 1041.

[0078] Moreover, as the estimated attached data, not only the area but also the annotation information is displayed in the estimated annotation information display area 1020. That is, the information processing device 100 estimates one or more areas of an object present in an image, and estimates the attached data of an object present in each estimated area. Furthermore, the information processing device 100 displays the area occupied by the detected object as an estimated area, and displays the estimated attached data of the detected object as estimated annotation information.

[0079] The annotation information displayed here is reference information for the worker, but it is merely the result of estimation by the pre-learning model and is not necessarily correct. Therefore, the worker checks whether the annotation displayed in the estimated annotation information display area 1020 is correct, and then inputs the displayed annotation if it is correct, or the correct annotation if it is incorrect, into the added data input area 330. Note that accuracy information may also be displayed.

[0080] 10 shows an example in which the estimated annotation information display area 1020 and the attached data input area 330 are separated, but they may be combined into one. Specifically, there is a method of displaying the annotation information included in the estimated attached data in a state in which it is input in the attached data input area 330.

[0081] Next, the process of determining an assignment target for concentration determination from among assignment targets performed by the assignment target determination unit 206 for concentration determination in the second embodiment will be described with reference to FIG.

[0082] In S1101 , the target for concentration assessment determining unit 206 acquires one target for concentration assessment from the target management unit 201 .

[0083] In S1102, the concentration determination target determination unit 206 acquires all of the estimated assigned data estimated for the target acquired in S1101 from the task result storage unit 204. Note that, as shown in FIG. 10, if it is necessary to assign assigned data to an area within the target, the estimated assigned data will also be information of a pair of the area within the target and the assigned data. Therefore, the assigned data estimation unit 205 also estimates the area to which the assigned data is to be assigned and the assigned data for that area. As a result, there may be multiple estimated assigned data for the target. In S1102, if there is multiple estimated assigned data, all of them are acquired.

[0084] In S1103, the concentration determination target determination unit 206 determines whether or not at least one of the estimated imparting data acquired in S1102 has a degree of accuracy equal to or less than the second threshold. If the result of the determination is that there is estimated imparting data whose degree of accuracy is equal to or less than the second threshold, the process proceeds to S1104, and if all the degrees of accuracy are greater than the second threshold, the process ends. Note that the second threshold may be, for example, a threshold set in advance by the user, or may use the tendency of probability included in the estimated imparting data for the imparting target managed by the imparting target management unit 201. The tendency of probability may be, for example, the average probability.

[0085] In this way, the information processing device 100 determines whether or not concentration is declining by using an estimated annotation with low accuracy, that is, an annotation target that is likely to require confirmation or correction by the worker. This makes it possible to accurately detect a decline in concentration, since when the worker's concentration is low, there is a high possibility that the estimated annotation will be approved without correction.

[0086] In S1104, the concentration determination target determination unit 206 sets a concentration determination flag for the target to be assigned and ends the process. Note that the concentration determination flag may be set as a column of the database when the target management unit 201 uses a database to manage the target, or may be assigned as metadata of the file when the target is a file.

[0087] The processes from S1101 to S1104 are performed for one grant target, and the grant target determination unit 206 for concentration ability assessment performs this process for all grant targets managed by the grant target management unit 201.

[0088] Next, the overall processing flow of the information processing apparatus 100 according to the second embodiment will be described with reference to FIG.

[0089] In S1201, the information processing device 100 stores in a queue of grant targets that the worker is working on from among the grant targets managed by the grant target management unit 201. The grant target storage queue may be created individually for each worker, or one queue may be shared by multiple workers. In the second embodiment, the information processing device 100 does not store grant targets in the queue of grant targets for concentration determination at the start of processing. This is because, in the second embodiment, a grant target that has a flag for concentration determination and has been presented to the worker at least once is determined to be a grant target for concentration determination. Details of the process of storing grant targets in the queue of grant targets for concentration determination will be described later with reference to FIG. 13.

[0090] In S1202, the information processing device 100 starts a timer for an interval of presentation of an object for concentration determination in order to obtain an elapsed time after presentation of the object for concentration determination. This timer is a timer for not performing the process of determining a decrease in concentration again until a predetermined period has elapsed.

[0091] In S1203, the information processing device 100 checks whether or not there is an assignment target in the assignment target storage queue managed by the assignment target management unit 201. If there is an assignment target, the process proceeds to S1204, and if there is no assignment target, the process ends.

[0092] In S1204, the information processing device 100 presents the attachment target and accepts the attachment data input by the worker. A detailed process flow will be described later with reference to FIG.

[0093] In S1205, the information processing device 100 judges whether it is time to present the concentration judgment target in the presentation timing judgment unit 207. Details will be described later with reference to Fig. 15. If it is judged that it is time to present the target as a result of the judgment, the process proceeds to S1206, and if it is judged that it is not time to present the target, the process proceeds to S1203.

[0094] In S1206, the information processing device 100 checks whether there is a target for concentration determination in the target storage queue for concentration determination managed by the target management unit 201. If there is a target for concentration determination, the process proceeds to S1207, and if there is not, the process proceeds to S1203.

[0095] In S1207, the concentration determination target presentation interval timer started in S1202 is reset.

[0096] In S1208, the information processing device 100 presents an assignment target for use in determining concentration and accepts assignment data input by the worker. Note that the process of S1204 and the process of S1208 are basically the same, but the assignment target presented to the worker is different. Specifically, a normal assignment target is presented in S1204, and an assignment target for use in determining concentration is presented in S1208. Also, the normal assignment target and the assignment target for use in determining concentration can be treated as the same by the worker. The details of the process flow will be described later with reference to FIG. 13 together with S1204.

[0097] In S1209, the information processing device 100 compares the attached data attached in S1208 with the attached data previously attached to the same attachment target. The comparison process will be described later with reference to FIG.

[0098] In S1210, if the comparison in S1209 shows that there is a difference between the two, the information processing apparatus 100 advances the process to S1211, and if they are the same, the information processing apparatus 100 advances the process to S1203.

[0099] In S1211, the information processing device 100 alerts the worker to the fact that his / her concentration is declining by using the target presentation unit 202. Note that the manner of the alert in S1211 is similar to the manner described in S510.

[0100] Next, the flow of processing for presenting an object to be given and accepting the given data input by the worker will be described with reference to FIG.

[0101] In S1301, the information processing device 100 acquires the grant target from the grant target management unit 201, and displays the grant target to the worker by the grant target presentation unit 202. When acquiring the grant target from the grant target management unit 201, the grant target is acquired from the grant target storage queue in the case of acquiring a normal grant target, and from the grant target storage queue for concentration determination in the case of acquiring a grant target for concentration determination.

[0102] In S1302, the information processing device 100 acquires the estimated attachment data estimated by the attachment data estimation unit 205 for the attachment target acquired in S1301 from the work result storage unit 204. If there are multiple estimated attachment data for the attachment target, all of them are acquired. In other words, if the attachment target is an image, if there are multiple estimated areas for the image, all of the multiple estimated attachment data related to these are acquired.

[0103] In S1303, the information processing device 100 displays the assignment target acquired in S1301 to the worker on the assignment target presentation unit 202, and at the same time, displays the estimated assignment data acquired in S1302 on the assignment target presentation unit 202 as shown in FIG.

[0104] In S1304, the information processing device 100 uses the task time measurement means 901 to start a task time timer in order to measure the task time of the worker.

[0105] In S1305, the information processing device 100 judges whether the worker has completed inputting the attachment data. If the inputting is completed, the process proceeds to S1306, and if the inputting is not completed, the determination is continued until the inputting is completed.

[0106] In S1306, the information processing apparatus 100 stops the task time timer started in S1304.

[0107] In S1307, the information processing device 100 acquires the additional data input by the worker using the additional data input unit 203.

[0108] In S1308, the information processing device 100 associates the assignment target presented in S1301, the assignment data acquired in S1307, and the elapsed time obtained by stopping the timer in S1306, and stores them as a work result in the work result storage unit 204. An example of the work result data will be described with reference to FIG.

[0109] In S1309, the information processing device 100 determines whether or not a concentration determination flag is set for the target presented in S1301. If the concentration determination flag is set, the process proceeds to S1310, and if not, the process ends.

[0110] In S1310, the information processing device 100 stores the grant target presented in S1301 in the grant target storage queue for concentration determination in the grant target management unit 201.

[0111] Examples of the estimated assignment data and the work result data will be described with reference to FIG.

[0112] 14(a), 14(b), 14(c), and 14(d) show examples of tables in the database stored in the work result storage unit 204. FIG.

[0113] The attachment target storage table 1410 shown in Fig. 14(a) is composed of three columns. Furthermore, one attachment target is represented by one row in the attachment target storage table 1410. Column 1411 is a column that stores an ID that identifies the image to which the attachment is to be applied. Column 1412 is a column that stores the file path of the image. Column 1413 is a column that stores a concentration determination flag that indicates whether the attachment target is for use in concentration determination.

[0114] The estimated attachment data storage table 1420 shown in FIG. 14(b) is composed of five columns. Furthermore, one piece of estimated attachment data is represented by one row in the estimated attachment data storage table 1420. Column 1421 is a column that stores an ID that identifies an estimation result. Column 1422 is a column that stores the estimation result. Column 1423 is a column that stores accuracy information of the estimation result. Column 1424 is a column that stores the ID of the image that is the estimated target. Column 1425 is a column that stores the ID of the area included in the image that is the estimated target. Note that the areas are managed in an area table 1440 described later.

[0115] The attached data storage table 1430 shown in FIG. 14(c) is composed of six columns. One piece of attached data is represented by one row in the attached data storage table 1430. Column 1431 is a column that stores an ID that identifies the attached data. Column 1432 is a column that stores the attachment result. Column 1433 is a column that stores the date and time when the attachment work of the attached data started. Column 1434 is a column that stores the date and time when the work of attaching the attached data ended. Column 1435 is a column that stores the ID of the image that is the estimated target. Column 1436 is a column that stores the ID of the area included in the image that is the attached target. The areas are managed in the area table 1440 described later.

[0116] Region table 1440 shown in FIG. 14(d) is composed of five columns. Furthermore, one region within the target is represented by one row in region table 1440. Column 1441 is a column that stores an ID that identifies the region. Column 1442 is a column that stores the X coordinate value of the starting point, starting from the top left point of the region. Column 1443 is a column that stores the Y coordinate value of the starting point, starting from the top left point of the region. Column 1444 is a column that stores the width value of the region. Column 1445 is a column that stores the height value of the region.

[0117] Next, a process flow for determining the timing for presenting the concentration determination target in the presentation timing determination unit 207 in the second embodiment will be described with reference to FIG.

[0118] In S1501, the information processing device 100 judges whether or not a third threshold value or more of time has passed since the last time the target for concentration judgment was displayed. If the third threshold value or more of time has passed, the process proceeds to S1502, and if the time is less than the third threshold value, the process ends. Note that the third threshold value is set so as not to continuously present the target for concentration judgment to the worker. If it is desired to frequently display the target for concentration judgment, the third threshold value may be set to 0, so that this judgment is not actually performed.

[0119] In S1502, the information processing device 100 acquires the most recent assigned data as a result of the work performed by the worker. An explanation will be given using the example of data shown in Fig. 14. The information processing device 100 acquires a row from the assigned data storage table 1430 in which the ID in column 1435 is the same as the ID of the assigned target on which the worker worked, the start date and time in column 1433 and the end date and time in column 1434 are the same, and the start date and time in column 1433 is the latest. In the example of the assigned data storage table 1430 shown in Fig. 14(c), the two IDs in column 1431, J and J+1, fall into this category.

[0120] In S1503, the information processing device 100 calculates the operation time by subtracting the start date and time in column 1433 from the end date and time in column 1434 of the attached data acquired in S1502. In the example of the attached data storage table 1430 shown in Fig. 14, the operation time is 88 seconds obtained by subtracting 11:23:52 on December 17, 2019 from 11:25:20 on December 17, 2019.

[0121] In S1504, the information processing device 100 extracts a work item from the attachment data acquired in S1502, which is the work result. The work item corresponds to, for example, labeling, area specification, and the like. In the example of FIG. 14, if there is at least one value in column 1432 in one or more extracted rows, labeling is extracted as a work item. Next, if there is at least one value in column 1436 in one or more extracted rows, area specification is extracted as a work item. Note that, although examples of labeling and area specification are shown in this embodiment, the work item is not limited to this. For example, if the attachment target is a video, the work item may be a section specification of the video, or if it is a character string, the work item may be a specification of a break between each phrase.

[0122] In S1505, the information processing device 100 calculates the amount of work by combining the number of pieces of assignment data assigned to the assignment target and the weighting according to the work item extracted in S1504. An explanation will be given using the example of data shown in FIG. 14. First, the number of pieces of assignment data is two from the assignment data acquired in S1502. Then, the weighting according to the work item extracted in S1504 is multiplied. Specifically, if the weighting for labeling is 1 and the weighting for area selection is 3, the amount of work required to generate assignment data is 4, which is the sum of 1 for labeling and 3 for area selection per piece of assignment data. In this explanation, since there are two pieces of assignment data, the amount of work is 4 x 2 = 8.

[0123] In S1506, the information processing device 100 divides the task time calculated in S1503 by the task amount calculated in S1505 to calculate a unit task time. In the example of Fig. 14, the task time is 88 seconds and the task amount is 8, so the unit task time is 11 seconds.

[0124] In S1507, the information processing device 100 determines whether the unit task time calculated in S1506 is equal to or greater than a fourth threshold. The fourth threshold may be set to one for the entire system, or may be set in advance for each worker. Alternatively, the unit task time may be calculated for each task, and the average or median may be used. If it is equal to or greater than the fourth threshold, the process proceeds to S1508, and if it is less than the fourth threshold, the process ends.

[0125] In S1508, since it was determined in S1507 that the unit work time is equal to or greater than the fourth threshold, the information processing device 100 determines that the worker may be less concentrated on the work than usual, and determines that it is time to present an annotation target for concentration determination. That is, the information processing device 100 determines the work efficiency based on the work time related to the annotation of the worker, and when the work efficiency is decreasing, determines whether the concentration is decreasing. In this way, it is possible to prevent mistakes in annotation by issuing a warning as necessary when the concentration starts to decrease.

[0126] Next, a process flow for determining whether or not concentration is decreasing in the second embodiment will be described with reference to FIG.

[0127] In S1601, the information processing device 100 acquires a first assigned data group assigned to the assignment target from the work result storage unit 204. The first assigned data is data most recently assigned by the worker, and is the assigned data assigned to the assignment target presented as the assignment target for concentration ability determination.

[0128] An example of data shown in FIG. 14 will be used for explanation. For the purpose of explanation, it is assumed that the current time is 15:36:12 on December 16, 2019, and the latest data in the attached data storage table 1430 shown in FIG. 14(c) is only up to the data with ID I+1. Under this premise, a row is obtained from the attached data storage table 1430 in which the ID in column 1435 is the same as the ID of the attached target worked on by the worker, the start date and time in column 1433 and the end date and time in column 1434 are the same, and the start date and time in column 1433 is the latest. In the example of the attached data storage table 1430 shown in FIG. 14, the two IDs in column 1431, I and I+1, fall into this category.

[0129] In S1602, the information processing device 100 acquires the second assigned data group assigned to the assignment target from the work result storage unit 204. The second assigned data is data previously assigned by the worker, and is assigned data assigned to the assignment target presented as a normal assignment target. An explanation will be given using the example of data shown in FIG. 14. For the sake of explanation, as in S1601, the explanation will be given on the assumption that the time is 15:36:12 on December 16, 2019. In addition, the assigned data storage table 1430 shown in FIG. 14(c) contains data up to ID I+1, and the assigned data assigned to image ID1 only has four IDs, 1, 2, I, and I+1.

[0130] From the attachment data storage table 1430, a row is obtained in which the ID in column 1435 is the same as the ID of the attachment target on which the worker worked, the start date and time in column 1433 and the end date and time in column 1434 are the same, and the start date and time in column 1433 is the latest. In the example of the attachment data storage table 1430 shown in Fig. 14, the two IDs in column 1431, 1 and 2, fall into this category.

[0131] In S1603, the information processing device 100 determines whether the number of pieces of appended data included in the first appended data group acquired in S1601 is the same as the number of pieces of appended data included in the second appended data group acquired in S1602. If they are the same, the process proceeds to S1604, and if they are different, the process proceeds to S1610. In the example of Fig. 14, the number of pieces of the first appended data group is two, and the number of pieces of the second appended data group is two, so they are the same.

[0132] In S1604, the information processing apparatus 100 stores the additional data included in the first additional data group acquired in S1601 in a determination queue.

[0133] In S1605, the information processing apparatus 100 judges whether or not the added data exists in the judgment queue. If the added data exists, the process proceeds to S1606, and if the added data does not exist, the process ends.

[0134] In S1606, the information processing apparatus 100 acquires the added data from the judgment queue as the first judgment target added data.

[0135] In S1607, the information processing device 100 judges whether or not the second group of attached data includes attached data having similar area information to the area information included in the first judgment target attached data acquired in S1606. If the area exists, the process proceeds to S1607. If the area does not exist, the process proceeds to S1610. Here, a similar area is judged based on whether or not the four pieces of information, the X and Y coordinates of the starting point, which is the upper left point of the area that defines the area, the width, and the height, are included within a certain width range.

[0136] An example of data shown in FIG. 14 will be used for explanation. In the above explanation, the data with IDs 1 and 2 in the data storage table 1430 are the first data group, and the data with IDs I and I+1 are the second data group. If the data with ID 1 in the first data group is the first target data, the data with ID 1 has an area ID of L, so the area data with ID L in the area table 1440 is referenced. In the area data with ID L, the X coordinate of the starting point is 8, the Y coordinate of the starting point is 14, the width is 489, and the height is 294. Next, these four values ​​are compared with the data included in the second data group. First, the data with ID I in the second data group has an area ID of M, so area M is confirmed. In the area M, the X coordinate of the starting point is 12, the Y coordinate of the starting point is 15, the width is 492, and the height is 297. If the conditions for similarity are set to a tolerance of ±10 for the four values ​​of the X coordinate of the origin, the Y coordinate of the origin, the width, and the height, then it can be said that the area with ID L is similar to the area with ID M. In the above explanation, the tolerance was set to ±10, but it is not limited to this.

[0137] In this way, when the comparison result shows that the previously input annotation being compared and the annotation most recently input by the worker are applied to different areas, the information processing device 100 can determine that the worker's concentration is declining.

[0138] In S1608, the information processing device 100 acquires the attached data determined to have a similar region in S1607 as the second determination target attached data. In the example of the data shown in Fig. 14, the attached data with ID I having the region with ID M determined to be a similar region in the above description becomes the second determination target attached data.

[0139] In S1609, the information processing device 100 judges whether the label of the first judgment target attached data is the same as the label of the second judgment target attached data. If they are the same, the process proceeds to S1605, and if they are different, the process proceeds to S1610. In the above example, the label of the first judgment target attached data is a standard car, and the label of the second judgment target attached data is a standard car, so it is judged that they match.

[0140] In S1609, if the comparison result between the previously input annotation and the most recently input annotation does not match, it is determined that the attached data is different and that the concentration has decreased, but this is not limited to the above. For example, if the comparison result between the previously input annotation and the most recently input annotation is a synonym or indicates a higher-level concept and a lower-level concept, it may be determined that the attached data matches and that the concentration has not decreased.

[0141] In S1610, the information processing device 100 determines that the additional data is different.

[0142] This makes it possible to know whether the assigned data that a worker assigned to the same target multiple times is different. Normally, the assigned data should be the same even if it is performed multiple times, but if it is different, it can be assumed that the worker was unable to make normal judgments. This makes it possible to determine that the worker's concentration is declining. Then, a warning is issued to the worker that their concentration is declining, and the worker is made to take a break in accordance with the warning, making it possible to prevent the worker from forcing themselves to work when their concentration is declining and making mistakes.

[0143] According to each embodiment described above, when a worker performs a task of assigning information to an object to be assigned, it is possible to judge whether the worker's recognition accuracy has decreased based on the result of the task performed by the worker, and if the recognition accuracy has decreased, a warning is issued. This allows the worker who receives the warning to stop the task, take a break, and then perform the task again in a state of concentration, making it possible to prevent work errors. In addition, since it is not necessary to use a device that is not closely related to the actual task to judge whether the worker's recognition accuracy has decreased, it is also possible to significantly suppress increases in costs.

[0144] <Other embodiments> The above-mentioned embodiment 1 and embodiment 2 may be appropriately combined. For example, the process of determining the decrease in concentration every predetermined number of times may be switched between the method of embodiment 1 and embodiment 2. Alternatively, the process of determining the decrease in concentration randomly may be switched between the method of embodiment 1 and embodiment 2.

[0145] In the above embodiment, the information processing device 100 is configured to determine the decrease in concentration, but the present invention is not limited to this. For example, the determination in S804 may be configured to determine the decrease in the recognition accuracy of the worker. That is, the above embodiment can be configured to detect not only the decrease in concentration but also the decrease in recognition accuracy caused by the decrease in eyesight due to eye fatigue of the worker, and to issue a warning or attention to the decrease.

[0146] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0147] 300 Grant target presentation window 310 Display area for granting 320 Change target button 330 Grant data input area

Claims

1. An acquisition means for acquiring a first annotation, which is an annotation for an annotation target, based on an input from a worker; a comparison means for comparing the first annotation acquired by the acquisition means with a second annotation to be compared; a determination means for determining a decrease in recognition accuracy of the worker based on a comparison by the comparison means when a unit work time calculated by dividing a work time related to the annotation of the worker by an amount of work becomes equal to or greater than a threshold value; and a warning means for issuing a warning when the determining means determines that the recognition accuracy of the worker has decreased.

2. The information processing device according to claim 1 , wherein the determination means determines that the recognition accuracy of the worker has deteriorated when the comparison result by the comparison means indicates that the first annotation and the second annotation do not match.

3. The method further includes estimating an annotation to be assigned to the annotation target, 3 . The information processing apparatus according to claim 1 , wherein the second annotation is an annotation estimated by the estimation means. 4 .

4. The information processing device according to claim 3 , wherein the estimation means estimates the annotation for the target by using a trained model that has been machine-learned.

5. 5 . The information processing apparatus according to claim 3 , wherein the second annotation is an annotation estimated by the estimation means with a degree of accuracy equal to or higher than a first threshold value.

6. The information processing apparatus according to claim 1 , wherein the second annotation is an annotation for the target based on an input from the worker prior to an input related to the first annotation.

7. The target of the grant is one in which the accuracy of the estimation by the estimation means is equal to or less than a second threshold value, The information processing apparatus according to claim 3 , wherein the second annotation is an annotation for the target based on an input from the worker prior to an input related to the first annotation.

8. 8. The information processing apparatus according to claim 1, wherein the warning means issues a warning by displaying information or outputting a sound.

9. The information processing device according to any one of claims 1 to 8, characterized in that the warning means displays information indicating that recognition accuracy may be declining or information indicating that there may be an error in the added annotation.

10. 10. The information processing apparatus according to claim 1, wherein the warning means displays information encouraging the user to take a break.

11. The information processing apparatus according to claim 1 , wherein the object to be tagged is an image or a text.

12. The information processing apparatus according to claim 1 , wherein the target to be tagged is a region including an object in an image.

13. The information processing apparatus according to claim 1 , wherein the annotation is a name of an object included in the target to which the annotation is to be added.

14. The information processing device according to any one of claims 1 to 13, characterized in that the determination means determines that the recognition accuracy of the worker has deteriorated when the comparison result by the comparison means indicates that the first annotation and the second annotation are assigned to different areas.

15. acquiring a first annotation based on an input from a worker, the first annotation being an annotation for an annotation target; when a unit work time calculated by dividing a work time related to the annotation of the worker by an amount of work becomes equal to or greater than a threshold value, a deterioration in recognition accuracy of the worker is determined based on a comparison between the acquired first annotation and a second annotation to be compared; A method for controlling an information processing apparatus, comprising: issuing a warning when it is determined that the recognition accuracy of the operator has decreased.

16. A program for causing a computer to function as the information processing device according to any one of claims 1 to 14.

Citation Information

Patent Citations

  • Article inspection device

    JP2007285705A

  • Learning model creation program, image identification information giving program, learning model creation device, image identification information giving device

    JP2012038244A

  • Annotation information application program and information processor

    JP2015166975A

  • Estimation system, learning device, learning method, estimation device, and estimation method

    JP2019159941A

  • Collaborative creation of annotation training data

    US20160147399A1