Information Processing Apparatus, Information Processing Method, and Information Processing Program

Abstract

To ensure the quality of processing results for multiple processing targets performed manually. [Solution] In an information processing system in which multiple worker devices, an administrator device, and an information processing device are connected to each other via a predetermined network N so that they can communicate via wired or wireless connections, the information processing device comprises a reception unit that receives processing results for multiple processing objects, including processing objects with correct answers linked to correct processing results, each time a worker completes processing for some of the processing objects, and a calculation unit that, when the processing object corresponding to the processing result received by the reception unit is a processing object with correct answers, calculates an evaluation value indicating an evaluation of the worker based on a comparison between the processing result received by the reception unit and the correct processing result.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Conventionally, there has been known a technique for ensuring the quality of processing results for a plurality of processing targets performed manually. For example, there is known a technique for assisting in creating teacher data for training a machine learning model. For example, there is known a technique for identifying a teacher image as a specific teacher image when there is a difference between a teaching class associated in advance with the teacher image and a class to which the teacher image is most frequently classified by a trained classifier.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is a demand for a technique that can ensure the quality of processing results for a plurality of processing targets performed manually.

Means for Solving the Problems

[0005] An information processing apparatus according to an embodiment includes a reception unit that receives, for each completion of processing on a part of a plurality of processing targets including a processing target with a correct processing result associated therewith, a processing result for the plurality of processing targets; and a calculation unit that calculates an evaluation value indicating an evaluation of the worker based on a comparison between the processing result received by the reception unit and the correct processing result when the processing target corresponding to the processing result received by the reception unit is the processing target with a correct processing result.

Brief Description of the Drawings

[0006]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Embodiments for Carrying Out the Invention

[0007] Hereinafter, embodiments for implementing the information processing apparatus, information processing method, and information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail with reference to the drawings. Note that the information processing apparatus, information processing method, and information processing program according to the present application are not limited by this embodiment. Also, in the following embodiments, the same parts are denoted by the same reference numerals, and duplicate descriptions are omitted.

[0008] (Embodiment) [1. Introduction] Conventionally, techniques for assisting in creating teacher data for supervised learning of a machine learning model have been known. Generally, teacher data for supervised learning consists of a pair of correct data and a label. Supervised learning is performed by giving an input (correct data) and its correct output (label) as teacher data and training a machine learning model so that it can produce the correct output when receiving a certain input. In the AI industry, the task of creating this teacher data (a pair of correct data and a label) is called annotation. The data to be annotated includes various types such as images, text, and audio depending on the application of the machine learning model. For example, when the application of the machine learning model is image recognition, the data to be annotated is an image.

[0009] Such annotation work is performed manually, but to create a high-precision machine learning model, a huge amount of teacher data, such as about 100,000 to 1,000,000 pieces, is required. Therefore, there are vendors who create teacher data. However, even if a requester who creates a machine learning model requests a vendor to create 500 sets of teacher data, there is a problem that the quality of the created teacher data (the accuracy of annotation is poor) is not guaranteed, for example, only 50 sets of teacher data at a level that can be used for learning are included in the delivered teacher data.

[0010] Therefore, in order to prevent the teacher data with poor annotation accuracy from being delivered as it is, the information processing apparatus 100 according to the present invention enables the annotator to evaluate the annotation accuracy during the process of creating the teacher data by the contractor, and to request the contractor to correct it from the requester when the accuracy is poor. Specifically, the information processing apparatus 100 periodically performs a progress review during the creation of the teacher data in order to evaluate the annotation accuracy. As a result, the information processing apparatus 100 can immediately detect a quality decline by periodically evaluating the quality through the progress review, so that an overall quality decline can be prevented. Therefore, the information processing apparatus 100 can ensure the quality of the processing results for a plurality of processing targets performed manually such as annotation.

[0011] [2. Outline of Information Processing] Hereinafter, the outline of the information processing method according to the embodiment will be described with reference to FIG. 1. FIG. 1 is an explanatory diagram showing the outline of the information processing method according to the embodiment. The annotation work according to the embodiment is performed by an operator of a contractor who has received a request from a requester selecting an area where an object is imaged from the received images. In FIG. 1, an example of evaluating the accuracy of teacher data created by an operator will be described by comparing the area of the image annotated by the requester (data with accurate annotation) with the area of the image annotated by the operator. Also, the requester manages the work by the operator. Therefore, hereinafter, the requester may sometimes be described as an administrator.

[0012] In FIG. 1, the requester requests the creation of teacher data for detecting a cat, which is an object included in the image. The requester selects five images in advance from 500 images and associates a correct selection area with each of the selected five images. The information processing apparatus 100 acquires 500 images including the five correct images (hereinafter, also referred to as images with answers) associated with the correct selection areas selected by the requester from the administrator device 20 used by the requester (administrator) (step S11).

[0013] Subsequently, when the information processing apparatus 100 receives 500 images, it determines the order in which the 500 images are to be processed by the operator. For example, the information processing apparatus 100 determines the order (hereinafter also referred to as the processing order) in which 5 correct images out of the 500 images are to be processed by the operator. Subsequently, when the information processing apparatus 100 determines the order in which the 5 correct images are to be processed by the operator, it determines the order in which the remaining 495 requested images are to be processed by the operator. For example, the information processing apparatus 100 determines to include correct images in the 1st, 101st, 201st, 301st, and 401st positions among the 500 images. Subsequently, when the information processing apparatus 100 determines the processing order of the 500 images received from the requester, it uploads the 500 images associated with the determined processing order to the cloud.

[0014] Also, the operator of the contractor who has received the annotation request from the requester accesses the cloud to which the image to be worked on has been uploaded via the operator device 10 used by the operator. The information processing apparatus 100 displays on the screen of the operator device 10 the 500 images one by one in the order in which the operator is to process them. The operator performs a selection process of selecting the area in which a cat is imaged from among the images displayed on the screen of the operator device 10 (step S12). For example, the operator performs a selection process of manually selecting seven rectangles (referred to as bounding boxes) that each enclose the areas in which seven cats are imaged from the 301st image. The information processing apparatus 100 receives the selected area of the image selected by the operator from the operator device 10 every time the operator completes the selection process of the area for one image.

[0015] In addition, when the image corresponding to the selection area received from the worker device 10 is an image with a correct answer, the information processing apparatus 100 calculates the degree of overlap (coincidence degree) between the selection area received from the worker device 10 and the correct selection area based on the comparison between the selection area received from the worker device 10 and the correct selection area (step S13). For example, the information processing apparatus 100 determines that the 301st image corresponding to the selection area received from the worker device 10 is an image with a correct answer. Subsequently, the information processing apparatus 100 calculates that the degree of overlap (coincidence degree) between the selection area of the worker and the correct selection area is 70% based on the comparison between the selection area received from the worker device 10 and the correct selection area.

[0016] When the information processing apparatus 100 calculates the degree of overlap (coincidence degree) between the selection area received from the worker device 10 and the correct selection area, it notifies the calculated result to the requester (step S14). In this way, the information processing apparatus 100 calculates the degree of overlap (coincidence degree) between the processing result of the selection process for the correct images included in the 1st, 101st, 201st, 301st, and 401st of the 500 images processed by the worker and the correct processing result and notifies the requester. The requester who has received the notification can see the result of the notification and know how the accuracy of the processing result (hereinafter also referred to as quality) of the selection process by the worker changes.

[0017] For example, when the quality of the processing result by the worker starts to decline during the work on a plurality of images, the information processing apparatus 100 detects the decline in quality and notifies the requester of the decline in quality. Thereby, the information processing apparatus 100 can, for example, prompt the requester to correct the decline in quality for the worker. Therefore, the information processing apparatus 100 can ensure the quality of the annotation for a plurality of images performed manually.

[0018] [3. Configuration of Information Processing System] First, with reference to FIG. 2, the configuration of the information processing system according to the embodiment will be described. FIG. 2 is a diagram showing a configuration example of the information processing system according to the embodiment. The information processing system 1 may include a worker device 10, a manager device 20, and an information processing device 100. The worker device 10, the manager device 20, and the information processing device 100 may be communicably connected by wire or wirelessly via a predetermined network N. Note that the information processing system 1 may include any number of worker devices 10, any number of manager devices 20, and any number of information processing devices 100.

[0019] The worker device 10 is an information processing device used by a worker. The worker device 10 is, for example, a smartphone, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), a head-mounted display, or the like. In the embodiment, an example in which the worker device 10 is a notebook PC or a desktop PC (hereinafter, may be simply referred to as a PC) will be described.

[0020] As shown in FIG. 2, depending on the worker who uses the worker device 10, the worker device 10 may be distinguished and described as worker devices 10-1 to 10-N. For example, the worker device 10-1 is a worker device 10 used by a worker U1 identified by the worker ID "U1". Also, for example, the worker device 10-2 is a worker device 10 used by a worker U2 identified by the worker ID "U2". Further, hereinafter, when the worker devices 10-1 to 10-N are described without particular distinction, they will be referred to as the worker device 10.

[0021] The management device 20 is an information processing device used by a manager who manages the work performed by workers. The management device 20 is, for example, a smartphone, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), a head-mounted display, or the like. In the embodiment, an example in which the management device 20 is a notebook PC or a desktop PC (hereinafter sometimes simply referred to as a PC) will be described.

[0022] The information processing device 100 is an information processing device that executes the processing described with reference to FIG. 1. The information processing device 100 receives from the worker device 10 a selection area for each of a plurality of images including a correct image with a correct selection area associated therewith when the worker completes the area selection process for one image. Further, when the image corresponding to the received selection area is a correct image, the information processing device 100 calculates an evaluation value indicating an evaluation of the worker based on a comparison between the received selection area and the correct selection area. Further, the information processing device 100 transmits information regarding the calculated evaluation value to the management device 20.

[0023] [4. Configuration of Information Processing Device] Next, the configuration of the information processing device 100 according to the embodiment will be described with reference to FIG. 3. FIG. 3 is a diagram showing a configuration example of the information processing device 100 according to the embodiment. The information processing device 100 may include a communication unit 110, a storage unit 120, and a control unit 130. Note that the information processing device 100 may include an input unit (for example, a keyboard, a mouse, etc.) that receives various operations from an administrator or the like of the information processing device 100, and a display unit (for example, a liquid crystal display, etc.) that displays various information.

[0024] (Communication Unit 110) The communication unit 110 is realized by, for example, a NIC (Network Interface Card) or the like. Then, the communication unit 110 is connected to the network by wire or wirelessly, and performs information transmission and reception, for example, between the worker device 10 and the management device 20.

[0025] (Memory unit 120) The memory unit 120 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. Specifically, the memory unit 120 stores, in association with each other, information that can identify each of a plurality of processing targets and the plurality of processing targets. For example, as an example of a processing target, the memory unit 120 stores, in association with each other, information (for example, an image ID) that can identify each of a plurality of images and the plurality of images. Further, the memory unit 120 stores, in association with each other, a processing target with an answer and the processing result of the answer. For example, as an example of a processing target, the memory unit 120 stores, in association with each other, a correct selection area selected by an annotation requester and an image with an answer.

[0026] (Control unit 130) The control unit 130 is a controller and is realized, for example, by various programs (corresponding to an example of an information processing program) stored in a storage device inside the information processing apparatus 100 being executed with the RAM as a work area by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like. Further, the control unit 130 is a controller and is realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0027] The control unit 130 has, as functional units, an acquisition unit 131, a determination unit 132, a provision unit 133, a reception unit 134, a calculation unit 135, a determination unit 136, and a notification unit 137, and may realize or execute the operations of information processing described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 2, and other configurations may be used as long as they can perform the information processing described later. Further, each functional unit indicates the function of the control unit 130 and does not necessarily have to be physically distinguished.

[0028] (Acquisition unit 131) The acquisition unit 131 acquires a plurality of processing targets including a processing target with a correct processing result associated therewith. Specifically, as an example of the processing target, the acquisition unit 131 acquires an image to be an object of an annotation operation. For example, the acquisition unit 131 acquires, from the administrator device 20 used by the requester (administrator), a plurality of images including an image with a correct answer associated with a selection area of the correct answer selected by the requester of the annotation.

[0029] Also, when the acquisition unit 131 acquires a plurality of processing targets including a processing target with a correct answer, the acquisition unit 131 associates information that can identify each of the plurality of processing targets with each of the plurality of processing targets and stores the same in the storage unit 120. For example, as an example of the processing target, when the acquisition unit 131 acquires a plurality of images to be an object of an annotation operation, the acquisition unit 131 associates information (for example, an image ID) that can identify each of the plurality of images with each of the plurality of images and stores the same in the storage unit 120.

[0030] Also, when the acquisition unit 131 acquires a processing target with a correct answer, the acquisition unit 131 associates the processing target with a correct answer with the correct processing result and stores the same in the storage unit 120. For example, as an example of the processing target, when the acquisition unit 131 acquires an image with a correct answer associated with a selection area of the correct answer selected by the requester of the annotation, the acquisition unit 131 associates the image with a correct answer with the selection area of the correct answer and stores the same in the storage unit 120.

[0031] (Determination unit 132) The determination unit 132 determines an order in which a worker processes a processing target with a correct answer among a plurality of processing targets. Specifically, when a plurality of processing targets are acquired by the acquisition unit 131, the determination unit 132 determines an order in which a worker processes a processing target with a correct answer among the plurality of processing targets acquired by the acquisition unit 131. For example, when a plurality of images are acquired by the acquisition unit 131, the determination unit 132 determines an order in which a worker processes an image with a correct answer among the plurality of images acquired by the acquisition unit 131.

[0032] More specifically, the determination unit 132 determines the order for having the operator process the processing targets with correct answers at predetermined intervals for a plurality of processing targets. For example, the determination unit 132 determines the order for having the operator process the images with correct answers at predetermined intervals for a plurality of images. In the example shown in FIG. 1, the determination unit 132 determines the order for having the operator process 5 images with correct answers for every 100 images out of 500 images. That is, the determination unit 132 determines that the order for having the operator process the images with correct answers among the 500 images is the 1st, 101st, 201st, 301st, and 401st.

[0033] Subsequently, when the determination unit 132 determines the order for having the operator process the processing targets with correct answers, the determination unit 132 determines the order for having the operator process other processing targets (hereinafter, also referred to as processing targets without correct answers) other than the processing targets with correct answers among the plurality of processing targets acquired by the acquisition unit 131. For example, when the determination unit 132 determines the order for having the operator process the images with correct answers, the determination unit 132 determines the order for having the operator process images other than the images with correct answers (hereinafter, also referred to as images without correct answers) among the plurality of images acquired by the acquisition unit 131. In the example shown in FIG. 1, when the determination unit 132 determines the order for having the operator process 5 images with correct answers, the determination unit 132 determines the order for having the operator process the remaining 495 images without correct answers among the 500 images.

[0034] When the determination unit 132 determines the processing order of a plurality of processing targets, the determination unit 132 uploads the plurality of processing targets associated with the determined processing order to a shared server (for example, a cloud server). For example, when the determination unit 132 determines the processing order of a plurality of images, the determination unit 132 uploads the plurality of images associated with the determined processing order to a shared server. In the example shown in FIG. 1, when the determination unit 132 determines the processing order of 500 images received from the requester, the determination unit 132 uploads the 500 images associated with the determined processing order to a shared server.

[0035] (Provision unit 133) The providing unit 133 provides a worker with a work screen for the worker to perform processing on a plurality of processing targets. Specifically, the providing unit 133 causes the worker device 10 to display a work screen for the worker to perform processing on a plurality of processing targets. More specifically, the providing unit 133 transmits content corresponding to a work screen for the worker to perform processing on a plurality of processing targets to the worker device 10. For example, the providing unit 133 transmits content corresponding to a work screen for the worker to perform a selection process of a selection area for a plurality of images to the worker device 10. The worker device 10 displays the work screen by displaying the content received from the information processing device 100 on the screen. For example, the providing unit 133 causes a plurality of images to be displayed one by one on the screen of the worker device 10 in the processing order determined by the determination unit 132. For example, the providing unit 133 causes content corresponding to the work screens shown in FIGS. 4 to 7 described later to be displayed on the screen of the worker device 10.

[0036] In addition, the providing unit 133 causes the worker device 10 to display a work screen for performing processing on a processing target with an answer and a work screen for performing processing on a processing target without an answer in a manner that cannot be visually distinguished. For example, in the images displayed on the screen of the worker device 10, the providing unit 133 causes a plurality of images to be displayed one by one on the screen of the worker device 10 in a manner that cannot be visually distinguished between an image with an answer and an image without an answer.

[0037] In addition, the providing unit 133 provides a manager with a management screen for displaying the processing status by the worker. Specifically, the calculation unit 135 calculates the ratio of the number of processed processing targets processed by each of a plurality of workers with respect to the number of a plurality of processing targets. The providing unit 133 provides a management screen for displaying, as an example of the processing status by the worker, the ratio of the number of processed processing targets by the worker calculated by the calculation unit 135. For example, when there are a plurality of workers, the providing unit 133 provides a management screen for displaying the ratio of the number of processed processing targets calculated by the calculation unit 135 for each of the plurality of workers.

[0038] In addition, the providing unit 133 provides a management screen for displaying the skill scores of workers. More specifically, the calculating unit 135 calculates a skill score indicating the processing skill of each of a plurality of workers based on the evaluation value. For example, when there are a plurality of workers, the providing unit 133 provides a management screen for displaying the skill scores of each of the plurality of workers. For example, the providing unit 133 transmits content corresponding to the management screen to the administrator device 20. The administrator device 20 displays the management screen by displaying the content received from the information processing device 100 on the screen.

[0039] Here, the management screen provided by the providing unit 133 will be described in detail with reference to FIGS. 12 to 13. FIGS. 12 to 13 are diagrams showing an example of the screen of the administrator device according to the embodiment.

[0040] In the example shown in FIG. 12, the providing unit 133 transmits content C21 including partial contents C211 to C216 in which the processing status (progress for today) and skill scores of each of a plurality of workers in charge of creating teacher data for the XX project are indicated by star marks to the administrator device 20.

[0041] In addition, the providing unit 133 provides content C21 including partial content C211 containing information visually notifying the administrator that the quality of the teacher data created by Mr. John, who is a worker, is high. For example, when the character "high quality" displayed in the partial content C211 is selected by the administrator, the providing unit 133 provides content C21 including the partial content C211 that can transition to a screen for displaying the detailed processing status of Mr. John.

[0042] In addition, the providing unit 133 provides a management screen including information for visually notifying an administrator of an operator whose evaluation value is lower than that of other operators among a plurality of operators. For example, as an example of a management screen including information for visually notifying an administrator of an operator whose evaluation value is lower than that of other operators, the providing unit 133 provides content C21 including partial content C214 including information for visually notifying an administrator that the quality of teacher data created by an operator, Mr. Bob, has deteriorated. For example, when the character "Security Detection" displayed in the partial content C214 is selected by the administrator, the providing unit 133 provides content C21 including partial content C214 that can transition to a screen displaying the detailed processing status of Mr. Bob.

[0043] In addition, the providing unit 133 provides content C21 including partial content C216 including information for visually notifying an administrator that the annotation work by an operator, Mr. Nigel, has stalled. For example, when the character "Work Stalled" displayed in the partial content C216 is selected by the administrator, the providing unit 133 provides content C21 including partial content C216 that can transition to a screen displaying the detailed processing status of Mr. Nigel.

[0044] In addition, the providing unit 133 provides content C21 for displaying the progress status of the entire XX project. For example, as an example of the progress status of the entire XX project, the providing unit 133 provides content C21 that displays that 386 out of 500 images to be processed in the entire XX project have been processed. In addition, the providing unit 133 provides content C21 for displaying the remaining working time. For example, the providing unit 133 provides content C21 that displays that the remaining working time of the XX project is 2 hours. When receiving the content C21, the administrator device 20 displays the received content C21 on the screen.

[0045] In the example shown in FIG. 13, the providing unit 133 transmits the content C22 including the partial content C221 that displays the detailed processing status by Mr. Nigel, who is an operator, to the administrator device 20. For example, when the character "work stagnation" displayed in the partial content C216 shown in FIG. 12 is selected by the administrator, the providing unit 133 transmits the content C22 corresponding to the screen that displays the detailed processing status of Mr. Nigel to the administrator device 20. When receiving the content C22, the administrator device 20 displays the received content C22 on the screen.

[0046] Also, the providing unit 133 provides the content C22 including the video C222 in which the processing by Mr. Nigel, who is an operator, is displayed in real time. For example, the providing unit 133 provides the content C22 including the video C222 in which the cursor of Mr. Nigel, who is an operator, is moving.

[0047] Also, the providing unit 133 provides the content C22 including the area C223 where the administrator and Mr. Nigel, who is an operator, can exchange messages through the screen. For example, the providing unit 133 provides the content C22 including the area F22 where messages can be input and the message sending button B22. Also, the providing unit 133 provides the content C22 including the area C223 where the reply message of Mr. Nigel, "Sorry, I won't do it anymore", for the administrator's message "You can't print it" is displayed.

[0048] (Reception unit 134) The reception unit 134 receives from the worker device 10 the processing results for a plurality of processing targets including the correct-answer processing targets associated with the correct processing results, each time the worker completes the processing for some of the processing targets. Specifically, the reception unit 134 receives from the worker device 10 the selection areas for a plurality of images including the image with the correct answer, for which the correct selection area is associated, each time the worker completes the area selection process for one image. More specifically, the reception unit 134 receives from the worker device 10 the processing result of the selection process by the worker, each time the worker completes the area selection process for one image, through the work screen provided by the provision unit 133.

[0049] In addition, the reception unit 134 receives from the worker device 10 the processing results for a plurality of processing targets including the correct-answer processing targets, in the order determined by the determination unit 132. Specifically, the reception unit 134 receives from the worker device 10 the processing results of the selection process for a plurality of images including the image with the correct answer, in the order determined by the determination unit 132. More specifically, the provision unit 133 transmits to the worker device 10 a work screen including one image at a time in the processing order determined by the determination unit 132, in the order determined by the determination unit 132. Then, the reception unit 134 receives from the worker device 10 the processing result of the selection process by the worker, each time the worker completes the area selection process for one image, through the work screen provided by the provision unit 133. In this way, the reception unit 134 receives from the worker device 10 the processing results of the selection process for the images, in the order determined by the determination unit 132.

[0050] Here, the process in which the reception unit 134 receives from the worker device 10 the processing result of the selection process for the image will be described in detail with reference to FIGS. 4 to 7. FIGS. 4 to 7 are diagrams showing an example of the screen of the worker device 10 according to the embodiment.

[0051] In the example shown in FIG. 4, the providing unit 133 transmits the content C11 corresponding to the work screen including the image G11 whose processing order determined by the determining unit 132 is the 15th to the worker device 10. When the worker device 10 receives the content C11, it displays the received content C11 on the screen. The worker device 10 accepts a selection process in which the worker selects the area O1 where the automobile is imaged from among the images G11 according to the operation of the worker. Subsequently, the worker device 10 accepts a selection operation of the transmission button B11 included in the content C11 according to the operation of the worker. When the worker device 10 accepts the selection operation of the transmission button B11, it transmits the information regarding the selection area O1 selected by the worker to the information processing device 100. The reception unit 134 accepts the selection area O1, which is the processing result of the selection process for the image G11 by the worker, from the worker device 10.

[0052] In the example shown in FIG. 5, the worker device 10 accepts a selection process in which the worker selects the area O2 where another automobile different from the automobile included in the selection area O1 is imaged from among the images G12 according to the operation of the worker. Subsequently, the worker device 10 accepts a selection operation of the skip button B12 included in the content C12 including the image G12 according to the operation of the worker. Here, the skip button B12 is a button for the worker to skip the selection process of the selection area for some reason. When the worker device 10 accepts the selection operation of the skip button B12, it transmits the information regarding the skip of the selection process of the selection area O2 selected by the worker to the information processing device 100. The reception unit 134 accepts the information regarding the skip of the selection process of the selection area O2 by the worker from the worker device 10.

[0053] In the example shown in FIG. 6, when the reception unit 134 receives information regarding skipping the selection process of the selection area O2, the providing unit 133 transmits the content C13 corresponding to the work screen on which the partial content G131 including the field F13 in which the operator can input the reason for skipping the selection process of the selection area O2 is displayed, to the operator device 10. Here, the partial content G131 includes a skip button B13 for transmitting the text input in the field F13 to the information processing device 100 and a cancel button B14 for not transmitting the text input in the field F13 to the information processing device 100.

[0054] Subsequently, in FIG. 7, the operator device 10 receives an operation of inputting, in accordance with the operator's operation, text indicating the reason for the operator to skip the selection process of the selection area O2, into the field F13'. In FIG. 7, the operator device 10 receives an input operation of the text "I want to check how far the inner car is processed." Subsequently, the operator device 10 receives a selection operation of the skip button B13 included in the partial content G131, in accordance with the operator's operation. When the operator device 10 receives the selection operation of the skip button B13, it transmits the text input by the operator to the information processing device 100. The reception unit 134 receives the text input by the operator from the operator device 10. When the text is received by the reception unit 134, the notification unit 137 transmits information regarding the text received by the reception unit 134, in association with the operator ID, to the administrator device 20.

[0055] Here, with reference to FIG. 8, the annotation according to the embodiment will be described in more detail. FIG. 8 is a diagram showing an example of the annotation according to the embodiment. In the example shown in FIG. 8, an image contains a plurality of objects such as "bottle", "cup", and three "cubes". At this time, the annotation work by the operator is performed by manually selecting five bounding boxes that respectively surround the areas where the "bottle", "cup", and three "cubes" included in the image are imaged. At this time, the reception unit 134 receives five bounding boxes corresponding to the "bottle", "cup", and three "cubes" selected by the operator from the operator device 10. Note that the teacher data generated by the annotation by selecting the bounding box as shown in FIG. 8 is used, for example, for training a machine learning model that performs object detection of a plurality of types included in the image.

[0056] Note that in the above-described example, the example in which the reception unit 134 receives the processing results for a plurality of processing targets including the processing target with a correct answer has been described, but the present invention is not limited to this. Specifically, the reception unit 134 may further receive the processing results for a plurality of processing targets that do not include the processing target with a correct answer each time the operator completes the processing for some of the processing targets.

[0057] (Calculation unit 135) When the processing target corresponding to the processing result received by the reception unit 134 is a processing target with a correct answer, the calculation unit 135 calculates an evaluation value indicating an evaluation of the operator based on a comparison between the processing result received by the reception unit 134 and the correct processing result. Specifically, when the processing result is received by the reception unit 134, the calculation unit 135 refers to the storage unit 120 to identify the processing target corresponding to the processing result. Subsequently, when the calculation unit 135 identifies the processing target corresponding to the processing result, it determines whether the identified processing target is a processing target with a correct answer. For example, the calculation unit 135 refers to the storage unit 120 to determine whether there is a correct processing result associated with the identified processing target. When the calculation unit 135 determines that there is a correct processing result associated with the identified processing target, it determines that the identified processing target is a processing target with a correct answer. Subsequently, when the calculation unit 135 determines that the processing target corresponding to the processing result received by the reception unit 134 is a processing target with a correct answer, it refers to the storage unit 120 to obtain information regarding the correct processing result associated with the processing target with a correct answer. Subsequently, when the calculation unit 135 obtains the information regarding the correct processing result, it calculates an evaluation value indicating an evaluation of the operator based on a comparison between the obtained correct processing result and the processing result received by the reception unit 134.

[0058] More specifically, based on a comparison between the processing result received by the reception unit 134 and the correct processing result, as an example of the evaluation value, the calculation unit 135 calculates the degree of overlap (degree of coincidence) between the processing result received by the reception unit 134 and the correct processing result. For example, when the image corresponding to the selection area received by the reception unit 134 is an image with a correct answer, the calculation unit 135 calculates an evaluation value indicating an evaluation of the operator based on a comparison between the selection area received by the reception unit 134 and the correct selection area. For example, based on a comparison between the selection area received by the reception unit 134 and the correct selection area, as an example of the evaluation value, the calculation unit 135 calculates the degree of overlap (degree of coincidence) between the selection area received by the reception unit 134 and the correct selection area.

[0059] For example, the calculation unit 135 calculates the IoU (Intersection over Union), which is the value obtained by dividing the area of the intersection of the selection area received by the reception unit 134 and the correct selection area by the area of the union of the selection area received by the reception unit 134 and the correct selection area. Here, the IoU is used as an evaluation index in object detection. For example, when the selection area received by the reception unit 134 and the correct selection area completely overlap (match), the value of the IoU is 1. On the other hand, when the selection area received by the reception unit 134 and the correct selection area do not overlap at all, the value of the IoU is 0. That is, it can be said that the larger the value of the IoU, the better the object detection. That is, the larger the value of the IoU calculated by the calculation unit 135, the higher the quality of the object detection by the operator. Therefore, the larger the value of the IoU calculated by the calculation unit 135, the higher the evaluation of the operator.

[0060] Note that in the above example, an example has been described in which the calculation unit 135 calculates an evaluation value indicating the evaluation of the operator based on the comparison between the processing result received by the reception unit 134 and the correct processing result, but it is not limited to this. Specifically, the calculation unit 135 may calculate an evaluation value indicating the evaluation of the operator based on the processing results for a plurality of processing targets that do not include the processing target with the correct answer received by the reception unit 134. For example, the calculation unit 135 may calculate another evaluation value indicating an evaluation other than the evaluation based on the comparison with the correct processing result based on the processing results for a plurality of processing targets that do not include the processing target with the correct answer received by the reception unit 134. For example, as an example of another evaluation value, the calculation unit 135 may calculate the processing speed at which the operator processes a plurality of processing targets. For example, the calculation unit 135 calculates the value obtained by dividing the number of a plurality of processing targets processed by the operator by the time required for the operator to process them as the processing speed.

[0061] (Determination unit 136) The determination unit 136 determines whether the quality of the processing result has deteriorated based on the evaluation value calculated by the calculation unit 135. Specifically, the determination unit 136 determines that the quality of the processing result has deteriorated when the evaluation value calculated by the calculation unit 135 is below the evaluation threshold. For example, the determination unit 136 determines that the quality of the processing result has deteriorated when the IoU value calculated by the calculation unit 135 is below the evaluation threshold.

[0062] (Notification unit 137) The notification unit 137 notifies an alert to the administrator who manages the work by the operator. Specifically, the notification unit 137 notifies an alert to the administrator when it is determined by the determination unit 136 that the quality has deteriorated. The alert notification process by the notification unit 137 will be described in detail with reference to FIG. 14. FIG. 14 is a diagram showing an example of the screen of the administrator device according to the embodiment.

[0063] In FIG. 14, the determination unit 136 determines that the quality of the processing result has deteriorated when the IoU value calculated by the calculation unit 135 is below the evaluation threshold of 50 points. For example, the calculation unit 135 calculates the IoU value for the 101st image as 45 points. Since the IoU value calculated by the calculation unit 135 is below the evaluation threshold of 50 points, the determination unit 136 determines that the quality of the annotation by the operator has deteriorated. Since it is determined by the determination unit 136 that the quality has deteriorated, the notification unit 137 notifies an alert to the administrator. In the example shown in FIG. 14, the notification unit 137 transmits the content C23 including the partial content C231 including the image G231 in which a surprised mark notifying the quality deterioration is displayed at the lower right of the thumbnail of the 101st image to the administrator device 20. Further, the notification unit 137 transmits the content C23 including the partial content C232 including the evaluation value S232 for the 101st image, the image G232 in which a surprised mark notifying the quality deterioration is displayed at the lower right of the thumbnail of the 101st image, and the message M23 regarding the quality deterioration detection to the administrator device 20. When receiving the content C23, the administrator device 20 displays the received content C23 on the screen.

[0064] [5. Modification Example] The information processing system 1 according to the above-described embodiment may be implemented in various different forms other than the above embodiment. Therefore, other embodiments of the information processing system 1 will be described below. Note that the same reference numerals are given to the same parts as in the embodiment, and the description thereof will be omitted.

[0065] [5-1. Processing Order of Processing Targets with Correct Answers] In the above-described embodiment, an example has been described in which the determination unit 132 determines the order so that the operator processes the processing targets with correct answers at a predetermined interval for a plurality of processing targets. However, the processing order of the processing targets with correct answers is not limited to this.

[0066] Specifically, the determination unit 132 determines to increase the number of processing targets with correct answers as the order to be processed by the operator becomes smaller. More specifically, the determination unit 132 determines to increase the number of processing targets with correct answers when the order to be processed by the operator is equal to or less than the first threshold value, rather than when the order to be processed by the operator exceeds the first threshold value. Thereby, the information processing apparatus 100 can confirm whether there is a discrepancy between the processing result by the operator and the correct processing result by the requester at an early stage of the processing for a plurality of processing targets.

[0067] Also, the determination unit 132 determines to increase the number of processing targets with correct answers as the order to be processed by the operator becomes larger. More specifically, the determination unit 132 determines to increase the number of processing targets with correct answers when the order to be processed by the operator is equal to or more than the second threshold value, rather than when the order to be processed by the operator is less than the second threshold value. Thereby, the information processing apparatus 100 can appropriately detect a decrease in quality due to the operator's fatigue, for example, by including more processing targets with correct answers in the latter half when the operator gets tired than in the first half among a plurality of processing targets.

[0068] Further, the determination unit 132 dynamically determines the order in which the correct-answer processing targets are to be processed by the operator based on the statistical information of the evaluation values calculated by the calculation unit 135. More specifically, as the statistical information, based on the change in the evaluation value, when the evaluation value falls below the third threshold value, the determination unit 132 determines to increase the number of correct-answer processing targets compared to before the evaluation value fell below the third threshold value. For example, the determination unit 132 determines different third threshold values for each operator based on the change in the evaluation value that is different for each operator. Subsequently, the determination unit 132 determines to increase the number of correct-answer processing targets compared to before the evaluation value fell below the third threshold value when the evaluation value falls below the third threshold value corresponding to each operator. Thereby, the information processing apparatus 100 can appropriately detect a quality degradation according to the operator by, for example, analyzing the tendency of quality degradation for each operator based on the change in the evaluation value and determining the third threshold value.

[0069] Also, as the statistical information, based on the change in the evaluation value, when the evaluation value shows a downward trend, the determination unit 132 determines to increase the number of correct-answer processing targets compared to before the evaluation value showed a downward trend. Thereby, the information processing apparatus 100 can prevent the quality degradation, for example, by including more correct-answer processing targets after the evaluation value shows a downward trend than before the evaluation value shows a downward trend.

[0070] Further, the determination unit 132 determines the number of correct-answer processing targets to be included with respect to the number of a plurality of processing targets based on the evaluation value calculated by the calculation unit 135. For example, based on the evaluation values of a plurality of operators, the determination unit 132 determines to increase the number of correct-answer processing targets to be included with respect to the number of a plurality of processing targets for an operator whose evaluation value is lower compared to other operators. Thereby, the information processing apparatus 100 can save the trouble of preparing a large number of correct-answer processing targets that are time-consuming to create and can efficiently allocate the limited number of correct-answer processing targets.

[0071] In addition, the reception unit 134 receives processing results for the same processing target from each of a plurality of workers. The calculation unit 135 calculates an evaluation value for a worker who has processed a processing result showing a tendency different from the processing results received from other workers among the plurality of workers to be lower than the evaluation value for other workers. Thereby, the information processing apparatus 100 can evaluate the work content of the worker even for a processing target without a correct answer. Further, the information processing apparatus 100 can save the trouble of preparing a large number of processing targets with a correct answer that are time-consuming to create, and can efficiently distribute the limited number of processing targets with a correct answer.

[0072] 〔5-2. Types of teacher data〕 In the above-described embodiment, an example in which the information processing apparatus 100 evaluates the quality of teacher data of a machine learning model that performs object detection among image recognition technologies has been described. However, the type of teacher data evaluated by the information processing apparatus 100 is not limited to this. Specifically, the above-described processing target may be text or voice in addition to an image according to the use of the machine learning model.

[0073] 〔5-3. Types of annotations〕 In the above-described embodiment, an example in which the information processing apparatus 100 evaluates the quality of teacher data created by annotation by selecting a bounding box shown in FIG. 8 has been described. However, the quality of teacher data created by other types of annotation may be evaluated. Here, the types of annotation will be described in detail with reference to FIGS. 9 to 11. FIGS. 9 to 11 are diagrams showing an example of annotation according to a modified example. The information processing apparatus 100 may evaluate the quality of teacher data created by the annotation shown in FIGS. 9 to 11.

[0074] In the example shown in FIG. 9, the image contains multiple objects such as "bottle", "cup", and three "cubes". At this time, the annotation work by the operator is performed by manually attaching labels indicating the types of objects such as "bottle", "cup", and "cube" contained in the image to the image. At this time, the reception unit 134 receives three labels corresponding to "bottle", "cup", and "cube" given by the operator from the operator device 10. Note that the teacher data generated by annotation by attaching labels as shown in FIG. 9 is used, for example, for training a machine learning model that performs image classification to classify an image into classes corresponding to the labels.

[0075] In the example shown in FIG. 10, the image contains multiple objects such as "bottle", "cup", and three "cubes". At this time, the annotation work by the operator is performed by color-coding and filling in the areas where "bottle", "cup", and the three "cubes" contained in the image are imaged into three types: "bottle", "cup", and "cube" for each object type. At this time, the reception unit 134 receives the areas color-coded and filled in by the operator into three types from the operator device 10. Note that the teacher data generated by annotation by color-coding areas for each object type as shown in FIG. 10 is used, for example, for training a machine learning model that performs semantic segmentation to classify all pixels in an image into classes.

[0076] In the example shown in FIG. 11, the image contains a plurality of objects such as "bottle", "cup", and three "cubes". At this time, the annotation work by the operator is performed by color-coding and filling the areas where "bottle", "cup", and three "cubes" included in the image are imaged into five types for each object. At this time, the reception unit 134 receives from the operator device 10 the areas color-coded and filled into five types by the operator. Note that the teacher data generated by the annotation that color-codes the areas for each object as shown in FIG. 11 is used, for example, for training a machine learning model that performs instant segmentation for classifying each pixel in the image for each object.

[0077] 〔6. Effect〕 As described above, the information processing apparatus 100 according to the embodiment includes a reception unit 134 and a calculation unit 135. The reception unit 134 receives the processing results for a plurality of processing targets including the processing targets with correct answers to which the correct processing results are associated, each time the operator completes the processing for some of the processing targets. When the processing target corresponding to the processing result received by the reception unit 134 is a processing target with a correct answer, the calculation unit 135 calculates an evaluation value indicating an evaluation of the operator based on a comparison between the processing result received by the reception unit 134 and the correct processing result.

[0078] In this way, the information processing apparatus 100 performs an interim review based on a comparison between the processing result by the operator and the processing target with a correct answer during the processing of a plurality of processing targets performed manually. Thereby, the information processing apparatus 100 can immediately detect when the quality deteriorates by periodically evaluating the quality through the interim review, and thus can prevent an overall deterioration in quality. Therefore, the information processing apparatus 100 can ensure the quality of the processing results for a plurality of processing targets performed manually such as annotation.

[0079] Further, the information processing apparatus 100 further includes a determination unit 132. The determination unit 132 determines the order in which the operator processes the processing targets with correct answers among the plurality of processing targets. The reception unit 134 receives the processing results for the plurality of processing targets including the processing targets with correct answers in the order determined by the determination unit 132. The calculation unit 135 calculates an evaluation value indicating the evaluation of the operator when the processing target corresponding to the processing result received by the reception unit 134 is a processing target with a correct answer.

[0080] Thereby, the information processing apparatus 100 can flexibly determine the order in which the processing targets with correct answers are processed according to the operator and the type of processing target, so that it is possible to more appropriately detect a decrease in quality according to the operator and the type of processing target.

[0081] Further, the determination unit 132 determines to increase the number of processing targets with correct answers as the order to be processed by the operator becomes smaller. Specifically, the determination unit 132 determines to increase the number of processing targets with correct answers when the order to be processed by the operator is equal to or less than the first threshold value rather than when the order to be processed by the operator exceeds the first threshold value.

[0082] Thereby, the information processing apparatus 100 can confirm whether there is a discrepancy between the processing result by the operator and the correct processing result by the requester at an early stage of the processing for the plurality of processing targets.

[0083] Further, the determination unit 132 determines to increase the number of processing targets with correct answers as the order to be processed by the operator becomes larger. Specifically, the determination unit 132 determines to increase the number of processing targets with correct answers when the order to be processed by the operator is equal to or greater than the second threshold value rather than when the order to be processed by the operator is less than the second threshold value.

[0084] Thereby, the information processing apparatus 100 can appropriately detect a decrease in quality due to the operator's fatigue, for example, by including more processing targets with correct answers in the latter half when the operator gets tired than in the first half among the plurality of processing targets.

[0085] Further, the determination unit 132 dynamically determines the order in which the correct-answer processing targets are to be processed by the operator based on the statistical information of the evaluation values calculated by the calculation unit 135. Specifically, as the statistical information, based on the change in the evaluation value, when the evaluation value falls below the third threshold, the determination unit 132 determines to increase the number of correct-answer processing targets compared to before the evaluation value fell below the third threshold.

[0086] Thereby, the information processing apparatus 100 can appropriately detect a decrease in quality according to the operator, for example, by analyzing the tendency of quality degradation for each operator based on the change in the evaluation value and determining the third threshold.

[0087] Also, as the statistical information, based on the change in the evaluation value, when the evaluation value shows a downward trend, the determination unit 132 determines to increase the number of correct-answer processing targets compared to before the evaluation value showed a downward trend.

[0088] Thereby, the information processing apparatus 100 can prevent a decrease in quality, for example, by including more correct-answer processing targets after the evaluation value shows a downward trend than before the evaluation value showed a downward trend.

[0089] Further, the determination unit 132 determines the number of correct-answer processing targets to be included with respect to the number of a plurality of processing targets based on the evaluation value calculated by the calculation unit 135. For example, the determination unit 132 determines to increase the number of correct-answer processing targets to be included with respect to the number of a plurality of processing targets for an operator whose evaluation value is lower than that of other operators based on the evaluation values of the plurality of operators.

[0090] Thereby, the information processing apparatus 100 can save the trouble of preparing a large number of correct-answer processing targets that are time-consuming to create and can efficiently allocate the limited number of correct-answer processing targets.

[0091] In addition, the reception unit 134 receives processing results for the same processing target from each of a plurality of workers. The calculation unit 135 calculates an evaluation value for a worker who has processed a processing result showing a tendency different from the processing results received from other workers among the plurality of workers to be lower than the evaluation value for other workers.

[0092] As a result, the information processing apparatus 100 can evaluate the work content of the worker even for a processing target without a correct answer. In addition, the information processing apparatus 100 can save the trouble of preparing a large number of processing targets with a correct answer that are time-consuming to create, and can efficiently distribute the limited number of processing targets with a correct answer.

[0093] In addition, the reception unit 134 further receives processing results for a plurality of processing targets not including the processing target with a correct answer each time the worker completes processing for some of the processing targets. The calculation unit 135 calculates an evaluation value indicating an evaluation of the worker based on the processing results received by the reception unit 134.

[0094] As a result, the information processing apparatus 100 can calculate another evaluation value (for example, the processing speed of the worker) indicating an evaluation other than the evaluation based on the comparison with the processing target with a correct answer.

[0095] In addition, the information processing apparatus 100 further includes a determination unit 136 and a notification unit 137. The determination unit 136 determines whether or not the quality of the processing result has deteriorated based on the evaluation value calculated by the calculation unit 135. The notification unit 137 notifies an alert to an administrator who manages the work by the worker. The determination unit 136 determines that the quality of the processing result has deteriorated when the evaluation value calculated by the calculation unit 135 is below the evaluation threshold. The notification unit 137 notifies an alert to the administrator when it is determined by the determination unit 136 that the quality has deteriorated.

[0096] As a result, when the information processing apparatus 100 detects a deterioration in the quality of the processing result by the worker, it can prompt the administrator to correct the deterioration in quality for the worker.

[0097] Further, the information processing apparatus 100 further includes a providing unit 133. The providing unit 133 provides a management screen for displaying the processing status of each of a plurality of workers to the administrator device 20 used by an administrator who manages the work of the workers. The calculating unit 135 calculates the ratio of the number of processed processing targets processed by each of the plurality of workers with respect to the number of a plurality of processing targets. The providing unit 133 provides a management screen that displays, as the processing status, the ratio calculated by the calculating unit 135 for each of the plurality of workers.

[0098] Thereby, the information processing apparatus 100 enables the administrator to check the processing status of the workers. For example, the administrator can allocate processing targets and processing targets with correct answers according to the processing status of the workers.

[0099] Further, the calculating unit 135 calculates a skill score indicating the processing skill of each of the plurality of workers based on the evaluation value. The providing unit 133 provides a management screen that displays the skill scores of each of the plurality of workers.

[0100] Thereby, the information processing apparatus 100 enables the administrator to check the skills of the workers. For example, the administrator can allocate processing targets and processing targets with correct answers according to the skills of the workers.

[0101] Further, the providing unit 133 provides a management screen including information for visually notifying the administrator of a worker among the plurality of workers whose evaluation value is lower than that of other workers.

[0102] Thereby, the information processing apparatus 100 enables the administrator to grasp a worker whose evaluation value is lower than that of other workers.

[0103] In addition, the information processing apparatus 100 further includes a providing unit 133 that causes a worker device used by a worker to display a work screen for the worker to perform processing on a plurality of processing targets. The providing unit 133 causes the worker device 10 to display, in a manner that cannot be visually distinguished, a work screen for performing processing on a processing target with a correct answer and a work screen for performing processing on other processing targets other than the processing target with a correct answer.

[0104] As a result, the information processing apparatus 100 cannot distinguish, from the worker, a processing target with a correct answer and other processing targets other than the processing target with a correct answer, so that it is possible to prevent the worker from performing processing carefully only when performing processing on the processing target with a correct answer.

[0105] In addition, the processing target is a teacher image for use in training a machine learning model that recognizes an object included in an image. Further, the processing is a selection process in which a worker selects an area where an object is imaged from the image. The reception unit 134 receives a selection area for a plurality of images including an image with a correct answer in which the correct selection area is associated, each time the worker completes the selection process for the area of a part of the images. The calculation unit 135 calculates an evaluation value indicating an evaluation of the worker based on a comparison between the selection area received by the reception unit 134 and the correct selection area when the image corresponding to the selection area received by the reception unit 134 is an image with a correct answer.

[0106] In this way, the information processing apparatus 100 performs a progress review based on a comparison between the selection area selected by the worker and the correct selection area during the annotation of a plurality of images performed manually. As a result, the information processing apparatus 100 can immediately detect a case where the quality of the annotation has deteriorated by periodically evaluating the quality of the annotation through the progress review, so that it is possible to prevent a decrease in the overall quality of the annotation. Therefore, the information processing apparatus 100 can ensure the quality of the annotation and the like.

[0107] 〔7. Hardware Configuration〕 Also, the information processing apparatus 100 according to the above-described embodiments is realized by a computer 1000 having a configuration as shown in FIG. 14, for example. FIG. 14 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing apparatus 100. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0108] The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400 and controls each part. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, a program dependent on the hardware of the computer 1000, and the like.

[0109] The HDD 1400 stores a program executed by the CPU 1100, data used by such a program, and the like. The communication interface 1500 receives data from other devices via a predetermined communication network and sends it to the CPU 1100, and sends data generated by the CPU 1100 to other devices via a predetermined communication network.

[0110] The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse via the input / output interface 1600. The CPU 1100 acquires data from the input device via the input / output interface 1600. Also, the CPU 1100 outputs the generated data to the output device via the input / output interface 1600.

[0111] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads such a program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0112] For example, when the computer 1000 functions as the information processing apparatus 100 according to the embodiment, the CPU 1100 of the computer 1000 realizes the functions of the control unit 130 by executing the program loaded onto the RAM 1200. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800. As another example, these programs may be acquired from another device via a predetermined communication network.

[0113] As described above, some of the embodiments of the present application have been described in detail with reference to the drawings. However, these are merely examples, and the present invention can be implemented in other forms with various modifications and improvements based on the knowledge of those skilled in the art, including the aspects described in the column of the disclosure of the invention.

[0114] [8. Others] Also, among the respective processes described in the above embodiments and modification examples, all or part of the processes described as being automatically performed can be manually performed, or all or part of the processes described as being manually performed can be automatically performed by a known method. In addition, regarding the processing procedures, specific names, and information including various data and parameters shown in the above documents and drawings, they can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

[0115] In addition, each component of each of the illustrated devices is functionally conceptual and does not necessarily have to be physically configured as shown in the drawings. That is, the specific form of distribution and integration of each device is not limited to that shown in the drawings, and all or part of it can be functionally or physically distributed and integrated in any unit according to various loads, usage situations, etc.

[0116] In addition, the above-described embodiments and modifications can be appropriately combined within a range that does not cause contradictions in the processing content.

[0117] In addition, the above-described "section (section, module, unit)" can be read as "means", "circuit", etc. For example, the calculation section can be read as a calculation means or a calculation circuit.

Explanation of Reference Numerals

[0118] 1 Information processing system 10 Operator device 20 Administrator device 100 Information processing device 110 Communication section 120 Storage section 130 Control section 131 Acquisition section 132 Decision section 133 Provision section 134 Reception section 135 Calculation section 136 Judgment section 137 Notification section

Claims

1. A reception unit that receives, from an operator, processing results for a plurality of processing targets including a processing target with a correct processing result associated therewith, each time the operator completes processing for a part of the processing targets; A determination unit that determines a processing order of the plurality of processing targets; A provision unit that transmits content for the operator to perform processing on each of the plurality of processing targets; A calculation unit that calculates a processing speed based on the number of the part of the processing targets processed by the operator and the time required for the operator to process the part of the processing targets until the operator completes processing for the part of the processing targets after transmitting the content with the youngest processing order; The information processing apparatus comprising: wherein the provision unit displays, on a management screen for a manager who manages the work of the operator, evaluation information based on the processing speed during the work of the operator on the plurality of processing targets on the management screen; An information processing apparatus.

2. wherein the provision unit further displays, on the management screen, notification information for visually notifying the manager that the work by the operator has stalled, based on the processing speed; The information processing apparatus according to claim 1.

3. The information processing apparatus further comprising a determination unit that determines whether or not the quality of processing results for the plurality of processing targets has deteriorated based on the processing speed; wherein the determination unit determines that the quality of processing results for the plurality of processing targets has deteriorated when the processing speed is lower than a speed threshold; The information processing apparatus according to claim 1.

4. The information processing apparatus further comprising a notification unit that notifies alert information to the manager; wherein the notification unit transmits the alert information to a manager device having the management screen when the determination unit determines that the quality of processing results for the plurality of processing targets has deteriorated; The information processing apparatus according to claim 3.

5. wherein the notification unit transmits, as the alert information, content including a thumbnail of the processing target being processed by the operator when it is determined that the quality of processing results for the plurality of processing targets has deteriorated and a mark for notifying that the quality of processing results for the plurality of processing targets has deteriorated to the manager device; The information processing apparatus according to claim 4.

6. wherein the notification unit transmits, as the alert information, content including a message for notifying that the deterioration in the quality of processing results for the plurality of processing targets has been detected to the manager device; The information processing apparatus according to claim 4.

7. The calculation unit when the processing target corresponding to the processing result received by the reception unit is the processing target with correct answer, further calculates an evaluation value for the processing result of the operator for the processing target with correct answer based on the comparison between the processing result received by the reception unit and the correct processing result, The providing unit in addition to the processing speed, causes the management screen to display evaluation information based on the evaluation value. The information processing apparatus according to claim 1.

8. An information processing method executed by a computer, a reception step of receiving, by an operator, processing results for a plurality of processing targets including a processing target with correct answer associated with a correct processing result each time the operator completes processing for some of the processing targets, a determination step of determining the processing order of the plurality of processing targets, a providing step of transmitting, to the operator, content for performing processing on each of the plurality of processing targets, a calculation step of calculating a processing speed calculated based on the number of the some of the processing targets processed by the operator and the time required for the operator to process the some of the processing targets until the operator's completion of processing for the some of the processing targets is received after transmitting the content with the youngest processing order, including The providing step displays, on a management screen for a manager who manages the work of the operator, evaluation information based on the processing speed during the work of the operator on the plurality of processing targets on the management screen. Information processing method.

9. a reception procedure of receiving, by an operator, processing results for a plurality of processing targets including a processing target with correct answer associated with a correct processing result each time the operator completes processing for some of the processing targets, a determination procedure of determining the processing order of the plurality of processing targets, a providing procedure of transmitting, to the operator, content for performing processing on each of the plurality of processing targets, a calculation procedure of calculating a processing speed calculated based on the number of the some of the processing targets processed by the operator and the time required for the operator to process the some of the processing targets until the operator's completion of processing for the some of the processing targets is received after transmitting the content with the youngest processing order, is executed by a computer, The providing procedure In a management screen for a manager who manages the work of the operator, evaluation information based on the processing speed is displayed on the management screen during the work of the operator on the plurality of processing targets. Information processing program.

Images