Contribution determination method, contribution determination device, and recording medium
By calculating the operator's contribution to image annotation work and calculating the reward based on the contribution, the problem of crowdsourcing workers being able to perform homework at home is solved, and the work progress and efficiency are improved.
Patent Information
- Application Number
- CN201910788142.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-08-29
- Filing Date
- 2019-08-26
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2039-08-26
AI Technical Summary
When using crowdsourcing methods to perform image annotation operations, the operators perform homework at home and there is a problem that it takes a long time to start and complete the homework. The incentive measures are not effective enough, resulting in slow progress of the work.
Through the contribution degree determination method executed by the computer, the update history and pre-determined weighting rules are used to calculate the operator's contribution in the job order, give the operator the first annotation job to the operator, and calculate the reward based on the contribution degree Amount to encourage operators to speed up the progress of their work.
It effectively shortens the completion time of annotation jobs, improves the progress of jobs, promotes active participation by operators, and saves energy consumption.
Smart Images

Figure CN110874562B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a contribution determination method, a contribution determination device, and a recording medium. Background Art
[0002] In recent years, a machine learning method called deep learning, a type of neural network, has attracted attention. In deep learning, learning data that is a set of bounding boxes representing the position of the recognized object on the image and correct labels representing the type of the recognized object are used to perform learning processing, thereby achieving high-precision object recognition.
[0003] As a method for preparing a large number of annotated images, there is a method of using crowdsourcing. Here, crowdsourcing is a mechanism for entrusting a job (task) to an unspecified number of people (operators) through the Internet. Therefore, by using crowdsourcing, multiple operators can perform annotation work, that is, find objects such as people required for learning processing from images such as image frames, and add a bounding box representing the area where the object is reflected and a label such as the type to the image of the recognition object. In this way, it is possible to suppress costs and prepare a large number of annotated images.
[0004] Here, for example, Patent Document 1 discloses a technology that determines the equivalent remuneration of each operator based on information indicating the location and time of each operator's work. This allows the operator who commissions the work to have multiple operators perform a larger amount of work without a set budget.
[0005] Prior art literature
[0006] Patent Literature
[0007] Patent Document 1: Japanese Patent Application Publication No. 2017-156815
[0008] However, most of the workers entrusted by crowdsourcing are people who work at home and work in their own free time, so it may take time to start and complete the entrusted work. In addition, even in the above-mentioned Patent Document 1, the incentive for accelerating the time it takes for the worker to start and complete the work does not work, so it may take time to start and complete the entrusted work. Summary of the invention
[0009] The present disclosure has been made in view of the above-mentioned situation, and an object of the present disclosure is to provide a contribution degree determination method, a contribution degree determination device, and a recording medium that can enable a worker utilizing crowdsourcing to start work more quickly.
[0010] In order to achieve the above-mentioned purpose, a contribution determination method of one embodiment of the present invention is a contribution determination method executed by a computer, comprising: an update history acquisition step, obtaining from a first storage device, an update history of annotation jobs corresponding to one or more object data of an annotation job, i.e., a job completed by multiple operators utilizing crowdsourcing; and a contribution calculation step, referring to the update history acquired in the update history acquisition step, calculating the job contribution of each of multiple operator IDs representing the multiple operators according to a predetermined weighting rule stored in a second storage device; in the contribution calculation step, the job contribution is calculated by weighting the operator ID representing the operator who first performed the annotation job on the one or more object data in the job sequence more heavily than the operator ID representing the operator who performed the annotation job later in the job sequence.
[0011] In addition, these overall or specific methods can be implemented through systems, methods, integrated circuits, computer programs, or recording media such as computer-readable CD-ROMs, or through any combination of systems, methods, integrated circuits, computer programs, and recording media.
[0012] According to the present disclosure, it is possible to realize a contribution degree determination method, a contribution degree determination device, and a recording medium that enable workers who utilize crowdsourcing to start work more quickly. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 This is a diagram showing an example of the overall configuration of the system according to the first embodiment.
[0014] Figure 2 This is a diagram showing an example of a still image stored in the sensor data DB according to the first embodiment.
[0015] Figure 3 This is a diagram showing an example of an annotation work screen provided by the annotation tool unit according to the first embodiment.
[0016] Figure 4A This is a diagram showing an example of an annotation work screen provided by the annotation tool unit according to the first embodiment.
[0017] Figure 4B This is a diagram showing an example of an annotation work screen provided by the annotation tool unit according to the first embodiment.
[0018] Figure 4C This is a diagram showing an example of an annotation work screen provided by the annotation tool unit according to the first embodiment.
[0019] Figure 5 This is a diagram showing an example of annotation work data stored in the annotation work data DB according to the first embodiment.
[0020] Figure 6 This is a diagram showing an example of the detailed configuration of the amount calculation unit according to the first embodiment.
[0021] Fig. 7A This is a flowchart showing an example of the operation of the contribution determination device according to the first embodiment.
[0022] Figure 7B Yes means Fig. 7A Flowchart of a detailed operation example of step S20 shown in FIG.
[0023] Fig. 8A This is a diagram showing another example of the annotation work screen provided by the annotation tool unit according to the first embodiment.
[0024] Figure 8B It means in Fig. 8A The illustrated annotation work screen is a diagram of an example of the annotation work screen when the annotation work is reflected.
[0025] Figure 8C It means in Fig. 8A The illustrated annotation work screen is a diagram of an example of the annotation work screen when the annotation work is reflected.
[0026] Fig. 9 This is a diagram showing an example of the overall configuration of a system according to the second embodiment.
[0027] Fig.10 Yes means Fig. 9 A diagram showing an example of a detailed configuration of a determination unit shown.
[0028] Fig.11 This is a diagram showing an example of an annotation work screen provided by the annotation tool unit according to the second embodiment.
[0029] Fig.12 This is a diagram showing an example of an annotation work screen provided by the annotation tool unit according to the second embodiment.
[0030] Fig.13 This is a flowchart showing the work completion determination processing of the contribution degree determination device according to the second embodiment.
[0031] Fig.14 This is a flowchart showing the inappropriate work determination processing of the contribution degree determination device according to the second embodiment. DETAILED DESCRIPTION
[0032] A contribution determination method according to one aspect of the present invention is a contribution determination method executed by a computer, comprising: an update history acquisition step, in which an update history of annotation jobs corresponding to one or more object data of an annotation job, i.e., a job completed by multiple operators utilizing crowdsourcing, is acquired from a first storage device; and a contribution calculation step, in which the job contribution of each of multiple operator IDs representing the multiple operators is calculated according to a predetermined weighting rule stored in a second storage device with reference to the update history acquired in the update history acquisition step; in the contribution calculation step, the job contribution is calculated by weighting the operator ID representing the operator who first performed the annotation job on the one or more object data in the job sequence more heavily than the operator ID representing the operator who performed the annotation job later in the job sequence.
[0033] This enables workers who utilize crowdsourcing to start work more quickly.
[0034] Here, for example, the one or more object data are one or more still images, and the annotation operation is a task of adding bounding boxes that respectively enclose one or more object objects respectively reflected in the one or more still images, and adding labels representing each of the one or more object objects to the bounding boxes.
[0035] Furthermore, for example, the one or more still images may be images captured by a vehicle-mounted camera mounted on a vehicle or a monitoring camera installed at a predetermined position, and the one or more target objects may include a person reflected in the still image.
[0036] In addition, for example, the one or more object data are one or more time series data, and the annotation operation is a job of adding time intervals to distinguish one or more states contained in the one or more time series data, and adding a label representing the corresponding state to each of the one or more time intervals.
[0037] In addition, for example, the one or more time series data may be sensor data about the vehicle obtained by the sensor simultaneously with a plurality of images taken by a vehicle-mounted camera mounted on the vehicle, and the one or more states may be at least one of the driving state of the vehicle, the driving location of the vehicle, the surrounding environment of the vehicle, and the condition of the road on which the vehicle is driving.
[0038] In addition, for example, in the contribution calculation step, the work contribution is calculated by weighting the worker ID of the worker who performed the annotation work last in the work sequence for the one or more object data more heavily than the worker ID of the worker who performed the annotation work from the later to the last in the work sequence.
[0039] In addition, for example, it may also include: an acceptance step of accepting the setting of an amount for each object data; and a calculation step of calculating, as a reward, an amount for each operator ID obtained by multiplying the work contribution of each operator ID calculated in the contribution calculation step by the amount accepted in the acceptance step.
[0040] In addition, for example, it may also include a determination step, in which it is determined whether the annotation work on each of the one or more object data is completed. In the determination step, when it is detected that an annotation work with a correction amount smaller than the correction amount of an annotation work previously performed in the work sequence has been performed a specified number of times on one of the object data, further annotation work on the one object data is locked, and it is determined that the annotation work on the one object data is completed.
[0041] In addition, for example, in the determination step, further, when a second annotation job is detected with respect to one of the object data and the amount of correction is greater than the amount of correction of the first annotation job performed previously in the job sequence, a notification is sent prompting confirmation of whether the second annotation job with respect to the one object data is an improper job.
[0042] In addition, for example, in the determination step, further, when it is detected that a second annotation operation has been performed on one of the object data with a correction amount larger than the correction amount of a first annotation operation performed previously in the operation sequence, when it is determined that the annotation operation on the one object data has been completed, the number of times regarding the one object data is notified.
[0043] In addition, a contribution determination device of one method of the present invention comprises: an update history acquisition unit, which acquires, from a first storage device, update records of annotation jobs corresponding to one or more object data of the annotation job, i.e., a job in which annotations are added and completed by multiple operators utilizing crowdsourcing; and a contribution calculation unit, which calculates the job contribution of each of multiple operator IDs representing the multiple operators according to a predetermined weighting rule stored in a second storage device, with reference to the update records acquired in the update history acquisition unit; the contribution calculation unit calculates the job contribution by weighting the operator ID representing the operator who first performed the annotation job on the one or more object data in the job sequence more heavily than the operator ID representing the operator who performed the annotation job later in the job sequence.
[0044] In addition, these overall or specific methods can be implemented through systems, methods, integrated circuits, computer programs, or recording media such as computer-readable CD-ROMs, or through any combination of systems, methods, integrated circuits, computer programs, or recording media.
[0045] The following specifically describes a contribution determination method of one mode of the present disclosure with reference to the accompanying drawings. In addition, the embodiments described below are all used to represent a specific example of the present disclosure. The numerical values, shapes, materials, constituent elements, configuration positions of constituent elements, etc. shown in the following embodiments are used as examples and are not intended to limit the present disclosure. In addition, the constituent elements that are not recorded in the independent claims representing the highest-level concepts among the constituent elements of the following embodiments are described as arbitrary constituent elements. In addition, in all embodiments, the contents can also be combined.
[0046] (Implementation Method 1)
[0047] [Overall structure of the system]
[0048] Figure 1 This is a diagram showing an example of the overall configuration of the system according to the first embodiment.
[0049] like Figure 1 As shown, the system of Embodiment 1 includes a contribution determination device 10, a server 20, and a plurality of operation terminals 30. The contribution determination device 10, the server 20, and the plurality of operation terminals 30 are connected via a network 40. The server 20 and the contribution determination device 10 may be connected via the network 40, or may be directly connected in a wired or wireless manner. In addition, it may be configured so that part or all of the contribution determination device 10 is included in the server 20.
[0050] [Structure of Server 20]
[0051] The server 20 includes an annotation tool unit 201, a sensor data DB 202, an annotation work data DB 203, and a remuneration DB 204. The server 20 is implemented by, for example, a computer including a processor (microprocessor), a memory, a communication interface, and the like.
[0052] <Sensor data DB202>
[0053] The sensor data DB 202 is a storage device that stores object data for multiple operators to perform annotation work using crowdsourcing. The sensor data DB 202 can be implemented by a semiconductor memory, a hard disk, etc. Here, the object data can be one or more still images. In this case, the one or more still images are images taken by, for example, a vehicle-mounted camera installed in a vehicle or a monitoring camera installed at a predetermined position.
[0054] Figure 2 This is a diagram showing an example of a still image stored in the sensor data DB 202 according to the first embodiment. Figure 2 The still image 51 shown is an image taken by a vehicle-mounted camera, and shows objects 511 and 512 representing people and an object 513 representing a car.
[0055] In addition, the object data is not limited to the case of one or more still images, but may also be one or more time series data. In this case, the one or more time series data may be motion images taken by a vehicle-mounted camera installed in the vehicle, or sensor data about the vehicle obtained by a sensor simultaneously with the continuous still images (hereinafter referred to as motion images). Here, the sensor data about the vehicle is, for example, the speed, acceleration, GPS or CAN data of the vehicle. In addition, the one or more time series data may be motion images taken by a monitoring camera, or may be sensor data obtained by life sensing or environmental sensing.
[0056] <Annotation Tool Section 201>
[0057] The annotation tool unit 201 selects object data for which an annotation operation is requested from a plurality of object data stored in the sensor data DB 202 , and provides an annotation operation screen for the selected object data to a plurality of operation terminals 30 via the network 40 .
[0058] Figure 3 is a diagram showing an example of an annotation work screen provided by the annotation tool unit 201 of the first embodiment. Figure 2 The same elements are denoted by the same reference numerals, and detailed description thereof will be omitted.
[0059] Figure 3The annotation operation screen 50 shown is provided by a web page or the like, and includes a Run button 50a, a data selection area 50b, and a Save button 50c. In addition, the annotation operation screen 50 includes an image area in which a still image 51 is displayed, and a label selection area 53. The label selection area 53 includes classification labels such as pedestrians, bicycles + passengers, cars, trucks, and motorcycles + passengers. The classification label is a label for a bounding box added to indicate the position of an object included in the still image 51, and one of pedestrians, bicycles + passengers, cars, trucks, and motorcycles + passengers can be selected. Furthermore, at least one of the multiple operators performing the annotation operation sends a label to the label selection area 53. Figure 3 The annotation operation screen 50 shown is an annotation operation in which a bounding box is added to a target object included in a still image 51 and a label of the added bounding box is selected from a label selection area 53 .
[0060] The space on the side of the "Worker ID" in the annotation work screen 50 is an area for inputting the worker's ID, and a unique ID for uniquely identifying the worker is input. In addition, in order to prevent others from using the ID, a password may be required after inputting the ID.
[0061] In addition, the space on the side of "Data Select" in the annotation operation screen 50 is an area for selecting a data set, and a data set in the target data can be selected by pulling down. And after the data set is selected, when the operation key 50a is pressed, a still image 51 is displayed in the image area of the annotation operation screen 50, and the annotation operation can be started.
[0062] In addition, it is also possible to prevent other operators from selecting the data set that is in the process of annotation. This can prevent multiple operators from performing annotation operations on the same data at the same time. In addition, for example, it is also possible to prevent the selection of data sets that no longer require annotation operations, such as data sets that have completed annotation operations. Here, for example, the annotation operation of the data set may be considered completed at the time when the data set is made public, or at a stage where a specified time has passed since the time when the initial annotation operation was completed. In addition, for example, the update of the annotation data operation a specified number of times may be used as a trigger to consider the annotation operation of the data set as completed.
[0063] In addition, by pressing the data selection area 50b in the annotation operation screen 50, it is possible to select which data to operate on from the data included in the selected data set ("Data_Set_001 / SUB006" in the figure). Figure 3As shown, it is a left and right arrow key, which can advance the data for annotation operation to the next data or return to the previous data. In addition, 32 / 50 displayed in the annotation operation screen 50 means that the selected data set includes 50 data and the 32nd data is currently selected.
[0064] When the save button 50 c in the annotation work screen 50 is pressed, the content of the annotation work currently being displayed on the annotation work screen 50 is registered (saved) in the annotation work data DB 203 .
[0065] In the present embodiment, for one object data, annotation work is performed by multiple operators. More specifically, for one object data, the work of adding annotations is performed by multiple operators using crowdsourcing, i.e., annotation work. That is, in other words, for one object data to which annotation work is entrusted through crowdsourcing, annotation work is performed by multiple operators who are considered to be able to perform annotation work on the one object data. Here, in the case where one or more object data is one or more still images, the annotation work is such a work that: a bounding box is attached to respectively enclose one or more object objects respectively reflected in one or more still images, and a label representing each of the one or more object objects is attached to the bounding box. One or more object objects include pedestrians reflected in the still image. One or more object objects may also include vehicles. In addition, in the case where one or more object data is one or more time series data, the annotation work is such a work that: a time interval is attached to distinguish one or more states contained in one or more time series data, and a label representing the corresponding state is attached to the one or more time intervals attached. When the time series data is data from a vehicle-mounted sensor, the one or more states are at least one of a driving state of the vehicle, a driving location of the vehicle, a surrounding environment of the vehicle, and a road condition on which the vehicle is driving.
[0066] The annotation tool unit 201 also acquires annotation work data indicating the annotation work performed on the annotation work screen provided via the network 40. The annotation tool unit 201 reflects the acquired annotation work data on the annotation work screen and provides it, and stores it in the annotation work data DB 203.
[0067] Figure 4A to Figure 4C is a diagram showing an example of an annotation work screen provided by the annotation tool unit 201 of the first embodiment. Figure 3 The same elements are denoted by the same reference numerals, and detailed description thereof will be omitted.
[0068] exist Figure 4A, an annotation work screen 50A is shown, which reflects the annotation work in which a bounding box 52 is added to an object 511 included in a still image 51, and a classification label of a pedestrian is selected for the bounding box 52 in a label selection area 53. Figure 4B , an annotation operation screen 50B is shown, which reflects the annotation operation of updating the bounding box 52 of the object 511 in the annotation operation screen 50A and adding a bounding box 54. Figure 4C , an annotation work screen 50C is shown, which reflects a plurality of annotation works performed on the still image 51 included in the annotation work screen 50A. More specifically, in the annotation work screen 50C, the bounding box 52 for the object 511 of the still image 51 is updated multiple times, and a bounding box 55 for the object 512 and a bounding box 56 for the object 513 are newly added. In addition, in the annotation work screen 50C, a certain worker mistook a tree for a person and a bounding box 57 was newly added.
[0069] <Annotation work data DB203>
[0070] The annotation work data DB 203 is an example of a first storage device, and stores the history of the annotation work data associated with each object data. The annotation work data DB 203 is realized by a semiconductor memory, a hard disk, or the like.
[0071] In the present embodiment, the annotation work data DB 203 stores a history of annotation work data indicating annotation work performed on the annotation work screen provided by the annotation tool unit 201 .
[0072] Figure 5 This is a diagram showing an example of annotation work data stored in the annotation work data DB 203 according to the first embodiment.
[0073] Annotation job data such as Figure 5 As shown in FIG. 1 , it includes the operator ID, object data ID, annotation ID, operation time, and annotation content. Figure 5 In the example shown, the annotation work data is composed of each line and exists for each data added to one still image (object data). The annotation work data is a record of the annotation work performed by one of a plurality of workers on each data.
[0074] The operator ID indicates the operator who performed the annotation work. More specifically, the operator ID is an identifier that uniquely indicates one of a plurality of operators using crowdsourcing. Figure 5 In the example shown, the ID of the operator who has performed annotation work on the same object data is shown. Figure 5In the example shown, Worker ID_0001, Worker ID_0002, Worker ID_0003, Worker ID_0004, etc. are shown as worker IDs.
[0075] The object data ID indicates the object data on which the annotation work is performed. More specifically, the object data ID is an identifier that uniquely indicates the object data on which the annotation work is performed, which is one of the object data entrusted by crowdsourcing. Figure 5 In the example shown, DataSet_001 / SUB006 / 32.jpg indicating one object data on which annotation work has been performed is shown as the object data ID.
[0076] The annotation ID indicates an annotation added to the object data indicated by the object data ID. More specifically, the annotation ID is an identifier that uniquely indicates an annotation added to the object data entrusted by crowdsourcing. Figure 5 In the example shown, 00001, 00002, 00003, and 00004 are shown, indicating annotation IDs such as bounding boxes or time intervals added to the object data indicated by the object data ID. Figure 5 In the example shown, the comment ID is made unique by combining with the object data ID, but the present invention is not limited to this.
[0077] The operation time indicates the time when the annotation operation was performed. More specifically, the operation time is the time when the annotation operation was performed on the object data represented by the object data ID of the same row, such as 2018 / 06 / 24 12:46:37. Figure 5 In the example shown, operation times such as 2018 / 06 / 24 12:46:37, 2018 / 06 / 24 18:10:24, 2018 / 06 / 25 10:31:57, and 2018 / 06 / 25 12:45:03 are shown, and the operation order can be known from the operation times.
[0078] The comment content indicates the content of the comment indicated by the comment ID. More specifically, the comment content indicates the content of the comment indicated by the comment ID in the same row.
[0079] exist Figure 5 In the example shown, "Create" indicates that the object data with the object data ID "DataSet_001 / SUB006 / 32.jpg" is annotated for the first time, and annotation IDs "00001" to "00004" are added. Figure 4C, then the annotation ID of 00001 corresponds to bounding box 52, and the annotation ID of 00002 corresponds to bounding box 55. The annotation ID of 00003 corresponds to bounding box 56, and the annotation ID of 00004 corresponds to bounding box 57. In addition, the bounding box of "(302, 209), (406, 374)" indicates that the upper left coordinates of the bounding box 52 of the annotation ID represented by "00001" are (302, 209) and the lower right coordinates are (406, 374). In addition, these coordinates are coordinates when the upper left corner of the still image 51 is the origin (0, 0). In addition, the bounding box of "(571, 246), (606, 360)" indicates that the upper left coordinates of the bounding box 55 of the annotation ID represented by "00002" are (571, 246) and the lower right coordinates are (606, 360). The bounding box of "(420, 262), (636, 334)" indicates that the upper left coordinates of the bounding box 56 of the annotation ID indicated by "00003" are (420, 262) and the lower right coordinates are (636, 334). The bounding box of "(219, 254), (242, 312)" indicates that the upper left coordinates of the bounding box 57 of the annotation ID indicated by "00004" are (219, 254) and the lower right coordinates are (242, 312).
[0080] in addition, Figure 5 The "Pedestrian" classification label shown in the figure indicates that a pedestrian is selected, and the "CAR" classification label indicates that a car is selected. Figure 4C , it indicates that the object 511 of the bounding box 52 with the annotation ID indicated by "00001" is a pedestrian. It indicates that the object 512 of the bounding box 55 with the annotation ID indicated by "00002" is a pedestrian. In addition, it indicates that the object of the bounding box 56 with the annotation ID indicated by "00003" is a car. In addition, it indicates that the object of the bounding box 57 with the annotation ID indicated by "00004" is mistakenly regarded as a pedestrian, but is actually a tree.
[0081] in addition, Figure 5 The "Update" shown in FIG. 1 indicates that an annotation operation is performed to update the annotations added to the object or the like included in the object data with the object data ID "DataSet_001 / SUB006 / 32.jpg". Figure 5 , Worker ID_0002, Worker ID_0001, and Worker ID_0003 are updated in this order in the direction of reducing the bounding box 52 indicated by the annotation ID "00001". Figure 5In the example shown, the upper left and lower right coordinates of the bounding box of the annotation ID represented by "00001" are updated from "(302, 202), (406, 374)" to "(316, 233), (382, 346)", and then to "(322, 209), (406, 374)". In addition, the classification label remains selected as pedestrian and is not updated.
[0082] in addition, Figure 5 The "Delete" shown in the figure indicates that an annotation operation is performed to delete the annotation added to the object or the like included in the object data with the object data ID "DataSet_001 / SUB006 / 32.jpg". Figure 5 , which indicates that an update has been performed to delete the bounding box 57 indicated by the annotation ID "00004" which was mistakenly attached by the worker of Worker ID_0004. Figure 4C , it means that the bounding box 57 represented by the annotation ID "00004" was added by mistake and was deleted from the still image 51.
[0083] <Remuneration amount DB204>
[0084] The remuneration amount DB 204 is a storage device that stores the calculation results output from the contribution degree determination device 10. The remuneration amount DB 204 is implemented by a semiconductor memory, a hard disk, or the like.
[0085] In this embodiment, the remuneration amount DB 204 stores the remuneration amount of each operator for all one or more object data, which is calculated using the contribution of each operator calculated for each object data. In addition, the contribution is standardized so that the total contribution of all operators for one object data becomes 1. Thus, the client using crowdsourcing can pay the operator according to the remuneration amount of each operator stored in the remuneration amount DB 204.
[0086] [Structure of the work terminal 30]
[0087] like Figure 1 As shown, the work terminal 30 includes a communication unit 301, a presentation unit 302, and an input unit 303. The work terminal 30 is realized by, for example, a computer including a processor (microprocessor), a memory, a sensor, a communication interface, etc. The work terminal is a portable terminal such as a personal computer or a tablet computer.
[0088] <Communication Department 301>
[0089] The communication unit 301 is implemented by a processor, a communication I / F, and the like, and communicates with the server 20 . More specifically, the communication unit 301 transmits an annotation work screen for the target data provided from the server 20 to the presentation unit 302 .
[0090] Furthermore, the communication unit 301 transmits annotation work data indicating the annotation work performed on the annotation work screen input by the input unit 303 to the server 20 .
[0091] <Prompt section 302>
[0092] The presenting unit 302 presents the annotation operation screen for the object data transmitted from the server 20 via the communication unit 301. For example, the presenting unit 302 presents the following Figure 3 The annotation operation screen 50 is shown.
[0093] In addition, the presenting unit 302 presents the annotation operation screen reflecting the annotation operation for the target data sent via the communication unit 301. For example, the presenting unit 302 presents Figure 4A The annotation operation screen 50A reflecting the annotation operation for the target data as shown in FIG. Figure 4B The annotation work screen 50B reflects the annotation work on the target data as shown.
[0094] <Input Unit 303>
[0095] The input unit 303 is an interface device for receiving input from the user. When the object data included in the annotation operation screen presented by the presentation unit 302 is an image, the input unit 303 receives input for annotation operations such as adding a bounding box to the object included in the object data, updating or deleting the object. Figure 4A Taking the illustrated annotation work screen 50A as an example, the input unit 303 may receive input of an annotation work in which a bounding box 52 indicating the position of the object 511 is added to the object 511 and a classification label is selected in the label selection area 53 .
[0096] Furthermore, when the object data included in the annotation work screen presented by the presentation unit 302 is time-series data, the input unit 303 may receive input of annotation work such as adding a time interval to a state included in the object data, updating or deleting the state.
[0097] [Contribution Determination Device 10]
[0098] The contribution determination device 10 includes an update history acquisition unit 101, a contribution calculation unit 102, a weighted DB 103, and a price calculation unit 104. The contribution determination device 10 is implemented by a computer including a processor (microprocessor), a memory, a sensor, a communication interface, and the like, for example.
[0099] <Update History Acquisition Unit 101>
[0100] The update history acquisition unit 101 acquires, from the annotation work data DB 203 , update history of annotation work corresponding to one or more target data respectively, that is, annotation work in which a plurality of workers have completed annotation by crowdsourcing.
[0101] For example, the update history acquisition unit 101 acquires the annotation work data DB 203 Figure 5 Multiple annotation work data as shown.
[0102] Furthermore, the update history acquisition unit 101 may acquire the annotation operation data which has passed a predetermined time from the operation time newly included in the annotation content as the annotation operation is completed among the plurality of annotation operation data stored in the annotation operation data DB 203. The operation time newly included in the annotation content is the operation time when the annotation operation was first performed on the object data, so the annotation operation can be considered to be completed when the predetermined time has passed from it.
[0103] <Contribution Calculation Unit 102>
[0104] The contribution calculation unit 102 refers to the update history acquired by the update history acquisition unit 101, and calculates the work contribution of each of the plurality of worker IDs representing the plurality of workers according to the predetermined weighting rule stored in the weighting DB 103. The contribution calculation unit 102 calculates the work contribution by weighting the worker ID representing the worker who first performed the annotation work on one or more object data in the work order more heavily than the worker ID representing the worker who performed the annotation work later in the work order.
[0105] Furthermore, when the completion trigger of the annotation work is time, since the number of workers is unknown, for example, the contribution ratio of new workers and update workers may be recorded in the weighting DB 103. This allows the contribution ratio to be calculated regardless of the number of update workers.
[0106] Furthermore, when the trigger for completing the annotation work is the number of updates, since the number of workers is fixed, the contribution degree may be directly recorded in the weighting DB for each new worker and updated worker, for example.
[0107] In this way, among the multiple operators assigned to one object data, the operator who first performs annotation work such as adding a bounding box is given a higher reward, so that the operator can start the work faster. As a result, the time until the annotation work is completed can be shortened. Furthermore, the total time for the server 20 to provide the annotation work screen can also be shortened, which also helps save energy.
[0108] Alternatively, the contribution calculation unit 102 may calculate the work contribution by weighting the worker ID of the worker who performed the annotation work on one or more object data last in the work sequence more heavily than the worker IDs of the workers who performed the annotation work from the later worker to the last worker in the work sequence.
[0109] Thus, among the multiple operators assigned to one object data, the operator who last updated the annotation work such as correcting the range of the bounding box can also receive more compensation. As a result, the correction of the annotation performed by the same operator or other operators can be valuable, so it is possible to promote the update work for one object data and shorten the time until the annotation work is completed. As a result, the total time for the server 20 to provide the annotation work screen can be shortened, which also helps save energy.
[0110] <Weighted DB103>
[0111] The weight DB 103 is an example of a second storage device, and is implemented by a semiconductor memory, a hard disk, etc. In the weight DB 103 , a predetermined weighting rule is registered.
[0112] In this embodiment, the weighting rule of adding a larger weight to the operator who first performed the annotation work in the work order is registered in the weighting DB 103. In addition, the weighting rule of adding the second largest weight to the operator who last performed the annotation work in the work order is registered in the weighting DB 103.
[0113] <Amount Calculation Unit 104>
[0114] Figure 6 This is a diagram showing an example of the detailed configuration of the amount calculation unit 104 according to the first embodiment.
[0115] The amount calculation unit 104 includes a receiving unit 1041 and a calculating unit 1042 , and calculates the remuneration amount for each worker for all of one or more object data.
[0116] The accepting unit 1041 accepts the setting of the amount for each object data.
[0117] Here, since the number of one or more still images or one or more states contained in each object data is unknown, it is also necessary to consider that if an amount corresponding to the amount is paid, the client's budget will be exceeded. Therefore, in this embodiment, by setting the amount for each object data, the increase in the amount paid by the client can be suppressed.
[0118] The calculation unit 1042 calculates the amount for each worker ID as remuneration by multiplying the work contribution for each worker ID calculated by the contribution calculation unit 102 by the amount accepted by the acceptance unit 1041 .
[0119] Thus, the sooner the operator starts annotation work on the object data, the more the amount of money as the operator's reward will be, and the sooner the operator starts annotation work, the more the amount of money as the operator's reward will be. Thus, the operator can be motivated to speed up the time until the annotation work on the object data is started or completed.
[0120] [Operation of Contribution Determination Device 10]
[0121] Next, the operation of the contribution degree determination device 10 configured as described above will be described.
[0122] Fig. 7A This is a flowchart showing an example of the operation of the contribution determination device 10 according to the first embodiment. Figure 7B Yes means Fig. 7A Flowchart of a detailed operation example of step S20 shown in FIG.
[0123] First, the contribution determination device 10 obtains the work history of the annotation work on the object data (S10). More specifically, the contribution determination device 10 selects one object data from the object data for which the annotation work has been completed, and obtains all the work history of the annotation work related to the selected object data. Figure 5 To explain, when the target data is DataSet_001 / SUB006 / 32.jpg, the contribution degree determination device 10 acquires all the work records in which DataSet_001 / SUB006 / 32.jpg is recorded in the target data ID from the annotation work data DB 203 .
[0124] Next, the contribution determination device 10 calculates the work contribution of each worker ID according to the weighting rule registered in the weighting DB 103 (S20). Figure 8B As shown, the contribution determination device 10 refers to the update history obtained in step S10 and calculates the work contribution of the object data (S201) according to the weighting rule registered in the weighting DB 103. Next, the contribution determination device 10 adds the work contribution of each operator ID (S202) to calculate the work contribution of each operator ID.
[0125] If the processing, i.e., calculation of the work contribution, is not completed for all the object data after step S20, the process returns to step S10 to calculate the work contribution of other object data. On the other hand, if the processing is completed for all the object data, the contribution determination device 10 ends the operation.
[0126] [Effects, etc.]
[0127] As described above, according to the present embodiment, it is possible to realize a contribution degree determination method and the like that can expedite the work of a worker entrusted by crowdsourcing to start the work.
[0128] More specifically, among the multiple operators assigned to one object data, the operator who first performs annotation work such as adding a bounding box is given a higher reward, so that the operator can start the work faster. As a result, the time until the annotation work is completed can be shortened. As a result, the total time for the server 20 to provide the annotation work screen can be shortened, which can also help save energy.
[0129] In addition, among the multiple operators assigned to one object data, the operator who last updated the annotation work such as correcting the range of the bounding box can also receive more compensation. In addition, when the payment amount is set for each object data, the amount allocated to each operator can be determined according to the ratio of contribution to the object data. In this way, the correction of the annotation performed by the same operator or other operators can be valuable, so it is not only possible to promote the update work for one object data, but also to shorten the time until the annotation work is completed. Furthermore, the total time for the server 20 to provide the annotation work screen can be shortened, which can also help save energy.
[0130] Furthermore, in the above-mentioned embodiment, using Figure 3 , Figure 4A and Figure 4B In the case where the target data is an image captured by a vehicle-mounted camera, an example of an annotation operation such as adding a bounding box to the object 511 representing a person has been described, but the present invention is not limited to this. Fig. 8A , Figure 8B , Figure 8C , an example of annotation work when the target data is time series data is described.
[0131] Fig. 8A This is a diagram showing another example of the annotation work screen provided by the annotation tool unit 201 according to the first embodiment. Fig. 8A The annotation operation screen 61 shown is provided by a web page and includes time series data 66 and time t 65 The time series data 66 is sensor data including the acceleration of the vehicle. Fig. 8AThe acceleration of the vehicle in the x, y, and z directions (horizontal, lateral, and vertical directions) is plotted in time series. In addition, at each time in the time series data 66, an image captured by a vehicle-mounted camera mounted on the vehicle is associated. Image 65 is captured at time t 65 Images captured by the onboard camera.
[0132] Thus, the annotation tool unit 201 can provide a time series data including the time series data 66 and the time t 65 The annotation work screen 61 of the image 65 is shown.
[0133] Figure 8B and Figure 8C It means in Fig. 8A The illustrated annotation operation screen is a diagram showing an example of an annotation operation screen in which an annotation operation is reflected. Fig. 8A The same elements are denoted by the same reference numerals, and detailed description thereof will be omitted.
[0134] exist Figure 8B In the annotation operation screen 61a shown, multiple operators have added time intervals that distinguish multiple states as annotations, and a label indicating the corresponding state is added to each of the multiple time intervals. That is, at least one of the operators checks the change in acceleration of the time series data 66 and the image 65 associated at each time, and adds the vehicle event, location, weather, etc. as annotations. Figure 8B In the example shown, as events, a time interval 661 when the vehicle turns right, a time interval 662 when the vehicle passes through a height difference, and a time interval 663 when the vehicle decelerates are added. In addition, labels are added for the vehicle's driving states such as right turn, height difference, and deceleration, as well as the road conditions on which the vehicle is driving.
[0135] In addition, Figure 8B In the example shown, a time period 664 in which the vehicle travels on a general road and a time period 665 in which the vehicle travels on a highway are added as locations, and labels indicating the states of the vehicle travel locations such as general roads and highways are also added.
[0136] Similarly, in Figure 8B In the example shown, a cloudy time period 666 is added as the weather, and a tag indicating the state of the vehicle surrounding environment such as cloudy is also added.
[0137] Figure 8C The annotation operation screen 61b shown includes time series data 66 and time t 67 Image 67. Figure 8C As shown in the annotation operation screen 61b, for example, at time t 65 The image 65 shows the time t 65The acceleration change of the time series data 66 is used to determine that the vehicle has passed the height difference. The other parts are as described above and the description is omitted.
[0138] (Implementation Method 2)
[0139] In the first embodiment, the completion of the annotation work is described as a certain time after the crowdsourcing operator first performs the work. However, in the case where the correction amount of the bounding box attached to an object data is almost zero, the annotation work can also be considered to be completed. In addition, in the first embodiment, it is assumed that multiple crowdsourcing operators correctly perform the annotation work. However, it is also necessary to consider the case where there are malicious bad people among the multiple operators. In this case, it is also necessary to consider the following situation, namely: the bad person is the person who enlarges or deletes the size of the bounding box attached to an object data, increases the number of updates of the annotation work, or is the first or last to perform the annotation work.
[0140] In the present embodiment, a contribution degree determination device and the like that can determine the completion of an annotation work and determine the possibility of an annotation work being performed by a defective person will be described, focusing on the differences from the first embodiment.
[0141] [Overall structure of the system]
[0142] Fig. 9 is a diagram showing an example of the overall structure of the system of Embodiment 2. Figure 1 The same elements are denoted by the same reference numerals, and detailed description thereof will be omitted.
[0143] The system of the second embodiment differs from the system of the first embodiment in the configuration of the server 20A and the configuration of the contribution degree determination device 10A. The other configurations are the same as those of the system of the first embodiment, and the description thereof will be omitted.
[0144] [Configuration of Server 20A]
[0145] The server 20A includes an annotation tool unit 201A, a sensor data DB 202, an annotation work data DB 203, and a remuneration DB 204. The server 20A is also implemented by a computer including, for example, a processor (microprocessor), a memory, a sensor, a communication interface, and the like.
[0146] Server 20A and Figure 1 Compared with the server 20 shown in FIG. 1 , the difference lies in the structure of the annotation tool unit 201A.
[0147] <Annotation Tool Section 201A>
[0148] The annotation tool unit 201A selects object data for which an annotation operation is requested from a plurality of object data stored in the sensor data DB 202 , and provides an annotation operation screen for the selected object data to the plurality of operation terminals 30 via the network 40 .
[0149] The annotation tool section 201A also acquires annotation work data indicating the annotation work performed on the annotation work screen provided via the network 40. The annotation tool section 201A reflects the acquired annotation work data on the annotation work screen and provides it, and stores it in the annotation work data DB 203.
[0150] In the present embodiment, the annotation tool unit 201A also locks further annotation work for one object data when receiving a notification indicating that the lock should be performed from the contribution degree determination device 10A. That is, the annotation tool unit 201A may stop acquiring annotation work data when receiving a notification indicating that the lock should be performed, and does not acquire further annotation work to thereby perform the lock. In addition, the annotation tool unit 201A may lock the provided annotation work screen when receiving a notification indicating that the lock should be performed, and display that no further input is accepted. Furthermore, the annotation tool unit 201A does not acquire further annotation work data for the annotation work screen.
[0151] [Contribution Determination Device 10A]
[0152] The contribution determination device 10A includes an update history acquisition unit 101A, a contribution calculation unit 102, a weighted DB 103, a price calculation unit 104, and a determination unit 105. The contribution determination device 10A is realized by a computer including, for example, a processor (microprocessor), a memory, a sensor, a communication interface, and the like.
[0153] Contribution determination device 10A and Figure 1 Compared with the contribution degree determination device 10 shown in the figure, the difference lies in the configuration of the update history acquisition unit 101A and the addition of the configuration of the determination unit 105.
[0154] <Determination Unit 105>
[0155] Fig.10 Yes means Fig. 9 FIG. 2 is a diagram showing an example of a detailed configuration of the determination unit 105 shown in FIG.
[0156] The determination unit 105 includes a work completion determination unit 1051 and an inappropriate work determination unit 1052 .
[0157] The work completion determination unit 1051 determines whether the annotation work for one or more object data is completed. More specifically, when the work completion determination unit 1051 detects that the annotation work for one object data with a smaller correction amount than the correction amount in the previous annotation work performed in the work sequence has been performed a predetermined number of times, it locks the further annotation work for one object data and determines that the annotation work for one object data is completed. Here, the so-called predetermined number of times may be, for example, two or more consecutive times or once.
[0158] Furthermore, the work completion determination unit 1051 may determine that the annotation work is completed when detecting that the update work on the target data has not been performed for a predetermined time.
[0159] Fig.11 is a diagram showing an example of an annotation operation screen provided by the annotation tool unit 201A according to the second embodiment. Figure 4B The same elements are denoted by the same reference numerals and detailed descriptions thereof are omitted.
[0160] Fig.11 5 shows an annotation work screen 50D reflecting a plurality of annotation work data. More specifically, the following situation is shown: the bounding box 52 added to the object 511 of the still image 51 is corrected to the bounding box 54, the bounding box 54 is corrected to the bounding box 71, and the bounding box 71 is further corrected to the bounding box 72. As can be seen from the figure, the correction amount after the bounding box 54 is small, and it is not a problem to regard the annotation work such as adding the bounding box 54 to the object 511 as completed.
[0161] That is, in the annotation work data DB 203, Fig.11 In the plurality of annotation operation data shown, there is almost no correction amount for the position of the bounding box according to the coordinates corresponding to the bounding box 54, the bounding box 71, and the bounding box 72 shown in the annotation content. Therefore, in the present embodiment, the operation completion determination unit 1051 first obtains the update history of the annotation operation on one object data from the annotation operation data DB 203. Next, for one object data, it is calculated whether the annotation operation with a smaller correction amount than the correction amount in the previous annotation operation performed in the operation sequence has been performed twice or more continuously. And, when it is detected by calculation that such an annotation operation has been performed twice or more continuously, the annotation tool unit 201 of the server 20 is notified to lock further annotation operations on the one object data. With this notification, the operation completion determination unit 1051 determines that the annotation operation on the one object data is completed.
[0162] When detecting that a second annotation operation has been performed on one object data with a larger correction amount than the first annotation operation performed immediately before in the operation sequence, the inappropriate operation determination unit 1052 sends a notification prompting confirmation of whether the second annotation operation on the one object data is an inappropriate operation.
[0163] Fig.12 is a diagram showing an example of an annotation operation screen 50E provided by the annotation tool unit 201A of the second embodiment. Figure 4B The same elements are denoted by the same reference numerals and detailed descriptions thereof are omitted.
[0164] Fig.12 The annotation operation screen 50E showing a plurality of annotation operation data is also shown. More specifically, the following situation is shown: the bounding box 52 added to the object 511 of the still image 51 is corrected to the bounding box 54, the bounding box 54 is corrected to the bounding box 71, and then corrected to the bounding box 83. As can be seen from the figure, although the correction amount from the bounding box 54 to the bounding box 71 is slight, the correction amount from the bounding box 71 to the bounding box 83 is large, and not only that, the bounding box added to the object 511 is inappropriate compared with the bounding box 71. Since it is unknown whether the correction from the bounding box 71 to the bounding box 83 is caused by the operator's mistake or by a bad annotation operation that deliberately increases the number of operations, the client or the administrator of the server 20 needs to confirm whether it is a bad situation.
[0165] Furthermore, in the annotation work data DB 203, Fig.12 In the plurality of annotation operation data shown, the correction amount for the position of the bounding box increases according to the coordinates corresponding to the bounding box 71 and the bounding box 83 shown in the annotation content. Therefore, in the present embodiment, the inappropriate operation determination unit 1052 first obtains the update history of the annotation operation on one object data from the annotation operation data DB 203. Next, it is calculated whether an annotation operation with a larger correction amount than the correction amount in the previous annotation operation performed in the operation sequence has been performed on one object data. And, in the case where such an annotation operation is detected by calculation, a notification is sent to the annotation tool unit 201 of the server 20 to prompt confirmation whether it is an inappropriate operation, thereby notifying the administrator of the server 20 and prompting them to confirm. Of course, it is also possible that the inappropriate operation determination unit 1052 directly notifies the administrator to prompt confirmation whether it is an inappropriate operation.
[0166] Furthermore, the inappropriate operation determination unit 1052 may detect the number of executions of the second annotation operation with a larger correction amount than the correction amount in the first annotation operation performed immediately before the previous one in the operation sequence for one object data. In this case, when the annotation operation for one object data is determined to be completed by the operation completion determination unit 1051, the inappropriate operation determination unit 1052 may notify the number of executions for one object data. The inappropriate operation determination unit 1052 may notify the annotation tool unit 201 of the server 20 of the number of executions, thereby notifying the administrator of the server 20, or may directly notify the administrator of the number of executions.
[0167] <Update History Acquisition Unit 101A>
[0168] The update history acquisition unit 101A acquires, from the annotation work data DB 203 , update history of annotation work corresponding to one or more target data for which annotation work by a plurality of workers using crowdsourcing has been completed.
[0169] In this embodiment, when the work completion determination unit 1051 determines that the annotation work on one or more object data has been completed, the update history acquisition unit 101A acquires the update history of the annotation work corresponding to the one or more object data determined to have completed the annotation work from the annotation work data DB 203. Other aspects are the same as those described in Embodiment 1, and further description is omitted.
[0170] [Operation of Contribution Determination Device 10A]
[0171] Next, the determination operation of the contribution degree determination device 10A configured as above will be described.
[0172] Fig.13 This is a flowchart showing the work completion determination process of the contribution degree determination device 10A according to the second embodiment.
[0173] First, the contribution determination device 10A detects whether the correction amount is smaller than the previous annotation work (S81). For example, the contribution determination device 10A can detect whether the correction amount is smaller than the previous annotation work by calculation based on the coordinates of the annotation content included in the annotation work data stored in the annotation work data DB203.
[0174] In step S81, if detected (S81: Yes), the contribution degree determination device 10A detects whether it has occurred twice or more consecutively. On the other hand, if not detected in step S81 (S81: No), step S81 is repeated again.
[0175] When it is detected that the annotation is performed two or more times in succession in step S82 (Yes in S82), the contribution degree determination device 10A locks further annotation work on the object data (S83). More specifically, the contribution degree determination device 10A notifies the annotation tool unit 201 of the server 20 to lock further annotation work on the object data. Thus, the annotation tool unit 201 of the server 20 can lock further annotation work on the object data.
[0176] Next, the contribution degree determination device 10A determines that the annotation work on the target data is completed (S84). More specifically, the contribution degree determination device 10A locks the annotation work on the one target data in the annotation tool unit 201 of the server 20 and determines that the annotation work on the one target data is completed.
[0177] Fig.14 This is a flowchart showing the inappropriate work determination process of the contribution degree determination device 10A according to the second embodiment.
[0178] First, the contribution determination device 10A detects whether the correction amount is greater than the previous annotation work (S91). For example, the contribution determination device 10A can detect whether the correction amount is greater than the previous annotation work by calculation based on the coordinates of the annotation content included in the annotation work data stored in the annotation work data DB203.
[0179] When it is detected in step S91 (Yes in S91), the contribution degree determination device 10A sends a notification to prompt confirmation of whether the annotation work is improper (S92). More specifically, the contribution degree determination device 10A may prompt confirmation by sending a notification to the annotation tool unit 201 of the server 20 to prompt confirmation of whether it is improper work, thereby notifying the administrator of the server 20 and the like. In addition, the contribution degree determination device 10A may directly send a notification to prompt confirmation of whether it is improper work to the administrator.
[0180] [Effects, etc.]
[0181] As described above, according to this embodiment, it is possible to allow a manager or the like to confirm whether or not a bad annotation operation is maliciously performed by a worker commissioned by crowdsourcing. This makes it possible to detect and manage bad workers.
[0182] Furthermore, according to the present embodiment, whether the annotation work is completed can be determined based on the amount of correction of the annotation work by the operator entrusted by crowdsourcing. Thus, the completion of the annotation work can be automatically determined.
[0183] (Other Implementation Possibilities)
[0184] The above is an explanation of the contribution determination method of one or more embodiments of the present disclosure based on the implementation mode, but the present disclosure is not limited to the implementation mode. The person skilled in the art implements the present embodiment with various variations that can be imagined without departing from the scope of the present disclosure, and the method of combining the constituent elements of different implementation modes is also included in the scope of one or more embodiments of the present disclosure. For example, the following situations are also included in the present disclosure.
[0185] (1) Specifically, each of the above-mentioned devices is a computer system composed of a microprocessor, ROM, RAM, hard disk unit, display unit, keyboard, mouse, etc. A computer program is stored in the above-mentioned RAM or hard disk unit. The above-mentioned microprocessor operates according to the above-mentioned computer program, so that each device realizes its function. Here, the computer program is composed of a plurality of command codes representing instructions to the computer in order to realize a predetermined function.
[0186] (2) Some or all of the components constituting each of the above devices may also be constituted by a system LSI (Large Scale Integration). System LSI is a multifunctional LSI manufactured by integrating multiple components on a single chip. Specifically, it is a computer system composed of a microprocessor, ROM, RAM, etc. A computer program is stored in the RAM. The microprocessor operates according to the computer program, so that the system LSI realizes its functions.
[0187] (3) A part or all of the components constituting each of the above-mentioned devices may also be constituted by an IC card or a single module that can be installed and removed relative to each device. The above-mentioned IC card or the above-mentioned module is a computer system composed of a microprocessor, ROM, RAM, etc. The above-mentioned IC card or the above-mentioned module may also include the above-mentioned super-multifunctional LSI. The microprocessor operates according to the computer program, so that the above-mentioned IC card or the above-mentioned module realizes its function. The IC card or the module may have tamper resistance.
[0188] (4) The present disclosure may be the methods shown above. In addition, the present disclosure may be a computer program that implements the methods using a computer, or a digital signal composed of the computer program.
[0189] (5) In addition, the present disclosure may record the above-mentioned computer program or the above-mentioned digital signal on a computer-readable recording medium, such as a floppy disk, a hard disk, a CD-ROM, an MO, a DVD, a DVD-ROM, a DVD-RAM, a BD (Blu-ray (registered trademark) Disc), a semiconductor memory, etc. In addition, the above-mentioned digital signal may be recorded on these recording media.
[0190] (6) In addition, the present disclosure may also transmit the above-mentioned computer program or the above-mentioned digital signal via an electronic communication line, a wireless or wired communication line, a network represented by the Internet, data broadcasting, and the like.
[0191] (7) In addition, the present disclosure may be a computer system including a microprocessor and a memory, wherein the memory stores the computer program and the microprocessor operates according to the computer program.
[0192] (8) In addition, the program or the digital signal may be recorded on the recording medium and transferred, or the program or the digital signal may be transferred via the network and the like, and implemented by another independent computer system.
[0193] Possibility of Industrial Application
[0194] The present disclosure can be used in a contribution degree determination method, a contribution degree determination device, and a recording medium, and in particular, can be used in a server, a system, etc. used when entrusting an annotation work to a crowdsourcing worker.
[0195] Explanation of symbols
[0196] 10. 10A Contribution Determination Device
[0197] 20, 20A Server
[0198] 30 Operation Terminal
[0199] 40 Network
[0200] 50a Run button
[0201] 50b Data selection area 50b
[0202] 50c Save button
[0203] 50, 50A, 50B, 50C, 50D, 50E Annotation work screen
[0204] 51 Still images
[0205] 52, 54, 71, 72, 83 Bounding box
[0206] 53 Label selection area
[0207] 65 images
[0208] 66 Time Series Data
[0209] 101, 101A Updated history acquisition department
[0210] 102 Contribution Calculation Department
[0211] 103 Weighted DB
[0212] 104 Amount Calculation Department
[0213] 105 Judgment Department
[0214] 201, 201A Annotation Tools Department
[0215] 202 Sensor data DB
[0216] 203 Annotation work data DB
[0217] 204 Remuneration Amount DB
[0218] 301 Ministry of Communications
[0219] 302 Tips Department
[0220] 303 Input
[0221] 511, 512, 513 objects
[0222] 661, 662, 663, 664, 665, 666 time interval
[0223] 1041 Reception Department
[0224] 1042 Computing Department
[0225] 1051 Job Completion Judgment Department
[0226] 1052 Improper Operation Judgment Department
Claims
1. A contribution determination method, performed by a computer, characterized in that: Include: An update history acquisition step of acquiring, from the first storage device, an update history of an annotation job in which a plurality of operators using crowdsourcing have completed a job of adding a plurality of annotations to one object data, that is, an annotation job; as well as a contribution calculation step, referring to the update history acquired in the update history acquisition step, and calculating, according to a predetermined weighting rule stored in the second storage device, a work contribution of each of a plurality of worker IDs representing the plurality of workers, which is standardized so that when the contribution of all workers for one object data is added up, it becomes 1; In the contribution calculation step, the work contribution is calculated by weighting the worker ID indicating the worker who first performed the annotation work on one or more object data in the work order more heavily than the worker ID indicating the worker who performed the annotation work later in the work order.
2. The contribution determination method according to claim 1, characterized in that: The one or more object data are one or more still images, The annotation task is a task of adding a bounding box that respectively encloses one or more target objects shown in the one or more still images, and adding a label representing each of the one or more target objects to the bounding box.
3. The contribution determination method according to claim 2, wherein: The one or more still images are images taken by a vehicle-mounted camera or a monitoring camera installed at a predetermined position. The one or more target objects include a person reflected in the still image.
4. The contribution determination method according to claim 1, wherein: The one or more object data are one or more time series data, The annotation task is a task of adding time intervals for distinguishing one or more states included in the one or more time series data, and adding a label indicating the corresponding state to each of the one or more time intervals.
5. The contribution determination method according to claim 4, wherein: The one or more time series data are sensor data about the vehicle acquired by a sensor simultaneously with a plurality of images captured by a vehicle-mounted camera mounted on the vehicle, The one or more states are at least one of a driving state of the vehicle, a driving location of the vehicle, a surrounding environment of the vehicle, and a condition of a road on which the vehicle is driving.
6. The contribution determination method according to any one of claims 2 to 5, characterized in that: In the contribution calculation step, the work contribution is calculated by weighting the worker ID of the worker who performed the annotation work last in the work sequence for the one or more object data more heavily than the worker ID of the worker who performed the annotation work from the later to the last in the work sequence.
7. The contribution determination method according to any one of claims 2 to 5, characterized in that: Also includes: An acceptance step for accepting the setting of the amount of each object data; and The calculation step calculates, as the reward, an amount for each of the worker IDs obtained by multiplying the work contribution of each of the worker IDs calculated in the contribution calculation step by the amount accepted in the acceptance step.
8. The contribution determination method according to any one of claims 1 to 5, characterized in that: The method further comprises a determination step of determining whether the annotation work on each of the one or more object data is completed. In the determination step, when it is detected that an annotation operation with a correction amount smaller than the correction amount of an annotation operation previously performed in the operation sequence has been performed for a predetermined number of times on the one object data, further annotation operations on the one object data are locked and it is determined that the annotation operation on the one object data is completed.
9. The contribution determination method according to claim 8, wherein: In the determination step, further, when a second annotation operation is detected with respect to the one object data, the correction amount of which is larger than the correction amount of the first annotation operation performed previously in the operation sequence, a notification is sent to prompt confirmation of whether the second annotation operation with respect to the one object data is an improper operation.
10. The contribution determination method according to claim 8, wherein: In the determination step, further, when it is detected that the number of times a second annotation operation has been performed on the one object data and the correction amount is larger than the correction amount of the first annotation operation performed previously in the operation sequence, when it is determined that the annotation operation on the one object data has been completed, the number of times on the one object data is notified.
11. A contribution determination device, characterized in that: have: An update history acquisition unit acquires, from the first storage device, an update history of an annotation job in which a plurality of operators using crowdsourcing have completed an annotation job of adding a plurality of annotations to one object data; and a contribution calculation unit, referring to the update history acquired by the update history acquisition unit, and calculating, according to a predetermined weighting rule stored in the second storage device, a work contribution of each of a plurality of worker IDs representing the plurality of workers, which is standardized so that when the contribution of all workers to one object data is added up, it becomes 1; The contribution degree calculation unit calculates the work contribution degree by weighting the worker ID indicating the worker who first performed the annotation work on one or more object data in the work order more heavily than the worker ID indicating the worker who performed the annotation work later in the work order.
12. A recording medium which is a non-transitory computer-readable recording medium and has recorded thereon a program for causing the computer to execute, characterized in that: The procedure consists of the following steps: An update history acquisition step of acquiring, from the first storage device, an update history of an annotation job in which a plurality of operators using crowdsourcing have completed a job of adding a plurality of annotations to one object data, that is, an annotation job; as well as a contribution calculation step, referring to the update history acquired in the update history acquisition step, and calculating, according to a predetermined weighting rule stored in the second storage device, a work contribution of each of a plurality of worker IDs representing the plurality of workers, which is standardized so that when the contribution of all workers for one object data is added up, it becomes 1; In the contribution calculation step, the work contribution is calculated by weighting the worker ID indicating the worker who first performed the annotation work on one or more object data in the work order more heavily than the worker ID indicating the worker who performed the annotation work later in the work order.
