Information processing method and information processing system
Through computer processing, the sensing data characteristics are determined based on the operator's annotation skills and the annotation skills are updated, which solves the problem of poor adaptability of operator skills to personal characteristics in the existing technology, and achieves continuous improvement of annotation skills and improvement of learning model quality. .
Patent Information
- Application Number
- CN201910665764.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-04-01
- Filing Date
- 2019-07-23
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2039-12-04
AI Technical Summary
The prior art is difficult to effectively improve the skills of the operator on personal characteristics, resulting in the correctness of image object detection greatly affected by the skills of the operator.
Through computer processing, the annotation skills of the operator assigned to the annotation information of the sensing data are obtained, and the characteristics of the sensing data are determined based on the annotation skills, the annotation results are obtained, and the annotation skills are updated using the positive annotation information corresponding to the sensing data characteristics, so as to achieve the Annotation training corresponding to the operator's personal characteristics.
It effectively improves the annotation skills of the operator, improves its adaptability to personal characteristics, and improves the continuity of the annotation skills and the quality of the learning model.
Smart Images

Figure CN110796156B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to an information processing method and an information processing system. Background Art
[0002] In recent years, methods for detecting objects from images have been implemented using deep learning. In order to accurately detect objects from images, for example, a large number of images are collected, and operators create supervision data based on the large number of images. Therefore, the accuracy of object detection in images varies depending on the skills of the operator.
[0003] As a technology related to the skills of the operator, there is, for example, the technology of Patent Document 1. The skill evaluation device of Patent Document 1 compares the position and area of the mark with the learning information in the operation of marking the part containing the face from the image as the object content to calculate the similarity. The skill evaluation device evaluates the operation as being performed by an operator with higher skills when the similarity to the learning information is higher. In this way, a highly skilled operator can be selected from a plurality of operators. In addition, the evaluation result is fed back to the operator.
[0004] Patent Document 1: Japanese Patent Application Publication No. 2015-200985 Summary of the invention
[0005] The conventional technology disclosed in Patent Document 1 has difficulty in effectively improving the skills of workers in accordance with their individual characteristics.
[0006] Therefore, the present invention provides an information processing method and an information processing system that effectively improve the annotation skills of operators according to individual characteristics.
[0007] In order to achieve the above-mentioned purpose, an information processing method related to a technical solution of the present invention uses a computer to perform the following processing: obtaining the annotation skills of an operator who assigns annotation information to sensing data; obtaining sensing data for annotation, the sensing data for annotation having the characteristics of the sensing data determined according to the above-mentioned annotation skills; obtaining the annotation results of the above-mentioned sensing data by the above-mentioned operator; and updating the above-mentioned annotation skills using the correct annotation information corresponding to the characteristics of the above-mentioned sensing data and the above-mentioned annotation results.
[0008] In addition, these inclusive or specific technical solutions may also be implemented by systems, methods, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, or by any combination of systems, methods, integrated circuits, computer programs, and recording media.
[0009] Effects of the Invention
[0010] According to the information processing method and the like of the present invention, the annotation skills of a worker can be effectively improved according to individual characteristics. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] Figure 1 This is a block diagram illustrating an information processing system and an input / output unit according to the first embodiment.
[0012] Figure 2 This is a diagram illustrating the transfer of information in the information processing system according to the first embodiment.
[0013] Figure 3A This is a diagram illustrating the first annotation information related to the first embodiment.
[0014] Figure 3B This is a diagram illustrating the annotation skills related to Implementation Example 1.
[0015] Figure 3C This is a diagram illustrating the second annotation information related to Implementation Example 1.
[0016] Figure 4 This is a diagram illustrating the operation of the information processing system according to the first embodiment.
[0017] Figure 5 This is a diagram illustrating a task of adding annotation information to an annotation image.
[0018] Figure 6 This is a block diagram illustrating an information processing system and an input / output unit according to the second embodiment.
[0019] Figure 7 This is a diagram illustrating the transmission of information in the information processing system related to embodiment 2.
[0020] Figure 8 This is a diagram illustrating the operation of the information processing system related to the second embodiment. DETAILED DESCRIPTION
[0021] An information processing method related to a technical solution of the present invention uses a computer to perform the following processing: obtaining the annotation skills of an operator who assigns annotation information to sensing data; obtaining sensing data for annotation, the sensing data for annotation having characteristics of the sensing data determined based on the above-mentioned annotation skills; obtaining the annotation results of the above-mentioned sensing data by the above-mentioned operator; and updating the above-mentioned annotation skills using the correct annotation information corresponding to the characteristics of the above-mentioned sensing data and the above-mentioned annotation results.
[0022] In this way, by determining the annotation sensing data corresponding to the operator's annotation skills, it is possible to train the operator to perform annotations corresponding to the operator's personal characteristics. For example, sensing data that the operator is not good at is determined as the annotation sensing data. In addition, by automatically updating the annotation skills, continuous improvement of the annotation skills can be achieved. Therefore, the operator's annotation skills can be effectively improved according to the personal characteristics.
[0023] An information processing system related to a technical solution of the present invention comprises: a first acquisition unit, which acquires sensing data for annotation having characteristics of the sensing data, wherein the characteristics of the sensing data are determined based on the annotation skills of an operator who assigns annotation information to the sensing data; a second acquisition unit, which acquires an annotation result of the sensing data for annotation by the operator; and a processing unit, which updates the annotation skill using correct annotation information corresponding to the characteristics of the sensing data and the annotation result.
[0024] The information processing system also has the same effects as described above.
[0025] In an information processing method according to one technical solution of the present invention, the annotation skills include skills related to the specification of the annotation information.
[0026] In this way, by targeting skills that are directly related to the quality of annotation information, it is possible to effectively improve the quality of a learning model that is learned using supervised data generated by annotation.
[0027] In the information processing method according to one aspect of the present invention, the specification of the annotation information includes at least one of specification of a position, specification of a size, and specification of a shape of the annotation.
[0028] In this way, the operator can be trained in at least one of specifying the position, size, and shape for annotation. Therefore, by training the operator in specifying annotations that the operator is not good at, the accuracy of the work of adding annotation information to the annotation sensing data is improved.
[0029] In an information processing method according to one technical solution of the present invention, the designation of the annotation information includes designation of a label of the annotation.
[0030] In this way, the operator can be trained to specify the annotation for each label (eg, category) of the annotation. Therefore, by training the operator on the category of annotation that the operator is not good at, the accuracy of the work of adding annotation information to the sensing data for annotation is improved.
[0031] In an information processing method related to a technical solution of the present invention, the annotation result includes the time spent on specifying the annotation information; and the annotation skill includes the time required for specifying the annotation information.
[0032] This allows the operator to be trained to improve the time required for the specified work of annotation, thereby improving the operator's skill of speed in the work of adding annotation information to the annotation sensing data.
[0033] In an information processing method according to one aspect of the present invention, the sensing data is image data; and the annotation sensing data is annotation image data.
[0034] This makes it possible to efficiently improve annotation skills even for image data with a large amount of information and individual differences in the improvement of annotation skills.
[0035] An information processing method related to a technical solution of the present invention determines the above-mentioned correct annotation information based on the above-mentioned annotation skills; and obtains the above-mentioned annotation image data by selecting the above-mentioned annotation image data that captures the object corresponding to the determined above-mentioned correct annotation information from a plurality of the above-mentioned annotation image data.
[0036] This makes it possible to select annotated image data with supervisory data that improves annotation skills. For example, annotated image data that requires the operator to be given annotation information that the operator is not good at can be selected. The operator can be trained in tasks that the operator is not good at.
[0037] In an information processing method related to a technical solution of the present invention, the correct annotation information is determined based on the annotation skill; and the image data for annotation is obtained by generating image data of an object corresponding to the determined correct annotation information.
[0038] This makes it possible to train the operator even if there is no annotation image data suitable for improving the annotation skills.
[0039] In an information processing method according to one technical solution of the present invention, the generation of the image data includes: synthesizing the image data in which the object corresponding to the determined correct annotation information is photographed with other image data.
[0040] In this way, it is possible to efficiently generate annotated image data with supervisory data.
[0041] The information processing method according to one technical solution of the present invention further prompts the operator with the difference between the correct annotation information and the annotation result.
[0042] As a result, the operator can intuitively understand his or her own annotation skills based on the difference between the correct annotation information and the annotation result.
[0043] The information processing method according to one technical solution of the present invention also prompts the operator with the updated annotation skills.
[0044] Thus, the operator can understand the status of his / her annotation skills. For example, the operator can understand whether the annotation skills have improved, changed, or declined. Thus, the operator can be directly or indirectly prompted to take actions to improve the annotation skills.
[0045] In addition, these inclusive or specific technical solutions may also be implemented by systems, methods, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, or by any combination of systems, methods, integrated circuits, computer programs, and recording media.
[0046] Hereinafter, the embodiments are described in detail with reference to the accompanying drawings. In addition, the embodiments described below all represent a specific example. The numerical values, shapes, materials, constituent elements, configuration positions and connection forms of constituent elements, steps, and the order of steps shown in the following embodiments are examples and are not intended to limit the present invention. In addition, among the constituent elements of the following embodiments, constituent elements that are not described in the independent claims representing the highest concept of the present invention are described as arbitrary constituent elements.
[0047] In addition, each figure is a schematic diagram and does not necessarily illustrate the exact details. In addition, in each figure, the same reference numerals are given to substantially the same structure, and repeated descriptions are omitted or simplified.
[0048] Hereinafter, an information processing method and an information processing system according to an embodiment of the present invention will be described.
[0049] (Implementation Method 1)
[0050] [structure]
[0051] Figure 1 This is a block diagram illustrating the information processing system 1 and the input / output unit 40 according to the first embodiment. Figure 2 This is a diagram illustrating the transfer of information in the information processing system 1 according to the first embodiment.
[0052] like Figure 1 and Figure 2 As shown, the operator (also referred to as an annotator) annotates the sensing data for machine learning. For example, the sensing data is image data, and the operator assigns annotation information related to the object captured in the annotation image to the image captured by the camera or the image synthesized by CG (Computer Graphics) (hereinafter referred to as the annotation image). The information processing system 1 evaluates the correctness of the annotation information assigned to the annotation image. The information processing system 1 evaluates the operator (determines the annotation skill) based on the annotation image to which the operator has assigned annotation information, and causes the operator to perform the task of assigning annotation information to the annotation image corresponding to the evaluation. The object is any object such as a person, an animal, a vehicle, or equipment.
[0053] Here, annotation refers to adding metadata associated with a specific object associated with an annotation image as a note (label), and is described in text. In addition, an annotation image refers to an image of an object that is a work target of an operator, and is used by the operator to add annotation information to the object.
[0054] The information processing system 1 is connected to an input / output unit 40 for the operator to add annotation information. The input / output unit 40 is, for example, a personal computer, and receives input from the operator via a screen displayed on a browser or the like on the personal computer. The information processing system 1 includes a first processing unit 10, a second processing unit 20, and a third processing unit 30.
[0055] The first processing unit 10 includes a storage unit 11, which stores information indicating an annotation image, first annotation information, and information indicating the annotation skills of the operator. The first annotation information mentioned here is annotation information (also called correct answer data) used as a model to evaluate whether the annotation image to which the operator has given annotation information, i.e., the annotated image, has been correctly annotated. The first annotation information is an example of correct answer annotation information.
[0056] In addition, when the storage unit 11 does not have an annotation image in which an object corresponding to the annotation skill is captured, the first processing unit 10 generates an annotation image in which an object corresponding to the annotation skill is captured. The generation of the annotation image includes synthesizing an image in which an object corresponding to the determined second annotation information is captured with other images.
[0057] use Figure 3A An example of the first annotation information will be described. Figure 3A This is a diagram illustrating the first annotation information related to the first embodiment.
[0058] like Figure 3A As shown, the first annotation information is when an object (equivalent to Figure 3A In the annotation image of the type (of the object), the text data as a template are arranged, such as the position of the object on the x and y coordinates, the size of the object, the shape of the object, the time required for the annotation information to be given, etc. When there are multiple objects in one annotation image, the first annotation information includes the text data corresponding to the multiple objects.
[0059] use Figure 3B An example of information indicating an annotation skill will be described. Figure 3B This is a diagram illustrating the annotation skills related to Implementation Example 1.
[0060] like Figure 3BAs shown, the annotation skill is a skill related to the specification (assignment) of annotation information, that is, the skill of the operator, which indicates the accuracy of each object, the correct answer rate of the options, the operation time required for the assignment of annotation information, etc., in the operation of the operator assigning annotation information to the annotation image. The annotation skill is, for example, an indicator obtained by evaluating the correctness of the operation of the operator assigning annotation information to the annotation image. The annotation skill is determined for each object. Accuracy means the correctness of the annotation information such as the position, range (size), shape, etc., assigned to the object contained in the annotation image.
[0061] The first processing unit 10 obtains the annotation skills of the operator who assigns the annotation information. The first processing unit 10 outputs the annotation image having the characteristics of the image determined according to the annotation skills of the operator to the input / output unit 40. The characteristics of the image are, for example, objects captured in the image. The input / output unit 40 obtains the annotation image and displays the annotation image in order for the operator to perform the task of assigning the annotation information.
[0062] Furthermore, the first processing unit 10 updates the operator's annotation skills when acquiring information indicating the operator's annotation skills evaluated by the evaluation unit 33 described later. In addition, the first processing unit 10 may acquire information indicating the operator's annotation skills evaluated by other devices from other devices.
[0063] The second processing unit 20 includes a first acquiring unit 21 and an imparting unit 22 .
[0064] The first acquisition unit 21 acquires, via the input / output unit 40 , an input by an operator for adding annotation information to an annotation image.
[0065] The assigning unit 22 generates an annotated image in which annotation information is assigned to the annotation image acquired by the first acquiring unit 21 and second annotation information described later, based on the input of assigning annotation information to the annotation image acquired by the first acquiring unit 21. The second annotation information is annotation information assigned to the annotation image by the operator, and is text data corresponding to the annotated image. The second annotation information also includes the time taken to specify (assign) the annotation information. The assigning unit 22 outputs the generated information representing the annotated image and the second annotation information to the third processing unit 30. The second annotation information is an example of an annotation result.
[0066] use Figure 3C The second annotation information will be described. Figure 3C This is a diagram illustrating the second annotation information related to Implementation Example 1.
[0067] like Figure 3CAs shown, the second annotation information is text data of the position, size, shape, time required for adding the annotation information, etc. of the object on the x and y coordinates input by the operator for the annotation image. When there are multiple objects in one annotation image, the second annotation information is the text data corresponding to each of the multiple objects.
[0068] The third processing unit 30 includes a second acquisition unit 32 and an evaluation unit 33 .
[0069] The second acquisition unit 32 acquires the annotated image and the second annotation information from the second processing unit 20, and outputs the second annotation information to the evaluation unit 33. In addition, the second acquisition unit 32 acquires the first annotation information to be compared with the second annotation information from the storage unit 11 of the first processing unit 10, and outputs the acquired first annotation information to the evaluation unit 33.
[0070] The evaluation unit 33 evaluates the second annotation information. Specifically, the evaluation unit 33 Figure 3C The position, size, shape, etc. of the object in the second annotation information is consistent with Figure 3A The evaluation unit 33 compares the position, size, shape, etc. of the object in the first annotation information with the first annotation information, and calculates the respective differences (similarity). The evaluation unit 33 generates information representing the annotation skills of the operator based on the calculated differences. The evaluation unit 33 outputs the information representing the generated new annotation skills to the first processing unit 10. The information representing the annotation skills stored in the storage unit 11 of the first processing unit 10 is automatically updated.
[0071] Furthermore, if the difference (similarity) calculated by the evaluation unit 33 is equal to or greater than a predetermined threshold, it can be considered that the image is annotated with high accuracy, and thus the annotated image and the second annotation information may be sent to an external device as a finished product.
[0072] [action]
[0073] Next, use Figure 4 The operation of the information processing system 1 according to the present embodiment will be described.
[0074] Figure 4 This is a diagram illustrating the operation of the information processing system 1 according to the first embodiment.
[0075] In the information processing system 1, it is assumed that the annotation skills of the operator are stored in advance in the storage unit 11 of the first processing unit 10. Furthermore, the operator performs a task of adding annotation information to the annotation image via the input / output unit 40.
[0076] like Figure 4 As shown, first, the first processing unit 10 acquires the annotation skill of the operator from the storage unit 11 ( S11 ).
[0077] Next, the first processing unit 10 outputs an annotation image having features of an image determined according to the acquired annotation skill (S12). Specifically, the first processing unit 10 retrieves an annotation image in which an object corresponding to the annotation skill is captured from the storage unit 11. When there is an annotation image in which an object corresponding to the annotation skill is captured, the first processing unit 10 selects the annotation image. In addition, when there is no annotation image in which an object corresponding to the annotation skill is captured in the storage unit 11, the first processing unit 10 generates an annotation image in which an object corresponding to the annotation skill is captured. The first processing unit 10 outputs the selected or generated annotation image to the input / output unit 40.
[0078] Next, the operator inputs annotation information via the annotation image displayed on the input / output unit 40. Figure 5 An example of a task of adding annotation information to an annotation image will be described. Figure 5 This is a diagram illustrating a task of adding annotation information to an annotation image.
[0079] For example, on the input / output unit 40, it is displayed
[0080] (1) “Raise your hands” or “Put your hands down”
[0081] (2) "Open your legs" or "Close your legs"
[0082] These are mutually contradictory and independent options. The operator performs the operation by selecting one of the options.
[0083] Furthermore, the operator sets the range of the person photographed in the annotation image in order to give annotation information to the position of the object (in the figure, the person) in the annotation image via the input / output unit 40. Furthermore, the position of the person's head is set as a point, and the skeleton of the person's hands, legs, etc. is set as a line.
[0084] like Figure 4 As shown, the input / output unit 40 outputs the input received by the operator to the second processing unit 20. If the first acquisition unit 21 of the second processing unit 20 acquires the input for the operator to add annotation information to the annotation image, the adding unit 22 of the second processing unit 20 adds the annotation information to the annotation image to generate the annotated image and the second annotation information. Furthermore, the second processing unit 20 outputs the annotated image and the second annotation information to the third processing unit 30 (S13).
[0085] Next, the second acquisition unit 32 of the third processing unit 30 acquires the second annotation information from the second processing unit 20, acquires the first annotation information from the storage unit 11 of the first processing unit 10, and outputs the second annotation information and the first annotation information to the evaluation unit 33. The evaluation unit 33 Figure 3CThe position, size, shape, etc. of the object in the second annotation information is consistent with Figure 3A The position, size, shape, etc. of the object in the first annotation information are compared with each other, and the respective differences (similarity) are calculated. The differences are performed for each object in the second annotation information. The evaluation unit 33 generates information representing the annotation skills of the operator based on the calculated respective differences. The evaluation unit 33 outputs the information representing the annotation skills of the operator to the first processing unit 10, so that the information representing the annotation skills stored in the storage unit 11 of the first processing unit 10 is updated (S14).
[0086] The evaluation unit 33 determines whether there is a predetermined object (task) that the operator is not good at in the task of adding annotation information to the annotation image (S15). Specifically, the evaluation unit 33 determines whether there is a skill below a predetermined threshold in the information indicating the operator's annotation skill.
[0087] When there is no skill whose annotation skill is less than the predetermined threshold value (No in S15 ), since there is no task that the operator is not good at, the information processing system 1 returns to step S11 .
[0088] On the other hand, if there is a skill whose annotation skill is less than the predetermined threshold value (Yes in S15), since there is a task that the operator is not good at, the evaluation unit 33 generates information indicating the evaluation result showing the annotation skill that is less than the predetermined threshold value, and presents the information indicating the evaluation result to the operator (S16). That is, the evaluation unit 33 outputs the information indicating the evaluation result to the input / output unit 40 or an external display device such as a personal computer.
[0089] Here, the evaluation results presented to the operator are exemplified.
[0090] The third processing unit 30 presents an annotated image as a model, options as a model, etc., and an annotated image performed by the operator, options selected by the operator, etc. In addition, the third processing unit 30 presents the difference between the annotation performed by the operator and the annotation as a model. The difference is represented by a statistical index that evaluates the plurality of objects included in the annotation image as a whole, an individual index that evaluates each object, an absolute index that evaluates when the goal is achieved, and a relative index among the plurality of objects. Based on the above indexes, the third processing unit 30 presents the operator with an operation that he is good at or an operation that he is not good at.
[0091] In addition, the third processing unit 30 prompts the respective working time for each object captured in the annotation image, comparison with other operators' annotation skills, the operator's annotation skill history, ranking of multiple operators, the operator's unit price for work, practice questions corresponding to the operator, etc.
[0092] Then, the information processing system 1 ends the processing and returns the flow to the beginning.
[0093] [Effects, etc.]
[0094] In this way, in the information processing system 1, an annotation image containing features of the image corresponding to the annotation skills of the operator is determined. This enables annotation training corresponding to the individual characteristics of the operator. For example, an image that the operator is not good at is determined as an annotation image.
[0095] In addition, the evaluation unit 33 updates the operator's annotation skills stored in the storage unit 11 of the first processing unit 10. Therefore, it is possible to determine an annotation image that includes features of the image corresponding to the operator's new (updated) annotation skills. As a result, the operator can continuously improve his annotation skills by performing training to give annotation information to the annotation image corresponding to his own annotation skills. Therefore, the operator's annotation skills can be effectively improved according to personal characteristics.
[0096] (Implementation Method 2)
[0097] [structure]
[0098] The configurations of the information processing method and the information processing system 200 of this embodiment are the same as those of the first embodiment unless otherwise specified, and the same reference numerals are given to the same configurations, and detailed descriptions of the configurations are omitted.
[0099] Figure 6 This is a block diagram illustrating an information processing system 200 and an input / output unit 40 according to the second embodiment. Figure 7 This is a diagram illustrating the transfer of information in the information processing system 200 according to the second embodiment.
[0100] like Figure 6 and Figure 7 As shown, the information processing system 200 of this embodiment mixes the evaluation image for evaluating the annotation skills of the operator with the multiple annotation images presented to the operator. The information processing system 200 mixes the evaluation image with the multiple annotation images in a form that looks the same as the annotation image, so that the operator does not notice that the image is used to evaluate his or her own annotation skills when performing the operation. In this way, the information processing system 200 can present multiple annotation images and evaluation images to the operator.
[0101] The information processing system 200 includes a first image selection unit 210 , a second image selection unit 220 , and a management unit 230 in addition to the first processing unit 10 , the second processing unit 20 , and the third processing unit 30 .
[0102] The first processing unit 10 has a storage unit 11, which stores information representing an evaluation image and first annotation information. If the first processing unit 10 obtains information representing the annotation skills of the operator from the management unit 230, it obtains an evaluation image corresponding to the annotation skills. For example, the first processing unit 10 processes the evaluation image to be presented to the operator so that it looks the same as the annotation image presented to the operator by the first image selection unit 210 (so that the operator does not notice that it is an evaluation image different from the annotation image). In addition, when the evaluation image does not exist in the storage unit 11, the first processing unit 10 newly generates an evaluation image to be presented to the operator, or processes the annotation image into an evaluation image. In this way, the first processing unit 10 obtains the evaluation image.
[0103] The first image selection unit 210 holds a plurality of annotation images for the operator to perform the operation of adding annotation information. The first image selection unit 210 has a storage unit 221 for storing the annotation images. The first image selection unit 210 obtains the annotation skills of the operator from the management unit 230, and determines the annotation images corresponding to the annotation skills. The first image selection unit 210 obtains the evaluation image from the first processing unit 10, mixes the obtained evaluation image with the plurality of annotation images, and outputs the evaluation image and the plurality of annotation images to the input / output unit 40.
[0104] The second image selection unit 220 obtains the information indicating the annotated image and the second annotation information generated by the second processing unit 20 via the input / output unit 40. The second image selection unit 220 outputs the information indicating the annotated image for evaluation and the second annotation information for evaluation that have been processed as the evaluation image, from among the information indicating the annotated image and the second annotation information, to the third processing unit 30.
[0105] The evaluation unit 33 of the third processing unit 30 evaluates the second annotation information and the first annotation information for evaluation. The evaluation unit 33 generates information indicating the annotation skill of the operator and outputs it to the management unit 230.
[0106] Furthermore, the second image selection unit 220 may transmit information indicating annotated images other than for evaluation and the second annotation information as finished products to an external device.
[0107] The management unit 230 includes a storage unit 231 for storing information indicating annotation skills, and manages the annotation skills of the operator. When the management unit 230 acquires the annotation skills of the operator, it updates the annotation skills stored in the storage unit 231.
[0108] Furthermore, the management unit 230 selects a worker based on the annotation skill. For example, a worker with overall low annotation skill may be selected so that the worker actively performs the work of adding annotation information to the annotation image.
[0109] [action]
[0110] Next, the operation of the information processing system 200 according to the present embodiment will be described.
[0111] Figure 8 This is a diagram illustrating the operation of the information processing system 200 according to the second embodiment.
[0112] In the information processing system 200 , it is assumed that the annotation skills of the operator are stored in advance in the management unit 230 .
[0113] like Figure 8 As shown, first, the first processing unit 10 acquires the annotation skill of the operator from the management unit 230 ( S21 ).
[0114] Next, the first processing unit 10 outputs an evaluation image corresponding to the acquired annotation skill (S22). Specifically, the first processing unit 10 retrieves an evaluation image in which an object corresponding to the annotation skill is captured from the storage unit 11. When there is an evaluation image in which an object corresponding to the annotation skill is captured, the first processing unit 10 selects the evaluation image. In addition, when there is no evaluation image in which an object corresponding to the annotation skill is captured in the storage unit 11, the first processing unit 10 generates an evaluation image in which an object corresponding to the annotation skill is captured, or processes the annotation image to be prompted to the operator into an evaluation image based on the annotation image acquired from the first image selection unit 210 in a manner that the operator does not notice that it is an evaluation image. The first processing unit 10 outputs information indicating the selected, generated or processed evaluation image to the first image selection unit 210.
[0115] The first image selection unit 210 obtains the annotation skill of the operator from the management unit 230 via the first processing unit 10, and selects an annotation image corresponding to the annotation skill. The first image selection unit 210 obtains the evaluation image from the first processing unit 10, mixes the obtained evaluation image with a plurality of annotation images, and outputs the evaluation image and the plurality of annotation images to the input / output unit 40 (S23).
[0116] The input / output unit 40 outputs the input received by the operator to the second processing unit 20. The second processing unit 20 generates an annotated image to which the operator has added annotation information and second annotation information for the annotation image. Furthermore, the second processing unit 20 outputs the annotated image and the second annotation information to the second image selection unit 220 (S24).
[0117] The second image selection unit 220 obtains the information indicating the annotated image and the second annotation information. The second image selection unit 220 outputs the information indicating the annotated image for evaluation and the second annotation information for evaluation, which have been processed as evaluation images, to the third processing unit 30 from among the information indicating the annotated image and the second annotation information (S25).
[0118] Next, if the third processing unit 30 obtains information representing the annotated image for evaluation and the second annotation information for evaluation, the evaluation unit 33 of the third processing unit 30 evaluates the second annotation information for evaluation and the first annotation information. That is, the evaluation unit 33 generates information representing the annotation skills of the operator based on the difference between the second annotation information for evaluation and the first annotation information for evaluation. The evaluation unit 33 outputs the information representing the annotation skills of the operator to the first processing unit 10, so that the information representing the annotation skills stored in the storage unit 11 of the first processing unit 10 is updated (S26). That is, the evaluation unit 33 determines whether there is a specified object (job) that the operator is not good at in the job of the operator to give annotation information to the annotation image (S27). Specifically, the evaluation unit 33 determines whether there is a skill in the information representing the annotation skills of the operator that is less than a specified threshold.
[0119] When there is no annotation skill whose skill is less than the predetermined threshold value (No in S27 ), since there is no task that the operator is not good at, the information processing system 200 returns to step S21 .
[0120] On the other hand, when there are annotation skills that are less than the specified threshold (yes in S27), since there are tasks that the operator is not good at, the evaluation unit 33 generates information indicating the evaluation results of the annotation skills that are less than the specified threshold, and prompts the operator with the information indicating the evaluation results (S28).
[0121] The evaluation unit 33 outputs information indicating the annotation skill to the management unit 230. When the management unit 230 obtains information indicating the annotation skill of the operator, it updates the annotation skill stored in the storage unit 231. Then, the information processing system 200 ends the processing and returns the flow to the beginning.
[0122] [Effects, etc.]
[0123] In this way, in the information processing system 200, the evaluation image is processed or selected in such a way that the evaluation image is not noticed as an evaluation image different from the annotation image. The information processing system 200 mixes the evaluation image with a plurality of annotation images and presents them to the operator. Therefore, it is difficult for the operator to notice that the evaluation image is included, so the operator's annotation skills can be evaluated.
[0124] The other effects of this embodiment are the same as those of the first embodiment.
[0125] (Other Modifications)
[0126] As mentioned above, the information processing method and the information processing system according to the first and second embodiments of the present invention have been described. However, the embodiments of the present invention are not limited to the first and second embodiments described above.
[0127] For example, in the above embodiment, the sensing data of the annotation object is described as an example of image data, but the sensing data is not limited to this. The sensing data of the annotation object may also be other sensing data such as distance data obtained from a distance sensor, sound data obtained from a microphone, point group data obtained from a radar such as LIDAR (Light Detection and Ranging), and odor data obtained from an odor sensor.
[0128] Furthermore, for example, the present invention may be realized as a program for causing a computer to execute the information processing method according to the above-described embodiment, or a storage medium storing the program.
[0129] In addition, each processing unit included in the information processing system according to the above-mentioned embodiment is typically implemented as an integrated circuit, that is, LSI. These may be formed individually as one chip, or a part or all of them may be formed as one chip.
[0130] Furthermore, integrated circuits are not limited to LSIs, and can be implemented by dedicated circuits or general-purpose processors. FPGAs (Field Programmable Gate Arrays) that can be programmed after LSI manufacturing, or reconfigurable processors that can reconfigure the connections and settings of circuit cells within LSIs can also be used.
[0131] In addition, in each of the above-mentioned embodiments, each component may be formed by dedicated hardware, or may be implemented by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory.
[0132] In addition, all the numbers used in the above description are exemplified in order to specifically describe the present invention, and the embodiments of the present invention are not limited to the exemplified numbers.
[0133] In addition, the division of the functional blocks in the block diagram is an example, and multiple functional blocks can also be implemented as one functional block, or one functional block can be divided into multiple blocks, or a part of the function can be transferred to other functional blocks. In addition, the functions of multiple functional blocks with similar functions can also be processed in parallel or in time-sharing by a single hardware or software.
[0134] In addition, the order in which each step in the flowchart is executed is exemplified for the purpose of specifically describing the present invention, and may be an order other than the above. In addition, a part of the above steps may be executed simultaneously (in parallel) with other steps.
[0135] The above describes the information processing method and information processing system related to one or more technical solutions based on Embodiments 1 and 2, but Embodiments 1 and 2 of the present invention are not limited to the multiple technical solutions. As long as it does not deviate from the main purpose of the present invention, various forms after deformation thought of by those skilled in the art to Embodiments 1 and 2, and forms constructed by combining constituent elements of different embodiments may also be included in the scope of one or more technical solutions.
[0136] The present invention can be applied to an evaluation system for acquiring the skill of adding annotation information to sensory data, a training system for improving the skill of adding annotation information to sensory data, and the like.
Claims
1. An information processing method, It is characterized in that Use a computer to process as follows: obtaining an annotation skill of the operator, the annotation skill being an index obtained by evaluating the correctness of the operator's work of adding annotation information to image data in which the object is captured; Acquiring annotation image data, the annotation image data capturing the object determined based on the annotation skill; obtaining the annotation result of the annotation image data by the operator; The annotation skill is updated using the correct annotation information corresponding to the feature of the image data and the annotation result.
2. The information processing method according to claim 1, It is characterized in that The annotation skills include skills related to the specification of the annotation information.
3. The information processing method according to claim 2, It is characterized in that The specification of the annotation information includes at least one of specification of a position, specification of a size, and specification of a shape of the annotation.
4. The information processing method according to claim 2, It is characterized in that The specification of the annotation information includes the specification of the annotation label.
5. The information processing method according to any one of claims 1 to 4, It is characterized in that The above annotation result includes the time taken to specify the above annotation information; The annotation skill includes the time required for specifying the annotation information.
6. The information processing method according to any one of claims 1 to 4, It is characterized in that According to the above annotation skills, determine the above correct annotation information; The annotation image data is acquired by selecting the annotation image data in which an object corresponding to the determined correct annotation information is photographed from among the plurality of annotation image data.
7. The information processing method according to any one of claims 1 to 4, It is characterized in that According to the above annotation skills, determine the above correct annotation information; The annotation image data is acquired by generating image data having an object corresponding to the determined correct annotation information.
8. The information processing method according to claim 7, It is characterized in that The generation of the image data includes combining image data in which an object corresponding to the determined correct annotation information is photographed with other image data.
9. The information processing method according to any one of claims 1 to 4, It is characterized in that The difference between the correct annotation information and the annotation result is also presented to the operator.
10. The information processing method according to any one of claims 1 to 4, It is characterized in that The updated annotation skills will also be prompted to the above operators.
11. An information processing system, It is characterized in that have: a first acquisition unit that acquires annotated image data in which an object is photographed and determined based on an operator's annotation skill, wherein the annotation skill is an indicator for evaluating the correctness of an operation of the operator to add annotation information to the image data in which the object is photographed; A second acquisition unit acquires an annotation result of the annotation image data by the operator; and The processing unit updates the annotation skill using the correct annotation information corresponding to the feature of the image data and the annotation result.
