Annotation method, annotation device, and annotation program

Abstract

To obtain an annotation method capable of, when performing annotation by using data in which non-annotated data and annotated data are mixed, determining accuracy of annotation for non-annotated data while suppressing a load of work of annotation of an operator compared with a conventional method.SOLUTION: An annotation method includes an image display step, an accuracy rate calculation step, and a determination step. The image display step displays a selection screen for allowing a user to select, from among annotated data and non-annotated data, data corresponding to classification based on a reference. The accuracy rate calculation step calculates an accuracy rate of a selection result of the annotated data, from the selection result of the annotated data and annotation information in which the annotated data and a result of the classification based on the reference are associated with each other. The determination step determines, on the basis of the accuracy rate, correctness of the selection result of the non-annotated data in the selection screen.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to an annotation method, an annotation device, and an annotation program for generating training data by labeling data that is a learning target. [Background technology]

[0002] In machine learning, training data in which data to be trained is labeled based on a predetermined classification is sometimes used. Labeling data to be trained in this way is called annotation. To improve the accuracy of inference using machine learning, it is necessary to prepare properly annotated training data. Patent Document 1 discloses a technique in which an operator performs annotation processing by combining annotation images with evaluation images for evaluating the operator's annotation skills, and the operator's skills are evaluated based on the accuracy of the operator's annotation results on the evaluation images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-24665 Summary of the Invention [Problem to be solved by the invention]

[0004] In the conventional technology, a mixture of unannotated images (images that have not been annotated) and annotated images (images that have already been annotated) is presented to the worker. This requires the worker to re-annotate the annotated images. In other words, in order to annotate the unannotated images, the worker must re-annotate not only the unannotated images but also the annotated images, which means that the worker must annotate more image data than the unannotated images. Thus, in the conventional technology, the re-annotation process is a wasteful task for the worker, placing a heavy burden on the worker. Furthermore, the conventional technology cannot determine the accuracy of the annotation results of the unannotated images obtained by the worker.

[0005] The present disclosure has been made in consideration of the above, and aims to provide an annotation method that, when annotating using data that is a mixture of unannotated data and annotated data, can determine the accuracy of annotations for unannotated data while reducing the annotation work burden on workers compared to conventional methods. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the present disclosure provides an annotation method in which an annotation device annotates data, the method including a screen display step, an accuracy rate calculation step, and a determination step. The screen display step displays a selection screen that allows a user to select data corresponding to a classification based on the criteria from annotated data, which is data classified based on a predetermined criterion, and unannotated data, which is data not classified based on the criterion. The accuracy rate calculation step calculates an accuracy rate for the user's selection of annotated data from the user's selection of annotated data on the selection screen and annotation information that associates the annotated data with the classification based on the criterion. The determination step determines the accuracy of the user's selection of unannotated data on the selection screen based on the accuracy rate. In the judgment process, if the accuracy rate is greater than a predetermined judgment reference value, the judgment result including the selection result of the unannotated data, the average accuracy rate, the variance, the number of collected data, and the authenticator information that identifies the user is stored in judgment result information associated with the unannotated data, and if multiple judgment results are obtained for the unannotated data, the selection result of the unannotated data is statistically processed to generate annotation result information. [Effects of the Invention]

[0007] According to the present disclosure, when annotating using data that is a mixture of unannotated data and annotated data, it is possible to determine the accuracy of annotations for unannotated data while reducing the annotation work burden on workers compared to conventional methods. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram schematically illustrating an example of the configuration of an annotation device according to a first embodiment. [Figure 2] FIG. 1 shows an example of annotation information according to the first embodiment. [Figure 3] FIG. 10 is a diagram showing an example of a selection screen according to the first embodiment; [Figure 4] FIG. 10 is a diagram showing an example of determination result information according to the first embodiment. [Figure 5] 1 is a flowchart showing an example of a procedure of an annotation method according to the first embodiment. [Figure 6] FIG. 10 is a diagram showing an example of a user's selection result on a selection screen according to the first embodiment; [Figure 7] FIG. 10 is a diagram showing another example of the determination result information according to the first embodiment. [Figure 8] FIG. 10 is a diagram showing another example of the determination result information according to the first embodiment. [Figure 9] FIG. 10 shows an example of a selection result of unannotated data by different users according to the first embodiment. [Figure 10] FIG. 10 is a diagram showing an example of determination result information according to the first embodiment. [Figure 11] FIG. 10 is a diagram showing an example of annotation information according to the second embodiment. [Figure 12] FIG. 10 is a diagram showing an example of a selection screen according to the second embodiment; [Figure 13] FIG. 10 is a diagram showing an example of a selection result on a selection screen according to the second embodiment; [Figure 14] FIG. 10 is a diagram showing an example of determination result information according to the second embodiment. [Figure 15] FIG. 10 is a diagram showing another example of the selection screen according to the second embodiment; [Figure 16] FIG. 10 is a diagram showing an example of determination result information according to the second embodiment. [Figure 17] FIG. 10 shows an example of the results of selection of unannotated data by different users in the second embodiment. [Figure 18] FIG. 10 is a diagram showing an example of determination result information according to the second embodiment. [Figure 19] FIG. 10 is a diagram showing an example of a selection screen according to the third embodiment; [Figure 20] FIG. 10 is a diagram showing an example of annotation information according to the third embodiment. [Figure 21] FIG. 13 is a diagram showing an example of determination result information according to the third embodiment. [Figure 22] FIG. 1 is a diagram showing an example of the configuration of an annotation device implemented using multiple computer systems. [Figure 23] FIG. 1 is a block diagram showing an example of the configuration of a computer system that realizes the annotation devices according to the first to third embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0009] An annotation method, an annotation device, and an annotation program according to embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0010] Embodiment 1 In the annotation method according to the first embodiment, during user authentication processing, a mixture of annotated data (data that has already been annotated) and unannotated data (data that has not yet been annotated) is presented to the user, and the user is prompted to select data from the mixture according to instructions, i.e., data classified based on a predetermined criterion. The selection result is then used to authenticate the user and annotate the unannotated data. When a user is required to perform a certain process and user authentication, such as logging in, is required to perform the process, the user typically performs the process carefully in order to be authenticated. In other words, the selection made during the user authentication process under such circumstances has a high accuracy rate, and the result of the authentication process can be used to annotate the unannotated data. The accuracy of the selection result for unannotated data can be assumed to have the same probability as the calculated accuracy rate for the annotated data, allowing the unannotated data to be annotated. Therefore, in the following embodiments, an annotation method, annotation device, and annotation program that combine authentication processing and annotation will be described.

[0011] 1 is a diagram schematically illustrating an example of the configuration of an annotation device according to Embodiment 1. The annotation device 10 includes an input unit 11, a display unit 12, an annotated information storage unit 13, an unannotated data storage unit 14, a data extraction unit 15, a playback processing unit 16, an input receiving unit 17, an authentication processing unit 18, an annotation processing unit 19, and a determination result information storage unit 20.

[0012] The input unit 11 is an input interface for a user who performs annotation in the annotation device 10. Examples of the input unit 11 include a keyboard and a mouse.

[0013] The display unit 12 is a display device that displays information to the user. An example of the display unit 12 is a liquid crystal display device.

[0014] The annotated information storage unit 13 stores annotated information, which is information about data that has already been annotated. The data includes image data, audio data, and video data. The annotated information includes annotated data, which is data that has been annotated, and annotation information that associates the annotated data with annotation results, which are results of classification based on predetermined criteria. Specifically, the annotation information associates the annotated data with tags, which are annotation results for the annotated data. Tags are information that indicate the classification of data based on predetermined criteria. One piece of data may have multiple tags depending on the use of the data.

[0015] FIG. 2 is a diagram showing an example of annotation information according to the first embodiment. As shown in FIG. 2, the annotation information is information in which information for identifying annotated data, such as a data name, is associated with a tag. In the example of FIG. 2, when the annotated data is image data, the tag indicates the result of classification according to the criterion of whether or not there is a scratch. Here, if the image data has a scratch, the tag is "Scratched," and if there is no scratch, the tag is "No Scratch." The annotation information is correct answer data or training data that indicates the result of classification of the annotated data based on a set criterion.

[0016] The unannotated data storage unit 14 stores unannotated data, which is data that has not been annotated, i.e., data that has not been classified based on a predetermined standard. The data includes image data, audio data, and video data.

[0017] The data extraction unit 15 extracts m pieces of annotated data from the annotated information storage unit 13 and n pieces of unannotated data from the unannotated data storage unit 14, where m and n are both natural numbers. The annotated data is used for user authentication. For this reason, it is desirable that the number of annotated data, m, be a number that allows authentication to be determined. When performing authentication, if the sum of the number of annotated data, m, and the number of unannotated data, n, is too large, authentication will take a long time and the burden on the user will be increased. For this reason, it is desirable that the total number of data (m+n) extracted by the data extraction unit 15 be set to an appropriate number depending on the type of data. In one example, it is desirable that the number of annotated data, m, is approximately "8," and the number of unannotated data, n, is approximately "2," which is about 1 / 4 of the annotated data.

[0018] The data extraction unit 15 preferably extracts annotated data from the annotated information storage unit 13 so as to include annotated data having a designated tag and annotated data not having the designated tag. However, the annotated data may not include annotated data having a designated tag, or may not include annotated data not having a designated tag. The designated tag is information indicating a classification to which unannotated data is to be annotated. In one example, if the tags associated with the annotated data are "scratched" and "not scratched," and the designated tag is "scratched," the data extraction unit 15 extracts annotated data with the tag "scratched" and annotated data with the tag "not scratched." In one example, the designated tag is set by an administrator of the annotation device 10 via the input unit 11.

[0019] The playback processing unit 16 generates a selection screen that allows the user to select data corresponding to a classification based on a set criterion from the annotated data and unannotated data extracted by the data extraction unit 15, and displays the selection screen on the display unit 12. If the data is audio data or video data, the playback processing unit 16 plays back the audio data or video data while displaying the selection screen on the display unit 12. The annotated data and unannotated data are arranged or displayed randomly on the selection screen. The selection screen includes an authentication instruction that is an instruction for authentication and also an instruction for annotation. The authentication instruction requests the selection of data containing a specified tag. The user views the data displayed on the selection screen and selects data in accordance with the authentication instruction via the input unit 11.

[0020] FIG. 3 is a diagram illustrating an example of a selection screen according to the first embodiment. The selection screen 200 includes an authentication instruction area 210, a data display area 220, and a confirmation button 230. The authentication instruction area 210 displays the authentication instruction. In this example, an authentication instruction such as "Please select all images with scratches" is displayed. The annotated data and unannotated data extracted by the data extraction unit 15 are randomly arranged in the data display area 220. Here, numbers 2, 3, 5-9, 11, and 12 represent annotated data, and numbers 1, 4, and 10 represent unannotated data. Each piece of data can be selected using a mouse or the like. After the user selects data that meets the authentication instructions and presses the confirmation button 230, the selection result is output to the input receiving unit 17. Note that in FIG. 3, a data name is displayed below each image for ease of explanation, but this data name is not displayed on the selection screen 200. The data names of the annotated data correspond to the data names in FIG. 2.

[0021] The input receiving unit 17 receives a selection result, which is data selected by the user on the selection screen 200 via the input unit 11. The input receiving unit 17 outputs the selection result to the authentication processing unit .

[0022] The authentication processing unit 18 acquires the selection result for the annotated data from the selection result. The authentication processing unit 18 refers to the annotated information in the annotated information storage unit 13, determines whether the selection result for the annotated data is correct, and calculates the accuracy rate. If the accuracy rate is greater than a reference value, which is a determination value for successful authentication, the authentication processing unit 18 determines that the authentication of the user is successful. If the accuracy rate is smaller than the reference value, the authentication processing unit 18 determines that the authentication of the user is unsuccessful. If the accuracy rate is equal to the reference value, the authentication of the user may be determined to be successful or unsuccessful. Note that the calculation method of the accuracy rate may be any method. In one example, the accuracy rate may be the ratio of the number of correct answers to the total number of annotated data. In another example, the accuracy rate may be the ratio of the number of correct answers to the total number of annotated data for each tag. In this case, the accuracy rate may differ for each tag, but the higher accuracy rate, the lower accuracy rate, or the average accuracy rate for each tag may be used as the accuracy rate. The authentication processing unit 18 corresponds to the accuracy rate calculation unit.

[0023] When the accuracy rate in the authentication processing unit 18 is greater than the judgment reference value, the annotation processing unit 19 judges the correctness of the selection result of the unannotated data by the user based on the accuracy rate. In the first embodiment, the selection result of the unannotated data made by the user on the selection screen 200 is considered to be correct with a probability of the accuracy rate. In other words, when the accuracy rate of the annotated data is 100%, the selection result of the unannotated data is also considered to be correct with a 100% probability. On the other hand, when the accuracy rate of the annotated data is smaller than the judgment reference value, the selection result of the unannotated data is also considered to have a low probability of being correct, and therefore is not adopted. In other words, when the accuracy rate in the authentication processing unit 18 is smaller than the judgment reference value, the annotation processing unit 19 does not adopt the judgment result of the unannotated data and discards it.

[0024] When the accuracy rate in the authentication processing unit 18 is greater than the judgment reference value, the annotation processing unit 19 acquires the selection results for the unannotated data and generates judgment result information that associates the selection results for the unannotated data with the unannotated data. The judgment result information includes the judgment results for the unannotated data. Furthermore, as will be described later, when multiple selection results are obtained for the unannotated data, the selection results for the unannotated data are statistically processed to generate judgment result information that includes statistical data. In this case, the judgment result information includes the selection results and statistical data that includes the accuracy rate. The annotation processing unit 19 stores the judgment result information in the judgment result information storage unit 20.

[0025] When multiple pieces of determination result information are obtained for unannotated data, the annotation processing unit 19 can store the determination result information for the unannotated data stored in the determination result information storage unit 20 and the new determination result information together as a single piece of determination result information. In this case, the selection results of the multiple pieces of unannotated data are statistically processed. In one example, the statistical data of the determination result information includes the average value, variance, number of collected data, etc. of the accuracy rate for the multiple selection results.

[0026] The criterion value can be any value. For example, if you want to increase the probability of accuracy in selecting unannotated data, you can set the criterion value to 0.8 or more. Since the accuracy rate is also affected by the number of annotated data, setting the criterion value too high may reduce the chances of obtaining a selection result for unannotated data. In this case, you can set the criterion value to 0.6 or more to ensure the number of selection results you can obtain.

[0027] The judgment result information storage unit 20 stores judgment result information. The judgment result information indicates the content of the judgment result made by the user for unannotated data. The judgment result includes, for each unannotated data, a selection result and statistical data including the accuracy rate of the selection result. The judgment result information stored in the judgment result information storage unit 20 ultimately becomes annotation result information, which is the annotation result for the unannotated data. In this case, the tag includes the selection result.

[0028] FIG. 4 is a diagram showing an example of determination result information according to the first embodiment. The determination result information is information indicating the annotation result of unannotated data by a user, i.e., the determination result of selection. Specifically, the determination result information is information that associates information identifying the unannotated data, such as the data name, with the determination result. In the example of FIG. 4, the determination result includes a selection result and an accuracy rate. The selection result is information indicating what selection the user made for the unannotated data on the selection screen 200. For example, if the user selects data number 1, i.e., image data G101, as "damaged" on the selection screen 200 of FIG. 3, "damaged" is stored as the selection result for the corresponding unannotated data in FIG. 4. The accuracy rate is stored as the accuracy rate calculated by the authentication processing unit 18.

[0029] Note that while the annotation information in FIG. 2 is shown as information in database format, this is merely an example. The annotation information may be attached to the annotated data itself. Furthermore, while the determination result information in FIG. 4 is shown as information in database format, this is merely an example. When the determination result information for certain unannotated data is to be regarded as final data, the final determination result information may be attached to the unannotated data itself as annotation result information. Unannotated data for which determination result information has been finalized as annotation result information cannot be called unannotated data, but in this specification, to distinguish it from annotated data, it will be referred to as unannotated data even if the determination result information has been finalized.

[0030] Next, an annotation method in such an annotation device 10 will be described. Fig. 5 is a flowchart showing an example of the procedure of the annotation method according to the first embodiment. As an example, the annotation method can be applied to a Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA), which is used in image authentication. In addition, the following description will be given taking an example where the data is image data.

[0031] First, when a user attempts to perform a certain process, execution of the annotation method is initiated. The certain process may be logging in to a computer system or connecting to certain data on a network. The data extraction unit 15 of the annotation device 10 extracts a predetermined number of annotated data from the annotated information storage unit 13 and extracts a predetermined number of unannotated data from the unannotated data storage unit 14 (step S11). Next, the playback processing unit 16 displays a selection screen 200 on the display unit 12, in which the extracted annotated data and unannotated data are randomly arranged (step S12). Here, it is assumed that the playback processing unit 16 displays the selection screen 200 shown in FIG. 3.

[0032] The user looks at the selection screen 200 displayed on the display unit 12 and selects data in accordance with the authentication instructions via the input unit 11. As described above, the user cannot access the desired information unless authenticated, so the selection of data in accordance with the authentication instructions during authentication is carried out carefully. In other words, it can be considered that humans distinguish data more accurately during authentication than in other tasks. In the first embodiment, this human tendency in situations where accurate distinction is required is utilized to use the selection results of unannotated data for annotation.

[0033] Fig. 6 is a diagram showing an example of a user's selection result on the selection screen of embodiment 1. In Fig. 6, the data selected by the user on selection screen 200 of Fig. 3 is surrounded by a rounded rectangle 221. Here, it is assumed that the user selected data numbers 1, 2, 7, 8, 10, and 11 as images with scratches.

[0034] When the user finishes selecting data in accordance with the authentication instructions and presses the confirmation button 230 on the selection screen 200, the input receiving unit 17 acquires the selection results on the selection screen 200 (step S13). Next, the authentication processing unit 18 acquires the selection results of the annotated data from the selection results (step S14), and calculates the accuracy rate indicating the correctness of the selection results of the annotated data by the user by referring to the annotation information in the annotated information storage unit 13 (step S15).

[0035] In the example of FIG. 6, the annotated data are numbers 2, 3, 5, 6, 7, 8, 9, 11, and 12. Of these, referring to the annotation information in FIG. 2, the data with scratches are numbers 2, 7, 8, and 11, and the data without scratches are numbers 3, 5, 6, 9, and 12. Furthermore, the user has selected numbers 2, 7, 8, and 11 from the data with scratches. This also means that the user has selected numbers 3, 5, 6, 9, and 12 as data without scratches.

[0036] Of the scratched data selected by the user, the correct answers are numbers 2, 7, 8, and 11. .a Notation done The total number of damaged data is four, and the number of correct answers is four, so the correct answer rate for the damaged data is 4 / 4=1.

[0037] Also, among the undamaged data that the user did not select, the correct answer is number 3,5. ,6, 9,12 .a Notation done The total number of flawless data is 5, and the number of correct answers is 5, so the accuracy rate for flawless data is 5 / 5=1.

[0038] Returning to FIG. 5, the authentication processing unit 18 determines whether the accuracy rate is equal to or greater than the judgment reference value (step S16). In the above example, the accuracy rate was "1" for both the scratched data and the scratch-free data, but it is possible that either one of them is less than the judgment reference value. In this case, the judgment may be made using the data with the higher accuracy rate, the data with the lower accuracy rate, or the average value of both. When more strict authentication is desired, it is desirable to make the judgment using the data with the lower accuracy rate.

[0039] If the accuracy rate is equal to or greater than the judgment reference value (Yes in step S16), the authentication processing unit 18 judges that the user authentication has been successful (step S17) and allows the user to perform a certain process. Thereafter, the annotation processing unit 19 acquires the selection result of unannotated data from the selection results (step S18). The annotation processing unit 19 judges the correctness of the selection result of unannotated data by the user on the selection screen 200 based on the accuracy rate (step S19). In other words, the annotation processing unit 19 considers that the selection result of unannotated data by the user is correct with the same probability as the accuracy rate of the selection result of annotated data. Specifically, the annotation processing unit 19 stores the acquired selection result of unannotated data and the accuracy rate corresponding to the selection result as judgment results in the judgment result information storage unit 20, in association with the acquired unannotated data.

[0040] In the example of FIG. 6, the unannotated data are numbers 1, 4, and 10. The user selects numbers 1 and 10 as images with scratches. In this case, the annotation processing unit 19 stores the selection result for numbers 1 and 10 as "scratched" and the accuracy rate of "1.0" in the determination result information. The annotation processing unit 19 also stores the selection result for number 4 as "no scratch," the opposite of "scratched," and the accuracy rate of "1.0" in the determination result information. FIG. 4 shows this content registered in the determination result information. In this way, when the authentication instruction is two-choice, not only can the one selected on the selection screen 200 be used as the selection result, but also the one not selected on the selection screen 200 can be used as the other option and as the selection result. This completes the annotation process of FIG. 5. When the annotation process for the unannotated data is completed and the determination result information is finalized, the determination result information becomes the annotation result information.

[0041] Returning to FIG. 5, if the accuracy rate is less than the judgment reference value in step S16 (No in step S16), the authentication processing unit 18 judges that user authentication has failed (step S20). As described above, in steps S16, S17, and S20, user authentication is performed using the accuracy rate calculated in step S15. If authentication fails, the accuracy rate does not satisfy the judgment reference value, and therefore the accuracy of the selection result of unannotated data cannot be guaranteed. For this reason, the process of saving the selection result of unannotated data as judgment result information is not performed. Then, the annotation process ends.

[0042] The unannotated data can be used multiple times as long as it is stored in the unannotated data storage unit 14. In other words, multiple pieces of determination result information may be obtained for the same unannotated data. In this case, multiple determination results for the same unannotated data may be stored in the determination result information in Fig. 4, or the determination results for the same unannotated data may be combined.

[0043] FIG. 7 is a diagram showing another example of the determination result information in the first embodiment. Note that the configuration of the determination result information is basically the same as that in FIG. 4, so the following will explain the differences. In the example of FIG. 7, the determination result includes the selection result, the average value and variance of the accuracy rate, and the number of collected data. The average value of the accuracy rate is the average value of the accuracy rates of multiple selection results for unannotated data. The variance of the accuracy rate is an index showing the variation in multiple accuracy rates. The number of collected data indicates the number of times unannotated data is used. Note that although an example using the average value and variance of the accuracy rate is shown here, other statistical values may also be used.

[0044] In this case, when the annotation processing unit 19 receives a new judgment result, it calculates the average value and variance of the accuracy rate and updates the average value and variance of the accuracy rate in the judgment result information. The annotation processing unit 19 also updates the number of collected data.

[0045] If an individual can be identified at the time of authentication, information such as authenticator information to prevent the same person from evaluating the same data may be added to the judgment results of unannotated data for which final annotation has not been determined.

[0046] FIG. 8 is a diagram showing another example of the determination result information in the first embodiment. The configuration of the determination result information is basically the same as that shown in FIG. 4, so differences will be described below. In the example shown in FIG. 8, the determination result includes a selection result, an accuracy rate, and authenticator information. The authenticator information is information for identifying the user who performed the authentication. In this case, the data extraction unit 15 acquires the authenticator information of the user who performs the authentication, refers to the determination result information, and extracts unannotated data from the unannotated data storage unit 14, excluding unannotated data having authenticator information that matches the acquired authenticator information. This prevents the same unannotated data from being used in the annotation process for the same user. That is, if the user to be authenticated can be identified as "A" at the time the selection screen 200 shown in FIG. 3 is displayed, the image data G101, G102, and G103 are not used in authenticating user "A," thereby preventing evaluations of the same image by the same person from being mixed in.

[0047] One piece of unannotated data can be judged by multiple users. Fig. 9 shows an example of the selection results of unannotated data by different users in embodiment 1. Here, the selection results by three users with authenticator information "A," "B," and "C" are shown for image data G101 and G102, which are unannotated data.

[0048] The annotation processing unit 19 performs statistical processing of the accuracy rates for the image data G101 and G102 shown in FIG. 9. Here, the annotation processing unit 19 calculates the average value, variance, and number of collected data of the accuracy rates as the statistical processing. The annotation processing unit 19 then registers the results of the statistical processing of the accuracy rates as the judgment result in the judgment result information. FIG. 10 is a diagram showing an example of the judgment result information of the first embodiment. As shown in FIG. 10, the judgment result includes the average value, variance, number of collected data, and authenticator information of the accuracy rates for the selection results, and it is possible to know the probability that the selection results are correct and the degree of variation that exists. Note that, when this judgment result information is used as the final annotation result information, the annotation processing unit 19 only needs to process the data so that it includes at least the selection results and the accuracy rates. In other words, the annotation processing unit 19 only needs to perform processing to delete unnecessary information from the judgment result.

[0049] By statistically processing the selection results of unannotated data by multiple users collected as described above, it is possible to obtain annotation results that can be judged in line with standard human sensibilities, despite the subjective variability of evaluations. Therefore, by taking into account the selection results of multiple users as shown in Fig. 9, it is possible to obtain final annotation result information, as shown in Fig. 10, in which the unannotated data is assigned judgment results calculated from statistical data.

[0050] The image data G101, G102 with annotation results can be used for various purposes by evaluating them based on the attached statistical data. For example, data with annotation information with a mean value less than 0.7 can be discarded as being unreliable, or data with large user variations can be extracted by looking at the variance.

[0051] The annotation method according to the first embodiment extracts a predetermined number of annotated data from the annotated information storage unit 13 and a predetermined number of unannotated data from the unannotated data storage unit 14, arranges them on the selection screen 200, and displays them to the user, allowing the user to select data that matches the authentication instruction. The annotation method according to the first embodiment calculates the accuracy rate of the selection result of the annotated data among the user's selection results on the selection screen 200, and if the accuracy rate is greater than a judgment reference value, determines that the user authentication has been successful and determines that the selection result of the unannotated data is correct based on the accuracy rate. Thus, the annotation method according to the first embodiment has the effect of being able to effectively utilize the annotation results of the data annotated by the worker while using the annotation results of the unannotated data as training data when annotating using data that is a mixture of unannotated data and unannotated data.

[0052] That is, in the first embodiment, the selection result of annotated data is not used for annotation, but for user authentication. Since it is used for user authentication, it does not impose a burden on the user. Furthermore, the accuracy rate of the selection result of annotated data is used as an index of the correctness of the selection result of unannotated data. This is because the judgment during user authentication is made more carefully than other processes and is expected to have a higher accuracy rate, so the selection result of unannotated data can be assumed to be correct with the same probability as the accuracy rate of annotated data. Furthermore, if the accuracy rate of annotated data is smaller than the judgment reference value, user authentication is deemed to have failed, and the selection result of unannotated data is discarded. Therefore, the selection result of unannotated data can be determined to be correct with the same probability as the accuracy rate corresponding to the judgment reference value. Thus, according to the annotation method of the first embodiment, the selection result of unannotated data is evaluated based on the correctness of the user's selection of annotated data, allowing the user performing authentication processing such as logging in to perform annotation without being aware that annotation is being performed.

[0053] Embodiment 2 In the example shown in the first embodiment, the data is image data and the authentication instruction has two options. In the second embodiment, an example is given in which the data is voice data and the authentication instruction has three or more options.

[0054] The configuration of the annotation device 10 of the second embodiment is the same as that described in the first embodiment, and therefore the description thereof will be omitted. The annotation method according to the second embodiment will be described below with reference to Fig. 5. First, in step S11, the data extraction unit 15 extracts a predetermined number of annotated data from the annotated information storage unit 13, and extracts a predetermined number of unannotated data from the unannotated data storage unit 14.

[0055] Here, the annotated information storage unit 13 stores audio data as annotated data. The annotation information of the annotated data also registers whether the data is a dog's bark, a cat's meow, or a bird's cry. FIG. 11 is a diagram showing an example of annotation information according to the second embodiment. Here, whether the annotated data is a dog's bark, a cat's meow, or a bird's cry is registered. A "1" is registered for those that match, and a "0" is registered for those that do not match. In one example, audio data A1 is "0" for dog and bird, and "1" for cat, so it is cat's meow data.

[0056] Next, in step S12, the playback processing unit 16 displays a selection screen 200 on the display unit 12, in which the extracted annotated data and unannotated data are randomly arranged. FIG. 12 is a diagram showing an example of the selection screen of the second embodiment. The selection screen 200 has an authentication instruction area 210, a data display area 220, a play again button 240, and a confirmation button 230. Note that the same components as those described in FIG. 3 are assigned the same reference numerals, and their description will be omitted. In this example, the authentication instruction area 210 displays an authentication instruction saying, "Please select the sound of a dog barking." The annotated data and unannotated data extracted by the data extraction unit 15 are randomly arranged in the data display area 220. However, since the data is audio data, an image 222 linked to the audio data is displayed. The image 222 of each data has a checkbox 223 for selecting the corresponding data if it matches the authentication instruction. Here, a filled-in checkbox 223 indicates that the data is selected, and an unfilled checkbox 223 indicates that the data is not selected. Here, numbers 1-5, 7, 9, 10, and 12 are annotated data, and numbers 6, 8, and 11 are unannotated data.

[0057] In this example, when the playback processing unit 16 displays the selection screen 200, it plays back the data in the data display area 220 in order, starting with the data numbered 1. The playback processing unit 16 displays the image 222 of the data being played back so that it can be distinguished from other data that is not being played back. In the example of Fig. 12, the data numbered 5 is being played back, and the periphery of the image data numbered 5 is displayed more emphasized than the others.

[0058] When the playback of the last audio data has finished, the playback processing unit 16 ends the playback of the audio data. When the user presses the play again button 240, the playback processing unit 16 plays the audio data in order, starting with data number 1. When the confirm button 230 is pressed, the selection result is output to the input receiving unit 17. Note that in FIG. 12, a data name is added below the image 222 of each data for the sake of convenience, but this data name is not displayed on the selection screen 200. The data names of the annotated data correspond to the data names in FIG. 11.

[0059] 13 is a diagram showing an example of a selection result on the selection screen of the second embodiment. Here, it is assumed that the user has selected numbers 2, 4, 6, and 7 as data indicating the barking of a dog. When the user finishes selecting data in accordance with the authentication instructions and presses the confirmation button 230 on the selection screen 200, the input receiving unit 17 acquires the selection result on the selection screen 200 in step S13.

[0060] In step S14, the authentication processing unit 18 acquires the selection result of the annotated data from the selection result, and in step S15, the authentication processing unit 18 refers to the annotation information in the annotated information storage unit 13 and calculates the accuracy rate indicating the correctness of the selection result for the annotated data. On the selection screen 200, numbers 1-5, 7, 9, 10, and 12 are annotated data. From the annotation information in FIG. 11, the data of the dog's bark are audio data A2, A4, A5, and A6, i.e., numbers 2, 4, 5, and 7 on the selection screen 200. Meanwhile, the user has selected data numbers 2, 4, and 7 from the annotated data as the dog's bark.

[0061] If the authentication processing unit 18 calculates the accuracy rate for the dog bark, the total number of audio data for dog barks is 4, and the user selected 3 of these as dog barks, so the accuracy rate is 3 / 4 = 0.75.

[0062] In step S16, the authentication processing unit 18 determines whether the accuracy rate is equal to or greater than the reference value. If the accuracy rate is equal to or greater than the reference value, in step S17, the authentication processing unit 18 determines that the user authentication has been successful. Thereafter, in step S18, the annotation processing unit 19 obtains the selection results of unannotated data from the selection results. Here, the data of numbers 6, 8, and 11 on the selection screen 200 are obtained.

[0063] In step S19, the annotation processing unit 19 determines the accuracy of the user's selection of unannotated data based on the accuracy rate. That is, if the accuracy rate exceeds the determination reference value, the annotation processing unit 19 registers the selection of unannotated data as being correct with the same probability as the accuracy rate in the determination result information. FIG. 14 is a diagram showing an example of determination result information in the second embodiment. In this example, the data numbered 6 on the selection screen 200, i.e., the audio data A101, was selected as a dog bark, so it is determined that the data numbered 6 is a dog bark with a probability of 0.75, which is the accuracy rate. Furthermore, the data numbers 8 and 11, i.e., the audio data A102 and A103, were not selected as a dog bark, so it is determined that the data numbered 8 and 11 are a dog bark with a probability of 0.25, which is 1 minus the accuracy rate.

[0064] In this manner, the determination result information for the dog bark in the unannotated data shown in FIG. 14 is obtained. However, because the annotated data is classified into dog barks, cat meows, and bird sounds, the above processing alone can only obtain information related to dog barks. Therefore, the playback processing unit 16 only needs to provide a selection screen 200 with authentication instructions for cat meows and bird sounds during other authentication processing. FIG. 15 is a diagram showing another example of a selection screen in the second embodiment. In FIG. 15, the authentication instruction has been changed to select cat meows. Similarly, the authentication processing unit 18 only needs to present a selection screen 200 with an authentication instruction for selecting bird sounds. In this manner, by using one unannotated data in the selection screen 200 with three different authentication instructions, the unannotated data can be annotated.

[0065] Fig. 16 is a diagram showing an example of determination result information according to embodiment 2. Here, the determination result information in Fig. 14 is shown with the addition of the user's determination results for the audio data A101, A102, and A103 as a cat's bark and a bird's cry. As a result, determination result information is obtained in which the accuracy of the audio data A101 being a dog's bark is 75%, the accuracy of the audio data A102 being a cat's meow is 100%, and the accuracy of the audio data A103 being a bird's cry is 80%.

[0066] Furthermore, when one piece of unannotated data is used on the selection screen 200 of multiple users, multiple results are obtained, as described in the first embodiment. FIG. 17 is a diagram showing an example of the selection results of unannotated data by different users in the second embodiment. Here, the selection screen 200 for authentication instructions to select a dog's bark is shown three times for the audio data A101. Furthermore, the probability that the sound is a dog's bark is subtracted from 1 to obtain the probability that the sound is a cat's or a bird's cry.

[0067] When a selection result such as that shown in FIG. 17 is obtained, the annotation processing unit 19 adds statistical data of these data to the audio data A101 to create final annotation result information. Here, the annotation processing unit 19 calculates the average value of the accuracy rate, the variance, and the number of data. FIG. 18 is a diagram showing an example of determination result information in the second embodiment. The audio data A101 records the average value of the accuracy rate, the variance, and the number of data for the dog bark, the average value of the accuracy rate, the variance, and the number of data for the cat meow, and the average value of the accuracy rate, the variance, and the number of data for the bird cry, all calculated from three data. Furthermore, this determination result information can be used as final annotation result information.

[0068] The second embodiment can also achieve the same effects as the first embodiment.

[0069] Embodiment 3 In the example shown in the first embodiment, the data is image data, and in the example shown in the second embodiment, the data is audio data. In the third embodiment, the data is video data.

[0070] The configuration of the annotation device 10 of the embodiment 3 is the same as that described in the embodiment 1. The annotation method is also the same as that described in the embodiments 1 and 2, so only the differences from the embodiments 1 and 2 will be described.

[0071] FIG. 19 is a diagram showing an example of a selection screen according to the third embodiment. In the third embodiment, the selection screen 200 includes an authentication instruction area 210, a data display area 220, a Yes button 251, and a No button 252. The authentication instruction area 210 displays an authentication instruction such as "Do you play soccer?" The data display area 220 is an area where one piece of video data from the annotated data and the unannotated data extracted by the data extraction unit 15 is played. The video data is distributed from the annotated information storage unit 13 or the unannotated data storage unit 14 and played in the data display area 220. The Yes button 251 is pressed by the user when the video data being played in the data display area 220 corresponds to the authentication instruction. The No button 252 is pressed by the user when the video data being played in the data display area 220 does not correspond to the authentication instruction.

[0072] When the user presses the Yes button 251 or the No button 252, the input accepting unit 17 accepts the user's selection result corresponding to the data being played in the data display area 220. Furthermore, if the predetermined number of videos have not been played, the playback processing unit 16 selects the next video data from the annotated information storage unit 13 or the unannotated data storage unit 14 and plays it in the data display area 220. The predetermined number is the sum of the number of annotated data extracted from the annotated information storage unit 13 and the number of unannotated data extracted from the unannotated data storage unit 14. When the predetermined number of videos have been played, the authentication processing unit 18 performs user authentication processing using the selection result, and if the user authentication is successful, the annotation processing unit 19 performs annotation processing using the accuracy rate.

[0073] Fig. 20 is a diagram showing an example of annotation information according to embodiment 3. As shown in Fig. 20, annotation information has already been generated for the video data of the annotated data so that it is clear whether soccer is being played or not. In this example, the tag includes a video classification indicating the classification of the video data and the result of the video classification.

[0074] In one example, the playback processing unit 16 plays back the video data V1, V2, and V3 in Fig. 20 and the unannotated video data V101 and V102 in the data display area 220 of the selection screen 200 in random order. Then, the input receiving unit 17 receives the selection result by the user. Here, it is assumed that the following selection result is obtained: Video Data V1: No Video Data V2: No Video data V3: Yes Video data V101: Yes Video data V102: No

[0075] The authentication processing unit 18 refers to the annotation information and calculates the accuracy rate for the video data V1, V2, and V3, which are annotated data. Here, the authentication processing unit 18 calculates the accuracy rate as the ratio of the number of correct answers to the total number of annotated data. Since the total number of annotated data is "3" and the number of correct answers is "2," the authentication processing unit 18 calculates the accuracy rate to be 2 / 3 = 0.67.

[0076] If the accuracy rate is greater than the judgment reference value, the accuracy of the selection result of the video data V101 and V102, which are unannotated data, is considered to be proportional to the accuracy rate of the video data V1, V2, and V3, which are annotated data. Therefore, the annotation processing unit 19 assigns the accuracy rate of the annotated data as the accuracy of the selection result of the unannotated data. FIG. 21 is a diagram showing an example of judgment result information in the third embodiment. In FIG. 21, the judgment result includes a selection result and an accuracy rate. The selection result also includes a video classification and a video classification result. Since the target video is soccer, the video classification is soccer. The result indicates whether the data corresponds to the video classification. In FIG. 21, the probability that the video data V101 is playing soccer is 0.67, and the probability that the video data V102 is not playing soccer is 0.67. This judgment result information is used as the final annotation information for the unannotated data.

[0077] In this way, even when the data is video data, it is possible to authenticate the user and also annotate the video data using the user authentication result. In other words, the third embodiment can also achieve the same effects as the first embodiment.

[0078] While the above description exemplifies the application of the annotation method to CAPTCHA, the annotation method can also be applied to general methods for authenticating users. While the above description exemplifies the addition of statistical data such as the mean value, variance, and number of data points of the accuracy rate, any statistical data can be used. Examples of other statistical data include the maximum value, minimum value, standard deviation, median, mode, geometric mean, and harmonic mean of the accuracy rate. The annotation processing unit 19 is configured to calculate required items depending on the purpose for which the judgment result information is used.

[0079] In one example, the annotation device 10 described above is realized by a computer system. In this case, the annotation device 10 shown in Fig. 1 may be realized by one computer system or by multiple computer systems.

[0080] An example of the annotation device 10 implemented using multiple computer systems will be described. Fig. 22 is a diagram schematically illustrating an example of the configuration of the annotation device implemented using multiple computer systems. The annotation device 10 includes a server device 30 and an information processing device 40, and the server device 30 and the information processing device 40 are connected via a network 50.

[0081] In this case, the server device 30 has an annotated information storage unit 13, an unannotated data storage unit 14, a data extraction unit 15, a reproduction processing unit 16, an input receiving unit 17, an authentication processing unit 18, an annotation processing unit 19, and a judgment result information storage unit 20. The server device 30 may be an on-premise server or a cloud server. The information processing device 40 has an input unit 11 and a display unit 12. The information processing device 40 is a computer system such as a personal computer owned by a user.

[0082] Next, the hardware configuration of the annotation device 10 according to the first to third embodiments will be described. In the annotation device 10 according to the first to third embodiments, a computer system functions as the annotation device 10 by executing a program, which is a computer program describing the processing in the annotation device 10, on the computer system. FIG. 23 is a block diagram showing an example of the configuration of a computer system that realizes the annotation device according to the first to third embodiments. As shown in FIG. 23, this computer system includes a control unit 501, an input unit 502, a storage unit 503, a display unit 504, a communication unit 505, and an output unit 506, which are connected via a system bus 507.

[0083] In FIG. 23 , the control unit 501 is a processor such as a CPU (Central Processing Unit) that executes a program describing the processing in the annotation device 10 of any one of the first to third embodiments, specifically, an annotation program that describes the annotation method shown in FIG. 5 . The input unit 502, for example, is configured with a keyboard, a mouse, and the like, and is used by a user of the computer system to input various information. The storage unit 503 includes various memories such as RAM (Random Access Memory) and ROM (Read Only Memory) and a storage device such as a hard disk, and stores programs to be executed by the control unit 501, necessary data obtained during processing, and the like. The storage unit 503 is also used as a temporary storage area for programs. The display unit 504 is configured with a display, a liquid crystal display device, and the like, and displays various screens to the user of the computer system. The communication unit 505 is a receiver and transmitter that performs communication processing. The output unit 506 is a printer, a speaker, and the like. Note that FIG. 23 is merely an example, and the configuration of the computer system is not limited to the example of FIG. 23 .

[0084] Here, an example of the operation of the computer system until the program is ready to be executed will be described. In the computer system having the above configuration, the program is installed in the storage unit 503 from, for example, a CD-ROM or DVD-ROM inserted in a CD (Compact Disc)-ROM drive or DVD (Digital Versatile Disc)-ROM drive (not shown). Then, when the program is executed, the program read from the storage unit 503 is stored in the main storage area of the storage unit 503. In this state, the control unit 501 executes the processing as the annotation device 10 of any of the first to third embodiments in accordance with the program stored in the storage unit 503.

[0085] In the above description, a program describing the processing in the annotation device 10 is provided using a CD-ROM or DVD-ROM as a recording medium, but this is not limited to this. Depending on the configuration of the computer system, the capacity of the program to be provided, etc., it is also possible to use a program provided via a transmission medium such as the Internet via the communication unit 505.

[0086] The data extraction unit 15, playback processing unit 16, input reception unit 17, authentication processing unit 18, and annotation processing unit 19 shown in Fig. 1 are realized by the control unit 501 shown in Fig. 23 executing programs stored in the storage unit 503 shown in Fig. 23. The storage unit 503 shown in Fig. 23 is also used to realize the data extraction unit 15, playback processing unit 16, input reception unit 17, authentication processing unit 18, and annotation processing unit 19. The annotated information storage unit 13, unannotated data storage unit 14, and determination result information storage unit 20 shown in Fig. 1 are realized by the storage unit 503 shown in Fig. 23.

[0087] Various aspects of the present disclosure are summarized below as appendices.

[0088] [Appendix 1] An annotation method in which an annotation device annotates data, comprising: a screen display step of displaying a selection screen that allows a user to select data corresponding to a classification based on the criteria from annotated data, which is data classified based on a predetermined criterion, and unannotated data, which is data not classified based on the criterion; an accuracy rate calculation step of calculating an accuracy rate for the selection result of the annotated data by the user from the selection result of the annotated data by the user on the selection screen and annotation information that associates the annotated data with the classification result based on the criteria; a determining step of determining the correctness of the selection result of the unannotated data by the user on the selection screen based on the accuracy rate; An annotation method comprising: [Appendix 2] The annotation method described in Appendix 1, characterized in that in the judgment step, if the accuracy rate is greater than a predetermined judgment reference value, the selection result of the unannotated data is set as annotation result information associated with the unannotated data. [Appendix 3] The annotation method described in Appendix 2, characterized in that in the judgment step, if multiple judgment results are obtained for the unannotated data, the selection results of the unannotated data are statistically processed to generate the annotation result information. [Appendix 4] further comprising an authentication step of authenticating the user using the accuracy rate calculated in the accuracy rate calculation step; An annotation method described in any one of Appendices 1 to 3, characterized in that in the authentication process, if the accuracy rate is greater than a predetermined judgment reference value, it is determined that authentication of the user is successful. [Appendix 5] The annotation method according to any one of claims 1 to 4, wherein the data is image data, audio data, or video data. [Appendix 6] a reproduction processing unit that displays a selection screen that allows a user to select data corresponding to a classification based on the criteria from annotated data, which is data classified based on a predetermined criterion, and unannotated data, which is data not classified based on the criterion; an accuracy rate calculation unit that calculates an accuracy rate for the selection result of the annotated data by the user from the selection result of the annotated data by the user on the selection screen and annotation information that associates the annotated data with the classification result based on the criteria; and an annotation processing unit that determines the correctness of a selection result of the unannotated data by the user on the selection screen based on the accuracy rate; An annotation device comprising: [Appendix 7] displaying a selection screen that allows a user to select data corresponding to a classification based on the criteria from among annotated data, which is data classified based on a predetermined criterion, and unannotated data, which is data not classified based on the criterion; calculating a correct answer rate for the selection result of the annotated data by the user from the selection result of the annotated data by the user on the selection screen and annotation information that associates the annotated data with the classification result based on the criteria; determining the accuracy of the selection result of the unannotated data by the user on the selection screen based on the accuracy rate; An annotation program characterized by causing a computer system to execute the above.

[0089] The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or different embodiments may be combined with each other. It is also possible to omit or modify parts of the configurations as long as they do not deviate from the gist of the invention. [Explanation of symbols]

[0090] 10 Annotation device, 11 Input unit, 12 Display unit, 13 Annotated information storage unit, 14 Unannotated data storage unit, 15 Data extraction unit, 16 Playback processing unit, 17 Input acceptance unit, 18 Authentication processing unit, 19 Annotation processing unit, 20 Judgment result information storage unit, 30 Server device, 40 Information processing device, 50 Network, 200 Selection screen, 210 Authentication instruction area, 220 Data display area, 221 Rounded rectangle, 222 Image, 223 Check box, 230 Confirmation button, 240 Play again button, 251 Yes button, 252 No button.

Claims

1. An annotation method in which an annotation device annotates data, comprising: a screen display step of displaying a selection screen that allows a user to select data corresponding to a classification based on the criteria from annotated data, which is data classified based on a predetermined criterion, and unannotated data, which is data not classified based on the criterion; an accuracy rate calculation step of calculating an accuracy rate for the selection result of the annotated data by the user from the selection result of the annotated data by the user on the selection screen and annotation information that associates the annotated data with the classification result based on the criteria; a determining step of determining the correctness of the selection result of the unannotated data by the user on the selection screen based on the accuracy rate; Including, In the judgment step, if the accuracy rate is greater than a predetermined judgment reference value, the judgment result including the selection result of the unannotated data, the average value of the accuracy rate, the variance, the number of collected data, and authenticator information identifying the user is stored in judgment result information associated with the unannotated data, and if multiple judgment results are obtained for the unannotated data, the selection result of the unannotated data is statistically processed to generate annotation result information.

2. The annotation method described in Claim 1, characterized in that in the judgment process, data processing is performed to include the selection result and the accuracy rate, and the annotation result information is generated.

3. further comprising an authentication step of authenticating the user using the accuracy rate calculated in the accuracy rate calculation step; 2. The annotation method according to claim 1, wherein in the authentication step, it is determined that the authentication of the user has been successful if the accuracy rate is greater than a predetermined reference value.

4. The annotation method according to claim 1 , wherein the data is image data, audio data, or video data.

5. a reproduction processing unit that displays a selection screen that allows a user to select data corresponding to a classification based on the criteria from annotated data, which is data classified based on a predetermined criterion, and unannotated data, which is data not classified based on the criterion; an accuracy rate calculation unit that calculates an accuracy rate for the selection result of the annotated data by the user from the selection result of the annotated data by the user on the selection screen and annotation information that associates the annotated data with the classification result based on the criteria; and an annotation processing unit that determines the correctness of a selection result of the unannotated data by the user on the selection screen based on the accuracy rate; Equipped with When the accuracy rate is greater than a predetermined judgment reference value, the annotation processing unit stores the judgment result, which includes the selection result of the unannotated data, the average value of the accuracy rate, the variance, the number of collected data, and authenticator information that identifies the user, in judgment result information associated with the unannotated data; and when multiple judgment results are obtained for the unannotated data, statistically processes the selection result of the unannotated data to generate annotation result information.

6. displaying a selection screen that allows a user to select data corresponding to a classification based on the criteria from among annotated data, which is data classified based on a predetermined criterion, and unannotated data, which is data not classified based on the criterion; calculating a correct answer rate for the selection result of the annotated data by the user from the selection result of the annotated data by the user on the selection screen and annotation information that associates the annotated data with the classification result based on the criteria; determining the accuracy of the selection result of the unannotated data by the user on the selection screen based on the accuracy rate; on a computer system, In the step of determining the correctness of the selection result, if the accuracy rate is greater than a predetermined judgment reference value, the judgment result including the selection result of the unannotated data, the average value of the accuracy rate, the variance, the number of collected data, and authentication person information that identifies the user is stored in judgment result information associated with the unannotated data, and if multiple judgment results are obtained for the unannotated data, the selection result of the unannotated data is statistically processed to generate annotation result information.

Images