Annotation support device and annotation support method

Abstract

To provide technology efficiently carrying out annotation as comparing with similar images.SOLUTION: An annotation support device (100) is provided with a clustering section (35) for clustering image data included in an image group (60) and an annotation receiving section (206) for receiving label assignment on the image data included in a cluster generated through the clustering.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an annotation support device and an annotation support method. [Background technology]

[0002] In the annotation process of good and bad products using AI, the initial image data group contains a large number of images with various variations, and each image needs to be uniformly and concentratedly labeled. In the judgment of good and bad products, the evaluation of good and bad and defective products is often ambiguous, and the work takes time.

[0003] For this reason, techniques for supporting annotation by users have been proposed (see, for example, Patent Document 1). Here, images that have not yet been annotated among images registered in the image database are displayed on a display device, text data or numerical data describing the image input by the user via an input device is received, and this data is recorded in the teacher data field of the image. Image data management information for each image, including a teacher data field, is registered in the image database. The image data management information includes an attribute 2 feature field, in which a numerical value or the like indicating a category based on the degree of similarity with any one of the images registered in the image database 422 is recorded. For images with the same attribute 2 feature, at the time the data is input into the teacher data field of any one of the images, the same data is also recorded in the teacher data field for images with the same attribute 2 feature, thereby reducing the number of annotations by the user. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-224184 A Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide a technique for efficiently performing annotation while comparing with similar images. [Means for solving the problem]

[0006] To solve the above problems, the present invention provides: A clustering unit that clusters image data included in an image data group; an annotation receiving unit that receives labels for image data included in the clusters generated by the clustering; The annotation support device is characterized by comprising:

[0007] This makes it possible to perform clustering on image data that has not been annotated beforehand. In addition, since annotation is performed on image data included in a cluster, annotation can be performed efficiently while comparing with similar images.

[0008] In the present invention, a reference value providing unit that provides a reference value indicating a feature of the image data when the label is assigned to the image data included in the cluster; It may be prepared.

[0009] This allows annotation to be performed while referring to reference values ​​that indicate the characteristics of image data, thereby enabling efficient annotation.

[0010] In the present invention, The annotation receiving unit may arrange the image data included in the cluster according to the reference value.

[0011] According to this, image data to be annotated is arranged according to the reference value, so that similar images are arranged close to each other, enabling more efficient annotation.

[0012] In the present invention, a reference data receiving unit that receives, from the image data group, a selection of the image data to which a predetermined label is to be added as a reference image data group; The reference value may be an index indicating a degree of similarity to reference image data included in the reference image data group.

[0013] According to this, since an index showing the similarity with the reference image data is used as the reference value, annotation based on the reference image data can be performed more effectively.

[0014] In the present invention, The image data may include an automatic annotation unit that assigns a predetermined label to the image data included in the image data group based on the index indicating a similarity between the image data and reference image data included in the reference image data group.

[0015] According to this, a predetermined label is automatically assigned based on an index indicating the degree of similarity with reference image data included in the reference image data group, thereby shortening the time required to annotate image data that is similar to the reference image data, enabling efficient annotation.

[0016] In the present invention, The reference data receiving unit is When the selection of the reference image data group is accepted, an auxiliary indicator representing the appropriateness of the selection may be provided.

[0017] This makes it possible to select a more appropriate reference data group based on the auxiliary index.

[0018] In the present invention, The image data collection system may further include a pre-processing unit that performs predetermined image processing for the clustering on the image data included in the image data group.

[0019] According to this, clustering can be performed on image data that has been subjected to a predetermined image processing, so that unnecessary information can be removed in advance, and highly accurate clustering can be achieved.

[0020] In the present invention, The annotation receiving unit is The label is assigned to the plurality of image data included in the cluster at once. It may be possible to accept the request.

[0021] This enables more efficient annotation than when labels are individually assigned to image data.

[0022] In the present invention, The image data acquisition system may further include a reference setting receiving section that receives a setting of the number of the image data to be selected as the reference image data group.

[0023] According to this, by appropriately setting the number of image data selected as the reference image data group, it is possible to adjust the accuracy of the clustering.

[0024] In the present invention, The annotation receiving unit is The display mode of the image data included in the cluster may be changed according to the distance from the center of the cluster.

[0025] This allows the distance between the image data and the center of the cluster to be recognized based on the display mode of the image data, thereby enabling efficient annotation.

[0026] In the present invention, The annotation receiving unit is A setting of a region of interest in the image data may be accepted, and the region of interest in the image data may be displayed.

[0027] This allows annotation while focusing on the image of the region of interest, making annotation more accurate.

[0028] In the present invention, a reclustering receiving unit that receives re-clustering of the image data to which the label has not been assigned, among the image data group; The annotation receiving unit may be configured to receive assignment of the label to image data included in a cluster generated by the second clustering.

[0029] According to this, since unannotated image data can be annotated based on the relationship with different image data included in a new cluster, annotation with high accuracy is possible.

[0030] In the present invention, The annotation receiving unit is The apparatus may provide test results of a learning model generated using the annotated image data group as learning data, and may also accept changes to the assigned labels.

[0031] This allows the assigned labels to be changed according to the test results of a learning model generated using annotated image data as learning data, thereby making annotation efficient.

[0032] The present invention also provides a method for producing a method for manufacturing a semiconductor device comprising the steps of: A step of clustering image data included in the image data group; receiving labels for image data included in the clusters generated by the clustering; The present invention relates to an annotation support method.

[0033] This makes it possible to perform clustering on image data that has not been annotated beforehand. In addition, since annotation is performed on image data included in a cluster, annotation can be performed efficiently while comparing with similar images. Effect of the Invention

[0034] According to the present invention, it is possible to provide a technique for efficiently performing annotation while comparing with similar images. [Brief description of the drawings]

[0035] [Figure 1] 1 is a functional block diagram of an annotation support device according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a diagram illustrating a hardware configuration of an annotation support device according to an embodiment of the present invention. [Diagram 3] 11 is a flowchart illustrating a procedure of a decision unit generating process according to an embodiment of the present invention. [Figure 4] 1 is a flowchart illustrating a procedure of annotation support processing according to an embodiment of the present invention. [Diagram 5] FIG. 13 is a diagram showing a display example of a typical non-defective product selection screen according to an embodiment of the present invention. [Figure 6] FIG. 2 is a diagram illustrating a concept of clustering according to an embodiment of the present invention. [Figure 7] FIG. 11 is a diagram showing a display example of an annotation screen according to an embodiment of the present invention. [Figure 8] FIG. 13 is a diagram showing a display example of an area selection screen according to an embodiment of the present invention. [Figure 9] FIG. 13 is a diagram showing a display example of a test result screen according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] [Application example] Hereinafter, application examples of the present invention will be described with reference to the drawings.

[0037] Fig. 1 shows a functional block diagram of an annotation support device 100. Fig. 4 is a flowchart illustrating an annotation support method.

[0038] The annotation support device 100 includes an input unit 10, a display unit 20, a processing unit 30, a data management unit 40, and an annotation data storage unit 50, and can be configured by a general computer device. The processing unit 30 also includes a preprocessing unit 31, a learning processing unit 32, a reference information assignment processing unit 33, an automatic annotation unit 34, a grouping processing unit 35, a ROI processing unit 36, and a screen information generating unit 37.

[0039] First, in response to an instruction from a user via the input unit 10, the data management unit 40 reads the image data group 60 (step S11).

[0040] Next, the images included in the read image data group 60 are displayed on the display unit 20, and the user is prompted to select 20 images of typical non-defective products (hereinafter referred to as "typical non-defective product images") (step S12). FIG. 5 shows an example of a non-defective product selection screen 201 generated by the screen information generating unit 37 and displayed on the display unit 20. The user switches between images to be displayed in the image display area 202 by operating scroll bars 203 and 204, selects one of the images displayed in the image display area 202 by right-clicking the mouse, and further selects an "OK" button 205 displayed on the non-defective product selection screen 201 to confirm the selection of the typical non-defective product image. .

[0041] In parallel with the user's selection of typical non-defective images in step S2, the pre-processing unit 31 of the processing unit 30 performs pre-processing on the image data of the read image data group 60 to delete resist and wiring pattern colors (step S13).

[0042] Next, the learning processing unit 32 of the processing unit 30 learns the typical good product image data selected by the user in step S2 (step S14) and generates a model for the input image data whose output value is the similarity to the typical good product image data.

[0043] Next, image data whose output value is equal to or less than a predetermined threshold is removed from the data population (step S15). The output value is smaller as the image is closer to a typical good product image. In this way, by removing image data whose output value is equal to or less than a predetermined threshold from the data population, it is possible to remove obviously good products from the annotation targets, thereby improving the efficiency of annotation.

[0044] Next, the grouping processing unit 35 of the processing unit 30 groups the remaining image data after removing the image data whose output value is equal to or less than the predetermined threshold value from the population in step S5 (step S16).

[0045] Next, the user performs annotation for each group of image data grouped in step S6. FIG. 7 shows an example of an annotation screen 206 displayed on the display unit 20. In the image display area 207, each image IM11 to IM19 is displayed. In the upper right corner of each image IM11 to IM19, an output value 217 is also displayed. When the user moves the pointer onto each image IM11 to IM19 displayed in the image display area 207 and left-clicks the mouse, the label of the image changes from OK to NG to Gray in that order, so that the user can select the desired label. In the image display area 207, the images IM11 to IM19 are arranged and displayed in the order of output value. When the output value indicates similarity, the images are displayed in order of similarity, so that images with an output value equal to or greater than a certain value can be collectively determined as NG or Gray.

[0046] The image IM17 displayed in the image display area 207 is highlighted HL with a different background color. The highlighting HL is performed on a predetermined percentage (10% in this example) of images that are out of the center of the cluster.

[0047] As described above, the user switches between groups of images displayed in the image display area 207 and performs annotations on at least some of the images in each group.

[0048] If there are images to which the user has not assigned a label, the user can select whether or not to regroup the images to which no label has been assigned (step S18). The user can press the above-mentioned "regroup unannotated images" button 222, which causes the grouping processing unit 35 of the processing unit 30 to regroup the images to which no label has been assigned, and the user can assign labels to the regrouped images.

[0049] In this manner, by performing annotation using the annotation supporting device 100, similar images can be annotated consecutively, thereby reducing the time required for annotation work.

[0050] Example 1 Hereinafter, an annotation support device 100 according to a first embodiment of the present invention will be described in more detail with reference to the drawings. However, the configurations of the device and system described in this embodiment should be appropriately modified depending on various conditions. In other words, it is not intended to limit the scope of the present invention to the following embodiment.

[0051] Fig. 1 shows a functional block diagram of annotation support device 100. Fig. 2 shows a hardware configuration diagram of annotation support device 100. Fig. 3 is a flowchart explaining a process of generating a classifier by machine learning. Fig. 4 is a flowchart explaining an annotation support method.

[0052] When using machine learning to generate a judger that judges pass / fail from images of components and lands, the process follows the flowchart shown in Figure 3.

[0053] First, images of target areas such as components and lands are collected (step S1). Next, annotation is performed to assign a label of good or bad to the collected images (step S2). Learning data is extracted from the annotated image data, and the extracted learning data is machine-learned into a learning model (step S3). This generates a classifier that outputs a pass / fail judgment result from the input image. Next, the performance of the generated classifier is tested using test data extracted from the annotated image data (step S4). If the performance of the generated classifier is sufficient and there are no problems with the test results, the trained classifier is applied to the teaching software and put into practical use (step S5). If the performance of the generated classifier is not sufficient and there is a problem with the test results, the process returns to step S2) and the annotation is redone.

[0054] To generate a high-performance classifier, it is necessary to prepare a large amount of high-quality training data, and annotation work, which assigns labels to collected image data, plays an important role in this endeavor.

[0055] An annotation support device 100, which will be described below, supports such annotation and efficiently generates high-quality learning data.

[0056] The annotation support device 100 includes an input unit 10, a display unit 20, a processing unit 30, a data management unit 40, and an annotation data storage unit 50.

[0057] The annotation support device 100 is configured by a general-purpose computer device including a processor 101 such as a CPU, a main storage unit (memory) 102, an auxiliary storage unit (hard disk, etc.) 103, an input unit (keyboard, mouse, controller, touch panel, etc.) 10, a display unit (display) 20, a communication unit 104, etc. The annotation support device 100 may be a computer system in which a plurality of computers work together. The processing unit 30 and the data management unit 40 are realized by developing a program stored in the auxiliary storage unit 103 into the main storage unit and executing it in the processor 101.

[0058] The annotation assistance method executed in the annotation assistance device 100 will be described below with reference to the flowchart in FIG. First, in response to an instruction from a user via the input unit 10, the data management unit 40 reads the image data group 60 (step S11). The image data group 60 may be read from an external storage device connected via a network via the communication unit 104, or may be read from the auxiliary storage unit 103.

[0059] Next, images included in the read image data group 60 are displayed on the display unit 20, and the user is prompted to select 20 images of typical non-defective products (hereinafter, referred to as "typical non-defective product images") (step S12). FIG. 5 shows an example of a non-defective product selection screen 201 displayed on the display unit 20. Approximately 1000 images included in the image data group 60 are displayed, and the user is prompted to select an image that is determined to be a typical non-defective product from among them. In FIG. 5, nine images, images IM1 to IM9, are displayed in the image display area 202. The user switches the images to be displayed in the image display area 202 by operating scroll bars 203 and 204, selects one of the images displayed in the image display area 202 by right-clicking the mouse, or the like, and further selects an "OK" button 205 displayed on the non-defective product selection screen 201 to confirm the selection of the typical non-defective product image. The number of typical non-defective images to be selected is an example, and the number of typical non-defective images to be selected can be input or selected using an input field or a spinner-type reference setting receiving section, allowing an appropriate number of images to be selected. At this stage, the selection of typical non-defective images is performed for each target part. When the user selects an image, an index indicating the appropriateness of the selection may be displayed. An example of such an index is an index indicating the similarity of the selected images. If images with similar scores for the index indicating the similarity are selected, it can be confirmed that the selection is appropriate. The non-defective selection screen 201 corresponds to the reference data receiving section of the present invention. Moreover, the typical non-defective images correspond to the reference image data group and reference image data of the present invention. The index indicating the appropriateness of the selection corresponds to the auxiliary index of the present invention.

[0060] In parallel with the user's selection of typical non-defective images in step S2, the preprocessing unit 31 of the processing unit 30 performs preprocessing on the image data of the read image data group 60 to delete resist and wiring pattern colors (step S13). By deleting parts that are unnecessary for judgment, such as resist and wiring pattern colors, it is possible to improve the accuracy of judgment. Subsequent processing is performed on the preprocessed image data. Here, deleting resist and wiring pattern colors corresponds to the predetermined image processing for clustering in the present invention.

[0061] Next, the learning processing unit 32 of the processing unit 30 learns the typical non-defective image data selected by the user in step S2 (step S14), and generates a model for the input image data, in which the similarity to the typical non-defective image data is the output value. As the learning model, for example, an autoencoder can be used, but is not limited to this. The similarity to the typical non-defective image data can be calculated by a similarity method between two images such as SSIM (Structural Similarity). The average distance may be calculated using a similarity calculation method. Here, the similarity or the average distance corresponds to an index indicating the similarity or a reference value indicating the feature.

[0062] Next, the reference information assignment processing unit 33 of the processing unit 30 calculates output values ​​such as similarity of image data included in the image data group 60 using a learned model or other methods, and the automatic annotation unit 34 removes image data whose output value is equal to or less than a predetermined threshold from the data population (step S15). The output value becomes smaller as the image is closer to a typical good image. In this way, by removing image data whose output value is equal to or less than a predetermined threshold from the data population, it is possible to remove obviously good products from the annotation targets, thereby improving the efficiency of annotation. Images whose output value is equal to or less than a predetermined threshold are collectively labeled as "OK", making it unnecessary to perform individual annotation work. As the threshold, for example, an average value can be set, but it is also possible to allow the user to set an appropriate threshold.

[0063] Next, the grouping processing unit 35 of the processing unit 30 performs grouping (clustering) on ​​the remaining image data, excluding the image data whose output value is equal to or less than the predetermined threshold value from the population in step S5 (step S16). FIG. 6 is a diagram conceptually explaining the grouping by the grouping processing unit 35. Here, the grouped image data is divided into two-dimensional Here, a typical non-defective image GIM with no defective features is located near the origin of the feature space. A colored circle IM corresponds to one image, and a circle (ellipse) CL containing multiple circles corresponds to one group (cluster). An arrow Ar1 indicates an output value, which is represented by the distance from the typical non-defective image in the feature space. An arrow Ar2 indicates the distance from a cluster center C of an image belonging to the cluster CL, and highlighting, which will be described later, is performed according to the distance from the cluster center C. The grouping processing unit 35 corresponds to the clustering unit of the present invention.

[0064] Next, the user performs annotation for each group of the image data grouped in step S6. FIG. 7 shows an example of an annotation screen 206 that is generated by the screen information generating unit 37 based on the grouping result and is displayed on the display unit 20. Here, the group name 210 including the image data displayed in the image display area 207 is displayed as "Group A" on the upper side of the image display area 207. Also displayed is the group number 211 of all groups, "1 / 20" (the first group out of the total of 20 groups). Then, a "Previous Group" button 212 and a "Next Group" button 213 are displayed for switching the group of image data to be displayed in the image display area 207 to the previous group or the next group, respectively. Under the group name 210, an "OK" button 214, a "NG" button 215, and a "Gray" button 216 are displayed for collectively labeling the image data included in the group with "OK", "NG", and "Gray". The annotation screen 206 corresponds to the annotation receiving unit of the present invention.

[0065] In the image display area 207, each of the images IM11 to IM19 is displayed. In the upper right corner of each of the images IM11 to IM19, an output value 217 is also displayed. When the user moves the pointer onto each of the images IM11 to IM19 displayed in the image display area 207 and left-clicks the mouse, the label of the image changes in order from OK to NG to Gray, so that the user can select the desired label. The selected label 218 is displayed as "OK" in the upper left corner of each of the images IM11 to IM19. In the image display area 207, the output values ​​217 of each of the images IM11 to IM19 are displayed, and the images IM11 to IM19 are arranged and displayed in the order of the output values. When the output value indicates the similarity, the images are displayed in order of similarity, so that images with an output value equal to or greater than a certain value can be collectively determined as NG or Gray. Here, images are displayed in image display area 207 in order of output value, but the user may select whether to arrange images in order of output value, or in other manner such as in order of scores from past judgment models. Again, images displayed in image display area 207 can be changed by operating scroll bars 219 and 220. Here, the output value corresponds to the reference value indicating the characteristics of image data of the present invention, and annotation screen 206 corresponds to the reference value providing unit of the present invention.

[0066] Image IM17 displayed in image display area 207 is highlighted HL with a different background color. Highlight display HL is performed on a predetermined percentage of images that are off the center of the cluster as described above. On the right side of image display area 207, a slider 221 is displayed for allowing the user to set the percentage of images to be highlighted. Here, it is set to highlight 10% of images that are off the center of the cluster. The different background color such as highlight display HL corresponds to the display mode of the present invention. The display mode that is changed according to the distance from the center of the cluster is not limited to this.

[0067] A “Regroup Unannotated” button 222 and a “Save Results” button 223 are displayed on the right side of the image display area 207. When the user presses the “Save Results” button 223, the annotations made by the user are finalized (step S17). The confirmed annotation results are stored in the annotation data storage unit 50 by the data management unit 40. The annotation data storage unit 50 stores the assigned labels and image IDs that identify the image data in association with each other. The user does not need to assign labels to all images in each group, and may confirm annotations for only some of the grouped images. When regrouping images to which the user has not assigned labels, the user presses the "Regroup Unannotated Images" button 222. The "Regroup Unannotated Images" button 222 corresponds to the reclustering receiving unit of the present invention.

[0068] A user may set a part of an image as a region of interest (ROI) and enlarge and display the region of interest in the image display area 235. FIG. 8 shows an example of a region of interest setting reception screen 230. The user may select a region of interest 233 by, for example, surrounding a part of an enlarged image 231 displayed as a pop-up with a rectangular frame 232. The setting of the region of interest 233 is confirmed by pressing an "OK" button 234. The ROI processing unit 36 ​​of the processing unit 30 generates images IM21 to IM29 of the region of interest based on this setting, and the image information generating unit 37 displays the region of interest setting reception screen 230 on the image display area 235, in which the images IM21 to IM29 of the region of interest are displayed, on the display unit 20. The setting of the region of interest 233 is not limited to the case where the user selects a desired range, and may be set by the ROI processing unit 36 ​​of the processing unit 30 recommending an appropriate region of interest 233 and the user accepting the recommended region of interest 233.

[0069] As described above, the user switches between groups of images displayed in the image display area 207 and performs annotations on at least some of the images in each group.

[0070] If there are images to which the user has not assigned a label, the user can select whether or not to regroup the images to which no label has been assigned (step S18). To regroup the images to which no label has been assigned, the user presses the above-mentioned "regroup unannotated" button 222. When the "regroup unannotated" button 222 is pressed, the process returns to step S6, and the grouping processing unit 35 of the processing unit 30 regroups the images to which no label has been assigned. Then, the regrouped images are displayed on the display unit 20 in the same manner as in FIG. 7, and the user can assign labels to the regrouped images (step S17).

[0071] When all data has been labeled, the annotation is complete (step S9).

[0072] As described above, training data and test data are extracted from the annotated image data. Then, the training data is machine-trained into a judgment model (step S3), and the test data is input to the trained judgment model to perform a test (step S4).

[0073] FIG. 9 shows an example of a test result screen 240. A judgment summary 241 is displayed on the test result screen 240. The summary 241 displays in table form the number of data judged as OK and the number of data judged as NG for each of the OK data, NG data, and gray data. Here, of the OK data, 1000 items have been judged as OK and 20 items have been judged as NG. Furthermore, of the NG data, 0 items have been judged as OK and 100 items have been judged as NG. Furthermore, for the gray data, there are 0 items that have been judged as both OK and NG.

[0074] The test result screen 240 displays a detailed result field 242 for each image. If there is a problem with the test result, that is, if the performance of the generated judgment model is not sufficient, the detailed result field 242 displays the target image IM31, the output value 243, and the annotation screen 244. 06 is displayed. Pressing the "Move" button 244 moves to the annotation screen 206 of the image, where the user can change the label to be added. The test result screen 240 also displays a histogram 245 showing each of the OK and NG measurement values. This histogram 245 allows the user to check whether the OK measurement value 245a and the NG measurement value 245b are separated. The test result screen 240 also displays an "Apply Model" button 246. If the test results show that there is no problem with the model generated by the AI ​​learning tool, the user can press the "Apply Model" button 246 to apply the generated model to the teaching software (step S6).

[0075] In the following, the components of the present invention will be described with reference to the reference numerals in the drawings in order to make it possible to compare the components of the present invention with the configurations of the embodiments. <Appendix 1> A clustering unit (35) that clusters image data included in the image data group (60); an annotation receiving unit (206) that receives labels for image data included in the clusters generated by the clustering; An annotation support device (100) comprising: <Appendix 2> a reference value providing unit (201) that provides a reference value indicating a feature of the image data when the label is assigned to the image data included in the cluster; An annotation support device (100) according to claim 1, comprising: <Appendix 3> The annotation support device (100) according to claim 2, wherein the annotation receiving unit (206) arranges the image data included in the cluster according to the reference value. <Appendix 4> a reference data receiving unit that receives, from the image data group (60), a selection of the image data to which a predetermined label is to be assigned as a reference image data group; The annotation support device (100) according to appendix 2 or 3, wherein the reference value is an index indicating a similarity to reference image data included in the reference image data group. <Appendix 5> The annotation support device (100) described in Appendix 4 is characterized in that it includes an automatic annotation unit (34) that assigns a predetermined label to the image data included in the image data group based on the index indicating the similarity with reference image data included in the reference image data group. <Appendix 6> The reference data receiving unit (201) The annotation support device (100) according to appendix 4 or 5, wherein when accepting a selection of the reference image data group, an auxiliary indicator representing the appropriateness of the selection is provided. <Appendix 7> 7. The annotation support device (100) according to any one of claims 1 to 6, further comprising a pre-processing unit (31) that performs predetermined image processing for the clustering on the image data included in the image data group (60). <Appendix 8> The annotation receiving unit (206) The annotation support device (100) according to any one of appendices 1 to 7, characterized in that the label assignment is accepted collectively for the plurality of image data included in the cluster. <Appendix 9> A reference setting unit that receives a setting of the number of the image data to be selected as the reference image data group. 7. An annotation support device (100) according to any one of claims 4 to 6, comprising a constant reception unit. <Appendix 10> The annotation receiving unit (206) 10. The annotation support device (100) according to any one of claims 1 to 9, characterized in that a display mode of the image data included in the cluster is changed according to a distance from the center of the cluster. <Appendix 11> The annotation receiving unit (206) The annotation support device (100) according to any one of appendices 1 to 10, characterized in that it accepts setting of an area of ​​interest in the image data, and displays the area of ​​interest in the image data. <Appendix 12> a reclustering receiving unit (222) that receives re-clustering for the image data to which the label has not been assigned, among the image data group (60); The annotation support device (100) described in any one of Appendices 1 to 11, characterized in that the annotation receiving unit (206) receives the assignment of the label to image data included in the cluster generated by the re-clustering. <Appendix 13> The annotation receiving unit (206) The annotation support device (100) according to any one of appendices 1 to 12, characterized in that it provides test results of a learning model generated using the annotated image data group as learning data, and accepts changes to the assigned labels. <Appendix 14> A step of clustering the image data included in the image data group (60) (step S16); A step of accepting assignment of labels to image data included in the clusters generated by the clustering (step S17); An annotation assistance method comprising: [Explanation of symbols]

[0076] 35: Grouping processing section 60: Image data group 100: Annotation support device 206:Annotation screen

Claims

1. A clustering unit that clusters image data included in an image data group; an annotation receiving unit that receives labels for image data included in the clusters generated by the clustering; An annotation support device comprising:

2. a reference value providing unit that provides a reference value indicating a feature of the image data when the label is assigned to the image data included in the cluster; 2. The annotation support device according to claim 1, further comprising:

3. The annotation support device according to claim 2 , wherein the annotation receiving unit arranges the image data included in the cluster according to the reference value.

4. a reference data receiving unit that receives, from the image data group, a selection of the image data to which a predetermined label is to be added as a reference image data group; 4. The annotation support device according to claim 2, wherein the reference value is an index indicating a similarity to reference image data included in the reference image data group.

5. The annotation support device according to claim 4, further comprising an automatic annotation unit that assigns a predetermined label to the image data included in the image data group based on the index indicating the degree of similarity with reference image data included in the reference image data group.

6. The reference data receiving unit is The annotation support device according to claim 4 , further comprising: an auxiliary indicator indicating appropriateness of the selection when the selection of the reference image data group is accepted.

7. The annotation support device according to claim 1 , further comprising a pre-processing unit that performs predetermined image processing for the clustering on the image data included in the image data group.

8. The annotation receiving unit is The annotation support device according to claim 1 , further comprising: a step of accepting assignment of the label to a plurality of the image data included in the cluster all at once.

9. The annotation support device according to claim 4 , further comprising a reference setting receiving unit that receives a setting of the number of the image data to be selected as the reference image data group.

10. The annotation receiving unit is 4. The annotation support device according to claim 1, wherein a display mode of the image data included in the cluster is changed according to a distance from a center of the cluster.

11. The annotation receiving unit is 4. The annotation support device according to claim 1, further comprising: receiving a setting of an area of ​​interest in the image data; and displaying the area of ​​interest in the image data.

12. The image data group is subjected to re-labeling for the image data to which the label is not assigned. A reclustering reception unit is provided for receiving the clustering of 4. The annotation support device according to claim 1, wherein the annotation receiving unit receives the assignment of the label to image data included in a cluster generated by the second clustering.

13. The annotation receiving unit is The annotation support device according to any one of claims 1 to 3, characterized in that it provides test results of a learning model generated using the annotated image data group as learning data, and accepts changes to the assigned labels.

14. A step of clustering image data included in the image data group; receiving labels for image data included in the clusters generated by the clustering; An annotation assistance method comprising:

Images