Training data creation assistance device, system, and training data creation assistance method

Abstract

The present invention aims to provide an auxiliary device for creating training data, an auxiliary system for creating training data, and an auxiliary method for creating training data that can create accurate training data while reducing the burden on the user. A first label to be assigned to the training data is estimated by an estimation unit using a pre-prepared learning model. The estimated first label is presented by a presentation unit. A second label to be assigned to the training data is received by a reception unit. Alternatively, a determination unit determines whether the first label and the second label are different. In the case where it is determined that the first label and the second label are different, a notification unit gives a notification. In response to an instruction from the user, the second label different from the first label is assigned to the training data by an assignment unit.

Description

Technical Field

[0001] The present invention relates to a training data creation assistance device, a training data creation assistance system, and a training data creation assistance method. Background Art

[0002] Training data used in supervised learning is generated by assigning a label indicating the correct answer to image data representing an object, characters, etc. When the assigned label is inaccurate, the accuracy of supervised learning decreases. Therefore, accurate label assignment (marking) is required.

[0003] For example, in the data marking operation inspection method described in Japanese Unexamined Patent Application Publication No. 2019-96319, multiple workers submit marking information related to one piece of image data. The marking information submitted by each worker is compared, and when it is not determined that all the marking information is the same, rework is requested.

[0004] In order to perform high-precision supervised learning, a large amount of training data needs to be created. However, when marking a large amount of training data, it takes a long time to perform the operation, and thus the judgment of the worker may become ambiguous due to a decrease in concentration or fatigue. Therefore, it is difficult to accurately mark a large amount of training data. Summary of the Invention

[0005] An object of the present invention is to provide a training data creation assistance device, a training data creation assistance system, and a training data creation assistance method that can create accurate training data while reducing the burden on the user.

[0006] (1) A training data creation assistance device according to one aspect of the present invention is a training data creation assistance device that assists in assigning a label to training data, and includes: a estimation unit that estimates, as a first label, a label that should be assigned to the training data using a pre-prepared learning model; a prompting unit that prompts the first label estimated by the estimation unit; and a receiving unit that receives, as a second label, a label that should be assigned to the training data.

[0007] In this training data creation assistance device, the first label estimated by the estimation unit using the learning model is prompted by the prompting unit as the label that should be assigned to the training data. Therefore, when the user designates the second label that should be assigned to the training data to the receiving unit, the user can refer to the first label prompted by the prompting unit. In addition, even when the user makes an incorrect designation due to a decrease in concentration or fatigue, the user can easily notice the error. Thus, accurate training data can be created while reducing the burden on the user.

[0008] (2) The estimation unit can evaluate the reliability of the estimated first label, and the prompting unit can further prompt the reliability evaluated by the estimation unit. In this case, the user only needs to carefully judge the second label to be specified for the training data with a relatively low reliability of the first label. Thus, the burden on the user can be further reduced.

[0009] (3) The estimation unit can also estimate multiple first labels to be assigned to multiple training data respectively, and evaluate the reliability of each estimated first label. The prompting unit can prompt the multiple first labels estimated by the estimation unit in the order of reliability. In this case, the user can easily identify the group of multiple training data for which the second label should be specified through careful judgment. Thus, the burden on the user can be further reduced.

[0010] (4) The training data can include image data representing an image, and the prompting unit can further prompt an image based on the training data in a manner corresponding to the first label. In this case, the user can easily identify the correspondence between the training data representing the image and the first label estimated for the training data.

[0011] (5) The training data creation assistance device according to another aspect of the present invention is a training data creation assistance device that assists in assigning labels to training data, and includes: an estimation unit that estimates a label to be assigned to training data as a first label using a pre-prepared learning model; a reception unit that receives a label to be assigned to training data as a second label; a determination unit that determines whether the first label is different from the second label; a notification unit that notifies when it is determined by the determination unit that the first label is different from the second label; and an assignment unit that, in response to an instruction from the user, assigns a second label different from the first label to the training data.

[0012] In this training data creation assistance device, when it is determined by the determination unit that the second label designated as the label to be assigned to the training data is different from the first label estimated by the estimation unit using the learning model, the notification unit makes a notification. Therefore, even when the user makes a wrong designation due to reduced concentration or fatigue, etc., the user can easily notice the mistake. In addition, when it is determined that the user has not made a wrong designation, the assignment unit assigns a second label different from the first label to the training data. Thus, accurate training data can be created while reducing the burden on the user.

[0013] (6) The training data creation assistance device can also include: a prompting unit that indicates the basis for the determination when it is determined by the determination unit that the first label is different from the second label. In this case, the user can consider the basis for the prompt and make a re-judgment on the second label to be assigned. Thus, the burden on the user can be further reduced.

[0014] (7) The training data may include image data representing an image, and the prompting unit may also represent the basis for determination by representing, in a recognizable manner, a part of the image based on the training data. According to this structure, when the user makes a re - judgment corresponding to the assigned second label, the user can easily identify the part of the image to be considered.

[0015] (8) The prompting unit may prompt an image based on the training data, and the receiving unit may also receive a second label for the training data representing the image prompted by the prompting unit. In this case, the user can easily identify the training data for which the second label should be specified.

[0016] (9) The estimating unit may estimate the first label as a first score, and the receiving unit may also receive the second label as a second score. When the difference between the first score and the second score is equal to or greater than a predetermined threshold, the determination unit may determine that the first label and the second label are different. According to this structure, even when the learning model is constructed by learning a regression problem, it is possible to easily determine whether the first label and the second label are different.

[0017] (10) The training data creation assistance device may further include: an updating unit that updates the learning model based on the second label and the training data corresponding to the second label in response to an instruction from the user. Thereby, it is possible to more accurately estimate the first label to be assigned to the training data.

[0018] (11) A training data creation assistance system according to another aspect of the present invention includes: the training data creation assistance device of the first invention, and a display device that displays the first label prompted by the prompting unit of the training data creation assistance device.

[0019] In this training data creation assistance system, the display device displays the first label prompted by the prompting unit of the above - mentioned training data creation assistance device. Therefore, when the user specifies the second label to be assigned to the training data, the user can refer to the first label displayed on the display device. In addition, even when the user makes a wrong specification due to reduced concentration or fatigue, etc., the user can easily notice the mistake. Thereby, it is possible to create accurate training data while reducing the burden on the user.

[0020] (12) A training data creation assistance system according to another aspect of the present invention includes: the training data creation assistance device of the second invention, and a display device that displays the basis for determination prompted by the prompting unit of the training data creation assistance device.

[0021] In the training data creation assistance system, a part of the image of the training data prompted by the prompting unit of the training data creation assistance device based on the above is displayed on the display device in a recognizable manner as the basis for determination. Thereby, accurate training data can be created while reducing the burden on the user. In addition, when the user re-judges the given second label, the part of the image that should be considered can be easily recognized.

[0022] (13) A training data creation assistance method according to another aspect of the present invention is a training data creation assistance method for assisting in assigning labels to training data, which includes the following steps: using a pre-prepared learning model to estimate the label to be assigned to the training data as a first label; prompting the estimated first label; and receiving the label to be assigned to the training data as a second label.

[0023] According to this training data creation assistance method, the first label estimated using the learning model as the label to be assigned to the training data is prompted. Therefore, when the user specifies the second label to be assigned to the training data, the prompted first label can be referred to. In addition, even if the user makes a wrong specification due to reduced concentration or fatigue, etc., the error can be easily noticed. Thereby, accurate training data can be created while reducing the burden on the user.

[0024] (14) A training data creation assistance method according to another aspect of the present invention is a training data creation assistance method for assisting in assigning labels to training data, which includes the following steps: using a pre-prepared learning model to estimate the label to be assigned to the training data as a first label; receiving the label to be assigned to the training data as a second label; determining whether the first label and the second label are different; notifying in the case where it is determined that the first label and the second label are different; and in response to an instruction from the user, assigning the second label different from the first label to the training data.

[0025] According to this training data creation assistance method, in the case where it is determined that the second label designated as the label to be assigned to the training data is different from the first label estimated using the learning model, a notification is made. Therefore, even if the user makes a wrong specification due to reduced concentration or fatigue, etc., the error can be easily noticed. In addition, in the case where it is determined that the user has not made a wrong specification, the second label different from the first label is assigned to the training data. Thereby, accurate training data can be created while reducing the burden on the user. Description of the Drawings

[0026] Figure 1 It is a diagram showing the structure of the assistance system according to the first embodiment of the present invention.

[0027] Figure 2It represents Figure 1 a diagram showing the structure of the auxiliary device.

[0028] Figure 3 It represents Figure 2 an example of a label designation screen of the display device shown in

[0029] Figure 4 It represents Figure 2 another example of the label designation screen of the display device shown in

[0030] Figure 5 It represents Figure 2 a flowchart of the auxiliary process performed by the auxiliary device.

[0031] Figure 6 It represents a diagram showing the structure of the auxiliary device according to the second embodiment of the present invention.

[0032] Figure 7 It represents Figure 6 an example of a label designation screen of the display device shown in

[0033] Figure 8 It represents Figure 6 an example of a notification screen of the display device shown in

[0034] Figure 9 It represents Figure 6 a flowchart of the auxiliary process performed by the auxiliary device. Detailed implementation manners

[0035] Hereinafter, a training data creation auxiliary device, a training data creation auxiliary system, and a training data creation auxiliary method according to embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the training data creation auxiliary device is abbreviated as the auxiliary device, the training data creation auxiliary system is abbreviated as the auxiliary system, and the training data creation auxiliary method is abbreviated as the auxiliary method.

[0036] [1] First embodiment

[0037] (1) Structure of the auxiliary system

[0038] Figure 1 It represents a diagram showing the structure of the auxiliary system according to the first embodiment of the present invention. As shown in Figure 1As shown in the figure, the auxiliary system 100 includes a processing device 10 and an inspection device 20. The processing device 10 is composed of a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, a storage device 14, an operation unit 15, a display device 16, and an input / output I / F (Interface) 17. The CPU 11, the RAM 12, the ROM 13, the storage device 14, the operation unit 15, the display device 16, and the input / output I / F 17 are connected to a bus 18.

[0039] The RAM 12 is used as a working area for the CPU 11. System programs are stored in the ROM 13. The storage device 14 includes a storage medium such as a hard disk or a semiconductor memory to store a training data creation auxiliary program (hereinafter, simply referred to as the auxiliary program). The auxiliary program may also be stored in the ROM 13 or other external storage devices. In addition, the storage device 14 stores a pre-prepared learning model.

[0040] An auxiliary device 30 for executing the training data creation auxiliary process (hereinafter, simply referred to as the auxiliary process) is composed of the CPU 11, the RAM 12, and the ROM 13. In the auxiliary process, based on the learning model, assistance is provided for assigning (labeling) labels to the training data. The auxiliary device 30 and the auxiliary process will be described in detail later.

[0041] The operation unit 15 is an input device such as a keyboard, a mouse, or a touch panel. By operating the operation unit 15, the user can specify the labels that should be assigned to the training data, which will be described later, for the auxiliary device 30. The display device 16 is a display device such as a liquid crystal display device, and displays images based on the training data. The input / output I / F 17 is connected to the inspection device 20.

[0042] The inspection device 20 is, for example, a substrate inspection device, which generates a plurality of image data respectively representing the images of a plurality of substrates by sequentially photographing the plurality of substrates to be inspected. An inherent identification number or the like is assigned to each of the generated image data. In addition, the inspection device 20 provides each of the generated image data as training data to the auxiliary device 30. In addition, the substrate refers to a substrate for an FPD (Flat Panel Display) such as a semiconductor substrate, a liquid crystal display device, or an organic EL (Electro Luminescence: light-emitting electronic version) display device, a substrate for an optical disc, a substrate for a magnetic disk, a substrate for an optical disk, a substrate for a photomask, a ceramic substrate, or a substrate for a solar cell.

[0043] Figure 2 It is a diagram showing Figure 1 the structure of the auxiliary device 30. As Figure 2As shown, the auxiliary device 30 includes an acquisition unit 31, a estimation unit 32, a presentation unit 33, a reception unit 34, and an assignment unit 35 as functional units. Figure 1 The CPU 11 of Figure 1 realizes the functional units of the auxiliary device 30 by executing an auxiliary program stored in the ROM 13 or the storage device 14 or the like. Part or all of the functional units of the auxiliary device 30 may also be realized by hardware such as an electronic circuit.

[0044] The acquisition unit 31 sequentially acquires a plurality of training data respectively representing images of a plurality of substrates from the inspection device 20. The estimation unit 32 uses a learning model stored in the storage device 14 to estimate a plurality of labels that should be respectively assigned to the plurality of training data acquired by the acquisition unit 31. In this example, the labels include "OK" or "NG" respectively indicating whether the substrates in the images represented by the training data are normal or abnormal. In addition, the estimation unit 32 evaluates the reliability (probability) of each estimated label based on the learning model stored in the storage device 14.

[0045] The presentation unit 33 causes the plurality of labels estimated by the estimation unit 32 and the images based on the training data to be displayed on the display device 16 in a corresponding manner. In addition, the presentation unit 33 can cause the reliability of each label evaluated by the estimation unit 32 to be displayed on the display device 16 in a manner corresponding to the label. Moreover, the presentation unit 33 can, based on an instruction from the operation unit 15, arrange the order of the plurality of labels displayed on the display device 16 in ascending or descending order of reliability.

[0046] The reception unit 34 receives from the operation unit 15 the labels that should be assigned to the respective training data acquired by the acquisition unit 31. The user can visually confirm the plurality of labels and the reliability based on the learning model in the label designation screen displayed on the display device 16, and designate the labels that should be assigned to the respective training data to the reception unit 34 by operating the operation unit 15. The assignment unit 35 assigns the labels received by the reception unit 34 to the respective training data.

[0047] (2) Display on the display device

[0048] Figure 3 is a diagram showing an example of a label designation screen of the display device 16 displayed on Figure 2 As shown in Figure 3 the label designation screen 40 includes: a selected image display area 41, an image selection area 42, an image list display area 43, a basic information display area 44, and a label information display area 45. In this example, the selected image display area 41 includes: an overall image display section 41a, an enlarged image display section 41b, and an estimated label display section 41c.

[0049] In the overall image display section 41a, based on the selected training data, a display representingFigure 2 The overall image of the part photographed by the inspection device 20. By visually confirming the image displayed on the overall image display unit 41a, the user can easily identify the training data to be labeled. In the enlarged image display unit 41b, an enlarged image of an arbitrary part of the overall image is displayed. In this example, the part of the image surrounded by the dotted rectangle in the overall image display unit 41a is enlarged and displayed in the enlarged image display unit 41b.

[0050] In the estimated label display unit 41c, the label estimated by the Figure 2 estimating unit 32 for the selected training data is displayed. By visually confirming the selection image display area 41, the user can easily identify the correspondence between the training data representing the image and the label estimated for the training data. In the selection image display area 41, the reliability evaluated by the estimating unit 32 for the selected training data can also be displayed.

[0051] In the image selection area 42, operation buttons 42a and 42b for selecting the training data of the operation object are displayed. By using the Figure 2 operation unit 15 to operate the operation button 42a or the operation button 42b, the user can select the desired training data. The image corresponding to the selected training data is displayed in the selection image display area 41. In addition, in the image selection area 42, the identification number of the selected training data and the progress of the operation are also displayed. In the Figure 3 example, among 2380 pieces of training data, the training data with the identification number "1115" is selected, and the operation progress is 47%.

[0052] In the image list display area 43, a plurality of thumbnail images respectively based on the Figure 2 multiple training data acquired by the acquisition unit 31 are displayed in the order of the identification numbers. In addition, the thumbnail image corresponding to the selected training data is displayed in a recognizable manner such as by a cursor 43a or highlighting. In the image list display area 43, the reliability evaluated by the estimating unit 32 for each training data can also be displayed corresponding to the thumbnail image. By using the operation unit 15 to select an arbitrary thumbnail image, the user can also select the training data of the operation object.

[0053] In the basic information display area 44, basic information such as the creation date of the selected training data is displayed. In the label information display area 45, check boxes 45a and 45b corresponding to the labels "OK" and "NG" respectively are displayed. The user can operate either of the check boxes 45a and 45b using the operation unit 15 to indicate the label that should be assigned to the selected training data. In addition, in the label information display area 45, a drop-down menu or the like with the same function can also be displayed instead of the check boxes 45a and 45b.

[0054] In addition, in the label information display area 45, a table 45c showing the total number of each label estimated by the estimation unit 32 is displayed. In Figure 3 the example, the total number of labels estimated as "OK" is 1763, and the total number of labels estimated as "NG" is 617. When the user indicates the label that should be assigned to the training data, the user can refer to the total number of each label displayed in the table 45c.

[0055] Figure 4 is a diagram showing another example of the label designation screen 40 of the display device 16 shown in Figure 2 . In Figure 4 the example, a plurality of thumbnail images of a plurality of training data acquired by the acquisition unit 31 are displayed in the image list display area 43 in descending order of the reliability evaluated by the estimation unit 32. The plurality of thumbnail images can also be displayed in the image list display area 43 in ascending order of reliability. By performing a prescribed operation using the operation unit 15, the user can change the display mode of the image list display area 43 between Figure 3 the example and Figure 4 the example.

[0056] For training data with a high reliability of the estimated label, the user can easily specify an accurate label. Therefore, the user only needs to carefully judge the label that should be specified only for the training data with a low reliability of the estimated label. Therefore, in Figure 3 the example, the user can also visually confirm the reliability and can easily identify the training data for which the label should be specified after careful judgment. Thereby, the burden on the user can be reduced.

[0057] In addition, in Figure 4 the example, a plurality of thumbnail images corresponding to a plurality of training data are displayed in the image list display area 43 in an order arranged according to the reliability. Therefore, the user can easily identify a group of a plurality of training data for which the label should be specified after careful judgment. Therefore, the burden on the user can be further reduced. In Figure 4 the example, the identification number corresponding to each thumbnail image can also be displayed in the image list display area 43.

[0058] (3) Auxiliary processing

[0059] Figure 5 It represents a flowchart of the auxiliary processing performed by the auxiliary device 30 of Figure 2 . The auxiliary processing of Figure 5 is performed by the CPU 11 of Figure 1 executing an auxiliary program stored in the ROM 13 or the storage device 14 etc. on the RAM 12. Hereinafter, the auxiliary device 30 of Figure 2 , Figure 3 or Figure 4 the label designation screen 40 of Figure 5 , and the flowchart of

[0060] are used to explain the auxiliary processing. First, the acquisition unit 31 sequentially acquires a plurality of training data from the inspection device 20 (step S1). Next, the estimation unit 32 estimates the label that should be assigned to each training data acquired in step S1 based on the learning model (step S2). In addition, the estimation unit 32 evaluates the reliability of each label estimated in step S2 based on the learning model (step S3). As a result, the label designation screen 40 is displayed on the display device 16.

[0061] Next, the prompting unit 33 causes a plurality of thumbnail images respectively corresponding to the plurality of training data acquired in step S1 to be displayed in the image list display area 43 of the label designation screen 40 in the display device 16 (step S4). In the initial setting of this example, the plurality of thumbnail images are displayed in the order of the identification numbers. In the image list display area 43, it is also possible to display in such a manner that the identification number and the reliability of the label evaluated in step S3 correspond to the corresponding thumbnail image.

[0062] Then, the prompting unit 33 determines whether a change in the arrangement order of the thumbnail images has been instructed (step S5). If a change in the order has not been instructed, the prompting unit 33 proceeds to step S7. If a change in the order has been instructed, the prompting unit 33 changes the arrangement order of the thumbnail images (step S6) and proceeds to step S7. Each time step S6 is executed, the arrangement order of the thumbnail images switches between the order of the identification numbers and the order of the reliability evaluated in step S3.

[0063] In step S7, the prompting unit 33 determines whether a certain training data has been selected (step S7). When no training data is selected, the prompting unit 33 proceeds to step S11. When training data is selected, the prompting unit 33 causes the image based on the selected training data and the label estimated in step S2 for the training data to be displayed in the selection image display area 41 of the label designation screen 40 in the display device 16 (step S8). In the selection image display area 41, the reliability of the label evaluated in step S3 for the selected training data can also be displayed.

[0064] Next, the accepting unit 34 determines whether a label designation has been accepted for the training data selected in step S7 (step S9). When the label designation is not accepted, the accepting unit 34 proceeds to step S11. When the label designation is accepted, the assigning unit 35 assigns the accepted label to the training data selected in step S7 (step S10), and proceeds to step S11.

[0065] In step S11, the assigning unit 35 determines whether an end is instructed (step S11). The user can instruct an end or continue by performing a prescribed operation using the operation unit 15. When an end is not instructed, the assigning unit 35 returns to step S5. Therefore, when there is remaining training data for which a label has not been assigned, or when changing a label that has been assigned once, the user instructs to continue. When an end is instructed, the assigning unit 35 ends the assistance process.

[0066] (4) Effects

[0067] In the assistance device 30 of the present embodiment, the label that should be assigned to the training data, which is estimated by the estimation unit 32 using the learning model, is prompted by the prompting unit 33. Therefore, when the user designates the label that should be assigned to the training data to the accepting unit 34, the user can refer to the label prompted by the prompting unit 33. In addition, even when the user makes a wrong designation due to reduced concentration or fatigue or the like, the user can easily notice the mistake. As a result, accurate training data can be created while reducing the burden on the user.

[0068] [2] Second Embodiment

[0069] (1) Structure of the Assistance System

[0070] Differences between the assistance device 30 of the second embodiment and the assistance device 30 of the first embodiment will be described. Figure 6 This is a diagram showing the structure of the assistance device 30 of the second embodiment of the present invention. As Figure 6As shown, in addition to the acquisition unit 31, the estimation unit 32, the presentation unit 33, the reception unit 34, and the assignment unit 35, the auxiliary device 30 further includes a determination unit 36, a notification unit 37, and an update unit 38 as functional units. By Figure 1 the CPU 11 executes an auxiliary program stored in the ROM 13 or the storage device 14, etc., to implement the functional units of the auxiliary device 30. A part or all of the functional units of the auxiliary device 30 may also be implemented by hardware such as an electronic circuit.

[0071] The determination unit 36 determines whether the label estimated by the estimation unit 32 is different from the label received by the reception unit 34. The notification unit 37 gives a notification when it is determined by the determination unit 36 that the labels are different. In this example, a sentence to the effect that the label received by the reception unit 34 is different from the label estimated by the estimation unit 32 is displayed on the display device 16 for notification. By recognizing the notification, the user can get an opportunity to re-judge whether the specified label is accurate.

[0072] Although the display device 16 is used for notification, the implementation mode is not limited thereto. For example, when the auxiliary system 100 includes a voice output device, it may also be notified by voice indicating the content to the effect that the label received by the reception unit 34 is different from the label estimated by the estimation unit 32, or by outputting a warning sound such as a buzzer. Or, when the auxiliary system 100 includes a display lamp such as a light, it may also be notified by turning the display lamp on, off, or flashing.

[0073] The presentation unit 33 presents the basis for the determination when it is determined by the determination unit 36 that the labels are different. In this case, the user can consider the basis for the presentation and make a re-judgment on the label to be assigned. In this example, the basis for the determination by the determination unit 36 is presented by displaying a part of the image based on the training data in a recognizable manner on the display device 16. According to this structure, when the user makes a re-judgment on the label to be assigned, the part of the image that should be considered can be easily recognized.

[0074] When the notification is given by the notification unit 37, the user re-judges whether the indicated label is accurate. As a result, when the user determines that the indicated label is incorrect, the user can re-indicate to the reception unit 34 the label that should be assigned to the training data. On the other hand, when the user determines that the specified label is accurate, the user can, by operating the operation unit 15, indicate to the assignment unit 35 the case of assigning the specified label.

[0075] Even when the label received by the receiving unit 34 is different from the label estimated by the estimating unit 32, the assigning unit 35 assigns the label received by the receiving unit 34 to the training data in response to an instruction from the user. The updating unit 38 updates the learning model stored in the storage device 14 based on the label received by the receiving unit 34 and the training data corresponding to the label in response to an instruction from the user. As a result, in subsequent assistance processing, the estimating unit 32 can estimate a more accurate label for the training data.

[0076] (2) Display of the display device

[0077] Figure 7 represents an example of Figure 6 the label specifying screen 40 of the display device 16 shown in Figure 7 The label specifying screen 40 of Figure 3 differs from the label specifying screen 40 of Figure 6 in the following respects. In the present embodiment, the label estimated by the estimating unit 32 and the reliability of the evaluation are not displayed on the label specifying screen 40. Therefore, the estimated label display unit 41c is not displayed in the selection image display area 41. In addition, the table 45c is not displayed in the label information display area 45.

[0078] In the label specifying screen 40, the user sequentially selects each training data and specifies the label to be assigned to the selected training data. Among them, when it is determined that the specified label is different from the label estimated by the estimating unit 32, a notification screen prompting the gist thereof is displayed on the display device 16.

[0079] Figure 8 represents an example of Figure 6 the notification screen of the display device 16 shown in Figure 8 As shown in

[0080] An image based on training data specifying a label different from the inferred label is displayed in the image display area 52. In the image, the part that serves as the basis for determining that the specified label is different from the inferred label is shown in a recognizable manner. In this example, by using Grad-CAM (Gradient-weighted Class Activation Mapping) or the like to label the corresponding part of the image with the marker 52a, the part that serves as the basis can be shown in a recognizable manner. In addition, the identification number of the training data, the inferred label, the reliability, and the specified label are also displayed in the image display area 52.

[0081] When the user determines through re-judgment that their previous judgment was incorrect, they operate the re-specification button 53 using the Figure 6 operation unit 15 to indicate re-specification of the label. In this case, the label specification screen 40 of Figure 7 is displayed again on the display device 16. Thereby, for the training data to be judged, the label to be assigned can be re-specified.

[0082] On the other hand, when the user determines through re-judgment that their judgment was correct, they operate the assignment button 54 using the operation unit 15 to indicate assignment of the label. Thereby, even if the specified label is different from the inferred label, the specified label is assigned to the training data. In addition, based on the specified label and the training data, the learning model stored in the Figure 6 storage device 14 is updated.

[0083] (3) Auxiliary processing

[0084] Figure 9 It is a flowchart showing the auxiliary processing performed by the Figure 6 auxiliary device 30. Figure 9 The auxiliary processing of Figure 1 is performed by the CPU 11 of Figure 6 executing an auxiliary program stored in the ROM 13 or the storage device 14 or the like on the RAM 12. Hereinafter, the auxiliary processing will be described using the Figure 7 auxiliary device 30 of Figure 8 the label specification screen 40 of Figure 9 the notification screen 50 and the

[0085] First, the acquisition unit 31 sequentially acquires a plurality of training data from the inspection device 20 (step S31). Thereby, the label designation screen 40 is displayed on the display device 16. Next, the estimation unit 32 determines whether a certain training data has been selected (step S32). If no training data is selected, the estimation unit 32 proceeds to step S43. If training data is selected, the estimation unit 32 estimates the label that should be assigned to the selected training data based on the learning model (step S33). In addition, the estimation unit 32 evaluates the reliability of the label estimated in step S33 based on the learning model (step S34).

[0086] Next, the acceptance unit 34 determines whether a label has been specified for the training data selected in step S32 (step S35). If no label has been specified, the acceptance unit 34 proceeds to step S43. If a label has been specified, the determination unit 36 determines whether the label accepted in step S35 is different from the label estimated in step S33 (step S36). In addition, steps S33 and S34 may also be executed between step S35 and step S36.

[0087] If the labels are the same, the assignment unit 35 assigns the accepted label to the training data selected in step S32 (step S37) and proceeds to step S43. If the labels are different, the notification unit 37 notifies by displaying a notification screen 50 on the display device 16 (step S38). The label estimated in step S33 and the reliability evaluated in step S34 may also be displayed in the notification screen 50.

[0088] Then, the acceptance unit 34 determines whether a re-specification of the label has been instructed (step S39). If a re-specification of the label has been instructed, the acceptance unit 34 returns to step S32. In this case, the label designation screen 40 is displayed again on the display device 16. Thereby, the user can select any training data and re-specify the label for the selected training data.

[0089] If a re-specification of the label has not been instructed, the assignment unit 35 determines whether a label assignment has been instructed (step S40). If a label assignment has not been instructed, the assignment unit 35 returns to step S39. The processing of steps S39 and S40 is repeated until a re-specification of the label or a label assignment is instructed.

[0090] If a label assignment has been instructed, even if the label accepted in step S35 is different from the label estimated in step S33, the assignment unit 35 assigns the label accepted in step S35 to the training data (step S41). In addition, the update unit 38 updates the learning model based on the training data to which the label has been assigned in step S41 and the label (step S42).

[0091] Finally, the assignment unit 35 determines whether the end has been indicated (step S43). The user indicates the end or continuation by performing a prescribed operation using the operation unit 15. In the case where the end is not indicated, the assignment unit 35 returns to step S32. Therefore, in the case where there is still training data for which a label has not been assigned, or in the case where a label that has been assigned once is changed, the user indicates continuation. In the case where the end is indicated, the assignment unit 35 ends the assistance process.

[0092] (4) Effects

[0093] In the assistance device 30 of the present embodiment, when the determination unit 36 determines that the label specified by the user is different from the label estimated by the estimation unit 32 using the learning model, the notification unit 37 performs notification. Therefore, even when the user makes a wrong specification due to reduced concentration, fatigue, etc., the user can easily notice the mistake. Further, in the case where it is determined that the user's specification is not wrong, the assignment unit 35 assigns a label different from the estimated label to the training data. Thereby, it is possible to create accurate training data while reducing the burden on the user.

[0094] [3] Other Embodiments

[0095] (1) In the above-described embodiment, the assistance system 100 includes the inspection device 20 for inspecting the substrate to assist in creating training data representing an image of the substrate, but the embodiment is not limited thereto. The assistance system 100 may also be used to assist in creating other training data. Further, the training data includes image data representing an image, but the embodiment is not limited thereto. The training data may be, for example, voice data for playing voice.

[0096] (2) In the above-described embodiment, the training data is classified into two types, and the label "OK" or "NG" is assigned to each of the classified training data, but the embodiment is not limited thereto. The training data may be classified into three or more types, and a label may be assigned to each of the classified training data.

[0097] Further, in the second embodiment, the learning model is constructed by learning a classification problem, but the embodiment is not limited thereto. The learning model may also be constructed by learning a regression problem. In this case, the estimation unit 32 estimates the first label as the first score, and the acceptance unit 34 accepts the second label as the second score. Further, in the case where the difference between the first score and the second score is equal to or greater than a predetermined threshold, the determination unit 36 determines that the first label is different from the second label.

[0098] (3)In the above-described embodiment, the estimation unit 32 evaluates the reliability of the estimated tag, but the embodiment is not limited thereto. The estimation unit 32 may not evaluate the reliability of the estimated tag. In this case, in the first embodiment, the plurality of thumbnail images are displayed in the image list display area 43 in the order of, for example, identification numbers instead of the order of reliability.

[0099] (4)In the second embodiment, when it is determined that the tags are different, the prompting unit 33 prompts the basis for the determination by prompting a part of the image, but the embodiment is not limited thereto. The prompting unit 33 may also prompt the basis for the determination by means of text or voice, etc. Alternatively, the prompting unit 33 may not prompt the basis for the determination.

[0100] (5)In the second embodiment, the learning model is updated based on the tag specified by the user and the training data corresponding to the specified tag, but the embodiment is not limited thereto. The learning model may not be updated. In this case, the auxiliary device 30 does not include the update unit 38.

[0101] [4]Corresponding relationship between each component of the claims and each part of the embodiment

[0102] Hereinafter, examples of the corresponding relationship between each component of the claims and each element of the embodiment will be described, but the present invention is not limited to the following examples. As each component of the claims, various other elements having the structure or function described in the claims can also be used.

[0103] In the above embodiment, the auxiliary device 30 is an example of a training data creation auxiliary device, the estimation unit 32 is an example of an estimation unit, the prompting unit 33 is an example of a prompting unit, the reception unit 34 is an example of a reception unit, the determination unit 36 is an example of a determination unit. The notification unit 37 is an example of a notification unit, the update unit 38 is an example of an update unit, the display device 16 is an example of a display device, and the auxiliary system 100 is an example of a training data creation auxiliary system.

Claims

1. An auxiliary device for creating training data, which assists in assigning labels to training data, wherein, It has: a presumption unit that presumes, using a pre-prepared learning model, a label that should be assigned to the training data as a first label; a reception unit that receives a label that should be assigned to the training data as a second label, a determination unit that determines whether the first label and the second label are different; a notification unit that notifies when it is determined by the determination unit that the first label and the second label are different; an assignment unit that, in response to an instruction from a user, assigns the second label that is different from the first label to the training data; and a prompting unit that indicates the basis for the determination when it is determined by the determination unit that the first label and the second label are different, wherein the training data includes image data representing an image, and the prompting unit indicates the basis for the determination by recognizably indicating a part of the image based on the training data.

2. The training data creation assistance device according to claim 1, wherein the prompting unit prompts an image based on the training data, and the reception unit receives the second label for the training data representing the image prompted by the prompting unit.

3. The training data creation assistance device according to claim 1 or 2, wherein the presumption unit presumes the first label as a first score, the reception unit receives the second label as a second score, and the determination unit determines that the first label and the second label are different when the difference between the first score and the second score is equal to or greater than a predetermined threshold.

4. The training data creation assistance device according to claim 1 or 2, wherein, It further has: an update unit that, in response to an instruction from a user, updates the learning model based on the second label and the training data corresponding to the second label.

5. A training data creation assistance system, wherein, It has: the training data creation assistance device according to any one of claims 1 to 4; and a display device that displays the basis for the determination prompted by the prompting unit of the training data creation assistance device.

6. A method for assisting in creating training data, which assists in assigning labels to training data, wherein, It includes: a step of presuming, using a pre-prepared learning model, a label that should be assigned to the training data as a first label; a step of receiving a label that should be assigned to the training data as a second label; a step of determining whether the first label and the second label are different; a step of notifying when it is determined that the first label and the second label are different; a step of, in response to an instruction from a user, assigning the second label that is different from the first label to the training data; a step of, when it is determined in the determination step that the first label and the second label are different, indicating the basis for the determination by recognizably indicating a part of the image based on the training data, wherein the training data includes image data representing an image.

Images