Quality maintenance management system, quality maintenance management method and program

Abstract

Provided is an accuracy maintenance management system, an accuracy maintenance management method, and a program that can improve the accuracy of detecting deterioration in accuracy of a trained model that outputs inference results based on explanatory variables. [Solution] The accuracy maintenance management system includes a display unit that displays on a display device a reception screen that accepts input operations for an inference result output from a trained model to which explanatory variables have been input or for an evaluation result for the explanatory variables, and a degradation determination unit that determines whether the accuracy of the trained model has deteriorated based on the evaluation result accepted on the reception screen. The reception screen may include an input field for the evaluation result and a display field for the explanatory variables. The reception screen may further include a display field for the inference result. The index calculation unit may calculate an evaluation index that represents the accuracy of the trained model based on the evaluation result accepted on the reception screen.

Description

[Technical field]

[0001] The present invention relates to a quality maintenance management system, a quality maintenance management method, and a program. [Background technology]

[0002] The accuracy of a trained model of machine learning may deteriorate due to various factors. For this reason, it is necessary to maintain and manage the accuracy of the trained model. Patent Document 1 discloses an image recognition device that judges the performance deterioration of a trained model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2023-174327 A Summary of the Invention [Problem to be solved by the invention]

[0004] In a trained model that uses structured numerical data (e.g., numerical data representing steel type) as explanatory variables, it is relatively easy to detect deterioration in the accuracy of the trained model based on changes in the input numerical data.

[0005] In contrast, images are unstructured data. Unstructured data is stored in its original format without being formatted into a predefined structure, and is not processed until it is used. For example, an image contains information such as RGB, but it is not in a format that allows interpretation of what it means (for example, whether it is a scratch), so in a trained model that uses images as explanatory variables, it is not easy to detect deterioration in the accuracy of the trained model based on changes in the input image.

[0006] Thus, there is a problem in that it is not possible to improve the accuracy of detecting deterioration in the accuracy of a trained model that outputs inference results based on explanatory variables (especially images).

[0007] The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide an accuracy maintenance management system, an accuracy maintenance management method, and a program that are capable of improving the accuracy of detecting deterioration in accuracy of a trained model that outputs inference results based on explanatory variables. [Means for solving the problem]

[0008] (1) One aspect of the present invention is an accuracy maintenance management system that includes a display unit that displays on a display device a reception screen that accepts input operations for an inference result output from a trained model to which explanatory variables have been input or for an evaluation result for the explanatory variables, and a deterioration determination unit that determines whether or not the accuracy of the trained model has deteriorated based on the evaluation result received on the reception screen.

[0009] (2) In one aspect of the present invention, in the quality maintenance management system described in (1) above, a reception screen includes an input field for the evaluation result and a display field for the explanatory variable.

[0010] (3) In one aspect of the present invention, in the accuracy maintenance management system described in (2) above, the reception screen further includes a display field for the inference result.

[0011] (4) In one aspect of the present invention, in the accuracy maintenance management system described in any one of (1) to (3) above, the system further includes an index calculation unit that calculates an evaluation index representing the accuracy of the trained model based on the evaluation result accepted on the reception screen, and the degradation determination unit determines whether the accuracy of the trained model has deteriorated based on the evaluation index.

[0012] (5) In one aspect of the present invention, in the accuracy maintenance management system described in (4) above, the degradation determination unit determines whether the accuracy of the trained model has deteriorated based on a change in the evaluation index.

[0013] (6) In one aspect of the present invention, in the accuracy maintenance management system described in any one of (1) to (3) above, the trained model outputs the inference result using an image as the explanatory variable.

[0014] (7) In one aspect of the present invention, in an accuracy maintenance management system described in any one of (1) to (3) above, the system further includes a degradation notification unit that notifies that the accuracy of the trained model has deteriorated when it is determined that the accuracy of the trained model has deteriorated.

[0015] (8) In one aspect of the present invention, in the accuracy maintenance management system described in (4) above, the evaluation index is the accuracy rate of the inference result, and the degradation determination unit determines that the accuracy of the trained model has deteriorated if the accuracy rate has decreased.

[0016] (9) In one aspect of the present invention, in the accuracy maintenance management system described in (8) above, the degradation determination unit determines that the accuracy of the trained model has deteriorated if the accuracy rate decreases even though the confidence of the inference result in the trained model is maintained above a confidence threshold.

[0017] (10) In one aspect of the present invention, in an accuracy maintenance management system described in any one of (1) to (3) above, the reception screen receives an annotation operation of a correct label to be used for re-learning the trained model.

[0018] (11) In one aspect of the present invention, in the accuracy maintenance management system described in any one of (1) to (3) above, the system further includes a learning unit that, when notified that the accuracy of the trained model has deteriorated, performs re-learning of the trained model using predetermined re-learning images as explanatory variables and a correct label accepted by an annotation operation as a new objective variable.

[0019] (12) One aspect of the present invention is an accuracy maintenance management method executed by an accuracy maintenance management system, the accuracy maintenance management method including the steps of: displaying, on a display device, a reception screen that accepts input operations for an inference result output from a trained model to which explanatory variables have been input, or for an evaluation result for the explanatory variables; and determining whether or not the accuracy of the trained model has deteriorated based on the evaluation result accepted on the reception screen.

[0020] (13) One aspect of the present invention is a program for causing a computer to execute the steps of: displaying on a display device a reception screen that accepts input operations for an inference result output from a trained model to which explanatory variables have been input or for an evaluation result for the explanatory variables; and determining whether or not accuracy of the trained model has deteriorated based on the evaluation result accepted on the reception screen. Effect of the Invention

[0021] According to the present invention, it is possible to improve the accuracy of detecting deterioration in accuracy of a trained model that outputs an inference result based on explanatory variables. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a quality maintenance management system in an embodiment. [Diagram 2] FIG. 11 is a diagram showing a display example of a reception screen in the embodiment. [Diagram 3] FIG. 13 is a diagram showing a display example of a details display screen in the embodiment. [Figure 4] 4 is a flowchart showing an example of the operation of the inference device in the embodiment. [Diagram 5] 10 is a flowchart illustrating an example of the operation of the evaluation terminal in the embodiment. [Figure 6] 4 is a flowchart showing an example of the operation of the quality maintenance management device in the embodiment. [Figure 7] 4 is a flowchart showing an example of the operation of the learning device in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram showing an example of the configuration of an accuracy maintenance management system 1 in an embodiment. The accuracy maintenance management system 1 is a system that maintains and manages the accuracy of one or more trained models. The trained model is a model that has been trained using a machine learning technique. The trained model includes, for example, a multi-layered neural network. The multi-layered neural network is, for example, a convolutional neural network. The multi-layered neural network may be a recurrent neural network. The machine learning technique is not limited to a specific technique, but is, for example, supervised learning.

[0024] The explanatory variables of the trained model may be either structured data or unstructured data, but in the following, unstructured data is used as an example. The unstructured data may be, for example, image data or sound data. The image is, for example, a captured image generated by a camera installed in a steelworks or the like. The captured image may be a video or a still image. The captured image is not limited to an image of a specific object, but is, for example, a captured image including an image of a manufactured product. The sound is, for example, a sound generated from a machine such as a belt conveyor. The sound generated from the machine is collected using a microphone installed near the machine. When an abnormality occurs in the machine, a sound different from a sound (normal sound) collected when no abnormality occurs is collected by the microphone. In the following, the explanatory variables of the trained model are, for example, images.

[0025] The quality maintenance management system 1 includes an inference device 2, a database device 3, an evaluation terminal 4 (terminal device), a quality maintenance management device 5, a learning device 6, and a display device 7. In the embodiment shown in FIG. 1, the above-mentioned devices (2 to 6) included in the quality maintenance management system 1 are separate from each other, but at least two of these devices may be integrated. That is, the quality maintenance management system 1 may be composed of multiple devices, or may be composed of one device. When composed of multiple devices, the multiple devices are connected to be able to communicate with each other to a necessary extent, but may be directly connected, or may be connected by a communication network including at least one of a wireless part and a wired part. In addition, the information stored (managed) in the database device 3 does not need to be concentrated in one device, and may be distributed and stored in multiple devices.

[0026] Each of the devices (2-6) is configured using one or more hardware processors such as a CPU (Central Processing Unit) and one or more memories (main storage devices). The memories are configured using storage devices such as a RAM (Random Access Memory) and a ROM (Read Only Memory). Each of the devices (2-6) functions by the one or more hardware processors executing one or more programs stored in the memory. Cloud computing technology may be used for each of the devices (2-6).

[0027] All or part of the functions of each device (2 to 6) may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array). The above programs may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, and semiconductor storage devices (e.g., solid state drives (SSDs)), hard disks built into computer systems, and storage devices such as semiconductor storage devices. The above programs may be transmitted via electric communication lines.

[0028] First, an overview of the quality maintenance control system 1 will be described. The inference device 2 inputs an image as an explanatory variable into the trained model. The inference device 2 acquires an inference result from the trained model into which the image is input. When there are a plurality of trained models to which accuracy is to be maintained and managed, the type of trained model may differ for each trained model. That is, the inference result by the trained model is not limited to a specific inference result. For example, the trained model may be a trained model that recognizes a target object in an input image, or a trained model that detects a target object in an input image. Detecting an object may, for example, detect the presence or absence of an object in an image, or may determine the area of ​​the object in an image. Also, recognizing an object means, for example, recognizing what the object is in an image.

[0029] The database device 3 stores various types of data, such as images (explanatory variables), inference results (objective variables; the same applies below), user evaluation results (registered results) for the inference results, user evaluation results (registered results) for the explanatory variables, a trained model, and correct answer labels of training data.

[0030] The evaluation terminal 4 displays a reception screen on the display device 7 for receiving the evaluation result by the user for the inference result or the explanatory variable. The explanatory variable input to the trained model is, for example, an image. The inference result output from the trained model is, for example, an object recognition result or a detection result. The evaluation terminal 4 receives the input operation of the evaluation result by the user via the reception screen, and records the input evaluation result in the database device 3. In addition to the input field for the evaluation result, the reception screen may display information useful for inputting the evaluation result (for example, the inference result and the explanatory variable). In this way, the user can efficiently evaluate the inference result by inputting the evaluation result by the user into the evaluation terminal 4 while checking the reception screen on which the inference result or the explanatory variable is displayed.

[0031] In addition, the validity of the inference result by the trained model can be judged based on the evaluation result by the user. Although it depends on the contents of the evaluation result, if the evaluation result is the evaluation result by the user on the inference result (whether the inference result is correct or not), the evaluation result directly indicates the correctness of the inference result, so that the appropriateness of the inference result can be judged based on the evaluation result. If the evaluation result is the evaluation result by the user on the inference result, and the evaluation result indicates the correct content of the inference result (for example, the presence or absence of an object), the evaluation terminal 4 can judge the correctness of the inference result by comparing the inference result with the evaluation result. On the other hand, if the evaluation result is the evaluation result by the user on the explanatory variable (for example, the judgment result by the user who viewed the image displayed on the reception screen that an object should be detected in the image), the evaluation terminal 4 can judge the correctness of the inference result by comparing the inference result with the evaluation result.

[0032] As such, the validity of each estimation result can be judged based on the evaluation results, and by using these judgment results, it is possible to improve the accuracy of detecting deterioration in the accuracy of a trained model.

[0033] Specifically, the accuracy maintenance management device 5 calculates an evaluation index representing the accuracy of the trained model based on the evaluation result (user evaluation result) received by the reception screen. The evaluation index will be described in detail later. The accuracy maintenance management device 5 determines whether the accuracy of the trained model has deteriorated based on the calculated evaluation index. By determining whether the accuracy of the trained model has deteriorated based on the evaluation index, it is possible to improve the recovery efficiency of the deteriorated accuracy. For example, if the recognition accuracy of the number "1" in a captured image is low, the user can analyze the cause of the deterioration of the recognition accuracy, especially for the number "1". Based on the analyzed cause of deterioration, the user can consider an action for recovering the accuracy, so it is possible to improve the recovery efficiency of the deteriorated accuracy.

[0034] When it is determined that the accuracy of the trained model has deteriorated, the accuracy maintenance management device 5 notifies the user that the accuracy of the trained model has deteriorated. The accuracy maintenance management device 5 may promptly notify the user, for example, by using an email and an alarm (such as an alarm display or an alarm sound). However, the accuracy maintenance management device 5 does not necessarily have to issue an alarm. For example, the user may check the result of the deterioration determination when the user periodically performs a checking operation via the evaluation terminal 4 or the like.

[0035] The learning device 6 receives, via the reception screen, an annotation operation (e.g., an operation of inputting an object name) by a user who has confirmed the inference result (e.g., an object recognition result) and the image on the reception screen. The learning device 6 may also receive, via the reception screen, an annotation operation (e.g., an operation of inputting a bounding box) by a user who has confirmed the inference result (e.g., an object detection result) and the image on another screen. Through the annotation operation, the learning device 6 acquires teacher data used for re-learning the trained model. The learning device 6 records the acquired teacher data in the database device 3.

[0036] When the accuracy maintenance management device 5 notifies the user that the accuracy of the trained model has deteriorated, the learning device 6 performs re-learning of the trained model using predetermined re-learning images (learning data) as explanatory variables and a correct label accepted by an annotation operation as a new objective variable. The learning device 6 records the trained model that has been re-learned in the database device 3. This allows the trained model to be efficiently re-learned, making it possible to improve the efficiency of recovery of deteriorated accuracy.

[0037] Next, each component of the quality maintenance control system 1 will be described in detail. <Inference device 2> The inference device 2 includes an acquisition unit 21, an inference unit 22, and an output unit 23. The acquisition unit 21 acquires an image (e.g., a captured image including an image of a product) as an explanatory variable from an input data storage device 31. The acquisition unit 21 may perform preprocessing on the acquired image. The preprocessing is, for example, a data corruption check, trimming, and brightness adjustment. The inference unit 22 acquires a trained model from a model storage device 32. The inference unit 22 inputs the acquired image to the trained model. The inference unit 22 acquires an inference result output from the trained model. For example, when a captured image including an image of a product is input to the trained model, the inference result by the trained model may be a variable representing a determination result of whether or not the product has a scratch. The output unit 23 records the inference result by the trained model in an inference result storage device 34.

[0038] <Database device 3> The database device 3 includes an input data storage device 31, a model storage device 32, a correct label storage device 33, an inference result storage device 34, and an evaluation result storage device 35. The input data storage device 31 stores an image as an explanatory variable (input data). The model storage device 32 stores a trained model. The correct label storage device 33 stores a correct label associated with an image by an annotation operation. The inference result storage device 34 stores an inference result of the trained model. The evaluation result storage device 35 stores a user's evaluation result of the inference result of the trained model and a user's evaluation result of the explanatory variables. The database device 3 stores the inference result of the trained model for the input image, the inference result or the user's evaluation result for the same image, and the correct label for the evaluation result, etc., in association with each other so that the correspondence between them can be understood.

[0039] <Evaluation terminal 4, learning device 6> The evaluation terminal 4 includes a display unit 41 and an evaluation unit 42. The display unit 41 displays (outputs) a reception screen in a display area 71 of the display device 7. The reception screen is a screen that receives input operations of the evaluation result by the user for the inference result by the trained model and the evaluation result by the user for the explanatory variables input to the trained model. The evaluation unit 42 receives the input operation of the evaluation result by the user via the reception screen. The reception screen may be a screen that receives the input operation of the evaluation result by the user for the explanatory variables input to the trained model. The evaluation unit 42 or the label input unit 61 (described later) may receive an annotation operation of a correct label for an image via the reception screen displayed in the display area 71. That is, the reception screen may include an area for displaying the image input to the trained model and an area for inputting the evaluation result by the user. Furthermore, the reception screen may include an area for showing the inference result. This allows the user to easily input the evaluation result while checking the image displayed on the reception screen. In addition, the user can easily input the evaluation result while checking the image and the inference result displayed on the reception screen. In this way, if not only the image but also the inference result is displayed on the reception screen, the user can easily check the inference result.

[0040] 2 is a diagram showing a display example of a reception screen 11 in an embodiment. The reception screen 11 includes a determination result search field 103. The reception screen 11 may further include a date and time search field 101, an actual product ID search field 102, a registration result search field 104, a determination result display field 105, an image display field 106, an annotation progress display field 107, a registration result display field 108, a registration selection field 109, and a search operation key 201.

[0041] The reception screen 11 displays, for example, in a table format, the status of annotation operations on a captured image group (learning data) including an image of an inference target (e.g., a product) for each identifier that can uniquely identify the captured inference target. The identifier of the inference target is, for example, a product ID (e.g., a product code).

[0042] The reception screen 11 may further include a download operation key 202 and an upload operation key 203. The evaluation unit 42 or the label input unit 61 may accept an operation (upload operation) for uploading new captured images (new learning data) as a re-learning image group to the input data storage device 31. The evaluation unit 42 or the label input unit 61 may accept, via the reception screen 11, an annotation operation for the uploaded re-learning image group.

[0043] The "actual product ID" is associated with a "registration time" (the time when the image is registered in the database device 3 or the time when the image is captured), a "pattern" (a pattern of packaging, etc.), a "location" (the location of the product), a "judgment result" (the inference result "OK" or "NG" by the trained model), a "registration result", a group of captured images of the product (training data), and the status of the annotation operation. The "registration result" is the registration result of the evaluation result by the user described above. In the "registration result" column, the inference result or the evaluation result by the user on the explanatory variable is displayed as, for example, "OK" or "NG". If the annotation operation is not required, the "registration result" column may display "not to be performed". In the "registration result" column, the presence or absence of an object such as a product may be displayed, or the object name of the product may be displayed, based on the evaluation result by the user. These notations may be selected by the user in a combo box, a list box, or an input box.

[0044] The date and time search field 101 is an input field for the date and time (the time when the captured image was registered in the input data storage device 31). This date and time is one of search keys for extracting a captured image from among the multiple captured images stored in the input data storage device 31.

[0045] The actual product ID search field 102 is an input field for searching for an image of a product by using the actual product ID as one of search keys. That is, the actual product ID search field 102 is an input field for extracting a captured image of a product to which the actual product ID is assigned from among the multiple captured images stored in the input data storage device 31 by using the actual product ID as one of search keys.

[0046] The judgment result search field 103 is an input field for searching for captured images using the "judgment result", which is an inference result by a trained model, as one of search keys. The registration result search field 104 is an input field for searching for captured images using the "registration result", which is an evaluation result by a user for the inference result or the explanatory variable, as one of search keys.

[0047] The judgment result display field 105 is a display field for the inference result based on the trained model (the judgment result based on the trained model is “OK” or “NG”). The image display field 106 is a field for displaying one or more captured images (trained data) as explanatory variables (display field for explanatory variables).

[0048] The annotation progress display field 107 is a field that displays whether annotation has been completed for the captured image group (learning data) displayed in the image display field 106. The user can check in the annotation progress display field 107 that there is a captured image group for which annotation operation has not been completed among the captured image group (learning data) associated with the actual product ID. This allows the user to easily check the annotation progress status.

[0049] The registration result display field 108 is a field that displays the registration status of the inference result or the user's evaluation result for the explanatory variable (for example, whether or not the inference result based on the trained model is correct, "OK"). The registration result display field 108 may be an input field for the evaluation result. The registration selection field 109 is a field that displays each check box for selecting a group of captured images associated with the actual product ID.

[0050] When the search operation key 201 is pressed by the user, the evaluation unit 42 executes a search process for captured image groups (learning data) associated with the same registration time based on search conditions defined using each input field displayed on the reception screen 11. The display unit 41 displays a list of captured image groups stored in the input data storage device 31 in the display area 71 based on the search results, so that each captured image group (registration time) associated with the actual product ID can be selected.

[0051] The evaluation unit 42 accepts an input operation of a user's evaluation result for the inference result by the trained model using the reception screen 11. The evaluation result by the user is, for example, whether or not the inference result by the trained model is correct, that is, "OK". The evaluation unit 42 may accept an input operation of a user's evaluation result for the explanatory variables input to the trained model using the reception screen 11. The evaluation result by the user is, for example, an evaluation result that an object should be detected in the image input to the trained model. The evaluation unit 42 records the evaluation result by the user in the evaluation result storage device 35.

[0052] Furthermore, the evaluation unit 42 accepts an annotation operation of a correct label for the captured image (re-learning image) downloaded from the input data storage device 31. The evaluation unit 42 records the accepted correct label in the correct label storage device 33 in association with the captured image as an explanatory variable.

[0053] When the user presses the download operation key 202 "download selected data (image / annotation)", the captured image as an explanatory variable selected from the list using the registration selection field 109 is downloaded from the input data storage device 31 to the display unit 41. The display unit 41 displays the downloaded captured image in the display area 71 (reception screen 11 or another screen). The user may perform an annotation operation to associate a correct label with the downloaded captured image.

[0054] When the upload operation key 203 “annotation upload” is pressed by the user, the evaluation unit 42 or the label input unit 61 records (uploads) a combination of the downloaded captured image (relearning image) and the correct label acquired by the annotation operation by the user to the database device 3. In this way, a plurality of combinations of the captured image (relearning image) and the correct label are collected in the database device 3.

[0055] In addition, when a plurality of similar captured images are present in the input data storage device 31 among the plurality of captured images collected, at least one captured image among the plurality of similar captured images may be excluded from candidates for re-learning images, for example, by the evaluation unit 42 or the learning unit 62. This makes it possible to re-learn the learned model using image data with less bias.

[0056] 3 is a diagram showing a display example of a details display screen in an embodiment. The details display screen is a screen that displays details of an inference result by a trained model for a captured image selected by a user. The details display screen is displayed in the display area 71 when the user selects (by performing a selection operation such as clicking or tapping) a captured image displayed in the image display field 106 of the reception screen 11, for example. This allows the user to easily check the displayed details.

[0057] The efficiency of the work of updating teacher data is improved by the user correcting the "registration result" displayed in the registration result display field 108 of the captured image selected by the user using the registration result selection field 110 on the detailed display screen. Here, when the user attempts to correct the "registration result" displayed in the registration result display field 108, the display unit 41 pops up a confirmation image 111 in the display area 71. This enables the evaluation unit 42 to reconfirm with the user whether or not to actually correct the "registration result" displayed in the registration result display field 108.

[0058] Returning to FIG. 1, the description of each component of the quality maintenance control system 1 will continue. <Accuracy maintenance control device 5> The accuracy maintenance management device 5 includes an index calculation unit 51, a deterioration determination unit 52, and a deterioration notification unit 53. The index calculation unit 51 calculates an evaluation index representing the accuracy of the trained model based on an evaluation result by a user who viewed the display on the reception screen 11. The evaluation index is, for example, a correct answer rate. The correct answer rate is the probability that the evaluation result by the user on the inference result is determined to be correct as "OK" when the inference result and the explanatory variable are displayed on the reception screen 11. In other words, the correct answer rate is the number of inference results determined to be correct as "OK" with respect to the total number of inference results. The correct answer rate may be the probability that the evaluation result by the user on the explanatory variable (the evaluation result that the inference result should be) matches the inference result output from the trained model with the explanatory variable (the image input to the trained model) displayed on the reception screen 11. In other words, the correct answer rate may be the number of inference results that match the evaluation result by the user on the explanatory variable with respect to the total number of inference results. The evaluation result of how the inference result should be is, for example, an evaluation result that an object should be detected in the image input to the trained model. The accuracy rate may be the result obtained by subtracting the incorrect answer rate from 1. The incorrect answer rate is the number of inference results that are incorrect, or "NG," relative to the total number of inference results.

[0059] Furthermore, the index calculation unit 51 may calculate an evaluation index representing the accuracy of the trained model based on the evaluation result of the trained model. The evaluation index is, for example, a confidence level. The confidence level is determined based on the evaluation result of the trained model for the eye inference result. Here, the evaluation result of the trained model is determined based on, for example, a probability distribution of the inference result. The probability distribution of the inference result is, for example, a distribution of weights of a node group constituting a predetermined layer in the neural network of the trained model. The index calculation unit 51 obtains the confidence level of the inference result by, for example, inputting the distribution of weights of the node group into a confidence level function.

[0060] The deterioration determination unit 52 determines whether the accuracy of the trained model has deteriorated based on the evaluation result received on the reception screen 11. Here, the deterioration determination unit 52 determines whether the accuracy of the trained model has deteriorated based on the evaluation index calculated using the received evaluation result. Specifically, the deterioration determination unit 52 may determine whether the accuracy has deteriorated based on a comparison between the calculated evaluation index and a threshold. For example, the accuracy rate is compared with a predetermined accuracy rate threshold, and when the accuracy rate has decreased to be less than the accuracy rate threshold, the deterioration determination unit 52 may determine that the accuracy of the trained model has deteriorated. Alternatively, the deterioration determination unit 52 may determine whether the accuracy of the trained model has deteriorated based on a change in the calculated evaluation index. For example, the deterioration determination unit 52 may determine that the accuracy of the trained model has deteriorated when the amount of decrease in the accuracy rate is equal to or greater than a predetermined quality threshold. Alternatively, such a plurality of determinations may be combined. For example, when the accuracy rate is equal to or greater than the accuracy rate threshold and the amount of decrease in the accuracy rate is equal to or greater than the quality threshold, the deterioration determination unit 52 may determine that the accuracy of the trained model has deteriorated. Also, for example, when the accuracy rate is equal to or greater than the accuracy rate threshold and is less than the accuracy rate threshold, the degradation determination unit 52 may determine that the accuracy of the trained model has deteriorated.

[0061] The evaluation index may be both the accuracy rate and the confidence level. When the accuracy rate decreases or the amount of decrease in the accuracy rate becomes large even though the confidence level of the inference result in the trained model is maintained at or above the confidence level threshold, the degradation determination unit 52 may determine that the accuracy of the trained model has deteriorated.

[0062] When it is determined that the accuracy of the trained model has deteriorated, the deterioration notification unit 53 notifies the user, for example by email, that the accuracy of the trained model has deteriorated.

[0063] <Learning Device 6> The learning device 6 includes a label input unit 61 and a learning unit 62. The label input unit 61 accepts, via the reception screen 11, an annotation operation by a user who has confirmed the inference result on the reception screen 11. The label input unit 61 records the acquired correct label in the correct label storage device 33.

[0064] For example, when the degradation notification unit 53 notifies the user that the accuracy of the trained model has deteriorated, the learning unit 62 performs re-learning of the trained model using teacher data including predetermined re-learning images (explanatory variables) and a correct label (new objective variable) accepted by the annotation operation. The learning unit 62 records the trained model after re-learning in the model storage device 32.

[0065] Next, an example of the operation of each device in the quality maintenance control system 1 will be described. 4 is a flowchart showing an example of the operation of the inference device 2 in the embodiment. The acquisition unit 21 acquires an image from the input data storage device 31 (step S101). The inference unit 22 executes an inference process for the image by inputting the image to a trained model (step S102). The output unit 23 records the inference result output from the trained model in the inference result storage device 34 (step S103).

[0066] 5 is a flowchart showing an example of the operation of the evaluation terminal 4 in the embodiment. The display unit 41 acquires the inference result from the inference result storage device 34 (step S201). The display unit 41 displays at least one of the inference result and the explanatory variables on the reception screen 11 that accepts an input operation of the inference result or the evaluation result for the explanatory variables (step S202). The evaluation unit 42 acquires the evaluation result input by the user's input operation via the reception screen 11 that accepts the input operation (step S203). The evaluation unit 42 records the evaluation result in the evaluation result storage device 35 (step S204).

[0067] 6 is a flowchart showing an example of the operation of the accuracy maintenance management device 5 in the embodiment. The index calculation unit 51 acquires the evaluation result from the evaluation result storage device 35 (step S301). The index calculation unit 51 calculates the evaluation index based on the evaluation result (step S302). The degradation determination unit 52 determines the accuracy degradation of the trained model based on the evaluation index (step S303). When it is determined that the accuracy of the trained model has deteriorated, the degradation notification unit 53 notifies the user of the accuracy deterioration of the trained model by, for example, email (step S304).

[0068] 7 is a flowchart showing an example of the operation of the learning device 6 in the embodiment. The learning unit 62 acquires one or more correct labels from the correct label storage device 33 (step S401). The learning unit 62 executes re-learning of the trained model using teacher data including images and correct labels (step S402). The learning unit 62 records the trained model after re-learning in the model storage device 32 (step S403).

[0069] As described above, the accuracy maintenance management system 1 judges whether the accuracy of the trained model has deteriorated based on the evaluation index calculated based on the evaluation result obtained via the reception screen 11. The evaluation result obtained via the reception screen 11 may be a user's evaluation result of the inference result output from the trained model, or a user's evaluation result of the explanatory variables input to the trained model. The user's evaluation result of the explanatory variables is, for example, an evaluation result of the inference result that should be as the user viewed the explanatory variables (images input to the trained model) displayed on the reception screen 11. This makes it possible to improve the accuracy of detecting deterioration in the accuracy of the trained model.

[0070] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included.

[0071] (Additional Note) <Appendix 1> A display unit that displays, on a display device, a reception screen that receives an input operation of an inference result output from a trained model to which explanatory variables have been input or an evaluation result for the explanatory variables; an index calculation unit that calculates an evaluation index representing accuracy of the trained model based on the evaluation result received on the reception screen; a degradation determination unit that determines whether or not the accuracy of the trained model has deteriorated based on the evaluation index; An accuracy maintenance management system equipped with <Appendix 2> 2. The quality maintenance management system according to claim 1, wherein the reception screen includes an input field for the evaluation result and a display field for the explanatory variable. <Appendix 3> The accuracy maintenance management system according to claim 1 or 2, wherein the reception screen further includes a display field for the inference result. <Appendix 4> An index calculation unit that calculates an evaluation index indicating accuracy of the trained model based on the evaluation result received on the reception screen, 4. The accuracy maintenance management system according to claim 1, wherein the degradation determination unit determines whether or not accuracy of the trained model has deteriorated based on the evaluation index. <Appendix 5> The accuracy maintenance management system according to claim 4, wherein the degradation determination unit determines whether or not the accuracy of the trained model has deteriorated based on a change in the evaluation index. <Appendix 6> 6. The accuracy maintenance management system according to claim 1, wherein the trained model uses an image as the explanatory variable and outputs the inference result. <Appendix 7> 7. The accuracy maintenance management system according to any one of Supplementary Note 1 to Supplementary Note 6, further comprising a degradation notification unit that notifies that the accuracy of the trained model has deteriorated when it is determined that the accuracy of the trained model has deteriorated. <Appendix 8> the evaluation index is a rate of accuracy of the inference result, 6. The accuracy maintenance management system according to claim 4 or 5, wherein the degradation determination unit determines that the accuracy of the trained model has deteriorated when the accuracy rate has decreased. <Appendix 9> The accuracy maintenance management system described in Appendix 8, wherein the degradation determination unit determines that the accuracy of the trained model has deteriorated when the accuracy rate has decreased even though the confidence of the inference result in the trained model is maintained above a confidence threshold. <Appendix 10> 10. The accuracy maintenance management system according to any one of appendix 1 to appendix 9, wherein the reception screen receives an annotation operation of a correct label to be used for re-learning the trained model. <Appendix 11> 11. The accuracy maintenance management system according to any one of Supplementary Note 1 to Supplementary Note 10, further comprising a learning unit that, when notified that the accuracy of the trained model has deteriorated, executes re-learning of the trained model by using predetermined re-learning images as explanatory variables and a correct label accepted by an annotation operation as a new objective variable. <Appendix 12> A quality maintenance management method executed by a quality maintenance management system, A step of displaying, on a display device, a reception screen for receiving an input operation of an inference result output from a trained model to which explanatory variables have been input or an evaluation result for the explanatory variables; determining whether or not accuracy of the trained model has deteriorated based on the evaluation result received on the reception screen; Methods for maintaining and managing accuracy, including: <Appendix 13> On the computer, A step of displaying, on a display device, a reception screen for receiving an input operation of an inference result output from a trained model to which explanatory variables have been input or an evaluation result for the explanatory variables; A step of determining whether or not the accuracy of the trained model has deteriorated based on the evaluation result received on the reception screen; A program for executing. [Explanation of symbols]

[0072] 1...Accuracy maintenance management system, 2...Inference device, 3...Database device, 4...Evaluation terminal, 5...Accuracy maintenance management device, 6...Learning device, 7...Display device, 11...Reception screen, 21...Acquisition unit, 22...Inference unit, 23...Output unit, 31...Input data storage device, 32...Model storage device, 33...Correct label storage device, 34...Inference result storage device, 35...Evaluation result storage device, 41...Display unit, 42...Evaluation unit, 51...Index calculation unit, 52...Deterioration determination unit, 53...Deterioration notification unit, 61...label input section, 62...learning section, 71...display area, 101...date and time search field, 102...actual item ID search field, 103...judgment result search field, 104...registration result search field, 105...judgment result display field, 106...image display field, 107...annotation progress display field, 108...registration result display field, 109...registration selection field, 110...registration result selection field, 111...confirmation image, 201...search operation key, 202...download operation key, 203...upload operation key

Claims

1. A display unit that displays on a display device a reception screen for receiving an input operation of an inference result output from a trained model to which explanatory variables have been input or an evaluation result for the explanatory variables; a degradation determination unit that determines whether or not accuracy of the trained model has deteriorated based on the evaluation result received on the reception screen; Equipped with The reception screen is an accuracy maintenance management system that accepts annotation operations of correct answer labels to be used for re-learning the trained model.

2. The accuracy maintenance management system described in claim 1, wherein the reception screen includes an input field for the evaluation result and a display field for the explanatory variable.

3. The accuracy maintenance management system described in Claim 2, wherein the reception screen further includes a display column for the inference result.

4. Further comprising an index calculation unit that calculates an evaluation index representing an accuracy of the trained model based on the evaluation result accepted on the reception screen, The accuracy maintenance management system according to claim 1 , wherein the deterioration determination unit determines whether or not accuracy of the trained model has deteriorated based on the evaluation index.

5. An accuracy maintenance management system as described in Claim 4, wherein the deterioration determination unit determines whether or not the accuracy of the trained model has deteriorated based on a change in the evaluation index.

6. An accuracy maintenance management system as described in any one of claims 1 to 3, wherein the trained model uses an image as the explanatory variable and outputs the inference result.

7. An accuracy maintenance management system as described in any one of claims 1 to 3, further comprising a degradation notification unit that notifies that the accuracy of the trained model has deteriorated when it is determined that the accuracy of the trained model has deteriorated.

8. The evaluation index is a rate of accuracy of the inference result, The accuracy maintenance management system according to claim 4 , wherein the degradation determination unit determines that the accuracy of the trained model has deteriorated when the accuracy rate has decreased.

9. An accuracy maintenance management system as described in Claim 8, wherein the deterioration determination unit determines that the accuracy of the trained model has deteriorated if the accuracy rate decreases even though the confidence of the inference result in the trained model is maintained above a confidence threshold.

10. An accuracy maintenance management system as described in any one of claims 1 to 3, further comprising a learning unit that, when notified that the accuracy of the trained model has deteriorated, performs re-learning of the trained model by using predetermined re-learning images as explanatory variables and a correct label accepted by an annotation operation as a new objective variable.

11. A quality maintenance management method executed by a quality maintenance management system, comprising: A step of displaying, on a display device, a reception screen for receiving an input operation of an inference result output from a trained model to which explanatory variables have been input or an evaluation result for the explanatory variables; determining whether or not accuracy of the trained model has deteriorated based on the evaluation result received on the reception screen; Including, The accuracy maintenance management method, wherein the reception screen receives an annotation operation of a correct answer label to be used for re-learning the trained model.

12. A computer comprising: A step of displaying, on a display device, a reception screen for receiving an input operation of an inference result output from a trained model to which explanatory variables have been input or an evaluation result for the explanatory variables; A step of determining whether or not the accuracy of the trained model has deteriorated based on the evaluation result received on the reception screen; Run the command, The reception screen receives an annotation operation of a correct answer label to be used for re-learning the trained model.

13. A display unit that displays, on a display device, a reception screen that receives an input operation of an inference result output from a trained model to which explanatory variables have been input or an evaluation result for the explanatory variables; a degradation determination unit that determines whether or not accuracy of the trained model has deteriorated based on the evaluation result received on the reception screen; and an index calculation unit that calculates an evaluation index that indicates accuracy of the trained model based on the evaluation result received on the reception screen; Equipped with the evaluation index is a rate of accuracy of the inference result, The deterioration determination unit determines that the accuracy of the trained model has deteriorated when the accuracy rate decreases even though the confidence of the inference result in the trained model is maintained above a confidence threshold.

14. The quality maintenance management system according to claim 13 , wherein the reception screen includes an input field for the evaluation result and a display field for the explanatory variable.

15. The quality maintenance management system according to claim 14 , wherein the reception screen further includes a display field for the inference result.

16. 16. The accuracy maintenance management system according to claim 13, wherein the degradation determination unit determines whether or not accuracy of the trained model has deteriorated based on a change in the evaluation index.

17. The accuracy maintenance management system according to claim 13 , wherein the trained model outputs the inference result using an image as the explanatory variable.

18. The accuracy maintenance management system according to any one of claims 13 to 15, further comprising a degradation notification unit that notifies the user that the accuracy of the trained model has deteriorated when it is determined that the accuracy of the trained model has deteriorated.

19. The accuracy maintenance management system according to claim 13 , wherein the reception screen receives an annotation operation of a correct answer label used for re-learning the trained model.

20. 16. The accuracy maintenance management system according to claim 13, further comprising a learning unit that, when notified that the accuracy of the trained model has deteriorated, executes relearning of the trained model by using a predetermined re-learning image as an explanatory variable and a correct label accepted by an annotation operation as a new objective variable.

21. A quality maintenance management method executed by a quality maintenance management system, A step of displaying, on a display device, a reception screen for receiving an input operation of an inference result output from a trained model to which explanatory variables have been input or an evaluation result for the explanatory variables; A step of determining whether or not accuracy of the trained model has deteriorated based on the evaluation result received on the reception screen; calculating an evaluation index representing accuracy of the trained model based on the evaluation result received on the reception screen; Including, the evaluation index is a rate of accuracy of the inference result, In the step of determining whether or not the accuracy has deteriorated, if the accuracy rate decreases even though the confidence of the inference result in the trained model is maintained above a confidence threshold, the accuracy maintenance management method determines that the accuracy of the trained model has deteriorated.

22. On the computer, A step of displaying, on a display device, a reception screen for receiving an input operation of an inference result output from a trained model to which explanatory variables have been input or an evaluation result for the explanatory variables; A step of determining whether or not accuracy of the trained model has deteriorated based on the evaluation result received on the reception screen; A step of calculating an evaluation index representing accuracy of the trained model based on the evaluation result received on the reception screen; Run the command, the evaluation index is a rate of accuracy of the inference result, In the step of determining whether or not the accuracy of the trained model has deteriorated, if the accuracy rate decreases even though the confidence of the inference result in the trained model is maintained at or above a confidence threshold, the program determines that the accuracy of the trained model has deteriorated.

Images