Accuracy maintenance management system, accuracy maintenance management method, and program

Abstract

To provide an accuracy maintenance management system, an accuracy maintenance management method, and a program that can improve the accuracy of detecting accuracy degradation in trained models that output inference results based on explanatory variables. [Solution] The accuracy maintenance management system includes a display unit that displays a reception screen on a display device for receiving input operations for inference results output from a trained model into which explanatory variables have been input, or for evaluation results for explanatory variables, and a degradation determination unit that determines whether or not the accuracy of the trained model has deteriorated based on the evaluation results received on the reception screen. The reception screen may include an input field for evaluation results and a display field for explanatory variables. The reception screen may further include a display field for inference results. The index calculation unit may calculate an evaluation index representing the accuracy of the trained model based on the evaluation results received on the reception screen.

Description

Technical Field

[0001] The present invention relates to an accuracy maintenance management system, an accuracy maintenance management method, and a program.

Background Art

[0002] The accuracy of a learned model of machine learning may deteriorate due to various factors. Therefore, it is required to maintain and manage the accuracy of the learned model. Patent Document 1 discloses an image recognition device that determines performance degradation of a learned model.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a learned model having numerical data (for example, numerical data representing a steel grade), which is structured data, as an explanatory variable, it is relatively easy to detect accuracy degradation of the learned model based on changes in the input numerical data.

[0005] On the other hand, an image is unstructured data. Unstructured data is not formatted into a predefined structure, is stored in its original format, and is not processed until use. For example, an image contains information such as RGB, but it is not in a format that can interpret what it means (for example, whether it is a scratch). Therefore, in a learned model having an image as an explanatory variable, it is not easy to detect accuracy degradation of the learned model based on changes in the input image.

[0006] Thus, there is a problem that the detection accuracy of accuracy degradation of a learned model that outputs an inference result based on an explanatory variable (especially an image) cannot be improved.

[0007] This invention has been made in view of the above circumstances, and aims to provide an accuracy maintenance management system, an accuracy maintenance management method, and a program that can improve the accuracy of detecting the degradation of the accuracy of a trained model that outputs inference results based on explanatory variables. [Means for solving the problem]

[0008] (1) One aspect of the present invention is an accuracy maintenance management system comprising: a display unit that displays on a display device an input screen for receiving input operations for inference results output from a trained model into which explanatory variables have been input, or for evaluation results for the explanatory variables; and a degradation determination unit that determines whether or not the accuracy of the trained model has deteriorated based on the evaluation results received on the input screen.

[0009] (2) In one aspect of the present invention, in the accuracy maintenance management system described in (1) above, the reception screen includes an input field for the evaluation result and a display field for the explanatory variable.

[0010] (3) In one aspect of the present invention, the accuracy maintenance management system described in (2) above, wherein the reception screen further includes a field for displaying the inference results.

[0011] (4) One aspect of the present invention is an accuracy maintenance management system according to any one of (1) to (3) above, further comprising an index calculation unit that calculates an evaluation index representing the accuracy of the trained model based on the evaluation result received on the reception screen, and the degradation determination unit determines whether or not the accuracy of the trained model has deteriorated based on the evaluation index.

[0012] (5) In one aspect of the present invention, in the accuracy maintenance management system described in (4) above, the degradation determination unit determines whether or not the accuracy of the trained model has deteriorated based on the change in the evaluation index.

[0013] (6) In one aspect of the present invention, in the accuracy maintenance management system described in any one of (1) to (3) above, the trained model outputs the inference result using the image as the explanatory variable.

[0014] (7) In one aspect of the present invention, the accuracy maintenance management system described in any one of (1) to (3) above further comprises a degradation notification unit that notifies the system that the accuracy of the trained model has deteriorated when it is determined that the accuracy of the trained model has deteriorated.

[0015] (8) In one aspect of the present invention, in the accuracy maintenance management system described in (4) above, the evaluation index is the accuracy rate of the inference result, and the deterioration determination unit determines that the accuracy of the trained model has deteriorated when the accuracy rate decreases.

[0016] (9) In one aspect of the present invention, in the accuracy maintenance management system described in (8) above, the degradation determination unit determines that the accuracy of the trained model has deteriorated if the accuracy rate decreases despite the confidence level of the inference result being maintained at or above a confidence threshold in the trained model.

[0017] (10) In one aspect of the present invention, in the accuracy maintenance management system described in any one of (1) to (3) above, the reception screen accepts annotation operations for the correct labels used for retraining the trained model.

[0018] (11) One aspect of the present invention is a precision maintenance management system described in any one of (1) to (3) above, further comprising a learning unit that, when notified that the precision of the trained model has deteriorated, performs retraining of the trained model using predetermined retraining images as explanatory variables and correct labels received by annotation operations as new target variables.

[0019] (12) One aspect of the present invention is a precision maintenance management method executed by a precision maintenance management system, the method including: displaying, on a display device, a reception screen for receiving an input operation of an evaluation result for an inference result output from a learned model to which an explanatory variable is input or for the explanatory variable; and determining, based on the evaluation result received on the reception screen, whether the precision of the learned model has deteriorated.

[0020] (13) One aspect of the present invention is a program for causing a computer to execute: a procedure of displaying, on a display device, a reception screen for receiving an input operation of an evaluation result for an inference result output from a learned model to which an explanatory variable is input or for the explanatory variable; and a procedure of determining, based on the evaluation result received on the reception screen, whether the precision of the learned model has deteriorated.

Advantages of the Invention

[0021] According to the present invention, it is possible to improve the detection precision of deterioration in the precision of a learned model that outputs an inference result based on an explanatory variable.

Brief Description of the Drawings

[0022] [Figure 1] It is a diagram showing a configuration example of a precision maintenance management system in an embodiment. [Figure 2] It is a diagram showing an example of display of a reception screen in an embodiment. [Figure 3] It is a diagram showing an example of display of a detailed display screen in an embodiment. [Figure 4] It is a flowchart showing an operation example of an inference device in an embodiment. [Figure 5] It is a flowchart showing an operation example of an evaluation terminal in an embodiment. [Figure 6] It is a flowchart showing an operation example of a precision maintenance management device in an embodiment. [Figure 7] It is a flowchart showing an operation example of a learning device in an embodiment. [Modes for carrying out the invention]

[0023] Embodiments of the present invention will be described in detail with reference to the drawings. Figure 1 shows an example configuration of the accuracy maintenance management system 1 in an embodiment. The accuracy maintenance management system 1 is a system that maintains and manages the accuracy of one or more trained models. A trained model is a model that has been trained using machine learning techniques. A trained model includes, for example, a multi-layer neural network. A multi-layer neural network is, for example, a convolutional neural network. A multi-layer neural network may also be a recurrent neural network. The machine learning technique is not limited to a specific technique, but is, for example, supervised learning.

[0024] The explanatory variables of a trained model can be either structured or unstructured data, but in the following example, unstructured data is used. Unstructured data can be, for example, image data or sound data. Images are, for example, images captured by cameras installed in a steel mill or the like. Captured images can be videos or still images. Captured images are not limited to images of specific objects, but for example, images that include images of manufactured products. Sounds are, for example, sounds produced by machinery such as a conveyor belt. Sounds produced by machinery are collected using microphones installed near the machinery. If a malfunction occurs in the machinery, a sound different from the sound collected when no malfunction occurs (normal sound) is collected by the microphone. In the following example, the explanatory variables of the trained model are images.

[0025] The accuracy maintenance management system 1 comprises an inference device 2, a database device 3, an evaluation terminal 4 (terminal device), an accuracy maintenance management device 5, a learning device 6, and a display device 7. In the embodiment shown in Figure 1, the above-mentioned devices (2 to 6) of the accuracy maintenance management system 1 are separate entities, but at least two of these devices may be integrated. That is, the accuracy maintenance management system 1 may consist of multiple devices or a single device. If it consists of multiple devices, the multiple devices are connected to each other so as to the necessary extent, but they may be directly connected or connected by a communication network that includes at least one of a wireless or wired portion. Furthermore, the information stored (managed) by the database device 3 does not need to be consolidated into a single device, but may be distributed and stored across multiple devices.

[0026] Each device (2-6) is composed of one or more hardware processors, such as a CPU (Central Processing Unit), and one or more memories (main memory). The memory is composed of storage devices such as RAM (Random Access Memory) and ROM (Read Only Memory). Each device (2-6) functions by having one or more hardware processors execute one or more programs stored in the memory. Cloud computing technology may be used in each of the devices (2-6).

[0027] All or part of the functions of each device (2-6) may be implemented using hardware such as ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), and FPGA (Field Programmable Gate Array). The above program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, semiconductor memory devices (e.g., solid-state drives (SSDs)), hard disks built into computer systems, or memory devices such as semiconductor memory devices. The above program may be transmitted via a telecommunications line.

[0028] First, I will explain the overview of the accuracy maintenance management system 1. Inference device 2 inputs an image as an explanatory variable into a trained model. Inference device 2 obtains the inference result from the trained model into which the image has been input. If there are multiple trained models that are subject to maintaining and managing accuracy, the type of trained model may differ for each trained model. That is, the inference result from the trained model is not limited to a specific inference result. For example, the trained model may be a trained model that recognizes a target object in the input image, or a trained model that detects a target object in the input image. Detecting an object may mean, for example, detecting the presence or absence of an object in the image, or determining the region of an object in the image. Recognizing an object may mean, for example, recognizing what the object in the image is.

[0029] The database device 3 stores various types of data. These types of data include, for example, images (explanatory variables), inference results (dependent variables; the same applies hereafter), user evaluation results for the inference results (registration results), user evaluation results for the explanatory variables (registration results), trained models, and correct labels for the training data.

[0030] The evaluation terminal 4 displays a reception screen on the display device 7 for receiving user evaluation results on inference results or explanatory variables. Explanatory variables input into the trained model are, for example, images. Inference results output from the trained model are, for example, object recognition results or detection results. The evaluation terminal 4 receives user input of evaluation results via the reception screen and records the input evaluation results in the database device 3. In addition to the input field for evaluation results, the reception screen may also display information useful for inputting evaluation results (for example, inference results and explanatory variables). In this way, by confirming the reception screen displaying the inference results or explanatory variables and inputting the evaluation results into the evaluation terminal 4, the user can efficiently evaluate the inference results.

[0031] Furthermore, the validity of the inference results from the trained model can be determined based on the user's evaluation results. Depending on what the evaluation results indicate, if the evaluation results are the user's evaluation of the inference results (whether the inference results are correct or incorrect), the evaluation results will directly indicate whether the inference results are correct or incorrect, and it will be possible to determine the appropriateness of the inference results based on those evaluation results. When the evaluation results are the user's evaluation of the inference results, and the evaluation results indicate the correct content of the inference results (for example, the presence or absence of an item), the evaluation terminal 4 can determine whether the inference results are correct or incorrect by comparing the inference results with the evaluation results. On the other hand, if the evaluation results are the user's evaluation of the explanatory variables (for example, a user's judgment that an object should be detected in an image displayed on the reception screen), the evaluation terminal 4 can determine whether the inference results are correct or incorrect by comparing the inference results with the evaluation results.

[0032] As described above, the validity of individual estimation results can be determined based on the evaluation results, and by utilizing these determination results, it is possible to improve the accuracy of detecting the degradation of the trained model's accuracy.

[0033] Specifically, the accuracy maintenance device 5 calculates an evaluation index representing the accuracy of the trained model based on the evaluation results (user evaluation results) received by the reception screen. Details of the evaluation index will be described later. Based on the calculated evaluation index, the accuracy maintenance device 5 determines whether or not the accuracy of the trained model has deteriorated. By determining whether or not the accuracy of the trained model has deteriorated based on the evaluation index, it is possible to improve the efficiency of recovering deteriorated accuracy. For example, if the recognition accuracy of the number "1" in an captured image is low, the user can analyze the factors causing the decrease in recognition accuracy, especially for that number "1". Based on the analyzed factors causing the decrease, the user can consider actions to restore accuracy, thus improving the efficiency of recovering deteriorated accuracy.

[0034] The accuracy maintenance device 5 notifies the user if it determines that the accuracy of the trained model has deteriorated. The accuracy maintenance device 5 may promptly notify the user using, for example, email and alarm (alarm display, alarm sound, etc.). However, the accuracy maintenance device 5 does not necessarily have to issue an alarm. For example, the user may check the result of the deterioration determination when performing periodic verification work via the evaluation terminal 4, etc.

[0035] The learning device 6 accepts annotation operations (e.g., inputting an object name) from a user who has viewed the inference results (e.g., object recognition results) and images on the reception screen, via the reception screen. The learning device 6 may also accept annotation operations (e.g., inputting a bounding box) from a user who has viewed the inference results (e.g., object detection results) and images on another screen, via the reception screen. Through the annotation operations, the learning device 6 acquires training data to be used for retraining the trained model. The learning device 6 records the acquired training data in the database device 3.

[0036] When the accuracy maintenance device 5 notifies the user that the accuracy of the trained model has deteriorated, the learning device 6 retrains the trained model using predetermined retraining images (training data) as explanatory variables and the correct labels received through annotation as new target variables. The learning device 6 records the retrained trained model in the database device 3. This allows for efficient retraining of the trained model, thereby improving the efficiency of recovering deteriorated accuracy.

[0037] Next, we will describe the details of each component of the accuracy maintenance management system 1. <Inference device 2> The inference device 2 comprises an acquisition unit 21, an inference unit 22, and an output unit 23. The acquisition unit 21 acquires an image (for example, an image of a product) as an explanatory variable from the input data storage device 31. The acquisition unit 21 may perform preprocessing on the acquired image. Preprocessing may include, for example, data corruption checking, cropping, and brightness adjustment. The inference unit 22 acquires a trained model from the model storage device 32. The inference unit 22 inputs the acquired image into the trained model. The inference unit 22 acquires the inference result output from the trained model. For example, if an image of a product is input into the trained model, the inference result from the trained model may be a variable representing the determination result of whether or not there are scratches on the product. The output unit 23 records the inference result from the trained model in the inference result storage device 34.

[0038] <Database device 3> The database device 3 comprises an input data storage device 31, a model storage device 32, a ground truth label storage device 33, an inference result storage device 34, and an evaluation result storage device 35. The input data storage device 31 stores images as explanatory variables (input data). The model storage device 32 stores the trained model. The ground truth label storage device 33 stores the ground truth labels associated with images through annotation operations. The inference result storage device 34 stores the inference results of the trained model. The evaluation result storage device 35 stores the user's evaluation results of the inference results by the trained model and the user's evaluation results of the explanatory variables. The database device 3 stores the data in a way that allows the correspondence between the inference results by the trained model for the input image, the user's evaluation results for this inference result or the same image, and the ground truth labels for these evaluation results, etc.

[0039] <Evaluation terminal 4, learning device 6> The evaluation terminal 4 comprises a display unit 41 and an evaluation unit 42. The display unit 41 displays (outputs) a reception screen to the display area 71 of the display device 7. The reception screen is a screen that accepts input operations for the user's evaluation results on the inference results of the trained model and the user's evaluation results on the explanatory variables input to the trained model. The evaluation unit 42 accepts the user's input operation for evaluation results via the reception screen. The reception screen may also be a screen that accepts the user's input operation for evaluation results on the explanatory variables input to the trained model. The evaluation unit 42 or the label input unit 61 (described later) may accept annotation operations for correct labels on images via the reception screen displayed in the display area 71. In other words, the reception screen may include an area for displaying images input to the trained model and an area for inputting the user's evaluation results. Furthermore, the reception screen may include an area for showing the inference results. This allows the user to easily input evaluation results while checking the images displayed on the reception screen. Furthermore, users can easily input evaluation results while viewing the images and inference results displayed on the reception screen. In this way, if not only the images but also the inference results are displayed on the reception screen, users can easily check the inference results as well.

[0040] Figure 2 shows an example of the display of the reception screen 11 in the embodiment. The reception screen 11 includes a judgment result search field 103. The reception screen 11 may further include a date and time search field 101, an item ID search field 102, a registration result search field 104, a judgment result display field 105, an image display field 106, an annotation progress display field 107, a registration result display field 108, a registration selection field 109, and a search operation key 201.

[0041] The reception screen 11 displays the status of annotation operations on the group of captured images (training data) containing images of the inference target (e.g., a product), for each unique identifier that identifies the captured inference target, for example, in a table format. The identifier of the inference target is, for example, the product ID (e.g., product code).

[0042] The reception screen 11 may further include a download operation key 202 and an upload operation key 203. The evaluation unit 42 or the label input unit 61 may accept an operation (upload operation) to upload new captured images (new training data) as a set of images for retraining to the input data storage device 31. The evaluation unit 42 or the label input unit 61 may also accept annotation operations on the uploaded set of images for retraining via the reception screen 11.

[0043] The "Product ID" is associated with the "Registration Time" (the time the image was registered to the database device 3, or the time the image was captured), the "Pattern" (the pattern of the packaging, etc.), the "Location" (the location of the product), the "Judgment Result" (the inference result by the trained model, either "OK" or "NG"), the "Registration Result", the product image set (training data), and the status of the annotation operation. The "Registration Result" is the registration result of the user's evaluation as described above. In the "Registration Result" column, the user's evaluation result for the inference result or explanatory variable is displayed, for example, as "OK" or "NG". If annotation operation is not required, "Not Applicable" may be displayed in the "Registration Result" column. In the "Registration Result" column, the presence or absence of an object such as a product may be indicated based on the user's evaluation result, or the object name of the product may be indicated. These indications may be selected by the user using a combo box, a list box, or entered arbitrarily using an input box.

[0044] The date and time search field 101 is an input field for the date and time (the time when the captured image was registered in the input data storage device 31). This date and time is one of the search keys for extracting an image from among multiple captured images stored in the input data storage device 31.

[0045] The product ID search field 102 is an input field for searching for images of a product, using the product ID as one of the search keys. In other words, the product ID search field 102 is an input field for extracting images of a product to which the product ID is assigned from among multiple images stored in the input data storage device 31, using the product ID as one of the search keys.

[0046] The judgment result search field 103 is an input field for searching captured images using "judgment result," which is the inference result from a trained model, as one of the search keys. The registration result search field 104 is an input field for searching captured images using "registration result," which is the user's evaluation result for the inference result or explanatory variables, as one of the search keys.

[0047] The judgment result display area 105 is the display area for the inference result by the trained model (judgment result by the trained model, "OK" or "NG"). The image display area 106 is the display area for displaying one or more captured images (training data) as explanatory variables (explanatory variable display area).

[0048] The annotation progress display area 107 is a field that shows whether annotation has been completed for the group of captured images (training data) displayed in the image display area 106. The user can check in the annotation progress display area 107 if there are any groups of captured images (training data) associated with the actual product ID that have not yet been annotated. This makes it easy for the user to check the progress of the annotation.

[0049] The registration result display field 108 is a field that displays the registration status of the user's evaluation results for the inference results or explanatory variables (for example, whether the inference result by the trained model is correct "OK" or not). The registration result display field 108 may also be an input field for evaluation results. The registration selection field 109 is a field that displays checkboxes for selecting the group of captured images associated with the actual product ID.

[0050] When the user presses the search operation key 201, the evaluation unit 42 performs a search process for image groups (learning data) associated with the same registration time, based on the search conditions defined using each input field displayed on the reception screen 11. Based on the search results, the display unit 41 displays a list of image groups stored in the input data storage device 31 in the display area 71, allowing selection for each image group (registration time) associated with the actual product ID.

[0051] The evaluation unit 42 accepts user input of evaluation results for the inference results of the trained model using the reception screen 11. The user's evaluation result is, for example, whether the inference result of the trained model is correct ("OK"). The evaluation unit 42 may also accept user input of evaluation results for the explanatory variables input to the trained model using the reception screen 11. The user's evaluation result is, for example, an evaluation result that an object should be detected in the image input to the trained model. The evaluation unit 42 records the user's evaluation results in the evaluation result storage device 35.

[0052] Furthermore, the evaluation unit 42 accepts annotation operations for ground truth labels on the captured images (retraining images) downloaded from the input data storage device 31. The evaluation unit 42 associates the received ground truth labels with the captured images as explanatory variables and records them in the ground truth label storage device 33.

[0053] When the user presses the download operation key 202 "Download Selected Data (Image / Annotation)", the captured image used as an explanatory variable, selected from the list using the registration selection field 109, is downloaded from the input data storage device 31 to the display unit 41. The display unit 41 displays the downloaded captured image in the display area 71 (reception screen 11, or another screen). The user may perform annotation operations to associate correct labels with the downloaded captured image.

[0054] When the user presses the upload operation key 203 "Annotation Upload," the evaluation unit 42 or the label input unit 61 records (uploads) the combination of the downloaded captured image (retraining image) and the correct label obtained by the user's annotation operation to the database device 3. In this way, multiple combinations of captured images (retraining images) and correct labels are collected in the database device 3.

[0055] Furthermore, if multiple similar images exist in the input data storage device 31 among the collected images, at least one of the similar images may be excluded from the list of images for retraining by, for example, the evaluation unit 42 or the learning unit 62. This makes it possible to retrain the trained model using image data with less bias.

[0056] Figure 3 shows an example of the display of the detailed display screen in the embodiment. The detailed display screen is a screen that displays the details of the inference results by the trained model for the captured image selected by the user. The detailed display screen is displayed in the display area 71 when the user selects (by clicking, tapping, or performing other selection operations) the captured image displayed in the image display field 106 of the reception screen 11. This makes it easy for the user to check the displayed details.

[0057] The efficiency of updating training data is improved by allowing the user to modify the "Registration Result" displayed in the Registration Result Display Field 108 of the user-selected image using the Registration Result Selection Field 110 on the detailed display screen. When the user attempts to modify the "Registration Result" displayed in the Registration Result Display Field 108, the display unit 41 pops up a confirmation image 111 in the display area 71. This allows the evaluation unit 42 to reconfirm with the user whether or not they really want to modify the "Registration Result" displayed in the Registration Result Display Field 108.

[0058] Returning to Figure 1, we will continue the explanation of each component of the accuracy maintenance management system 1. <Accuracy maintenance control device 5> The accuracy maintenance device 5 comprises an index calculation unit 51, a degradation determination unit 52, and a degradation notification unit 53. The index calculation unit 51 calculates an evaluation index representing the accuracy of the trained model based on the evaluation results by the user who viewed the display on the reception screen 11. The evaluation index is, for example, the accuracy rate. The accuracy rate is the probability that the user's evaluation result for the inference result was correct ("OK") when the inference result and explanatory variables are displayed on the reception screen 11. In other words, the accuracy rate is the number of inference results that were correct ("OK") relative to the total number of inference results. The accuracy rate may also be the probability that the user's evaluation result for the explanatory variable (the evaluation result of what the inference result should be) matches the inference result output from the trained model when the explanatory variable (the image input to the trained model) is displayed on the reception screen 11. In other words, the accuracy rate may also be the number of inference results that matched the user's evaluation result for the explanatory variable relative to the total number of inference results. The evaluation result, which indicates how the inference result should be, is, for example, an evaluation result that an object should be detected in the image input to the trained model. The accuracy rate can also be the result obtained by subtracting the error rate from 1. The error rate is the number of inference results that were judged as incorrect ("NG") relative to the total number of inference results.

[0059] Furthermore, the index calculation unit 51 may calculate an evaluation index representing the accuracy of the trained model based on the evaluation results of the trained model. The evaluation index is, for example, the confidence score. The confidence score is determined based on the evaluation results of the trained model on the inference result. Here, the evaluation results of the trained model are determined, for example, based on the probability distribution of the inference result. The probability distribution of the inference result is, for example, the weight distribution of the node groups constituting a predetermined layer in the neural network of the trained model. The index calculation unit 51 obtains the confidence score of the inference result by, for example, inputting the weight distribution of the node groups into the confidence score function.

[0060] The degradation determination unit 52 determines whether the accuracy of the trained model has deteriorated based on the evaluation results received on the reception screen 11. Here, the degradation determination unit 52 determines whether the accuracy of the trained model has deteriorated based on an evaluation index calculated using the received evaluation results. Specifically, the degradation determination unit 52 may also determine whether accuracy has deteriorated based on a comparison between the calculated evaluation index and a threshold. For example, if the accuracy rate is compared with a predetermined accuracy threshold and the accuracy rate falls below the accuracy threshold, the degradation determination unit 52 may determine that the accuracy of the trained model has deteriorated. Alternatively, the degradation determination unit 52 may determine whether the accuracy of the trained model has deteriorated based on a change in the calculated evaluation index. For example, if the decrease in accuracy rate is greater than or equal to a predetermined quality threshold, the degradation determination unit 52 may determine that the accuracy of the trained model has deteriorated. Or, a combination of such determinations may be used. For example, if the accuracy rate is greater than or equal to the accuracy threshold, and the decrease in accuracy rate is greater than or equal to the quality threshold, the degradation determination unit 52 may determine that the accuracy of the trained model has deteriorated. For example, if the accuracy rate is above the accuracy threshold, and the accuracy rate is below the accuracy threshold, the degradation determination unit 52 may determine that the accuracy of the trained model has deteriorated.

[0061] The evaluation metrics may include both accuracy and confidence. The degradation determination unit 52 may determine that the accuracy of the trained model has deteriorated if, despite the confidence level of the inference results in the trained model remaining above the confidence threshold, the accuracy decreases or the decrease in accuracy becomes significant.

[0062] If the degradation notification unit 53 determines that the accuracy of the trained model has deteriorated, it notifies the user of the deterioration in the accuracy of the trained model, for example, by email.

[0063] <Learning device 6> The learning device 6 comprises a label input unit 61 and a learning unit 62. The label input unit 61 receives annotation operations from the user who has confirmed the inference results on the reception screen 11, via the reception screen 11. The label input unit 61 records the acquired correct labels in the correct label storage device 33.

[0064] For example, if the user is notified by the degradation notification unit 53 that the accuracy of the trained model has deteriorated, the learning unit 62 retrains the trained model using training data that includes predetermined retraining images (explanatory variables) and correct labels (new target variables) received through annotation operations. The learning unit 62 then records the retrained trained model in the model storage device 32.

[0065] Next, we will describe examples of the operation of each device in the accuracy maintenance management system 1. Figure 4 is a flowchart showing an example of the operation of the inference device 2 in the embodiment. The acquisition unit 21 acquires an image from the input data storage device 31 (step S101). The inference unit 22 performs inference processing on the image by inputting the image to the trained model (step S102). The output unit 23 records the inference result output from the trained model in the inference result storage device 34 (step S103).

[0066] Figure 5 is a flowchart showing an example of the operation of the evaluation terminal 4 in the embodiment. The display unit 41 acquires the inference result from the inference result storage device 34 (step S201). The display unit 41 displays at least one of the inference result and / or explanatory variable on the reception screen 11, which accepts input operations for the inference result or evaluation result for the explanatory variable (step S202). The evaluation unit 42 acquires the evaluation result entered by the user's input operation via the reception screen 11 (step S203). The evaluation unit 42 records the evaluation result in the evaluation result storage device 35 (step S204).

[0067] Figure 6 is a flowchart showing an example of the operation of the accuracy maintenance device 5 in the embodiment. The index calculation unit 51 acquires the evaluation result from the evaluation result storage device 35 (step S301). The index calculation unit 51 calculates the evaluation index based on the evaluation result (step S302). The degradation determination unit 52 determines the accuracy degradation of the trained model based on the evaluation index (step S303). If the degradation notification unit 53 determines that the accuracy of the trained model has deteriorated, it notifies the user of the accuracy degradation of the trained model, for example, by email (step S304).

[0068] Figure 7 is a flowchart showing an example of the operation of the learning device 6 in the embodiment. The learning unit 62 acquires one or more correct labels from the correct label storage device 33 (step S401). The learning unit 62 performs retraining of the trained model using training data including images and correct labels (step S402). The learning unit 62 records the trained model after retraining in the model storage device 32 (step S403).

[0069] As described above, the accuracy maintenance management system 1 determines whether the accuracy of the trained model has deteriorated based on an evaluation index calculated based on the evaluation results obtained via the reception screen 11. The evaluation results obtained via the reception screen 11 may be the user's evaluation of the inference results output from the trained model, or the user's evaluation of the explanatory variables input into the trained model. The user's evaluation of the explanatory variables is, for example, the evaluation result of a user who saw the explanatory variables (images input into the trained model) displayed on the reception screen 11, indicating what the inference results should be. This makes it possible to improve the accuracy of detecting the deterioration of the trained model's accuracy.

[0070] While embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention.

[0071] (Note) <Note 1> A display unit that displays a reception screen on a display device for receiving input operations for inference results output from a trained model into which explanatory variables have been input, or for evaluation results for said explanatory variables, An index calculation unit calculates an evaluation index representing the accuracy of the trained model based on the evaluation results received on the reception screen, A degradation determination unit that determines whether or not the accuracy of the trained model has deteriorated based on the evaluation index. A precision maintenance management system equipped with the following features. <Note 2> The reception screen includes an input field for the evaluation results and a display field for the explanatory variables, as described in Appendix 1, for the accuracy maintenance management system. <Note 3> The reception screen further includes a display field for the inference results, as described in Appendix 1 or Appendix 2, for accuracy maintenance management systems. <Note 4> The system further includes an index calculation unit that calculates an evaluation index representing the accuracy of the trained model based on the evaluation results received on the reception screen, The accuracy maintenance management system according to any one of the appendices 1 to 3, wherein the degradation determination unit determines whether or not the accuracy of the trained model has deteriorated based on the evaluation index. <Note 5> The accuracy maintenance management system described in Appendix 4, wherein the degradation determination unit determines whether or not the accuracy of the trained model has deteriorated based on the change in the evaluation index. <Note 6> The accuracy maintenance management system described in any one of Appendix 1 to Appendix 5, wherein the trained model outputs the inference result using the image as the explanatory variable. <Note 7> The accuracy maintenance management system according to any one of the appendices 1 to 6, further comprising a degradation notification unit that notifies the system of the degradation of the accuracy of the trained model when it is determined that the accuracy of the trained model has deteriorated. <Note 8> The aforementioned evaluation index is the accuracy rate of the inference result, The accuracy maintenance management system described in Appendix 4 or Appendix 5, wherein the deterioration determination unit determines that the accuracy of the trained model has deteriorated when the accuracy rate decreases. <Note 9> The accuracy maintenance management system described in Appendix 8, wherein the degradation determination unit determines that the accuracy of the trained model has deteriorated if the accuracy rate decreases despite the confidence level of the inference result being maintained at or above a confidence threshold in the trained model. <Note 10> The reception screen is an accuracy maintenance management system described in any one of the appendices 1 to 9, which accepts annotation operations for the correct labels used to retrain the trained model. <Note 11> The accuracy maintenance management system described in any one of the appendices 1 to 10, further comprising a learning unit that, when notified that the accuracy of the trained model has deteriorated, performs retraining of the trained model using predetermined retraining images as explanatory variables and ground truth labels received through annotation operations as new target variables. <Note 12> A method for maintaining accuracy performed by an accuracy maintenance management system, The steps include displaying on a display device a reception screen that accepts input operations for inference results output from a trained model into which explanatory variables have been input, or for evaluation results for said explanatory variables, A step of determining whether the accuracy of the trained model has deteriorated based on the evaluation results received on the reception screen. A method for maintaining accuracy, including the following: <Note 13> On the computer, A procedure for displaying a screen on a display device that accepts input operations for inference results output from a trained model into which explanatory variables have been input, or for evaluation results for said explanatory variables, A procedure for determining whether the accuracy of the trained model has deteriorated based on the evaluation results received on the reception screen, and A program to execute. [Explanation of symbols]

[0072] 1…Accuracy maintenance management system, 2…Inference device, 3…Database device, 4…Evaluation terminal, 5…Accuracy maintenance management device, 6…Learning device, 7…Display device, 11…Reception screen, 21…Acquisition unit, 22…Inference unit, 23…Output unit, 31…Input data storage device, 32…Model storage device, 33…Correct label storage device, 34…Inference result storage device, 35…Evaluation result storage device, 41…Display unit, 42…Evaluation unit, 51…Index calculation unit, 52…Degradation judgment unit, 53…Degradation notification unit, 61…Label input section, 62…Learning section, 71…Display area, 101…Date and time search field, 102…Item ID search field, 103…Judgment result search field, 104…Registration result search field, 105…Judgment result display field, 106…Image display field, 107…Annotation progress display field, 108…Registration result display field, 109…Registration selection field, 110…Registration result selection field, 111…Confirmation image, 201…Search operation key, 202…Download operation key, 203…Upload operation key

Claims

1. A display unit that displays a reception screen on a display device for receiving input operations for inference results output from a trained model into which explanatory variables have been input, or for evaluation results for said explanatory variables, A degradation determination unit determines whether or not the accuracy of the trained model has deteriorated based on the evaluation results received on the reception screen. A precision maintenance management system equipped with the following features.

2. The accuracy maintenance management system according to claim 1, wherein the reception screen includes an input field for the evaluation results and a display field for the explanatory variables.

3. The accuracy maintenance management system according to claim 2, wherein the reception screen further includes a field for displaying the inference results.

4. The system further includes an index calculation unit that calculates an evaluation index representing the accuracy of the trained model based on the evaluation results received on the reception screen, The accuracy maintenance management system according to any one of claims 1 to 3, wherein the degradation determination unit determines whether or not the accuracy of the trained model has deteriorated based on the evaluation index.

5. The accuracy maintenance management system according to claim 4, wherein the degradation determination unit determines whether or not the accuracy of the trained model has deteriorated based on the change in the evaluation index.

6. The accuracy maintenance management system according to any one of claims 1 to 3, wherein the trained model outputs the inference result using the image as the explanatory variable.

7. The accuracy maintenance management system according to any one of claims 1 to 3, further comprising a degradation notification unit that notifies the system of the degradation of the accuracy of the trained model when it is determined that the accuracy of the trained model has deteriorated.

8. The aforementioned evaluation index is the accuracy rate of the inference result, The accuracy maintenance management system according to claim 4, wherein the degradation determination unit determines that the accuracy of the trained model has deteriorated when the accuracy rate decreases.

9. The accuracy maintenance management system according to claim 8, wherein the degradation determination unit determines that the accuracy of the trained model has deteriorated if the accuracy rate decreases despite the confidence level of the inference result being maintained at or above a confidence threshold in the trained model.

10. The accuracy maintenance management system according to any one of claims 1 to 3, wherein the reception screen accepts annotation operations for correct labels used to retrain the trained model.

11. The accuracy maintenance management system according to any one of claims 1 to 3, further comprising a learning unit that, when notified that the accuracy of the trained model has deteriorated, performs retraining of the trained model using predetermined retraining images as explanatory variables and the correct labels received through annotation operations as new target variables.

12. A method for maintaining accuracy performed by an accuracy maintenance management system, The steps include displaying on a display device a reception screen that accepts input operations for inference results output from a trained model into which explanatory variables have been input, or for evaluation results for said explanatory variables, A step of determining whether the accuracy of the trained model has deteriorated based on the evaluation results received on the reception screen. A method for maintaining accuracy, including the following:

13. On the computer, A procedure for displaying a screen on a display device that accepts input operations for inference results output from a trained model into which explanatory variables have been input, or for evaluation results for said explanatory variables, A procedure for determining whether the accuracy of the trained model has deteriorated based on the evaluation results received on the reception screen, and A program to execute.

Images