Inspection system, method, and program

Abstract

To prevent a possible erroneous determination that a machine learning model has deteriorated even if the machine learning model has not deteriorated.SOLUTION: An inspection system performs an inspection using a machine learning model. The machine learning model is a learned machine learning model having performed machine learning for predicting a predetermined event for an inspection target. An inspection unit repeatedly executes prediction about the inspection target using the machine learning model upon receipt of data about the inspection target as input data, and outputting prediction result data indicating a predicted result. An accumulation unit accumulates the prediction result data output from the inspection unit. An evaluation unit evaluates whether or not the input data falls within a range assumed at the time of learning of the machine learning model from the tendency of time series variation of the result predicted for the inspection target indicated by the prediction result data accumulated in the accumulation unit, and outputs an evaluation result.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an inspection system, method, and program. [Background technology]

[0002] In recent years, from the viewpoint of labor saving, machine learning models are used for quality control in the field of manufacturing. For example, a computer in which a machine learning model is built predicts whether a product satisfies a predetermined inspection standard. Patent Document 1 also describes that a computer in which a machine learning model is built predicts whether the quality of a product that has undergone a manufacturing process satisfies a predetermined condition based on input equipment data in the manufacturing process.

[0003] It is known that machine learning models deteriorate with continued use. When a machine learning model deteriorates, the prediction accuracy decreases. For this reason, Patent Document 1 describes that when the prediction accuracy of a machine learning model changes, it is determined that the machine learning model has deteriorated, and a new machine learning model is generated. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-87101 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, even if the machine learning model is not actually degraded, the prediction accuracy of the machine learning model may decrease due to other conditions. Therefore, as described in Patent Document 1, if a change in the prediction accuracy of the machine learning model is uniformly regarded as a deterioration of the machine learning model, there is a problem that the machine learning model may be erroneously determined to have deteriorated even if it has not deteriorated. [Means for solving the problem]

[0006] The present disclosure can be realized in the following forms.

[0007] (1) According to an embodiment of the present disclosure, an inspection system is provided that performs inspection using a machine learning model. The machine learning model is a trained machine learning model that has undergone machine learning to predict a predetermined event for an inspection target. The inspection system includes an inspection unit that repeatedly performs a prediction for the inspection target using the machine learning model with data for the inspection target as input data, and outputs prediction result data indicating the predicted result, a storage unit that stores the prediction result data output by the inspection unit, and an evaluation unit that evaluates whether the input data is within a range assumed at the time of learning the machine learning model based on a trend of time-series change in the predicted result for the inspection target indicated by the prediction result data stored in the storage unit, and outputs an evaluation result. According to this aspect, when determining whether or not the machine learning model has deteriorated, not only the prediction accuracy of the machine learning model but also other conditions, such as whether or not the input data is within an expected range, can be taken into consideration. Therefore, even if the machine learning model has not actually deteriorated, it is possible to prevent a mistaken determination that the machine learning model has deteriorated when the prediction accuracy of the machine learning model has decreased due to other conditions. (2) In the inspection system of the above aspect, the machine learning model may be one that has undergone machine learning to predict which of a plurality of preset classifications the inspection object will be classified into. The inspection unit predicts a predicted probability indicating the possibility that the inspection object will be classified into each of the plurality of classifications. The inspection unit determines that the inspection object will be classified into the classification with the highest predicted probability value. The inspection unit outputs prediction result data including a predicted predicted probability for the classification determined for the inspection object. The storage unit stores the prediction result data output by the inspection unit together with time information indicating when the prediction was made for the inspection object. The evaluation unit may evaluate whether or not the input data is within a range assumed at the time of learning the machine learning model, based on a trend of time-series change in the predicted predicted probability for the classification determined for the inspection object, which is indicated by the prediction result data stored in the storage unit. According to this aspect, it is possible to evaluate whether the input data is within the range assumed when training the machine learning model based on the trend of time-series changes in the predicted prediction probability for the classification determined for the test subject. (3) In the inspection system of the above aspect, the machine learning model may be one that has undergone machine learning to predict whether or not the inspection object is within a normal range. The inspection unit predicts whether or not the inspection object is within a normal range. The inspection unit outputs prediction result data including a value representing the ratio of the number of inspection objects predicted to be within the normal range to the total number of inspection objects inspected in a set period. The storage unit stores the prediction result data output by the inspection unit together with time information indicating when the prediction was made for the inspection object. The evaluation unit may evaluate whether or not the input data is within a range assumed at the time of learning the machine learning model, based on the trend of time-series changes in the ratio indicated by the prediction result data stored in the storage unit. According to this aspect, it is possible to evaluate whether the input data is within the range assumed when training the machine learning model from the trend of time-series changes in the ratio of the number of test subjects predicted to be within the normal range to the total number of test subjects tested. (4) In the inspection system of the above aspect, the input data may be image data representing the appearance of the inspection target acquired by a sensor. According to this aspect, the inspection system can perform visual inspection of the inspection target using the image data, and can evaluate whether the input data is within the range assumed at the time of learning the machine learning model based on the trend of time-series changes in the prediction result predicted by the machine learning model. (5) The input data may be voice data representing the operating state of the inspection target obtained by a sensor. According to this aspect, the inspection system can inspect the inspection target for abnormal sounds, and can evaluate whether the input data is within the range assumed at the time of learning the machine learning model, based on the tendency of time-series changes in the prediction results predicted by the machine learning model. The present disclosure may be realized in various forms other than an inspection system, for example, a method executed by a computer that performs an inspection using a machine learning model, or a computer program that realizes the method. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing a functional configuration of an inspection system according to a first embodiment. [Diagram 2] 1 is a diagram illustrating a hardware configuration of an inspection device according to a first embodiment. [Diagram 3] FIG. 2 is a diagram illustrating a hardware configuration of an evaluation device according to the first embodiment. [Figure 4] 4 is a diagram showing an example of data stored in a storage unit according to the first embodiment; FIG. [Diagram 5] 4 is a flowchart of a classification process according to the first embodiment. [Figure 6] 4 is a flowchart of an evaluation process according to the first embodiment. [Figure 7] FIG. 11 is a diagram showing a functional configuration of an inspection system according to a second embodiment. [Figure 8] FIG. 11 is a diagram showing an example of data stored in a storage unit according to the second embodiment. [Figure 9] 10 is a flowchart of an inspection process according to the second embodiment. [Figure 10] 10 is a flowchart of an evaluation process according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A. First embodiment 1 is a diagram showing a functional configuration of an inspection system 1000 according to a first embodiment. The inspection system 1000 includes an inspection device 100 and an evaluation device 200. The inspection device 100 and the evaluation device 200 are different computers installed in a server room in a factory.

[0010] A sensor 50 is connected to the inspection device 100 via a network. In this embodiment, an example in which the sensor 50 is a camera will be described. The sensor 50 photographs a part to be inspected by the inspection device 100 at a predetermined position on a production line. The inspection object is, for example, an automobile part such as a body panel, an engine cylinder block, or a gear incorporated in a transmission. In this embodiment, the inspection object is two types of parts, part A1 and part A2. Therefore, the object photographed by the sensor 50 is either part A1 or A2. The sensor 50 transmits image data of the photograph of either part A1 or A2 to the inspection device 100.

[0011] The inspection device 100 uses image data supplied from the sensor 50 as input data, and judges the classification and pass / fail of the inspection object using a machine learning model. In this embodiment, the machine learning model is a trained model that has undergone machine learning to predict a predetermined event for the inspection object. The evaluation device 200 evaluates the result of the judgment made by the inspection device 100 for the inspection object.

[0012] 2 is a diagram showing a hardware configuration of the inspection device 100. The inspection device 100 includes a processor 11, a main memory unit 12, an auxiliary memory unit 13, an interface 14, and a communication unit 15. The main memory unit 12, the auxiliary memory unit 13, the interface 14, and the communication unit 15 are connected to the processor 11 via an internal bus 19.

[0013] The processor 11 is, for example, a CPU (Central Processing Unit). The processor 11 loads a program stored in the auxiliary storage unit 13 into the main storage unit 12 and executes the program, thereby implementing each function of the inspection device 100.

[0014] The main memory unit 12 is, for example, a main storage device including a RAM (Random Access Memory). Programs are loaded into the main memory unit 12 from the auxiliary memory unit 13. The main memory unit 12 is also used as a working area when the processor 11 executes the programs.

[0015] The auxiliary storage unit 13 is, for example, a hard disk drive (HDD), a solid state drive (SSD), or a solid state hybrid drive (SSHD). The auxiliary storage unit 13 stores programs executed by the processor 11 and various data used to execute the programs. For example, the auxiliary storage unit 13 stores a program for the inspection device 100 to inspect an inspection target. Under the control of the processor 11, the auxiliary storage unit 13 supplies the processor 11 with data used by the processor 11 and stores the data supplied from the processor 11.

[0016] The interface 14 is, for example, an I / O (Input / Output) port such as a serial port or a USB (Universal Serial Bus) port. Input devices such as a keyboard, a mouse, and a touch panel are connected to the interface 14. The interface 14 outputs information input by a user via the input device to the processor 11. A display device 61 is also connected to the interface 14. The display device 61 is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display. The interface 14 outputs an image supplied from the processor 11 to the display device 61.

[0017] The communication unit 15 includes a network interface circuit for communicating with an external device. The communication unit 15 receives a signal from the outside and outputs data indicated by the signal to the processor 11. The communication unit 15 also transmits a signal indicating the data output from the processor 11 to the external device.

[0018] 3 is a diagram showing a hardware configuration of the evaluation device 200. The evaluation device 200 includes a processor 21, a main memory unit 22, an auxiliary memory unit 23, an interface 24, and a communication unit 25. The main memory unit 22, the auxiliary memory unit 23, the interface 24, and the communication unit 25 are connected to the processor 21 via an internal bus 29.

[0019] The processor 21 has the same functions as the processor 11 of the inspection device 100. The main memory unit 22 has the same functions as the main memory unit 12 of the inspection device 100. The auxiliary memory unit 23 has the same functions as the auxiliary memory unit 13 of the inspection device 100. The interface 24 has the same functions as the interface 14 of the inspection device 100. A display device 62 is connected to the interface 24. The display device 62 is, for example, a liquid crystal display or an organic EL display. The interface 24 outputs an image supplied from the processor 21 to the display device 62. The communication unit 25 has the same functions as the communication unit 15 of the inspection device 100.

[0020] Next, the function of the inspection device 100 will be described with reference to Fig. 1. The inspection device 100 includes a data acquisition unit 110, a classification unit 120, and an inspection unit .

[0021] When the data acquisition unit 110 receives image data of an object to be inspected from the sensor 50, the data acquisition unit 110 stores the image data in a determined area. The data acquisition unit 110 transmits a path of a storage destination of the image data to the classification unit 120. The function of the data acquisition unit 110 is realized by the processor 11 executing a program for processing related to data acquisition.

[0022] The classification unit 120 uses a trained machine learning model to determine which of two types of parts the inspection target corresponds to. The machine learning model is generated by any machine learning method. Examples of machine learning techniques include random forest and SVM (support vector machine). The machine learning model used by the classification unit 120 is generated by machine learning for predicting whether the inspection target corresponds to part A1 or part A2. This machine learning is performed using a large number of image data obtained by photographing parts A1 and A2 as training data.

[0023] The reason why a large number of image data are used as the training data is as follows. In the manufacturing line, the sensor 50 captures the parts transported by the conveyor. The position where the sensor 50 is installed is fixed. On the other hand, the position where the parts are arranged on the conveyor is not always the same. For example, when comparing multiple images showing the same type of parts, the position and inclination of the parts in each image may differ. Therefore, it is preferable that the image data used as the training data is image data with various variations. For example, images taken while shifting the position where the parts are arranged little by little and images taken while rotating the parts little by little may be included. By using image data with various variations as the training data, the machine learning model can correctly determine the type of the parts to be inspected.

[0024] In this embodiment, the prediction accuracy of the machine learning model used in the classification unit 120 is assumed to meet a predetermined standard. For example, it is a machine learning model that has been used for a short period of time since its generation and whose index value for evaluating the prediction accuracy meets a predetermined standard.

[0025] The classification unit 120 uses the image data of the inspection target as input data and utilizes a trained machine learning model to predict the probability that the inspection target corresponds to each of the parts A1 and A2. Hereinafter, this probability may be referred to as a predicted probability.

[0026] Furthermore, the classification unit 120 determines whether the inspection target is part A1 or part A2 based on the predicted probability. For example, it is assumed that the classification unit 120 predicts that the probability that the inspection target corresponds to part A1 is 70 percent and the probability that the inspection target corresponds to part A2 is 30 percent. In this case, the classification unit 120 determines that the inspection target is part A1. It is also assumed that the classification unit 120 predicts that the probability that the inspection target corresponds to part A1 is 15 percent and the probability that the inspection target corresponds to part A2 is 85 percent. In this case, the classification unit 120 determines that the inspection target is part A2. The classification unit 120 notifies the evaluation device 200 of the results of the prediction and determination.

[0027] Moreover, the classification unit 120 outputs the determination result as to which part the inspection target corresponds to and the path of the storage destination of the image data to the inspection unit 130. The function of the classification unit 120 is realized by the processor 11, for example, by executing application software in which a trained machine learning model is implemented. The classification unit 120 is also called an inspection unit.

[0028] The inspection unit 130 performs a visual inspection of the inspection target using the trained machine learning model. The visual inspection is an inspection to check for defects in appearance such as foreign matter attached to the surface of the component and deformation of the component.

[0029] The machine learning model used by the inspection unit 130 is generated by machine learning to predict whether an inspection target is a pass or fail product. This machine learning is performed using image data of parts associated with a judgment value of pass or fail as training data. The function of the inspection unit 130 is realized by the processor 11, for example, by executing application software in which a learned machine learning model is implemented. Note that the machine learning model used by the inspection unit 130 and the machine learning model used by the classification unit 120 are different.

[0030] The inspection unit 130 receives as input data the image data and the result of the determination by the classification unit 120 as to which part the inspection target corresponds to, and predicts the probability that the part shown in the image data corresponds to a good product and a defective product. In this embodiment, the inspection unit 130 determines that the inspection target is a good product if the probability that it is a good product is higher than the probability that it is a defective product. The inspection unit 130 determines that the inspection target is a defective product if the probability that it is a defective product is higher than the probability that it is a good product.

[0031] For example, it is assumed that the inspection unit 130 predicts that the part A1 to be inspected has an 80 percent probability of being a good product and a 20 percent probability of being a defective product. In this case, the inspection unit 130 judges that the inspection target is a good product. The inspection unit 130 outputs to the display device 61 an image showing the probability that the inspection target corresponds to a good product and a defective product, respectively, and the judgment result that the inspection target is a good product or a defective product.

[0032] In this embodiment, it is assumed that the prediction accuracy of the machine learning model used in the inspection unit 130 meets a predetermined level. In this embodiment, the evaluation device 200 does not evaluate the judgment result of the inspection unit 130. Therefore, the inspection unit 130 does not output the judgment result to the evaluation device 200.

[0033] Next, a description will be given of the function of the evaluation device 200. The evaluation device 200 includes a storage unit 210 and an evaluation unit 220. In this embodiment, an example will be described in which the evaluation device 200 evaluates the prediction result by the classification unit 120 of the inspection device 100.

[0034] 4 is a diagram showing an example of data stored in the storage unit 210. The storage unit 210 stores data supplied from the classification unit 120. The data supplied from the classification unit 120 to the storage unit 210 includes a date and time, a determination result as to which part the inspection target corresponds to, a predicted probability that the inspection target corresponds to each part, and a path to a storage destination of image data. The function of the storage unit 210 is realized by the processor 11 executing software that stores and reads data on request, such as a database management system.

[0035] The evaluation unit 220 shown in FIG. 1 evaluates whether or not data input to the machine learning model used in the classification unit 120 during a specified evaluation period is within the range assumed when the machine learning model was trained.

[0036] In this embodiment, the evaluation unit 220 detects anomalies by detecting whether outliers are included in the time series data of the predicted probabilities. For example, the evaluation unit 220 detects outliers included in the time series data of the predicted probabilities using an existing method for outlier detection.

[0037] When the evaluation unit 220 detects that the time series data of the prediction probability includes an outlier, it determines that the data input to the machine learning model is not within the range assumed when the machine learning model was trained. As described above, in this embodiment, the prediction accuracy of the machine learning model meets a predetermined level. That is, the machine learning model has not deteriorated. In addition, when input data is input, the machine learning model outputs result data regarding the prediction. Therefore, when an abnormality is detected in the trend of time series changes in the output result data, it is considered that the input data was data that was not within the assumed range. The function of the evaluation unit 220 is realized by the processor 21 executing a program for processing related to the evaluation.

[0038] According to the above-described aspect, when the test object is determined to be one of the classifications, the evaluation device 200 can evaluate whether or not the input data is within the range assumed at the time of learning the machine learning model, based on the tendency of the time-series change in the predicted probability predicted for that classification.

[0039] 5 is a flowchart of a classification process in which the inspection device 100 classifies parts based on image data. Every time the inspection device 100 receives image data from the sensor 50, it executes the following classification process.

[0040] In step S101, the data acquisition unit 110 receives image data from the sensor 50. The data acquisition unit 110 performs preprocessing such as noise removal and binarization on the received image data. The data acquisition unit 110 stores the preprocessed image data in a predetermined area in the auxiliary storage unit 13 of the inspection device 100. The data acquisition unit 110 transmits to the classification unit 120 the path of the storage destination of the preprocessed image data.

[0041] In step S102, first, the classification unit 120 reads out image data stored in the path notified by the data acquisition unit 110. The classification unit 120 uses the image data as input data and predicts the predicted probability that the part to be inspected shown in the image corresponds to each of the two types of parts. In step S103, the classification unit 120 determines which part the inspection target is based on the predicted predicted probability. Specifically, the classification unit 120 determines that the inspection target corresponds to the part with the higher predicted probability value.

[0042] In step S104, the classification unit 120 transmits data to the evaluation device 200, including the predicted probability that the inspection target corresponds to each part, the determination result as to which part the inspection target corresponds to, the path to the storage destination of the image data, and date and time information. This data is also called prediction result data. The date and time indicate when the classification unit 120 made the prediction. The date and time are also called time information. The above is a series of flows of the classification process.

[0043] 6 is a flowchart of the evaluation process by the evaluation device 200. Every time the evaluation device 200 receives prediction result data from the inspection device 100, it executes the following evaluation process.

[0044] In step S111, the storage unit 210 receives data including the predicted probability that the inspection target corresponds to each part, the determination result as to which part the inspection target corresponds to, the path of the storage destination of the image data, and the date and time from the classification unit 120. In step S112, the storage unit 210 stores the data received from the classification unit 120.

[0045] In step S113, when it is time to evaluate the input data (step S113; YES), the evaluation unit 220 executes the process of step S114. On the other hand, in step S113, when it is not time to evaluate the input data (step S113; YES), the accumulation unit 210 executes the process of step S211 again.

[0046] In step S114, the evaluation unit 220 detects outliers in the time series data of the predicted probability. First, the evaluation unit 220 reads out from the storage unit 210 the values ​​of the predicted probability of the inspection object predicted to correspond to the part A1 in the past week, sorted by date and time. Next, the evaluation unit 220 reads out from the storage unit 210 the values ​​of the predicted probability of the inspection object predicted to correspond to the part A2 in the past week, sorted by date and time. The evaluation unit 220 determines whether or not an outlier is included in the time series data of the predicted probability of the inspection object predicted to correspond to the part A1 in the past week. Furthermore, the evaluation unit 220 determines whether or not an outlier is included in the time series data of the predicted probability of the inspection object predicted to correspond to the part A2 in the past week. When the evaluation unit 220 detects that the time-series data of the predicted probability contains an outlier, it determines that the input data to the machine learning model is not within the range assumed when the machine learning model was trained.

[0047] In step S115, the evaluation unit 220 presents the evaluation result for the input data based on the outlier detection result. For example, the evaluation unit 220 outputs to the display device 62 an image indicating that there was input data that was not within the expected range, the value of the predicted probability detected as the outlier, the date and time when the predicted probability value was predicted, and a path indicating the storage destination of the input data used. This allows the user to know that data that is not within the range expected when the machine learning model was trained was input to the machine learning model. This is the flow of the evaluation process.

[0048] As described above, in the first embodiment, it is evaluated whether the input data is within the range assumed at the time of learning the machine learning model based on the trend of time-series changes in the predicted probability that the test object corresponds to the classification.

[0049] In this way, whether or not the input data is within the range expected at the time of learning the machine learning model is evaluated based on the trend of time-series changes in the prediction results predicted by the machine learning model. In this way, when determining whether or not the machine learning model has deteriorated, not only the prediction accuracy of the machine learning model but also other conditions, such as whether or not the input data is within the expected range, can be taken into consideration. Therefore, even if the machine learning model has not actually deteriorated, it is possible to prevent erroneous determination that the machine learning model has deteriorated when the prediction accuracy of the machine learning model has decreased due to other conditions.

[0050] B. Second embodiment 7 is a diagram showing the functional configuration of an inspection system 1001 according to the second embodiment. The following description will focus on configurations different from those of the first embodiment. The inspection system 1001 includes an inspection device 100 and an evaluation device 200.

[0051] The inspection device 100 performs an abnormal sound inspection. The inspection target of the inspection device 100 is, for example, an automobile engine. In this embodiment, the sensor 50 is a microphone. The sensor 50 is connected to the inspection device 100 via a network. The sensor 50 collects the sound of the automobile engine before shipment at a predetermined position on the production line. In this embodiment, audio data indicating the operating state of the engine to be inspected becomes input data for the abnormal sound inspection. The sensor 50 transmits the collected audio data to the inspection device 100.

[0052] The inspection device 100 includes a data acquisition unit 110 and an inspection unit 130. It is to be noted that the inspection device 100 does not include a classification unit 120 in the second embodiment.

[0053] When the data acquisition unit 110 receives engine sound audio data from the sensor 50, the data acquisition unit 110 stores the audio data in a determined area. The data acquisition unit 110 transmits to the inspection unit 130 the path of the destination where the audio data is stored.

[0054] The inspection unit 130 predicts whether the inspection target is normal or not by using a trained machine learning model. The machine learning model used by the inspection unit 130 is generated by machine learning for predicting whether the inspection target is normal or not. This machine learning uses a large amount of voice data that collects normal engine sounds as training data. Many machines are in operation at the manufacturing site, and sounds emitted from these machines are also included in the voice data that is the inspection target. Therefore, it is preferable that the voice data used as training data is voice data with various variations. For example, it may include voice data that collects normal engine sounds in an environment where surrounding noise is blocked, and voice data that collects normal engine sounds in an environment where there is surrounding noise. By using voice data with various variations as training data, the machine learning model can correctly determine whether the inspection target is good or bad.

[0055] The inspection unit 130 uses the audio data supplied from the data acquisition unit 110 as input data to predict the degree of abnormality, which is an index showing how far the engine sound being inspected is from a normal engine sound. In this embodiment, the degree of abnormality is an index showing that the inspection object is closer to an abnormal state the higher the numerical value. Furthermore, the inspection unit 130 judges whether the inspection object is normal or not. When the calculated degree of abnormality is equal to or less than a preset threshold, the inspection unit 130 judges that the inspection object is normal. On the other hand, when the calculated degree of abnormality exceeds a preset threshold, the inspection unit 130 judges that the inspection object is abnormal.

[0056] Furthermore, the inspection unit 130 counts the number of inspections performed during the set period and the number of inspection subjects determined to be normal during that period. The set period is, for example, one day.

[0057] The inspection unit 130 calculates the ratio of the number of test subjects determined to be normal to the number of tests performed during a set period at a set timing. For example, after all tests for a day are completed, the inspection unit 130 calculates the ratio of the number of test subjects determined to be normal to the total number of test subjects performed on that day. Hereinafter, this ratio may be referred to as the ratio of normal determinations. The classification unit 120 notifies the evaluation device 200 of the calculated ratio. The function of the inspection unit 130 is realized by the processor 11, for example, by executing application software in which a trained machine learning model is implemented.

[0058] The evaluation device 200 includes a storage section 210 and an evaluation section 220 .

[0059] 8 is a diagram showing an example of data stored in the storage unit 210. The storage unit 210 stores data supplied from the inspection unit 130. The data supplied from the inspection unit 130 to the storage unit 210 includes a date and a proportion determined to be normal.

[0060] 7, the evaluation unit 220 evaluates whether or not data input to the machine learning model used by the inspection unit 130 is within a range assumed at the time of learning of the machine learning model. The evaluation unit 220 evaluates data input to the machine learning model used by the inspection unit 130 during a specified evaluation period. The evaluation period is, for example, the past month.

[0061] First, the evaluation unit 220 reads out the values ​​of the ratio of normal judgments for the period to be evaluated sorted by date from the storage unit 210. Therefore, the data read out from the storage unit 210 is time-series data of the ratio of normal judgments.

[0062] Next, the evaluation unit 220 detects an abnormality in the trend of time-series changes in the proportion of normal judgments based on the time-series data of the proportion of normal judgments.

[0063] In this embodiment, the evaluation unit 220 detects anomalies based on the presence or absence of outliers. For example, the evaluation unit 220 uses an existing method for outlier detection to determine whether or not an outlier is included in the time-series data of the proportion of cases determined to be normal in the past month.

[0064] If the evaluation unit 220 detects that the time series data of the proportion of normal judgments contains an outlier, it determines that the input data to the machine learning model is not within the range assumed when the machine learning model was trained.

[0065] According to the above-described aspect, the evaluation device 200 can evaluate whether or not the input data is within the range assumed at the time of learning the machine learning model, based on the tendency of time-series change in the ratio of the number of test subjects predicted to be within the normal range to the total number of test subjects that were tested.

[0066] 9 is a flowchart of an inspection process in which the inspection device 100 inspects an inspection target from image data. Every time the inspection device 100 receives voice data from the sensor 50, it executes the following inspection process.

[0067] In step S201, the data acquisition unit 110 of the inspection device 100 receives voice data from the sensor 50. The data acquisition unit 110 performs preprocessing on the acquired voice data. The data acquisition unit 110 stores the preprocessed image data in a determined area in the auxiliary storage unit 13 of the inspection device 100. The data acquisition unit 110 transmits to the inspection unit 130 the path of the storage destination of the preprocessed image data.

[0068] In step S202, the inspection unit 130 judges whether the test object is normal or not. Specifically, first, the inspection unit 130 predicts the degree of abnormality of the test object using the voice data supplied from the data acquisition unit 110 as input data. Next, the inspection unit 130 judges whether the test object is normal or not based on the value of the degree of abnormality. The inspection unit 130 outputs an image displaying the judgment result of whether the test object corresponds to either normal or abnormal and the value indicating the degree of abnormality to the display device 61. Furthermore, the inspection unit 130 counts the number of tests performed during a set period and the number of test objects judged to be normal during that period.

[0069] In step S203, when it is time to calculate the proportion of test subjects determined to be normal (step S203; YES), the test unit 130 executes the process of step S204. On the other hand, in step S203, when it is not time to calculate the proportion of test subjects determined to be normal (step S203; NO), the test unit 130 executes the process of step S201 again.

[0070] In step S204, the inspection unit 130 calculates the ratio of the number of test objects determined to be normal to the total number of test objects tested during the set period. In step S205, the inspection unit 130 transmits data including the calculated ratio of test objects determined to be normal and a date to the evaluation device 200. This data is also called prediction result data. The date indicates when the inspection unit 130 predicted whether or not the test objects were normal. The date is also called time information.

[0071] 10 is a flowchart of the evaluation process by the evaluation device 200. Every time the evaluation device 200 receives prediction result data from the inspection device 100, the evaluation device 200 executes the following evaluation process.

[0072] In step S211, the storage unit 210 receives data including the proportion of test objects determined to be normal and the date from the inspection unit 130. In step S212, the storage unit 210 stores the received data.

[0073] In step S213, when it is time to evaluate the input data (step S213; YES), the evaluation unit 220 executes the process of step S214. On the other hand, in step S213, when it is not time to evaluate the input data (step S213; NO), the accumulation unit 210 executes the process of step S211 again.

[0074] In step S214, the evaluation unit 220 detects outliers in the time series data of the proportion of normal judgments. First, the evaluation unit 220 reads out the value of the proportion of normal judgments in the past month from the accumulation unit 210. The evaluation unit 220 determines whether or not the read time series data includes an outlier.

[0075] In step S215, the evaluation unit 220 presents the evaluation result for the input data based on the outlier detection result. When the evaluation unit 220 detects that an outlier is included in the time series data of the normal judgment ratio, it outputs to the display device 62 an image indicating that there was input data that was not within the expected range, the value of the normal judgment ratio detected as the outlier, and the examination date. This allows the user to know that data that was not within the expected range when the machine learning model was trained was input to the machine learning model. This is the flow of the evaluation process.

[0076] In the second embodiment, when inspecting for abnormal sounds using voice data, it is possible to evaluate whether or not the input data is within the range assumed at the time of learning the machine learning model, based on the tendency of time-series changes in the prediction results predicted by the machine learning model.

[0077] In this way, whether or not the input data is within the range expected at the time of learning the machine learning model is evaluated based on the trend of time-series changes in the prediction results predicted by the machine learning model. In this way, when determining whether or not the machine learning model has deteriorated, not only the prediction accuracy of the machine learning model but also other conditions, such as whether or not the input data is within the expected range, can be taken into consideration. Therefore, even if the machine learning model has not actually deteriorated, it is possible to prevent erroneous determination that the machine learning model has deteriorated when the prediction accuracy of the machine learning model has decreased due to other conditions.

[0078] C. Other Embodiments (C1) In the first embodiment, an example has been described in which the evaluation device 200 evaluates the result of the classification process by the classification unit 120 of the inspection device 100. However, the evaluation device 200 may not only evaluate the result of the classification process by the classification unit 120, but also evaluate the result of the inspection process by the inspection unit 130 in a similar manner.

[0079] As described above, the inspection unit 130 performs an appearance inspection of the inspection object using the trained machine learning model. For example, the inspection unit 130 predicts the probability that the inspection object is determined to be a good product and the probability that the inspection object is determined to be a defective product. The inspection unit 130 determines that the inspection object is a good product when the value of the probability that the inspection object is determined to be a good product exceeds a threshold. The inspection unit 130 determines that the inspection object is a defective product when the value of the probability that the inspection object is determined to be a defective product exceeds a threshold. The inspection unit 130 transmits the value of the probability that the inspection object determined to be a good product is determined to be a good product to the evaluation device 200. Furthermore, the inspection unit 130 transmits the value of the probability that the inspection object determined to be a defective product is determined to be a defective product to the evaluation device 200. The evaluation device 200 may determine whether the input data is within a range expected at the time of learning the machine learning model from the trend of change in the time series data of the value of the probability that the inspection object is determined to be a good product and the trend of change in the time series data of the value of the probability that the inspection object is determined to be a defective product.

[0080] According to this aspect, when performing visual inspection using image data, it is possible to evaluate whether the input data is within the range assumed when the machine learning model was trained, based on the trend of time-series changes in the prediction results predicted by the machine learning model.

[0081] (C2) In the first embodiment, the prediction result data transmitted by the classification unit 120 to the evaluation device 200 included the date and time when the classification unit 120 made the prediction. In the second embodiment, the prediction result data transmitted by the inspection unit 130 to the evaluation device 200 included the date when the inspection unit 130 made the prediction. Thus, in the first and second embodiments, the side transmitting the prediction result data acquired the time information. However, the side receiving the prediction result data may also acquire the time information.

[0082] For example, the classification unit 120 in the inspection system 1000 does not need to include the date and time in the prediction result data to be transmitted to the evaluation device 200. In this case, the classification unit 120 transmits the prediction result data to the evaluation device 200 immediately after determining which part the inspection target corresponds to. When the storage unit 210 receives the prediction result data from the classification unit 120, it acquires the date and time when the data was received. The storage unit 210 stores the data received from the classification unit 120 together with information indicating the date and time.

[0083] Since the classification unit 120 transmits the prediction result data to the evaluation device 200 immediately after the judgment, the date and time stored in the storage unit 210 can be regarded as time information indicating when the prediction was made for the test object. Therefore, the data read from the storage unit 210 by the evaluation unit 220 is time-series data of the prediction probability.

[0084] The inspection unit 130 in the inspection system 1001 does not need to include the date and time in the prediction result data to be transmitted to the evaluation device 200. In this case, the inspection unit 130 calculates the ratio of the number of test objects determined to be normal to the number of tests performed in a set period, and then immediately transmits the prediction result data to the evaluation device 200. When the storage unit 210 receives the prediction result data from the inspection unit 130, it acquires the date on which the data was received, and stores the received data together with information indicating the date.

[0085] Since the inspection unit 130 transmits the prediction result data to the evaluation device 200 immediately after calculating the ratio, the date stored in the storage unit 210 can be regarded as time information indicating when the prediction was made for the inspection target. Therefore, the data read from the storage unit 210 by the evaluation unit 220 is time-series data of the ratio of normal judgments.

[0086] Alternatively, neither the side that transmits the prediction result data nor the side that receives the prediction result data may acquire time information. The storage unit 210 stores the prediction result data in the order in which the prediction result data is received. The data read from the storage unit 210 by the evaluation unit 220 can be said to be time-series data of the ratio of normal judgments.

[0087] (C3) In the first embodiment, the classification unit 120 has been described as determining whether the inspection object is one of two types of parts. The classification unit 120 may determine whether the inspection object is one of three or more types of parts. In this case, the machine learning model used by the classification unit 120 is a model that has undergone memory machine learning to predict which of a plurality of preset classifications the inspection object will be classified into.

[0088] (C4) In the first and second embodiments, the inspection systems 1000 and 1001 are configured to include the inspection device 100 and the evaluation device 200. However, the functions of the inspection device 100 and the evaluation device 200 may be realized by one computer.

[0089] (C5) In the first embodiment, an example has been described in which the data acquisition unit 110 transmits a path to a storage destination of the image data on which preprocessing has been performed to the classification unit 120. Alternatively, the data acquisition unit 110 may not perform preprocessing. In this case, the data acquisition unit 110 may store the image data received from the sensor 50 in a determined area in the auxiliary storage unit 13 of the inspection device 100. Alternatively, the data acquisition unit 110 may transmit image data on which preprocessing has been performed or image data on which preprocessing has not been performed to the classification unit 120.

[0090] In the second embodiment, an example has been described in which the data acquiring unit 110 transmits a path to a storage destination of the preprocessed voice data to the inspection unit 130. Alternatively, the data acquiring unit 110 may not perform preprocessing. In this case, the data acquiring unit 110 may store the voice data received from the sensor 50 in a determined area in the auxiliary storage unit 13 of the inspection device 100. Alternatively, the data acquiring unit 110 may transmit the preprocessed voice data or the unpreprocessed voice data to the inspection unit 130.

[0091] As a recording medium for recording a program that realizes the functions of the above-mentioned inspection device 100 and evaluation device 200, a computer-readable recording medium including a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, a semiconductor memory, and a magnetic tape can be used.

[0092] Furthermore, the means for realizing the functions of the inspection device 100 and the evaluation device 200 are not limited to software, and may be partly or entirely realized by dedicated hardware. For example, the dedicated hardware may be a circuit such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

[0093] The present disclosure is not limited to the above-mentioned embodiment, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features in the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention column can be appropriately replaced or combined to solve some or all of the above-mentioned problems or to achieve some or all of the above-mentioned effects. Furthermore, if the technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0094] 11, 21...processor, 12, 22...main memory unit, 13, 23...auxiliary memory unit, 14, 24...interface, 15, 25...communication unit, 19, 29...internal bus, 50...sensor, 61, 62...display unit, 100...inspection device, 110...data acquisition unit, 120...classification unit, 130...inspection unit, 200...evaluation device, 210...storage unit, 220...evaluation unit, 1000, 1001...inspection system, A1, A2...component

Claims

1. An inspection system that performs inspection using a machine learning model, The machine learning model is a trained machine learning model that has undergone machine learning to predict a predetermined event regarding an object to be inspected, An inspection unit that repeatedly executes the steps of: using data about the inspection target as input data, making a prediction about the inspection target using the machine learning model, and outputting prediction result data indicating the predicted result; a storage unit that stores the prediction result data output by the inspection unit; an evaluation unit that evaluates whether or not the input data is within a range assumed at the time of learning of the machine learning model based on a trend of a time series change of the result predicted for the test object indicated by the prediction result data stored in the storage unit, and outputs an evaluation result; An inspection system comprising:

2. 2. The inspection system of claim 1, The machine learning model is a model obtained by performing the machine learning for predicting which of a plurality of preset classifications the test object will be classified into, The inspection unit includes: predicting a predicted probability indicating the likelihood that the test subject will be classified into each of the plurality of classes; determining that the test object is classified into the classification having the highest predicted probability value; outputting the prediction result data including the predicted probability for the classification determined for the test object; The storage unit stores the prediction result data output by the inspection unit together with time information indicating when the prediction was made for the inspection target, The evaluation unit evaluates whether or not the input data is within a range assumed at the time of learning of the machine learning model, based on a trend of a time-series change in the predicted prediction probability for the classification determined for the test object, which is indicated by the prediction result data stored in the storage unit. Inspection system.

3. 2. The inspection system of claim 1, The machine learning model is a model obtained by performing machine learning to predict whether the test subject is within a normal range, The inspection unit includes: predicting whether the test subject is within the normal range; outputting the prediction result data including a value representing a ratio of the number of the test subjects predicted to be within the normal range to the total number of the test subjects tested during a set period; The storage unit stores the prediction result data output by the inspection unit together with time information indicating when the prediction was made for the inspection target, The evaluation unit evaluates whether or not the input data is within a range assumed at the time of learning of the machine learning model, based on a trend of a time-series change in the ratio indicated by the prediction result data stored in the storage unit. Inspection system.

4. 4. The inspection system according to claim 1, further comprising: The input data is image data representing an appearance of the inspection object acquired by a sensor. Inspection system.

5. 4. The inspection system according to claim 1, further comprising: The input data is voice data representing an operating state of the inspection target acquired by a sensor. Inspection system.

6. 1. A computer-implemented method for performing testing using a machine learning model, comprising: The machine learning model is a trained machine learning model that has undergone machine learning to predict a predetermined event regarding an object to be inspected, The computer, A step of predicting the test object using the machine learning model with data about the test object as input data, and outputting prediction result data indicating the predicted result; accumulating the prediction result data; A step of evaluating whether or not the input data is within a range assumed at the time of learning of the machine learning model based on a trend of time-series change of the result predicted for the test subject indicated by the accumulated prediction result data, and outputting an evaluation result; How to do it.

7. A program executed by a computer, The computer performs the inspection using a machine learning model; The machine learning model is a trained machine learning model that has undergone machine learning to predict a predetermined event regarding an object to be inspected, The computer includes: A function of predicting the test subject using the machine learning model with data about the test subject as input data, and outputting prediction result data indicating the predicted result; A function of accumulating the prediction result data; A function of evaluating whether the input data is within a range assumed at the time of learning of the machine learning model based on a trend of time-series changes in the results predicted for the test subject indicated by the accumulated prediction result data, and outputting an evaluation result; A program to achieve this.

Images