Evaluation method

Abstract

To accurately evaluate appropriateness of an abnormality determination model to be used in abnormality determination.SOLUTION: An evaluation method for evaluating appropriateness of a first abnormality determination model which is machine-learnt in order to output first abnormality degrees of a plurality of inspection objects after digitizing the respective degrees includes the steps for: acquiring a plurality of periods of datasets where a plurality of first abnormality degrees output from the first abnormality determination model is collected for each predetermined period; generating a second abnormality determination model which is the model machine-learnt by inputting the acquired plurality of periods of datasets and is the second abnormality determination model to output second abnormality degrees of the datasets after digitizing the degrees; outputting the second abnormality degrees by inputting the new datasets to the second abnormality determination model; and evaluating appropriateness of the first abnormality determination model based on the output second abnormality degrees.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to evaluating the suitability of a machine-learned anomaly detection model. [Background technology]

[0002] Various methods have been proposed for testing a workpiece or the like after machining, outputting the degree of anomaly of the test object using a machine-learned model (anomaly determination model), and determining whether the test object has an anomaly based on the degree of anomaly. In anomaly determination using machine learning, the change in the degree of anomaly output may be due to an anomaly of the test object, or it may be due to a cause other than the anomaly of the test object, for example, a difference between the environment when the learning data was acquired and the environment when the anomaly determination is performed. In the latter case, there is a risk of misjudging that an abnormality is present even when it is not actually abnormal. Therefore, in Patent Document 1, a change pattern of abnormal values ​​expected to occur under normal circumstances is specified in advance, and a filtering process according to the change pattern is performed on the degree of anomaly output, thereby determining whether an abnormality is present based on abnormal values ​​obtained only when the abnormality is essentially abnormal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2018 / 230645 Summary of the Invention [Problem to be solved by the invention]

[0004] In the configuration in which a filtering process is performed in advance according to a change pattern of an abnormal value as described above, when an unexpected change pattern occurs, an appropriate filtering process cannot be performed, which may result in an erroneous judgment. An unexpected change pattern may occur, for example, when the environment when the input data for learning is acquired differs from the environment when an abnormality judgment is performed to an unexpected degree. In such a case, the abnormality judgment model used for the abnormality judgment is not appropriate. Therefore, a technology capable of accurately evaluating the appropriateness of an abnormality judgment model used for an abnormality judgment is desired. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms. [Mode 1] A method for evaluating the suitability of a first anomaly judgment model that has been machine-learned using only images of inspection subjects in normal states in order to digitize and output each of a first degree of abnormality of a plurality of inspection subjects, the method comprising the steps of: acquiring a data set consisting of only a plurality of first degrees of abnormality output in a normal state, the data set being a compilation of a plurality of first degrees of abnormality output from the first anomaly judgment model for each predetermined period, for a plurality of the periods; generating a second anomaly judgment model that has been machine-learned by inputting the acquired data set for the plurality of periods, the second anomaly judgment model being for digitizing and outputting a degree of second abnormality of the data set; inputting a new data set obtained by compiling a plurality of the degrees of first abnormality output from the first anomaly judgment model for the period when there is no abnormality into the second anomaly judgment model to output the second degree of abnormality; and evaluating the suitability of the first anomaly judgment model based on the output second degrees of abnormality.

[0006] According to one embodiment of the present disclosure, there is provided an evaluation method for evaluating the suitability of a first anomaly judgment model that has been machine-learned to digitize and output each of the first degrees of anomaly of a plurality of test objects. The evaluation method includes the steps of: acquiring a data set for a plurality of periods in which the plurality of first degrees of anomaly are compiled from the first anomaly judgment model; generating a second anomaly judgment model that is machine-learned by inputting the acquired data sets for the plurality of periods, the second anomaly judgment model being for digitizing and outputting the second degrees of anomaly of the data set; inputting the new data set obtained by compiling the plurality of first degrees of anomaly output from the first anomaly judgment model for the periods into the second anomaly judgment model to output the second degrees of anomaly; and evaluating the suitability of the first anomaly judgment model based on the output second degrees of anomaly. According to the evaluation method of this embodiment, a data set in which a plurality of first abnormality degrees output from the first anomaly judgment model are compiled for each predetermined period is acquired for a plurality of periods, the data set is input to generate a second anomaly judgment model by machine learning, and a new data set is input to the second anomaly judgment model to evaluate the suitability of the first anomaly judgment model based on the second anomaly degree obtained, so that the first anomaly judgment model can be evaluated with high accuracy. When data (data when there is no abnormality) obtained in an environment similar to the data input when the first anomaly judgment model is generated by machine learning is input to the second anomaly judgment model as a new data set, the second anomaly degree becomes a relatively low value. In this case, the suitability of the first anomaly judgment model is high. On the other hand, when data (data when there is no abnormality) obtained in an environment significantly different from the data input when the first anomaly judgment model is generated by machine learning is input to the second anomaly judgment model as a new data set, a higher value is output as the second anomaly. In this case, the suitability of the first anomaly judgment model can be evaluated as being lower. Therefore, by evaluating the suitability of the first abnormality determination model based on the second degree of abnormality output from the second abnormality determination model, the first abnormality determination model can be evaluated with high accuracy. The present disclosure can be realized in various forms, for example, in the form of an evaluation device for an anomaly determination model, a workpiece inspection system, a computer program for evaluating an anomaly determination model, a computer program for generating a machine learning model for evaluating an anomaly determination model, a storage medium storing these computer programs, etc. [Brief description of the drawings]

[0007] [Figure 1] 1 is a block diagram showing a schematic configuration of an inspection system to which an evaluation method according to an embodiment of the present disclosure is applied; [Diagram 2] FIG. 2 is a block diagram showing a detailed configuration of the abnormality determination device; [Diagram 3] FIG. 2 is a block diagram showing a detailed configuration of an evaluation device. [Figure 4] 4 is a flowchart showing the procedure of an evaluation process in the first embodiment. [Diagram 5] 5 is a flowchart showing a procedure of a response process in the first embodiment. [Figure 6] 10 is a flowchart showing a procedure of a response process in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] A. First embodiment: A1. System Configuration: 1 is a block diagram showing a schematic configuration of an inspection system 10 to which an evaluation method according to an embodiment of the present disclosure is applied. The inspection system 10 determines the presence or absence of an abnormality in a workpiece W machined on a production line LP using machine learning. The inspection system 10 also evaluates the suitability of a machine learning model (a first abnormality determination model 122 described below) used for this abnormality determination using machine learning. The inspection system 10 includes an abnormality determination device 100 and an evaluation device 200.

[0009] In this embodiment, the manufacturing line LP is provided with a processing device 20, a conveying device 30, and an imaging device 40. The processing device 20 includes a press die for performing press processing, and a press device that presses the press die against the workpiece W to plastically deform the workpiece W into a predetermined shape. In this embodiment, in the manufacturing line LP, the processing device 20 processes 500 workpieces W per day. Note that the number of workpieces W per day is not limited to 500, and any number of workpieces W may be processed per day. Moreover, the processing device 20 may be a device that performs any type of processing, such as cutting or die casting, instead of a device for performing press processing.

[0010] The transport device 30 transports the workpiece W after processing (pressing) by the processing device 20 to a predetermined position for the next process. In this embodiment, the transport device 30 is configured by a belt conveyor. Note that the transport device 30 may be configured by a transport machine or a robot that moves on a track instead of the belt conveyor.

[0011] The imaging device 40 captures images of the workpieces W one after another while they are being transported by the transport device 30. In this embodiment, the imaging range of the imaging device 40 is a range including the entire workpiece W. Objects present behind the transport device 30 and the workpiece W may be reflected in this imaging range. When capturing the image, the workpiece W may be temporarily stopped from being transported or may be moving. The captured image obtained by the imaging device 40 is used when the abnormality determination device 100 determines an abnormality. The captured image data may be, for example, data represented by gradation values ​​of R (red), G (green), and B (blue) for each pixel. Alternatively, the captured image data may be data consisting of Y (luminance), U (first color difference), and V (second color difference), and may be YCbCr image data or YPbPr image data.

[0012] 2 is a block diagram showing a detailed configuration of the abnormality determination device 100. The abnormality determination device 100 quantifies and outputs the degree of abnormality of each workpiece W to be inspected, and determines the presence or absence of an abnormality in the workpiece W based on the degree of abnormality. In this embodiment, the abnormality determination device 100 is configured as a computer, and includes a CPU 110, a storage device 120, a display unit 130, an image data acquisition unit 140, a first communication unit 150, and an internal bus 190. The CPU 110, the storage device 120, the display unit 130, the image data acquisition unit 140, and the first communication unit 150 can exchange data with each other via the internal bus 190.

[0013] The storage device 120 includes a ROM, a RAM, and a hard disk drive (HDD). A control program is stored in the ROM in advance, and the CPU 110 functions as a first inspection unit 111, a first determination unit 112, and a first notification unit 113 by expanding the control program into the RAM and executing it.

[0014] The first inspection unit 111 outputs the degree of abnormality of each work W by using machine learning. The "degree of abnormality" is a value indicating the degree of deviation from a normal state. In this embodiment, the degree of abnormality in a state closer to normal is a smaller value, and the degree of abnormality in a more abnormal state, that is, a state more deviated from the normal state, is a larger value. For example, in the normal case, 0 to 30 is output as an abnormal value, and in the abnormal case, a value greater than 50 is output. Hereinafter, the degree of abnormality output by the first inspection unit 111 is referred to as the "first degree of abnormality." In this embodiment, the first inspection unit 111 determines the first degree of abnormality using the first abnormality determination model 122. The first abnormality determination model 122 is a machine learning model in a neural network. Specifically, in this embodiment, the first abnormality determination model 122 is a machine learning model used in a DAGMM (Deep Auto Encoder Gaussian Mixture Model), which is one of so-called hybrid models. The first abnormality determination model 122 has completed learning in advance using only images of the workpiece W in a normal state after press processing by the processing device 20. The first inspection unit 111 inputs captured image data obtained by the image data acquisition unit 140 (captured image data obtained by the imaging device 40) to the first abnormality determination model 122, and outputs the degree of deviation from the normal state as a first degree of abnormality.

[0015] The first determination unit 112 determines whether or not the workpiece W is abnormal based on the first abnormality degree output by the first inspection unit 111. In the abnormality determination device 100, a first determination threshold 121 is set in advance as a determination threshold for determining whether or not the workpiece W is abnormal. When the first abnormality degree output from the first inspection unit 111 is equal to or greater than the first determination threshold 121, the first determination unit 112 determines that the workpiece W is "abnormal." When the first abnormality degree is smaller than the first determination threshold 121, the first determination unit 112 determines that the workpiece W is "normal." For example, "50" may be set as the first determination threshold 121. Note that the first determination threshold 121 is not limited to 50, and any value that can distinguish between the first abnormality degree in a normal state and the first abnormality degree in an abnormal state may be used.

[0016] The first notification unit 113 notifies the determination result by the first determination unit 112. Specifically, the determination result is displayed on the display unit 130. At this time, the determination result may be displayed as text information or an image. Furthermore, the captured image on which the determination is based may be displayed together with the determination result. Note that, in a configuration in which the abnormality determination device 100 includes a speaker, the determination result may be notified by outputting it as sound from the speaker. Furthermore, the abnormality determination device 100 may notify a predetermined computer of the determination result by email or the like.

[0017] The storage device 120 stores the above-mentioned control program, the first judgment threshold value 121, and the first anomaly judgment model 122. The storage device 120 includes a first anomaly degree storage unit 123. The first anomaly degree storage unit 123 stores the first anomaly degrees output from the first inspection unit 111. At this time, the first anomaly degree storage unit 123 stores the first anomaly degrees obtained in a predetermined period together as a data set. In this embodiment, the "predetermined period" is "days." That is, the first anomaly degree storage unit 123 stores the first anomaly degrees obtained each day, that is, a total of 500 first anomaly degrees, as one data set. Note that the above-mentioned "predetermined period" may be any period other than "days," such as "weeks," "months," or "three days."

[0018] The display unit 130 is configured, for example, with a liquid crystal display. As described above, the display unit 130 displays the judgment result by the first judgment unit 112, that is, the presence or absence of an abnormality in each workpiece W. The image data acquisition unit 140 is connected to the imaging device 40 and acquires the captured image obtained by the imaging device 40. The image data acquisition unit 140 may acquire the captured image data by performing communication with the imaging device 40, for example, in a Universal Serial Bus (USB) system. The first communication unit 150 executes communication with the evaluation device 200. Such communication may be communication by any system, such as communication via a LAN (Local Area Network) defined in IEEE802.3 or a wireless LAN defined in IEEE802.11, or a so-called WAN (Wide Area Network) or MAN (Metropolitan Area Network).

[0019] FIG. 3 is a block diagram showing a detailed configuration of the evaluation device 200. The evaluation device 200 evaluates the validity of the first abnormality judgment model 122 included in the abnormality judgment device 100. The "validity of the first abnormality judgment model 122" means a property indicating how appropriate (valid) the first abnormality judgment model 122 is as a machine learning model used to judge the presence or absence of an abnormality in the workpiece W. "Appropriate" means that the first abnormality judgment model 122 is capable of outputting a first abnormal value that allows the first judgment unit 112 to judge the workpiece W as "normal" when the workpiece W is normal and the first judgment unit 112 to judge the workpiece W as "abnormal" when the workpiece W is abnormal. On the other hand, "inappropriate" means that the first abnormality judgment model 122 is capable of outputting a first abnormal value that allows the first judgment unit 112 to judge the workpiece W as "abnormal" when the workpiece W is normal, or the first judgment unit 112 to judge the workpiece W as "normal" when the workpiece W is abnormal.

[0020] In this embodiment, the evaluation device 200 is configured as a computer, and includes a CPU 210, a storage device 220, a display unit 230, a second communication unit 250, and an internal bus 290. The CPU 210, the storage device 220, the display unit 230, and the second communication unit 250 can exchange data with each other via the internal bus 290.

[0021] The storage device 220 includes a ROM, a RAM, and a hard disk drive (HDD). A control program is stored in the ROM in advance, and the CPU 210 functions as a second inspection unit 211, a second determination unit 212, and a second notification unit 213 by expanding the control program in the RAM and executing it.

[0022] The second inspection unit 211 uses machine learning to output the degree of anomaly of the data set of the first degree of anomaly output from the first inspection unit 111. This "degree of anomaly" has the same meaning as the above-mentioned first degree of anomaly. That is, it is a value indicating the degree of deviation from a data set in a normal state. Hereinafter, the degree of anomaly output by the second inspection unit 211 is called the "second degree of anomaly". In this embodiment, as with the first degree of anomaly, the second degree of anomaly is such that, in the normal case, 0 to 30 is output as an abnormal value, and in the abnormal case, a value greater than 50 is output. In this embodiment, the second inspection unit 211 determines the second degree of anomaly using the second abnormality determination model 222. The second abnormality determination model 222 is a machine learning model in a neural network. Specifically, in this embodiment, the second inspection unit 211 is a machine learning model used in a DAGMM, similar to the first abnormality determination model 122. The generation of the first abnormality determination model 122 will be described later.

[0023] The second judgment unit 212 evaluates (judges) the suitability of the first abnormality judgment model 122 based on the second abnormality degree output by the second inspection unit 211. Specifically, the evaluation device 200 is set in advance with a second judgment threshold 221 as a judgment threshold for judging the suitability of the first abnormality judgment model 122, and judges the first abnormality judgment model 122 as "inappropriate" when the second abnormality degree output from the second inspection unit 211 is equal to or greater than the second judgment threshold 221, and judges the first abnormality judgment model 122 as "appropriate" when the second abnormality degree is smaller than the second judgment threshold 221. In this embodiment, the second judgment threshold 221 is set to the same value as the first judgment threshold 121. Therefore, for example, "50" may be set as the second judgment threshold 221. Note that the second judgment threshold 221 is not limited to 50, and any value that can distinguish between the second abnormality degree when the first abnormality judgment model 122 is appropriate and the second abnormality degree when the first abnormality judgment model 122 is inappropriate may be used as the second judgment threshold 221. The reason why the suitability of the first abnormality determination model 122 can be determined based on the second abnormality degree will be described later.

[0024] The second notification unit 213 notifies the judgment result by the second judgment unit 212, that is, the suitability of the first abnormality judgment model 122. Specifically, the judgment result is displayed on the display unit 230. At this time, the judgment result may be displayed as text information or an image. In addition, the first abnormality degree on which the judgment is based may be displayed together with the judgment result. In addition, in a configuration in which the evaluation device 200 includes a speaker, the judgment result may be notified by outputting it as a sound from the speaker. In addition, the judgment result may be notified from the evaluation device 200 to a predetermined computer by e-mail or the like.

[0025] The storage device 220 stores the above-mentioned control program, the second judgment threshold value 221, and the second anomaly judgment model 222. The storage device 220 includes a second anomaly degree storage unit 223. The second anomaly degree storage unit 223 stores the second anomaly degree output from the second inspection unit 211. At this time, the judgment result by the second judgment unit 212 may also be stored in association with the second anomaly degree storage unit 223.

[0026] The display unit 230 is configured, for example, with a liquid crystal display. As described above, the display unit 230 displays the judgment result by the second judgment unit 212, that is, the judgment result of the suitability of the first abnormality judgment model 122. The second communication unit 250 executes communication with the abnormality judgment device 100. This communication is the same type of communication as the communication executed by the first communication unit 150 described above.

[0027] In the inspection system 10 having the above configuration, the suitability of the first abnormality determination model 122 is evaluated by the evaluation device 200 executing an evaluation process described below.

[0028] A2. Evaluation process: 4 is a flowchart showing the procedure of the evaluation process in the first embodiment. The evaluation process is a process for evaluating the suitability of the first anomaly determination model 122, and is executed when the inspection system 10 is powered on. The evaluation process is executed to realize an evaluation method as an embodiment of the present disclosure.

[0029] In the evaluation device 200, the second inspection unit 211 acquires a data set of the first abnormality degree for learning the second abnormality determination model 222 from the abnormality determination device 100 for a predetermined period (step S105). In this embodiment, the "predetermined period" in step S105 is "100 days". Note that it is not limited to 100 days, and may be any period that may include a plurality of data sets. Here, in this embodiment, the data set acquired in step S105 is a data set consisting of a value at which the first abnormality degree of each day is judged to be "normal", that is, a value smaller than the first judgment threshold value 121, "50". As described above, the second abnormality determination model 222 is a machine learning model used in DAGMM, and is generated by unsupervised learning. Therefore, in this embodiment, a data set consisting of only the first abnormality degree output in a normal state is used as a learning model when generating the second abnormality determination model 222.

[0030] FIG. 4 shows an example of a learning data set for 100 days acquired in step S105. In this example, (11, 21, 12, 15, 13, . . . 17) is acquired as a data set of the first abnormality degree on the first day. In addition, (20, 17, 14, 11, 19, . . . 18) is acquired as a data set of the first abnormality degree on the second day. In addition, (12, 12, 15, 19, 22, . . . 21) is acquired as a data set of the first abnormality degree on the 100th day. All of these data sets are groups of values ​​of a single type of parameter, namely, the first abnormality degree, that is, the "abnormality degree determined based on the image of the workpiece W after press processing." Therefore, these data sets can be said to be one-dimensional vectors of 1×500. In the DAGMM, such one-dimensional vectors can be input and the abnormality degree (degree of deviation from the learned state) can be output.

[0031] The second inspection unit 211 inputs the data set for the predetermined period acquired in step S105 and performs machine learning to generate the second abnormality determination model 222 (step S110). This step S110 and the above-mentioned step S105 can be called a "learning process" for generating the second abnormality determination model 222. In this embodiment, this learning process is executed only once after the evaluation device 200 is started up.

[0032] The second inspection unit 211 acquires a new data set of the first abnormality degree from the abnormality determination device 100, and outputs the second abnormality degree by inputting the data set to the second abnormality determination model 222 (step S115). FIG. 4 shows an example of the new data set acquired in step S115 and an example of the second abnormality degree obtained when the data set is input. As a new data set of the first abnormality degree, (92, 75, 88, 80, 67, . . . 78) is acquired. And, as an example of the second abnormality degree, "70" is output. The newly acquired data set of the first abnormality degree is (92, 75, 88, 80, 67, . . . 78), and all of the first abnormality degrees are greater than the first determination threshold value 121 "50". In this way, the case where the first abnormality value output for many of the total 500 workpieces W processed on that day exceeds the first determination threshold value 121 "50" corresponds to a case where the environment during the transportation of the workpieces W changes, etc. The "change in the environment" may occur for various reasons, such as when the lighting environment in the production line LP changes in brightness or color due to a breakdown of a fluorescent lamp or the addition or modification of a lighting device, or when the wallpaper reflected in the background of the work W is replaced. When such an environment change occurs, that is, when the environment when the learning data set used to generate the first anomaly determination model 122 was obtained is significantly different from the current environment, the first anomaly degree for many works W may become a value larger than the first determination threshold value 121, which is "50," as shown in the example of FIG. 4.

[0033] The second judgment unit 212 evaluates the suitability of the first abnormality judgment model 122 based on the second abnormality degree output in step S115 (step S120). Specifically, when the second abnormality degree is greater than the second judgment threshold 221, the second judgment unit 212 evaluates (judges) the first abnormality judgment model 122 as "inappropriate", and when the second abnormality degree is equal to or less than the second judgment threshold 221, the first abnormality judgment model 122 is evaluated as "appropriate". Therefore, as in the example of FIG. 4, when the second abnormality degree is "70", it is evaluated as "inappropriate". As described above, when the second abnormality degree is greater than the second judgment threshold 221, it is assumed that the environment when the learning data set used to generate the first abnormality judgment model 122 was obtained is significantly different from the current environment. In such a case, when the normality of the work W is judged using the first abnormality judgment model 122, there is a risk of erroneously determining that the normal work W is "abnormal" or erroneously determining that the abnormal work W is "normal". Therefore, in this case, it is determined that the first abnormality determination model 122 is no longer appropriate (is inappropriate) as an abnormality determination model to be used in determining the normality of the workpiece W. The above steps S115 and S120 can be called an evaluation process (of the first abnormality determination model 122).

[0034] When step S120 is completed, the process returns to step S115. Therefore, a new data set of the first abnormality degree is acquired from the abnormality determination device 100. As described above, the data set of the first abnormality degree corresponds to the determination result of the normality of the work W for one day (a total of 500 work pieces W). Therefore, a new data set of the first abnormality degree is acquired on the next day or later.

[0035] 5 is a flowchart showing the procedure of the response process in the first embodiment. The response process is a process executed in the evaluation device 200 in response to the result of the evaluation process. In the evaluation device 200, when step S120 of the above-mentioned evaluation process is completed, the response process is executed.

[0036] The second notification unit 213 judges whether "the first abnormality determination model 122 is appropriate" based on the evaluation result obtained in step S120 of the above-mentioned evaluation process (step S205). If it is judged that the first abnormality determination model 122 is appropriate (step S205: YES), the response process is completed. On the other hand, if it is judged that the first abnormality determination model 122 is not appropriate (inappropriate) (step S205: NO), the second notification unit 213 causes the display unit 230 to display that the first abnormality determination model 122 is inappropriate (step S210).

[0037] By displaying on the display unit 230 that the first abnormality determination model 122 is inappropriate, it is possible to prompt the manager of the inspection system 10 to take the following measures (i) or (ii), for example. (i) Using a recently obtained data set, the first anomaly determination model 122 is regenerated by machine learning (switching the anomaly determination model). (ii) Return the altered environment to its original state. Specific examples of responses to (ii) above include repairing fluorescent lights, restoring the number and types of lighting devices, and restoring the color of the lighting to its original state.

[0038] According to the inspection system 10 of the first embodiment described above, a data set in which a plurality of first abnormality degrees output from the first inspection unit 111 (first abnormality determination model 122) are compiled for each predetermined period (days) is acquired for a plurality of periods (100 days), which is input to generate the second abnormality determination model 222 by machine learning, and a new data set is input to the second abnormality determination model 222 to obtain a second abnormality degree, and the suitability of the first abnormality determination model 122 is evaluated based on the second abnormality degree obtained, so that the first abnormality determination model 122 can be evaluated with high accuracy. When data (data without abnormality) obtained in an environment similar to the data input when the first abnormality determination model 122 is generated by machine learning is input to the second abnormality determination model 222 as a new data set, the second abnormality degree becomes a relatively low value. In this case, the suitability of the first abnormality determination model 122 is high. On the other hand, when data (data without anomaly) obtained in an environment significantly different from the data input when the first anomaly determination model 122 was generated by machine learning is input to the second anomaly determination model 222 as a new data set, a higher value is output as the second degree of anomaly. In this case, the suitability of the first anomaly determination model 122 can be evaluated as being lower. Therefore, by evaluating the suitability of the first anomaly determination model 122 based on the second degree of anomaly output from the second anomaly determination model 222, the first anomaly determination model 122 can be evaluated with high accuracy.

[0039] Furthermore, if the first anomaly determination model 122 is determined to be inappropriate, this is displayed on the display unit 230, thereby encouraging the administrator of the inspection system 10, etc., to take measures such as (i) regenerating the first anomaly determination model 122 through machine learning using a recently obtained data set, or (ii) restoring the changed environment to its original state.

[0040] B. Second embodiment: FIG. 6 is a flowchart showing the procedure of the response process in the second embodiment. The inspection system of the second embodiment differs from the inspection system 10 of the first embodiment in that the abnormality determination device 100 and the evaluation device 200 are realized by the same computer. For convenience, in the inspection system of the second embodiment, the same functional units as those in the inspection system 10 (the abnormality determination device 100 and the evaluation device 200) of the first embodiment are given the same reference numerals, and detailed descriptions thereof are omitted. The response process of the second embodiment shown in FIG. 6 differs from the response process of the first embodiment shown in FIG. 5 in that steps S220 to S225 are executed instead of step S210. The other procedures of the response process of the second embodiment are the same as those of the first embodiment, so the same procedures are given the same reference numerals, and detailed descriptions thereof are omitted.

[0041] If the first anomaly determination model 122 is determined to be inappropriate (inappropriate) in the above-mentioned step S205 (step S205: NO), the first inspection unit 111 subtracts the second degree of anomaly output by the second inspection unit 211 from each of the first degrees of anomaly in the new data set of the first degrees of anomaly to correct the data (step S220). For example, as shown in FIG. 4, when (92, 75, 88, 80, 67, . . . 78) are obtained as the new data set of the first degrees of anomaly and "70" is obtained as the second degree of anomaly, the data set of the first degrees of anomaly is corrected to (22, 5, 18, 10, 0, . . . 8) as a result of step S220. Note that in this embodiment, if the subtraction results in a negative value, "0" (zero) is set as the corrected first degree of anomaly.

[0042] The first inspection unit 111 re-determines whether or not each workpiece W has an abnormality based on each corrected first abnormality degree obtained in step S220 (step S225). Therefore, for example, each of the corrected first abnormality degrees (22, 5, 18, 10, 0, . . . 8) is compared with the first determination threshold 121, and many of them are changed from "abnormal" to "normal". As described above, when (92, 75, 88, 80, 67, . . . 78) is obtained as a new data set of first abnormality degrees, there is a possibility that the first abnormality degree is outputted large due to the change in the environment. However, in this case, the first abnormality degree is corrected downward by subtracting the value "70" of the second abnormality degree, so that the first abnormality degree can be corrected by the value corresponding to the increase in the abnormality degree due to the environment. This makes it possible to suppress erroneous determination of the presence or absence of an abnormality of the workpiece W.

[0043] The inspection system of the second embodiment described above has the same effects as the inspection system 10 of the first embodiment. In addition, when it is determined that the first anomaly determination model 122 is inappropriate, each of the first anomaly degrees constituting a new data set of the first anomaly degrees is corrected by subtracting the second anomaly degree, and whether or not the workpiece W is abnormal is re-determined based on the corrected first anomaly degrees, so that it is possible to cancel the increase in the first anomaly degree caused by the change in the environment and determine whether or not the workpiece W is abnormal.

[0044] C. Other embodiments: (C1) In the second embodiment, the response process is performed by subtracting the second abnormality degree from each first abnormality degree in the data set of the new first abnormality degree, but the present disclosure is not limited to this. For example, if the first abnormality determination model 122 is determined to be inappropriate (inappropriate) (step S205: NO), the first determination threshold value 121 may be corrected by adding the second threshold value to the first determination threshold value 121. With this configuration, the same effect as in the second embodiment is achieved.

[0045] (C2) In each embodiment, the first anomaly determination model 122 is a machine learning model used in DAGMM, but the present disclosure is not limited to this. For example, the first anomaly determination model 122 may be any type of machine learning model, such as a so-called hybrid model such as AE+GMM (Auto Encoder + Gaussian Mixture Model), DeepSVDD, or DeepDAD, a so-called generative adversarial network such as AutoGAN (Auto Generative Adversarial Network), ADGAN, f-AnoGAN, or EfficientGAN, or a machine learning model used in an autoencoder such as AutoEncorder, SSIM AutoEncorder, or AE-Grad.

[0046] (C3) In each embodiment, the presence or absence of an abnormality in the workpiece W is determined based on the captured image of the workpiece W obtained by the imaging device 40, but the present disclosure is not limited to this. For example, the presence or absence of an abnormality in the workpiece W may be determined based on any information that can determine the normality of the workpiece W or the processing by the processing device 20, such as a measured value of a current or resistance value obtained when a current is passed through the workpiece W, a measured value of the surface temperature of the workpiece W, or a measured value of the position or size of the workpiece W.

[0047] (C4) In the second embodiment, the anomaly determination device 100 and the evaluation device 200 are realized by the same computer, but the present disclosure is not limited to this. As in the first embodiment, the anomaly determination device 100 and the evaluation device 200 may be configured by different computers. In such a configuration, after the second anomaly degree is output, the evaluation device 200 may notify the second anomaly degree to the anomaly determination device 100. Then, in the anomaly determination device 100, the notified second anomaly degree may be subtracted from the data set stored in the first anomaly degree storage unit 123 to correct the data set.

[0048] (C5) In each embodiment, step S120 may be omitted. That is, the result of "appropriate" or "inappropriate" may not be output. In such a configuration, a corresponding process may be executed after step S115, and in step S205, the second abnormality degree output in step S115 may be compared with the second judgment threshold 221, and if it is greater than the second judgment threshold 221, step S210 or steps S220, S225 may be executed. In such a configuration, the execution of step S210 or steps S220, S225 means that the appropriateness of the first abnormality judgment model 122 is evaluated as "inappropriate". On the other hand, the non-execution of step S210 or steps S220, S225 means that the appropriateness of the first abnormality judgment model 122 is evaluated as "appropriate".

[0049] (C6) Instead of the response process of each embodiment, the following process may be executed as the response process. That is, the second abnormality degree output by the second inspection unit 211 may be recorded every time it is output, and the time when the second abnormality degree exceeds the second judgment threshold value 221 of "50" may be estimated from the history of the second abnormality degree, and when the time reaches a time a predetermined period before such time, the fact may be notified. For example, the aging of fluorescent lights and wallpaper progresses little by little as time passes. In the above configuration, such a change may be detected as the second abnormality degree, and it may be possible to change the fluorescent lights or wallpaper or switch the first abnormality judgment model 122 before the first inspection unit 111 starts to frequently make erroneous judgments.

[0050] (C7) In each embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. For example, at least one of the functional units of the first inspection unit 111, the first judgment unit 112, the first notification unit 113, the second inspection unit 211, the second judgment unit 212, and the second notification unit 213 may be realized by an integrated circuit, a discrete circuit, or a module combining these circuits. In addition, when a part or all of the functions of the present disclosure are realized by software, the software (computer program) can be provided in a form stored in a computer-readable recording medium. The term "computer-readable recording medium" is not limited to portable recording media such as flexible disks and CD-ROMs, but also includes internal storage devices in a computer such as various RAMs and ROMs, and external storage devices fixed to a computer such as a hard disk. In other words, the term "computer-readable recording medium" has a broad meaning including any recording medium that can fix data packets not temporarily.

[0051] The present disclosure is not limited to the above-mentioned embodiments, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features in each embodiment corresponding to the technical features in each form described in the Summary of the Invention column can be appropriately replaced or combined in order to solve some or all of the above-mentioned problems or to achieve some or all of the above-mentioned effects. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted.

[0052] The present disclosure can also be realized, for example, by the following aspects. [Aspect 1] A computer program for evaluating the suitability of a first abnormality determination model trained by machine learning in order to digitize and output first abnormality degrees of a plurality of test objects, a function of acquiring a data set for a plurality of predetermined periods, the data set being a compilation of the plurality of first anomaly degrees output from the first anomaly determination model; a function of generating a second anomaly determination model that is machine-learned by inputting the acquired data sets for the plurality of periods, the second anomaly determination model being for quantifying and outputting a second degree of anomaly of the data set; a function of inputting a new data set obtained by collating a plurality of the first anomaly degrees output from the first anomaly determination model for the period into the second anomaly determination model and outputting the second anomaly degree; a function of evaluating the suitability of the first anomaly determination model based on the output second anomaly degree; A computer program that enables a computer to realize the above. [Aspect 2] An inspection system for inspecting a plurality of inspection objects, comprising: a first anomaly determination model that has been machine-learned to quantify and output the first anomaly degrees of a plurality of test objects; a first inspection unit that outputs first abnormality degrees of a plurality of inspection objects using the first abnormality determination model; A storage unit that stores a data set in which the plurality of first anomaly degrees output from the first anomaly determination model are compiled for each predetermined period; a second anomaly determination model that is machine-learned by inputting the data sets for the plurality of periods, the second anomaly determination model being configured to quantify and output a second degree of anomaly of the data set; a second inspection unit that inputs a new data set obtained by collating the multiple first abnormality degrees output from the first abnormality determination model for the period into the second abnormality determination model to output the second abnormality degree; A determination unit that evaluates the suitability of the first abnormality determination model based on the output second abnormality degree; An inspection system comprising: [Aspect 3] A computer program for generating a second abnormality determination model that is machine-learned to evaluate suitability of a first abnormality determination model that is machine-learned to quantify and output first abnormality degrees of a plurality of test objects, The second anomaly determination model is a machine-learned model that quantifies and outputs a second anomaly degree of the data set, a function of acquiring a data set for a plurality of predetermined periods, the data set being a compilation of the plurality of first anomaly degrees output from the first anomaly determination model; a function of generating the second anomaly determination model by inputting the acquired data sets for the multiple periods and performing machine learning; A computer program that enables a computer to realize the above. [Explanation of symbols]

[0053] 10... inspection system, 20... processing device, 30... conveying device, 40... imaging device, 100... abnormality determination device, 110... CPU, 111... first inspection section, 112... first determination section, 113... first notification section, 120... storage device, 121... first determination threshold, 122... first abnormality determination model, 123... first abnormality degree storage section, 130... display section, 140... image data acquisition section, 150... first communication section, 190... internal bus, 200... evaluation device, 210... CPU, 211... second inspection section, 212... second determination section, 213... second notification section, 220... storage device, 221... second determination threshold, 222... second abnormality determination model, 223... second abnormality degree storage section, 230... display section, 250... second communication section, 290... internal bus, LP... production line, W... work

Claims

[Claim 1] 1. An evaluation method for evaluating suitability of a first abnormality determination model that has been machine-learned using only images of inspection objects in a normal state in order to digitize and output first abnormality degrees of a plurality of inspection objects, the method comprising: acquiring a data set consisting of only the plurality of first abnormality degrees output in a normal state, the data set being a compilation of the plurality of first abnormality degrees output from the first abnormality determination model for each predetermined period, for a plurality of the periods; generating a second anomaly determination model that is machine-learned by inputting the acquired data sets for the plurality of periods, the second anomaly determination model being for quantifying and outputting a second degree of anomaly of the data set; a step of inputting a new data set obtained by aggregating a plurality of the first abnormality degrees output from the first abnormality determination model for the period when no abnormality is present into the second abnormality determination model to output the second abnormality degree; evaluating the suitability of the first abnormality determination model based on the output second abnormality degree; The evaluation method comprises:

Images