Analysis system, analysis method, and program

Abstract

To provide an analysis system, an analysis method, and a program capable of providing information that facilitates analysis of factors of an event.SOLUTION: An analysis system 10 for analyzing factors of an event predicted by a machine-learned model, comprises: a prediction-related information acquisition unit 104 for acquiring a prediction result of the event by a black-box type machine learning model, condition data, and correct-answer data corresponding to the prediction result; a data extraction unit 105 for extracting the condition data used for the factor analysis from an acquired set of condition data based on a result of comparison between the prediction result and the correct-answer data; a white-box model generation unit 106 for generating a white-box type machine learning model using the extracted condition data; and a visualization unit 107 for visualizing and outputting a model structure of the white-box type machine learning model.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an analysis system, an analysis method, and a program.

Background Art

[0002] In recent years, it has become common to predict events using machine learning models. In such predictions, black box type machine learning models such as deep learning that enable highly accurate predictions are used. In a black box type machine learning model, the configuration of the model is black boxed, and it is difficult for a person to consider the process of the prediction process of the model. On the other hand, in a white box type machine learning model in which the configuration of the model is not black boxed, it is easy for a person to consider the process of the prediction process of the model.

[0003] Patent Document 1 discloses a technique that combines a black box type machine learning model and a white box type machine learning model. In this document, when the accuracy of the predicted value by the black box type machine learning model changes, a technique for analyzing the factor that causes the change in the accuracy of the predicted value by the black box type machine learning model using the white box type machine learning model is disclosed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the technology described in Patent Document 1, it is possible to consider measures for improving the accuracy of a black-box type machine learning model. However, in this technology, factors that affect the event itself predicted by the black-box type machine learning model, that is, factors that affect the prediction target itself, cannot be analyzed. For this reason, it is difficult to consider measures for improving the event itself in this technology. Hereinafter, factors that affect the event itself (factors that affect the prediction target itself) will be referred to as factors of the event.

[0006] The present disclosure has been made against the background of the above circumstances, and an object thereof is to provide an analysis system, an analysis method, and a program that can provide information that facilitates the analysis of factors of an event.

Means for Solving the Problem

[0007] One aspect of the present disclosure for achieving the above object is an analysis system for analyzing factors of an event predicted by a machine-learned model, the prediction result of the event by a black-box type machine learning model, and the input data that is the data input to the black-box type machine learning model to obtain the prediction result or the related data that is the data associated with the input data input to the black-box type machine learning model to obtain the prediction result, a prediction-related information acquisition unit that acquires the correct answer data corresponding to the prediction result, a data extraction unit that extracts the condition data to be used for factor analysis from the set of the acquired condition data based on the comparison result between the prediction result and the correct answer data, a white-box model generation unit that generates a white-box type machine learning model using the extracted condition data, and a visualization unit that visualizes and outputs the model structure of the white-box type machine learning model. According to this analysis system, since the conditional data is extracted using the comparison result between the prediction result and the correct data, only a data group whose prediction accuracy meets a predetermined standard can be extracted. Therefore, a white-box type machine learning model can be generated and visualized by excluding factors that affect the prediction accuracy. Thus, according to this analysis system, information that facilitates the analysis of the factors of an event can be provided.

[0008] In the above aspect, the prediction-related information acquisition unit may further acquire the likelihood of the prediction result, and the data extraction unit may extract the conditional data based on the comparison result between the prediction result and the correct data and the likelihood. According to such an analysis system, since the conditional data is extracted using the likelihood in addition to the comparison result between the prediction result and the correct data, only a data group whose prediction accuracy considering the likelihood meets a predetermined standard can be extracted. Therefore, it is expected that a white-box type machine learning model can be generated by more appropriately excluding factors that affect the prediction accuracy.

[0009] In the above aspect, after visualizing the model structure, the system further has a data determination unit that receives an instruction from the user and determines the conditions of the data used for training the black-box type machine learning model based on the instruction. The prediction-related information acquisition unit acquires the prediction result of the event by the new black-box type machine learning model trained using data that meets the determined conditions, the conditional data, and the correct data corresponding to the newly acquired prediction result. The data extraction unit extracts the conditional data for factor analysis from the set of the acquired conditional data based on the comparison result between the newly acquired prediction result and the correct data. The white-box model generation unit generates a new white-box type machine learning model using the extracted conditional data, and the visualization unit may visualize and output the model structure of the new white-box type machine learning model. According to such an analysis system, a new white-box type machine learning model can be generated using a data group organized from a new perspective, and the model structure can be visualized. Therefore, even if the model structure desired by the user cannot be obtained, a new model structure can be obtained, improving the convenience for the user.

[0010] In one aspect described above, the data extraction unit receives an instruction from the user after visualizing the model structure, extracts the conditional data used for factor analysis based on the instruction, the white-box model generation unit generates a new white-box type machine learning model using the newly extracted conditional data, and the visualization unit may visualize and output the model structure of the new white-box type machine learning model. According to such an analysis system, a new white-box type machine learning model can be generated using a data group organized from a new perspective, and the model structure can be visualized. Therefore, even if the model structure desired by the user cannot be obtained, a new model structure can be obtained, improving the convenience for the user.

[0011] In one aspect described above, the conditional data may be data indicating the manufacturing conditions of a product, and the event may be the quality of the product. According to such an analysis system, it is possible to assist in the analysis of manufacturing conditions that affect the quality of a product. Therefore, it can contribute to the improvement of the quality of the product.

[0012] Another aspect of the present disclosure for achieving the above object is an analysis method for analyzing factors of an event predicted by a machine-learned model, including: obtaining prediction results of the event by a black-box type machine learning model and conditional data which is either input data input to the black-box type machine learning model to obtain the prediction results or related data associated with the input data input to the black-box type machine learning model to obtain the prediction results, and correct data corresponding to the prediction results; extracting the conditional data for factor analysis from the obtained set of conditional data based on a comparison result between the prediction results and the correct data; generating a white-box type machine learning model using the extracted conditional data; and visualizing and outputting a model structure of the white-box type machine learning model. According to this analysis method, since the conditional data is extracted using the comparison result between the prediction results and the correct data, only a data group whose prediction accuracy satisfies a predetermined standard can be extracted. Therefore, a white-box type machine learning model can be generated and visualized by excluding factors that affect the prediction accuracy. Thus, according to this analysis method, information that facilitates the analysis of the factors of an event can be provided.

[0013] Another aspect of the present disclosure for achieving the above object is a program for analyzing factors of an event predicted by a machine-learned model, including a prediction result of the event by a black-box type machine learning model, and input data that is data input into the black-box type machine learning model to obtain the prediction result or associated data associated with the input data input into the black-box type machine learning model to obtain the prediction result. A prediction-related information acquisition step of acquiring condition data, which is any of the above, and correct answer data corresponding to the prediction result; a data extraction step of extracting the condition data used for factor analysis from the acquired set of condition data based on a comparison result between the prediction result and the correct answer data; a white-box model generation step of generating a white-box type machine learning model using the extracted condition data; and a visualization step of visualizing and outputting a model structure of the white-box type machine learning model, and causing a computer to execute the program. According to this program, since the condition data is extracted using the comparison result between the prediction result and the correct answer data, only a data group whose prediction accuracy satisfies a predetermined standard can be extracted. Therefore, a white-box type machine learning model can be generated and visualized by excluding factors that affect the prediction accuracy. Thus, according to this program, information that facilitates the analysis of the factors of an event can be provided.

Effects of the Invention

[0014] According to the present disclosure, an analysis system, an analysis method, and a program that can provide information that facilitates the analysis of the factors of an event can be provided.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Mode for Carrying Out the Invention

[0016] Hereinafter, with reference to the drawings, a technique for analyzing factors of an event predicted by a machine-learned model will be described. In the embodiment described below, for ease of understanding, as one specific example, a technique for analyzing factors that affect the quality of a casting product (that is, a product manufactured through a casting process) will be described. That is, in the following description, the event predicted by the machine-learned model is the quality of the casting product, and the analysis system according to the embodiment enables analysis of factors that affect the quality of the casting product. As will be described later, the analysis system according to the present disclosure may be used for analysis of products manufactured through other processes such as painting products (products manufactured through a painting process). Further, the analysis system according to the present disclosure may be used for analysis of factors of an event predicted by a machine-learned model, and the event handled by the analysis system according to the present disclosure does not necessarily have to be an event related to a product.

[0017] FIG. 1 is a block diagram showing an example of the configuration of the analysis system 10 according to the present embodiment. As shown in FIG. 1, the analysis system 10 includes a data collection unit 100, a database 101, a black box model generation unit 102, a prediction execution unit 103, a prediction-related information acquisition unit 104, a data extraction unit 105, a white box model generation unit 106, a visualization unit 107, and a data determination unit 108. Note that all of the components shown in FIG. 1 may be implemented in one device, or these components may be distributed and implemented in a plurality of devices. For example, the black box model generation unit 102 and the prediction execution unit 103 may be implemented in a device different from the device implementing the other components.

[0018] The data collection unit 100 collects data used for the processes performed in the analysis system 10. The data collection unit 100 may collect data by receiving data transmitted from other devices, may collect data by reading data stored in other devices, or may collect data by acquiring data input by the user to the analysis system 10. In this embodiment, the data collection unit 100 collects data indicating the manufacturing conditions of products (hereinafter referred to as manufacturing condition data). Specifically, for example, the data collection unit 100 collects data such as the mold temperature, the amount of cooling water, and the piston speed, which are data indicating the casting conditions. These data may be measured values measured using sensors or the like, or may be set values of manufacturing apparatuses. Further, the data collection unit 100 collects product data for each individual product. In this embodiment, as the product data, the data collection unit 100 specifically collects an image of the product. More specifically, this image is an image for determining the presence or absence of casting defects by a black box type machine learning model, and is, for example, an image of the surface of the product. For example, the data collection unit 100 acquires an image of the product taken by an imaging device. Further, the data collection unit 100 further acquires, for each individual, data indicating the quality inspection result by the inspection operator (hereinafter referred to as inspection result data) as product data. The inspection operator inspects the quality of the product, for example, visually. The inspection result data is, for example, data indicating whether or not casting defects have occurred, that is, data indicating whether the product is a good product or a defective product. The inspection result data corresponds to the correct answer data used for the learning of the black box type machine learning model. The data collection unit 100 acquires, for example, inspection result data input by the inspection operator using a terminal device such as a tablet.

[0019] The data collection unit 100 stores the collected data in the database 101. Note that the data collection unit 100 stores the data related to the same product (sample) in the database 101. That is, the data collection unit 100 associates the manufacturing condition data for manufacturing a certain product with the product data (image and inspection result data) of this product, and stores them in the database 101. The data collection unit 100 collects data for a large number of samples and stores them in the database 101. As a result, data for a large number of samples are accumulated in the database 101.

[0020] The black box model generation unit 102 generates a black box type machine learning model using the data collected by the data collection unit 100. Here, the black box type machine learning model refers to a machine learning model in which the model configuration is black boxed. For example, neural networks such as deep learning correspond to this, but it is not limited to this.

[0021] In the present embodiment, the black box model generation unit 102 acquires the product image and the product inspection result data from the database 101, and uses these as learning data to generate a black box type machine learning model. The generated black box type machine learning model receives the product image as input and outputs information indicating whether the product is a good product or a defective product as a prediction result. Note that this model may output the likelihood of the prediction result in addition to the prediction result. That is, not only information indicating whether the product is a good product or a defective product, but also the probability that the product is a good product or the probability that the product is a defective product may be output.

[0022] The prediction execution unit 103 performs prediction processing using the data collected by the data collection unit 100 and the black box type machine learning model generated by the black box model generation unit 102, and predicts an event. Note that the prediction execution unit 103 inputs data (in this embodiment, an image of a product) for a sample for which correct answer data (inspection result data) is obtained into the black box type machine learning model and performs prediction processing. Therefore, the prediction result of the event obtained by the prediction processing by the prediction execution unit 103 can be determined as correct or incorrect by comparing it with the correct answer data (inspection result data). The prediction execution unit 103 stores the prediction result output by the black box type machine learning model in the database 101 in association with the data (in this embodiment, an image of a product) input for the prediction. Therefore, in this embodiment, for the sample for which the prediction processing by the prediction execution unit 103 has been performed, product data (that is, an image and inspection result data), product manufacturing condition data, and a prediction result are stored in association with each other. Note that the prediction execution unit 103 performs prediction processing for a large number of samples.

[0023] The prediction-related information acquisition unit 104 acquires the prediction result of the event by the black box type machine learning model generated by the black box model generation unit 102, the condition data corresponding to this prediction result, and the correct answer data corresponding to this prediction result. Here, the condition data is data indicating the preconditions of the event, and the manufacturing condition data corresponds to this. Note that in this embodiment, since the manufacturing condition data is data associated with the image, the condition data can also be described as follows. That is, in this embodiment, the condition data is data associated with the input data (that is, an image) input to the black box type machine learning model to obtain the prediction result, and is also referred to as associated data. Further, the prediction-related information acquisition unit 104 may further acquire the likelihood of the prediction result. The prediction-related information acquisition unit 104 acquires the prediction result, the likelihood of the prediction result, the condition data, and the correct answer data by reading them from the database 101. The prediction-related information acquisition unit 104 acquires these pieces of information for a large number of samples.

[0024] Based on the comparison result of the prediction result and the correct data, the data extraction unit 105 extracts the conditional data (manufacturing condition data) used for factor analysis from the set of conditional data (manufacturing condition data) acquired by the prediction-related information acquisition unit 104. Specifically, for example, the data extraction unit 105 extracts only the conditional data for the products (samples) for which correct prediction results are obtained from the set of conditional data (manufacturing condition data) acquired by the prediction-related information acquisition unit 104. In other words, the data extraction unit 105 excludes the conditional data for the products (samples) for which incorrect prediction results are obtained from the acquired set of conditional data. For example, in a prediction using a black box type machine learning model, if an inspection operator also determines that a product predicted to be a defective product is indeed a defective product, the conditional data for this product will be extracted by the data extraction unit 105. Similarly, in a prediction using a black box type machine learning model, if an inspection operator also determines that a product predicted to be a non-defective product is indeed a non-defective product, the conditional data for this product will be extracted by the data extraction unit 105. That is, the data extraction unit 105 extracts the manufacturing condition data of the products for which the prediction results using the black box type machine learning model match the actual inspection results.

[0025] When the prediction-related information acquisition unit 104 also acquires the likelihood of the prediction result, the data extraction unit 105 may extract conditional data (manufacturing condition data) used for factor analysis based on the comparison result between the prediction result and the correct data and the likelihood. Specifically, for example, the data extraction unit 105 extracts only the conditional data for the products (samples) for which the correct prediction result is obtained together with a likelihood equal to or higher than a predetermined threshold from the set of conditional data (manufacturing condition data) acquired by the prediction-related information acquisition unit 104. In other words, even if the conditional data for the product (sample) for which the correct prediction result is obtained, if the likelihood of this prediction result is less than the threshold, the conditional data for this product (sample) is not extracted. For example, when the threshold is 80%, in the prediction using the black box type machine learning model, if the inspection operator also determines that the product predicted to be a defective product with a probability of 85% is a defective product, the conditional data for this product will be extracted by the data extraction unit 105. On the contrary, in the prediction using the black box type machine learning model, even if the inspection operator also determines that the product predicted to be a defective product with a probability of 51% is a defective product, the conditional data for this product is not extracted by the data extraction unit 105.

[0026] The white box model generation unit 106 generates a white box type machine learning model for factor analysis using the conditional data extracted by the data extraction unit 105. Here, the white box type machine learning model refers to a machine learning model whose model configuration is not black boxed. For example, a decision tree such as a classification tree corresponds to this, but it is not limited to this. Specifically, for example, the white box model generation unit 106 generates a decision tree using the extracted conditional data and the correct data corresponding to the conditional data. In the present embodiment, more specifically, the white box model generation unit 106 uses the conditional data (manufacturing condition data) as an explanatory variable and the correct data (inspection result data) as an objective variable to generate a classification tree. Thereby, a classification tree for predicting whether a product is a non-defective product or a defective product from manufacturing conditions is generated.

[0027] The visualization unit 107 visualizes and outputs the model structure of the white box model generated by the white box model generation unit 106. In the present embodiment, the visualization unit 107 visualizes and outputs the tree structure of a decision tree (classification tree). For example, the visualization unit 107 outputs to an output device such as a display. That is, the visualization unit 107 visualizes the model structure of the white box model and displays it on the display.

[0028] The data determination unit 108 determines the conditions of data for re-training the black box model. Specifically, after the visualization unit 107 visualizes the model structure, the data extraction unit 105 receives an instruction from the user and determines the conditions of the data used for training the black box model based on the instruction. When the user determines that the visualized model structure of the white box model is not the desired model structure, in order to obtain a new model structure of the white box model, the user instructs the analysis system 10 about the conditions of the data for re-training the black box model. For example, when it is difficult for the user to identify the cause of an event from the visualized model structure of the white box model, or when the user wants to identify further causes, the user instructs the conditions of the data for re-training the black box model. The data determination unit 108 extracts data that meets the conditions instructed by the user from the database 101 as the data for re-training the black box model.

[0029] When the conditions for data for retraining the black box type machine learning model are determined, the black box model generation unit 102 generates a new black box type machine learning model using only the data that satisfies the conditions. Thereafter, the above-described processing is repeated by each component of the analysis system 10. That is, the prediction execution unit 103 executes prediction processing using a new black box type machine learning model trained using data that satisfies the determined conditions. Then, the prediction-related information acquisition unit 104 acquires the prediction result of an event by the new black box type machine learning model, the conditional data corresponding to the newly acquired prediction result, and the correct answer data corresponding to the newly acquired prediction result. In this case as well, the prediction-related information acquisition unit 104 may further acquire the likelihood of the prediction result. The data extraction unit 105 extracts conditional data for factor analysis from the set of acquired conditional data based on the comparison result between the newly acquired prediction result and the correct answer data. The white box model generation unit 106 generates a new white box type machine learning model using the extracted conditional data. The visualization unit 107 visualizes and outputs the model structure of the new white box type machine learning model.

[0030] Next, the hardware configuration of the analysis system 10 will be described. FIG. 2 is a block diagram showing an example of the hardware configuration of the analysis system 10. As shown in FIG. 2, the analysis system 10 includes an input / output interface 150, a network interface 151, a memory 152, and a processor 153.

[0031] The input / output interface 150 is an interface for connecting the analysis system 10 and an input / output device. For example, input devices such as a mouse and a keyboard, and output devices such as a display are connected to the input / output interface 150.

[0032] The network interface 151 is used to communicate with any other device as necessary. The network interface 151 may include, for example, a network interface card (NIC).

[0033] The memory 152 is constituted by, for example, a combination of a volatile memory and a non-volatile memory. The memory 152 is used to store software (computer program) including one or more instructions executed by the processor 153, and data used for various processes of the analysis system 10. For example, the above-described database 101 may be realized by a storage device such as the memory 152.

[0034] The processor 153 reads and executes software (computer program) from the memory 152 to perform the processing of each component shown in FIG. 1. The processor 153 may be, for example, a microprocessor, an MPU (Micro Processor Unit), or a CPU (Central Processing Unit). The processor 153 may include a plurality of processors. Thus, the analysis system 10 has the function as a computer.

[0035] The program includes a group of instructions (or software code) for causing a computer to perform one or more functions described in the embodiments when loaded into the computer. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray (registered trademark) disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0036] Next, the processing flow of the analysis system 10 will be described. FIG. 3 is a flowchart showing an example of the processing flow of the analysis system 10. Hereinafter, the processing flow will be described with reference to FIG. 3.

[0037] In step S100, the data collection unit 100 collects data used for the processing of the analysis system 10, such as manufacturing condition data (condition data), product images, and inspection result data (correct answer data), and stores them in the database 101.

[0038] Next, in step S101, the black box model generation unit 102 generates a black box type machine learning model using the product image and the product inspection result data as learning data.

[0039] Next, in step S102, the prediction execution unit 103 inputs the product image for which the correct answer data is obtained into the black box type machine learning model generated in step S101 and performs prediction processing.

[0040] Next, in step S103, the prediction related information acquisition unit 104 acquires the prediction result obtained in step S102, the condition data corresponding to this prediction result, and the correct answer data corresponding to this prediction result. As described above, the likelihood of the prediction result may further be acquired.

[0041] Next, in step S104, the data extraction unit 105 extracts the condition data (manufacturing condition data) for factor analysis from the set of condition data (manufacturing condition data) based on the comparison result between the prediction result and the correct answer data. That is, in this step, the data extraction unit 105 extracts the condition data used for generating the white box type machine learning model. As described above, in this step, the condition data may also be extracted using the likelihood of the prediction result.

[0042] Next, in step S105, the white box model generation unit 106 generates a classification tree using the condition data extracted in step S104 and the correct answer data corresponding to the condition data.

[0043] Next, in step S106, the visualization unit 107 visualizes and outputs the model structure of the white box type machine learning model generated in step S105. FIGS. 4 and 5 are schematic diagrams showing examples of the model structure visualized by the visualization unit 107. In any of the examples shown in FIGS. 4 and 5, in step S104, the data extraction unit 105 extracts manufacturing condition data for 15,300 samples (products). In the examples shown in FIGS. 4 and 5, each manufacturing condition data is two-dimensional data consisting of data on the temperature of the mold and data on the amount of cooling water. Note that in FIGS. 4 and 5, for ease of understanding, the manufacturing condition data is two-dimensional data, but the number of dimensions is not limited to 2. Also, in any of the examples shown in FIGS. 4 and 5, among the 15,300 samples, correct answer data indicating good products is associated with the manufacturing condition data for 15,000 samples, and correct answer data indicating defective products is associated with the manufacturing condition data for 300 samples.

[0044] When the model structure of the generated white-box machine learning model is the model structure shown in FIG. 4, for example, by manufacturing a product with the mold temperature set to 50 or higher, the user can understand from the model structure that the occurrence of defective products can be eliminated. Also, for example, the user can understand from the model structure that even if the mold temperature is less than 50, by manufacturing a product with the cooling water volume set to 50 or less, the occurrence of defective products can be reduced. In this way, the user can analyze the factors that affect the quality of the product by checking the output of the visualization unit 107. A user who has performed such an analysis changes the manufacturing conditions of the product so as to suppress the occurrence of defective products. In the example shown in FIG. 4, for example, the user implements a countermeasure so that the product is manufactured with the mold temperature set to 50 or higher. If it is difficult to implement such a countermeasure for some reason, such as due to reasons related to the operation of the equipment, the user may implement a countermeasure so that the product is manufactured with the cooling water volume set to 50 or less.

[0045] When the model structure of the generated white-box type machine learning model is the model structure shown in FIG. 5, for example, by manufacturing a product with the mold temperature set to 50 or higher, the user can understand from the model structure that the occurrence of defective products can be eliminated. However, in the example shown in FIG. 5, for 300 defective product samples manufactured under manufacturing conditions where the mold temperature is less than 50, based on the cooling temperature, they cannot be classified so that there is sufficient bias in the number of defective products. Specifically, out of the 300 samples of defective products manufactured under manufacturing conditions where the mold temperature is less than 50, 130 were manufactured under manufacturing conditions where the cooling water volume is 50 or less, and 170 were manufactured under manufacturing conditions where the cooling water volume is more than 50. Therefore, it is difficult to perform a cause analysis on defective products manufactured under manufacturing conditions where the mold temperature is less than 50. Thus, a user who determines that sufficient cause analysis cannot be performed from the model structure visualized in step S106 instructs the analysis system 10 to output the model structure of another white-box type machine learning model. At this time, the user instructs the analysis system 10 regarding the data conditions. For example, a user who has confirmed the model structure shown in FIG. 5 may instruct to generate a model using only the data of products (samples) manufactured under manufacturing conditions where the cooling water volume is 50 or less, or may instruct to generate a model using only the data of products (samples) manufactured under manufacturing conditions where the cooling water volume is more than 50. Of course, the user may also instruct the data specified by the manufacturing conditions corresponding to the branch nodes with a shallower depth than the branch nodes that classify the leaf nodes. For example, in the example shown in FIG. 5, the user may specify the mold temperature as the data condition, or may specify the mold temperature and the cooling water volume. Also, of course, the user may instruct the analysis system 10 to output the model structure of another white-box type machine learning model not only when it is determined that sufficient cause analysis cannot be performed from the visualized model structure, but also in any case such as when it is difficult to implement countermeasures.

[0046] When the analysis system 10 receives an instruction from the user to output the model structure of another white-box type machine learning model (Yes in step S107), the process proceeds to step S108. On the other hand, when such an instruction is not received (No in step S107), the process ends.

[0047] In step S108, the data determination unit 108 determines the conditions for the data for the retraining of the black-box type machine learning model according to the instruction received from the user, and extracts the data that meets the conditions from the database 101. After step S108, the process returns to step S101, and the processes after step S101 are repeated.

[0048] Next, the effects of the analysis system 10 will be described. Generally, it is expected that there are different characteristics in the prerequisite conditions (manufacturing conditions) of events between samples (products) with high prediction accuracy and low prediction accuracy of the black-box type machine learning model. For the group of samples having the first characteristic, the group of samples having the second characteristic can be noise. Therefore, when a white-box type machine learning model is generated using such samples with different characteristics together, it is likely to cause the complication of the model structure. For example, in the case of a casting product, when the product image of a product with obvious shrinkage cavity, which is a kind of defect, is input into the black-box type machine learning model, it can be judged as defective with high accuracy, while for a product with less obvious shrinkage cavity, the prediction accuracy tends to decrease. It is considered that the manufacturing conditions of the product affect whether the shrinkage cavity generated in the product is obvious or not. Therefore, if a classification tree with manufacturing conditions as explanatory variables is generated for all samples, a complex model will be generated, and it may be difficult to understand the factors affecting the event.

[0049] On the other hand, according to the analysis system 10, since the conditional data is extracted using the comparison result of the prediction result and the correct answer data, it is possible to extract only a group of data that satisfies a predetermined standard for the prediction accuracy. That is, it is expected to generate a white-box type machine learning model using only a group of samples mixed with samples having similar characteristics, rather than a group of samples mixed with samples having different characteristics. That is, according to the analysis system 10, it is possible to generate a white-box type machine learning model by excluding factors that affect the prediction accuracy. Therefore, it is possible to suppress the complication of the model structure. And according to the analysis system 10, such a model can be visualized. Therefore, according to the analysis system 10, it is possible to provide information that facilitates the analysis of the cause of an event.

[0050] Also, as described above, in addition to the comparison result of the prediction result and the correct answer data, the conditional data may be extracted using the likelihood. In this case, it is possible to extract only a group of data that satisfies a predetermined standard for the prediction accuracy considering the likelihood. Therefore, it is expected to generate a white-box type machine learning model by more appropriately excluding factors that affect the prediction accuracy.

[0051] Also, through step S108 and subsequent processes, a new black-box type machine learning model is generated after specifying the data to be used for model generation. Then, a new white-box type machine learning model is generated using the data extracted based on the prediction using this model. Therefore, according to the analysis system 10, a new white-box type machine learning model can be generated using a data group organized from a new perspective, and its model structure can be visualized. Thus, even if the model structure desired by the user cannot be obtained, a new model structure can be obtained, improving the convenience for the user. In the flowchart described above, after step S108, both a black-box type machine learning model and a white-box type machine learning model are newly generated, but only the new generation of the white-box type machine learning model may be performed. In this case, the data extraction unit 105 receives an instruction from the user after visualizing the model structure, and based on the instruction, extracts the conditional data to be used for factor analysis, that is, the conditional data to be used for generating a new white-box type machine learning model, and the white-box model generation unit 106 may generate a new white-box type machine learning model using the newly extracted conditional data. For example, in this case, the data extraction unit 105 further extracts only the data that matches the conditions instructed by the user from the data extracted in step S104.

[0052] Also, as described above, the conditional data may be data indicating the manufacturing conditions of the product, and the event may be the quality of the product. By using such conditional data and events, it is possible to assist in the analysis of the manufacturing conditions that affect the quality of the product. Therefore, it can contribute to the improvement of the quality of the product.

[0053] The present invention is not limited to the above-described embodiment and may be modified as appropriate without departing from the spirit and scope of the present invention. For example, in the above-described embodiment, a black-box machine learning model is generated that receives an image of a product as input and outputs information indicating whether the product is good or bad. However, other models may be generated. For example, a black-box machine learning model that predicts a numerical value, such as the paint film thickness of a product manufactured through a painting process, may be generated. In this case, the data collection unit 100 acquires, for each individual product, measurement data (e.g., measured film thickness) of a predetermined measurement item of the product as product data in addition to manufacturing condition data. Note that the quality of the product may be determined by the numerical value of the predetermined measurement item. The black-box model generation unit 102 then generates a black-box machine learning model using the manufacturing condition data and measurement data collected by the data collection unit 100 as training data. In this case, specifically, the black-box model generation unit 102 receives product manufacturing condition data as input and generates a black-box machine learning model that outputs a numerical value of a predetermined measurement item of the product (e.g., paint film thickness) as a prediction result. Therefore, the event predicted by the model is the numerical value of the predetermined measurement item. In this case, the condition data can also be referred to as follows: That is, the condition data is data input into the black-box machine learning model to obtain a prediction result, and is also referred to as input data. Furthermore, the white-box model generation unit 106 may generate a regression tree as the white-box machine learning model instead of the classification tree described above. For example, the white-box model generation unit 106 may generate a regression tree by using the manufacturing condition data as explanatory variables and the correct data (numerical values for specified measurement items) as objective variables. [Explanation of symbols]

[0054] 10 Analysis System 100 Data Collection Department 101 Database 102 Black box model generation unit 103 Prediction Execution Department 104 Prediction-related information acquisition unit 105 Data extraction unit 106 White box model generation unit 107 Visualization unit 108 Data determination unit 150 Input / output interface 151 Network interface 152 Memory 153 Processor

Claims

1. An analysis system for analyzing factors of events predicted by a machine learning model, a prediction-related information acquisition unit that acquires a prediction result of the event by a black-box machine learning model, a likelihood of the prediction result, condition data that is either input data that is data input to the black-box machine learning model to obtain the prediction result or related data that is data associated with the input data that is input to the black-box machine learning model to obtain the prediction result, and correct answer data corresponding to the prediction result; a data extraction unit that extracts the condition data to be used for factor analysis from the acquired set of condition data based on the comparison result between the prediction result and the correct answer data and the likelihood; a white-box model generation unit that generates a white-box machine learning model using the extracted condition data; a visualization unit that visualizes and outputs the model structure of the white-box machine learning model; An analysis system having:

2. a data determination unit that receives an instruction from a user after the visualization of the model structure and determines conditions for data to be used for training the black-box machine learning model based on the instruction; the prediction-related information acquisition unit acquires a prediction result of the event by a new black-box machine learning model trained using data that satisfies the determined condition, the condition data, and correct answer data corresponding to the newly acquired prediction result; the data extraction unit extracts the condition data to be used for factor analysis from the acquired set of condition data based on a comparison result between the newly acquired prediction result and the correct answer data; the white-box model generation unit generates a new white-box machine learning model using the extracted condition data; The visualization unit visualizes and outputs the model structure of the new white-box machine learning model. The analytical system of claim 1 .

3. the data extraction unit receives an instruction from a user after visualizing the model structure, and extracts the condition data to be used in factor analysis based on the instruction; the white-box model generation unit generates a new white-box machine learning model using the newly extracted condition data; The visualization unit visualizes and outputs the model structure of the new white-box type machine learning model. The analysis system according to claim 1 or 2.

4. The condition data is data indicating the manufacturing conditions of the product, The event is the quality of the product The analysis system according to any one of claims 1 to 3.

5. An analysis method for analyzing the factors of an event predicted by a machine-learned model, A computer obtains the prediction result of the event by a black-box type machine learning model, the likelihood of the prediction result, and condition data that is either input data that is the data input to the black-box type machine learning model to obtain the prediction result or related data that is data associated with the input data input to the black-box type machine learning model to obtain the prediction result, and correct data corresponding to the prediction result, Based on the comparison result between the prediction result and the correct data and the likelihood, the computer extracts the condition data to be used for factor analysis from the set of the obtained condition data, The computer generates a white-box type machine learning model using the extracted condition data, The computer visualizes and outputs the model structure of the white-box type machine learning model Analysis method.

6. A program for analyzing the factors of an event predicted by a machine-learned model, A prediction-related information acquisition step of obtaining the prediction result of the event by a black-box type machine learning model, the likelihood of the prediction result, and condition data that is either input data that is the data input to the black-box type machine learning model to obtain the prediction result or related data that is data associated with the input data input to the black-box type machine learning model to obtain the prediction result, and correct data corresponding to the prediction result, A data extraction step of extracting the condition data to be used for factor analysis from the set of the obtained condition data based on the comparison result between the prediction result and the correct data and the likelihood, A white-box model generation step of generating a white-box type machine learning model using the extracted condition data, A visualization step of visualizing and outputting the model structure of the white-box type machine learning model A program for causing a computer to execute.

Images