Data generation device

Abstract

In order to provide a data generation device that generates data to be inserted into a system to be verified so as to improve the accuracy of a statistical model or a machine learning model for identifying the cause of a system failure, a data generation device according to the present invention optimizes training of a first model, which is a machine learning model or a statistical model for identifying the cause of a system failure, and comprises: a cause identification unit that identifies the cause of a system failure using the first model; an embedding unit that embeds an observation value of the system in a low-dimensional space; an uncertainty estimation unit that estimates an uncertainty of the cause of the failure identified by the cause identification unit with respect to sets of failures and parameters; an embedding estimation unit that estimates an embedded value of the embedding unit with respect to the sets of failures and parameters; and a failure selection unit that, on the basis of the estimation results of the uncertainty estimation unit and the embedding estimation unit, determines the validity of the sets of failures and parameters, and selects a set of a failure and a parameter to be inserted into the system.

Description

Data Generator

[0001] The present invention relates to a data generating device.

[0002] A technique is known in which a fault (e.g., a fault type, a fault location, etc.) is inserted into a system, a factor identifier is machine-learned from the observed values ​​(fault data), and the observed data at the time of the fault occurrence is input to the machine-learned factor identifier to identify the fault factor (e.g., see Non-Patent Document 1).

[0003] J. Kawasaki, G. Moura, and Y. Suzuki, "Comparative analysis of network fault classification using machine learning," in NOMS 2020-2020 IEEE / IFIP Network Operations and Management Symposium. IEEE, 2020, pp. 1-6.

[0004] In conventional fault injection techniques, including those described in non-patent literature, the type of fault and various parameters are often randomly set to generate data to be injected into the system under verification. However, faults that are difficult to identify generally require more training data than faults that are easy to identify. Furthermore, random fault injection can result in biased data toward specific faults and parameters, making it impossible to generate a wide range of data. All of these factors reduce the accuracy of factor identifiers built based on supervised learning.

[0005] An embodiment of the present invention has been made in consideration of the above-mentioned problems, and provides a data generation device that generates data to be inserted into a system to be verified so as to improve the accuracy of a statistical model or machine learning model that identifies the causes of system failures.

[0006] In order to solve the above problems, a data generation device according to an embodiment of the present invention is a data generation device that optimizes learning of a first model, which is a machine learning model or a statistical model that identifies a fault factor of a system, and includes: a factor identification unit that identifies a fault factor of the system using the first model; an embedding unit that embeds observed values ​​of the system into a low-dimensional space; an uncertainty estimation unit that estimates uncertainty in the identification of the fault factor by the factor identification unit for a pair of a fault and a parameter; an embedding estimation unit that estimates an embedded value of the embedding unit for the pair of the fault and the parameter; and a fault selection unit that determines the validity of the pair of the fault and the parameter based on the estimation results of the uncertainty estimation unit and the embedding estimation unit, and selects the pair of the fault and the parameter to be inserted into the system.

[0007] According to an embodiment of the present invention, a data generation device can be provided that generates data to be inserted into a system to be verified so as to improve the accuracy of a statistical model or machine learning model that identifies the cause of a system failure.

[0008] FIG. 1 is a diagram illustrating an example of the configuration of a data generation device according to the present embodiment; FIG. 2 is a diagram illustrating an overview of a framework according to the present embodiment; FIG. 3 is a diagram illustrating the processing of a framework according to the present embodiment; FIG. 4 is a diagram illustrating an example of the processing of a framework according to the present embodiment; FIG. 5 is a flowchart illustrating an example of a data generation process according to the present embodiment; and FIG. 6 is a diagram illustrating an example of the hardware configuration of a computer.

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0010] <Outline of the Present Embodiment> In the operation of ICT (Information and Communication Technology) systems, it is extremely important to assume that failures will occur and prepare for them. Fault injection is an approach to accumulating knowledge about failures. In recent years, chaos engineering, a framework for fault injection experiments aimed at accumulating knowledge about troubleshooting in real environments or verifying system robustness, has also been implemented. Artificial fault injection is relatively easy, particularly in virtualization and container platforms, and many tools for fault injection have been developed. Data obtained through fault injection can be used as training data for supervised learning to build a cause identifier. A cause identifier captures the relationship between the cause of a failure and the observed values ​​(fault data) at that time in advance. When actual observed values ​​at the time of the failure are given, the cause of the failure can be determined as a classification problem. Non-cited Document 1 creates a cause identifier using supervised learning using various types of machine learning models and compares and verifies its performance.

[0011] (Problem) In conventional fault injection techniques, such as those described in Non-Patent Document 1, data is often generated by randomly setting fault types and various parameters. However, faults that are difficult to identify generally require more training data than faults that are easy to identify. Furthermore, random fault injection may result in data bias toward specific faults and parameters, making it impossible to generate a wide range of data. Both of these factors reduce the accuracy of a fault identifier built based on supervised learning. This embodiment has been made in consideration of the above points, and one of its objectives is to provide a fault insertion experiment optimization framework that prioritizes fault data generation for faults that are difficult to identify while also ensuring the diversity of the fault data. In particular, it provides a method for efficiently finding pairs of faults and parameters that are likely to be effective for training a fault identifier and generating observed values.

[0012] (Solution) In this embodiment, we focus on the uncertainty of fault identification and preferentially insert faults with high uncertainty, i.e., faults whose causes are thought to be difficult to identify. Furthermore, we embed fault data in a low-dimensional space and use the distance in that space to acquire a highly diverse fault data set. Then, we perform supervised learning using the acquired fault data set to acquire a cause identifier that can handle a wide range of faults while increasing the accuracy rate for faults that are difficult to identify.

[0013] <Functional Configuration> Fig. 1 is a diagram showing an example of the configuration of a data generation device according to this embodiment. The data generation device 100 is an information processing device having a computer configuration, or a system including multiple computers. The data generation device 100 realizes, for example, each functional configuration shown in Fig. 1 by executing a predetermined program on the computer included in the data generation device 100.

[0014] 1, the data generating device 100 realizes various functional components, such as a cause identifying unit 110, an embedding unit 120, an uncertainty estimating unit 130, an embedding estimating unit 140, a fault selecting unit 150, a system control unit 160, and a learning database 170. Note that at least a part of the above-described functional components may be realized by hardware.

[0015] The factor identification unit 110 executes a factor identification process to identify a failure factor of the system 10. For example, the factor identification unit 110 identifies (estimates) a failure factor of the system 10 by using a first model 111, which is a machine learning model or a statistical model that has learned the relationship between the observed values ​​of the system 10 or values ​​based on the observed values ​​of the system 10 and the failure factor of the system 10.

[0016] The embedding unit 120 performs an embedding process to embed the observed values ​​of the observed system 10 into a low-dimensional space. For example, the embedding unit 120 compresses the observed values ​​of the system 10 or values ​​based on the observed values ​​of the system 10 into a low-dimensional vector.

[0017] The uncertainty estimation unit 130 executes an uncertainty estimation process to estimate the uncertainty of the identification of the fault factor by the factor identification unit 110 for a pair of the fault and the parameter. For example, the uncertainty estimation unit 130 estimates the uncertainty of the identification of the fault factor by using a second model 131 which is a machine learning model or a statistical model that has learned the relationship between the pair of the fault and the parameter and the uncertainty of the identification of the fault factor based on the probability vector that is the output value of the factor identification unit 110.

[0018] The embedding estimation unit 140 executes an embedding estimation process to estimate an embedding value for a pair of a fault and a parameter of the embedding unit 120. For example, the embedding estimation unit 140 estimates the embedding value of the embedding unit 120 using a third model 141, which is a machine learning model or a statistical model that has learned the relationship between the pair of a fault and a parameter and a low-dimensional vector that is an output value of the embedding unit 120.

[0019] The fault selection unit 150 executes a fault selection process to determine the validity of pairs of faults and parameters based on the estimation results of the uncertainty estimation unit 130 and the embedding estimation unit 140, and to select pairs of faults and parameters to be inserted into the system 10. For example, based on the output of the uncertainty estimation unit 130 and the output of the embedding estimation unit 140, the fault selection unit 150 determines pairs of faults and parameters that are estimated to have large uncertainty in the estimation of the fault cause by the factor identification unit 110 and large diversity between data, as pairs of faults and parameters to be inserted into the system 10.

[0020] The system control unit 160 includes, for example, a fault insertion unit 161 and a current status restoration unit 162. The fault insertion unit 161 executes a fault insertion process to insert a pair of a fault and a parameter selected by the fault selection unit 150 into the system 10 to be verified. The current status restoration unit 162 executes a current status restoration process to restore the system 10 to its current status before the fault insertion unit 161 inserts a new pair of a fault and a parameter. The system control unit 160 inserts the pair of a fault and a parameter selected by the fault selection unit 150 into the system 10 and acquires observed values ​​of the system 10.

[0021] The training database 170 stores training data for training the first model 111, the second model 131, the third model 141, and the like.

[0022] <Problem Setting> Here, the problem setting of this embodiment will be described, however, the scope of application of the present invention is not strictly limited to the following problem setting.

[0023] Let y∈Y be the fault index that can be inserted into the system to be verified. The fault index is discrete and is the target of estimation (classification) in this embodiment during testing. The fault index is defined based on the fault type, such as packet loss or CPU (Central Processing Unit) load, and the location of the fault. In addition to the fault index, let u∈U be the background parameter (hereinafter simply referred to as the parameter) that defines the state of the system to be verified. u is a vector including continuous and discrete values, and its value can be freely set during fault insertion experiments. The parameter u is, for example, the number of users or a service tuning parameter, and is not necessarily the target of estimation during testing. u may be extended to time-series data.

[0024] The state of the system to be verified is expressed by the observed value x∈X. x is a vector including continuous and discrete values, and may be the output value of the system to be verified itself, or may be something that can be calculated from the output value. The observed value x may be arbitrarily defined to be used as an input feature of the factor identification module described later. For example, the observed value x may be the CPU usage rate or traffic volume of each component. In this formulation, the system to be verified can be regarded as a function f (or stochastic process) that gives the observed value x from the fault and parameter pair (y, u).

[0025] In this embodiment, we aim to build a factor specifier h. This is a method that calculates the probability (certainty) h of each fault index y when an observation value x is input. y This is a function that gives (x)∈[0,1]. You can think of this as a function that returns the vector value expressed by the following formula (1).

[0026] For example, when a neural network model is used as the factor specifier h, each value of the softmax function is output, which is generally expressed as the following equation (2):

[0027] When identifying the cause, for example, the estimated failure index is calculated using the following equation (3).

[0028] The factor specifier h is a classifier and requires supervised learning.

[0029] Building a fault identification tool based on fault injection consists of several phases. First, in the selection phase, pairs (y,u)∈Y×U that are expected to have a high insertion effect are selected based on certain criteria from a large number of pairs. In the subsequent insertion phase, the selected {(y,u)} is injected into the target system to obtain {x}. Then, in the training phase, a fault identification tool h is trained based on the obtained data {(x,y)}. These phases are repeated to build a fault identification tool h. After building a fault identification tool h, we evaluate whether h can correctly estimate the fault index y when given x, which is the value when a fault actually occurs. This final evaluation is called the test phase. These stages can be performed in a verification environment or in a commercial environment, similar to chaos engineering.

[0030] The task in this embodiment is as follows: Y, U, X, and f are given. Assume that obtaining observed values ​​in a fault injection experiment, i.e., sampling the value of f, is expensive or time-consuming. Therefore, how should each phase be designed to achieve a highly accurate factor identifier h with as few injections as possible? In particular, how should {(y, u)} be selected to be given to f in the selection phase?

[0031] (Processing Overview) Fig. 2 is a diagram showing an overview of the framework according to this embodiment. The key feature of the framework 200 according to this embodiment is the introduction of an embedder emb 201, an embedding estimator emb_est 202, and an uncertainty estimator unc 203. Here, the embedder emb 201 corresponds to the embedding unit 120 in Fig. 1. The embedding estimator emb_est 202 corresponds to the third model 141 in Fig. 1. The uncertainty estimator unc 203 corresponds to the second model 131 in Fig. 1. Furthermore, the factor identifier h 204 corresponds to the first model 111 in Fig. 1.

[0032] The embedder emb201 takes an observation value x of the system 10 to be observed as input, converts it into a lower-dimensional value e∈E, and outputs it. Here, the low-dimensional value is typically, for example, two-dimensional or three-dimensional, but this is not necessarily required. Generally, observations with similar properties are embedded so that they are close to each other, and observations with different properties are distant from each other. The embedder emb201 may use, for example, a dimensionality reduction method such as UMAP (Uniform Manifold Approximation and Projection) or t-SNE (t-distributed stochastic neighbor embedding), but this embodiment does not limit the model. Below, we assume that an unsupervised learning model is used for the embedder emb201. However, it is also possible to use the intermediate layer of a supervised learning model, such as the factor identifier h204.

[0033] The embedded estimator emb_est 202 takes a fault and a parameter pair (y, u) as input and returns a value ^e on E. That is, emb_est(y, u) = ^e∈E. This mimics the behavior of emb(f(y, u)) = e. The embedded estimator emb_est 202 is a regression model acquired through supervised learning. Note that the superscript "^" of ^e should actually be written directly above the "e", but due to the notational constraints of the specification, it is written as "^e" here.

[0034] The uncertainty estimator unc203 receives a fault and parameter pair (y, u) as input and returns an estimate of the uncertainty index ^UM, which is a real value. Note that the superscript "^" of ^UM should actually be written directly above "UM", but due to limitations in the description, it is written as "^UM" here.

[0035] To explain the estimated value of the uncertainty index ^UM, we first describe the uncertainty index UM. The uncertainty index UM is the output of the factor specifier h204 [0,1] |Y| The vector value h(x)=(h1(x),・・・,h |Y|(x)). This represents the uncertainty of the estimation result of the factor specifier h204, and the larger the value, the more uncertain the estimation. Although the specific definition is arbitrary, the uncertainty index UM can be expressed as, for example, y' (x)) y' The maximum value among the values ​​h for the true value y y The following equation (4), which is the difference between (x), may be used.

[0036] The uncertainty index UM is expressed as a vector (h y' (x)) y' The entropy of may be used.

[0037] The value of UM defined in this way is estimated by inputting a fault and parameter pair (y, u), resulting in the uncertainty index estimate ^UM. In other words, the uncertainty estimator unc203 mimics the behavior of the function UM(h(f(y, u))). Here, UM is written as a function with h() as an argument. The uncertainty estimator unc203 is a regression model acquired by supervised learning using a fault and parameter pair (y, u) as input and UM as output. One of the key features of this embodiment is that the embedded estimator emb_est202 and the uncertainty estimator unc203 can provide embedded estimates and uncertainty estimates, respectively, without the need for time-consuming and costly fault insertion experiments on the target system.

[0038] (Framework Processing) First, the framework 200 performs fault injection into the system 10 to be verified a number of times to obtain observed values ​​and obtain initial data D1={(y, u, x)}. The framework 200 uses this initial data D1 to train the factor identifier h204. Separately, it also creates evaluation data D2={(y, u, x)}. Then, it inputs {x} of D2 to the factor identifier h204 to obtain {h(x)}. Here, the factor identifier h204 is not trained. The framework 200 also trains the uncertainty estimator unc203 using {(y, u, h(x))}. Furthermore, the framework 200 inputs {x} of D1 and D2 to the (unsupervised) embedder emb201 to obtain {emb(x)}, and trains the embedding estimator emb_est202 using {(y, u, emb(x))}.

[0039] Now, let us consider a method for selecting {(y,u)} with a high learning effect. First, as shown in FIG. 3, the framework 200 inputs the candidate set {y}={(y,u)} to the uncertainty estimator unc203 to obtain the uncertainty estimate {^h(y)}. Here, y is the vector (y,u), and in FIG. 3, there are 100 of them. Finally, y1,...,y 100 Consider a situation where we want to select three from among the following. The superscript "^" of ^h should be written directly above the "h", but due to the notational constraints of the specification, it is written as "^h" here. Based on the value of this uncertainty estimate {^h(y)}, the framework 200 calculates the uncertainty of each y i The framework 200 calculates the estimated value ^UMi of the uncertainty index for {^UMi}. Since the larger the uncertainty index, the more valuable it is as learning data. a pieces(N a In Figure 3, it is assumed that ^UMi with i=1 to 5 is selected.

[0040] The framework 200 inputs the {y_i} selected in this way into the embedding estimator emb_est 202 as shown in Figure 4 to obtain the embedding estimate {^ei}. Since the embedding estimate is considered to represent the proximity of each piece of data, selecting data that are as far apart as possible is expected to provide more diversity in the data and to increase the accuracy of the factor identifier h204 that is ultimately obtained. The framework 200 assumes that the embedding space E contains distances and selects data points that are as far apart as possible as N b pieces (N b is an arbitrary constant). If the distance is not specifically defined, Euclidean distance may be used. Any algorithm may be used for the selection method, but for example, the algorithm used when determining the initial value of k-means++, a type of clustering method, can be applied. This is an algorithm designed to make it more likely that points that are far from already selected points will be selected probabilistically. Figure 4 shows how three points, i=1, 3, 5, were selected as a result. N selected in this way b y's i Since this data has a large degree of uncertainty and ensures diversity, it should be highly valuable as training data.

[0041] The framework 200 is based on the N b y's i is inserted into the system 10 to be verified as shown in FIG. 5, and verification data x i The framework 200 also acquires the acquired verification data x i is added to the initial data D1, and the factor identifier h204 is retrained. Furthermore, the framework 200 recreates the evaluation data D2 again using the retrained new factor identifier h204, and retrains the uncertainty estimator unc203. Furthermore, the framework 200 retrains the embedding estimator emb_est202 using the data obtained by inputting the observation data {x} thus far into the embedder emb201. The framework 200 improves the accuracy of the factor identifier h204 by repeating the above loop.

[0042] In this way, in this embodiment, the uncertainty (difficulty) of factor identification for each data item is estimated based on the uncertainty estimator unc 203, data items with large uncertainty are selected, and the diversity of the training data is ensured based on the embedding estimator emb_est 202. In this way, it is possible to estimate the value of the data before actually conducting a fault insertion experiment in the system 10 to be verified, and it is expected that an experiment with a high learning effect can be conducted to obtain data, thereby further improving the accuracy of the factor identification unit h 204.

[0043] <Processing Flow> Figure 6 is a flowchart showing the processing flow of the data generation method according to this embodiment. This processing illustrates an example of data generation processing executed by the data generation device 100, which is an example of the framework 200 according to this embodiment. Here, we will mainly explain the operation of each functional component in the data generation processing that narrows down the fault and parameter pairs (y, u) to be inserted into the system 10 to be verified. Note that at the start of the processing in Figure 6, the first model 111, which is the factor identifier h 204, the second model 131, which is the uncertainty estimator unc 203, and the third model 141, which is the embedded estimator emb_est 202, have already undergone initial training.

[0044] In step S601, the fault selection unit 150 uses the uncertainty estimation unit 130 to estimate an uncertainty index from candidate pairs of faults and parameters (y, u). For example, the fault selection unit 150 inputs candidate pairs of faults and parameters {(y, u)} to the uncertainty estimation unit 130. The uncertainty estimation unit 130 estimates an uncertainty index for each candidate pair of faults and parameters (y, u) using the second model 131. Preferably, the uncertainty estimation unit 130 returns to the fault selection unit 150 the candidate pair of faults and parameters {(y, u)} that has the largest estimated uncertainty index value.

[0045] In step S602, the fault selection unit 150 estimates embedded estimate values ​​by estimating the embedded values ​​of the embedding unit 120 from candidate fault and parameter pairs (y, u) with the highest estimated uncertainty index values. For example, the fault selection unit 150 inputs candidate fault and parameter pairs {(y, u)} with the highest estimated uncertainty index values, acquired from the uncertainty estimation unit 130, to the embedding estimation unit 140. The embedding estimation unit 140 estimates embedded estimate values ​​for each input candidate fault and parameter pair (y, u) using a third model 141. Preferably, the embedding estimation unit 140 returns to the fault selection unit 150 a highly diverse combination calculated based on the embedded estimate values ​​from among the input candidate fault and parameter pairs {(y, u)}.

[0046] In step S603, the fault selector 150 selects, from among the candidate fault-parameter pairs, a fault-parameter pair (y, u) with a large uncertainty index and high diversity as the pair (y, u) to be inserted into the system 10. For example, in step S602, the fault selector 150 selects the fault-parameter pair (y, u) acquired from the embedding estimation unit 140 as the fault-parameter pair (y, u) to be inserted into the system 10, and passes the selected fault-parameter pair (y, u) to the system control unit 160. Note that this process is an example of a process of selecting, based on the uncertainty index and the embedded estimation value, a fault-parameter pair (y, u) with a large uncertainty in the estimation of the fault cause and high diversity among the data as the fault-parameter pair (y, u) to be inserted into the system 10 to be verified.

[0047] In step S604, the system control unit 160 inserts the fault and parameter pair (y, u) selected by the fault selection unit 150 into the system 10 to be verified. For example, the system control unit 160 restores the system 10 to its original state using the current state restoration unit 162, and then inserts the selected invitation and parameter pair (y, u) into the system 10. The observed value {x} output by the system 10 is passed to the cause identification unit 110 and the embedding unit 120.

[0048] The factor identification unit 110 uses the first model 111 to estimate an estimated value h(x) of the fault factor from the observation value {x} output by the system 10. The factor identification unit 110 also outputs the fault and parameter pair (y, u), the estimated value h(x) of the fault factor, and the uncertainty index UM to the training database 170. The embedding unit 120 also embeds {x} into a low-dimensional space and outputs the resulting {emb(x)} to the training database 170.

[0049] The data stored in the training database 170 is used as training data for further training the first model 111, the second model 131, and the third model 141.

[0050] By the processing of Figure 6, the data generation device 100 can generate data to be inserted into the system 10 to be verified so as to improve the accuracy of the first model, which is a statistical model or machine learning model that identifies the cause of failure in the system 10.

[0051] <Hardware Configuration> The data generating device 100 has, for example, the hardware configuration of a computer 700 as shown in Fig. 7. Alternatively, the data generating device 100 is configured by a plurality of computers 700. Note that the data generating device 100 may be realized by a program executed on a virtual machine on the cloud.

[0052] 7 is a diagram showing the hardware configuration of a computer 700. In the example of FIG. 7, a computer 700 includes a processor 701, a memory 702, a storage device 703, a communication device 704, an input device 705, an output device 706, a bus B, and the like.

[0053] The processor 701 is, for example, an arithmetic unit such as a CPU (Central Processing Unit) that executes predetermined programs to realize various functions. The memory 702 is a storage medium readable by the computer 700, and includes, for example, a RAM (Random Access Memory) and a ROM (Read Only Memory). The storage device 703 is a computer-readable storage medium, and may include, for example, a HDD (Hard Disk Drive), an SSD (Solid State Drive), various optical disks, and magneto-optical disks.

[0054] The communication device 704 includes one or more pieces of hardware (communication devices) for communicating with other devices via a wireless or wired network. The input device 705 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The output device 706 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that outputs to the outside.

[0055] The bus B is commonly connected to the above components and transmits, for example, address signals, data signals, and various control signals. The processor 701 is not limited to a CPU, and may be, for example, a DSP (Digital Signal Processor), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array).

[0056] (Supplementary Note) The data generating device 100 in this embodiment is not limited to being realized by a dedicated device, but may also be realized by a general-purpose computer. In this case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed to realize the function. Note that the term "computer system" here includes hardware such as an OS and peripheral devices.

[0057] Furthermore, "computer-readable recording media" includes various storage devices such as portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines when transmitting programs via networks such as the Internet or communication lines such as telephone lines, and devices that store programs for a certain period of time, such as volatile memory within computer systems that serve as servers or clients in such cases.

[0058] Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in a computer system, or may be one that is realized using hardware such as a PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

[0059] <Effects of the Present Embodiment> The data generation device 100 according to the present embodiment can generate data (insertion data and learning data) with higher learning value compared to conventional random fault injection and learning data generation, etc., and is therefore expected to improve the accuracy of the first model 111 that identifies the cause of a fault. Furthermore, the data generation device 100 according to the present embodiment can also reduce the number of fault injections, the experiment time, and the experiment cost required for the accuracy of the first model 111 to reach a predetermined accuracy. This is expected to speed up the introduction of machine learning models to respond to system faults and improve their performance.

[0060] As described above, according to this embodiment, a data generation device 100 can be provided that generates data to be inserted into the system 10 to be verified so as to improve the accuracy of the first model 111, which is a statistical model or machine learning model that identifies the cause of a failure in the system 10.

[0061] Summary of Embodiments This specification discloses at least the data generation device described in the following paragraphs. (Item 1) A data generation device that optimizes learning of a first model that is a machine learning model or a statistical model that identifies a fault factor of a system, the data generation device comprising: a factor identification unit that identifies a fault factor of the system using the first model; an embedding unit that embeds observed values ​​of the system into a low-dimensional space; an uncertainty estimation unit that estimates uncertainty of the factor identification unit's identification of the fault factor for a pair of a fault and a parameter; an embedding estimation unit that estimates an embedded value of the embedding unit for the pair of the fault and the parameter; and a fault selection unit that determines the validity of the pair of the fault and the parameter based on estimation results of the uncertainty estimation unit and the embedding estimation unit, and selects the pair of the fault and the parameter to be inserted into the system. (Item 2) The data generation device described in item 1, wherein the embedding unit compresses observed values ​​of the system or values ​​based on observed values ​​of the system into a low-dimensional vector. (Clause 3) The data generation device described in clause 1 or 2, wherein the factor identification unit identifies a fault factor of the system using the first model that has learned the relationship between an observation value of the system or a value based on the observation value of the system and the fault factor of the system; the uncertainty estimation unit estimates the uncertainty of identifying the fault factor using a second model that is a machine learning model or a statistical model that has learned the relationship between a pair of the fault and a parameter and the uncertainty of identifying the fault factor based on a probability vector that is an output value of the factor identification unit; and the embedding estimation unit estimates an embedded value of the embedding unit using a third model that is a machine learning model or a statistical model that has learned the relationship between the pair of the fault and a parameter and a low-dimensional vector that is an output value of the embedding unit. (4) The data generation device according to any one of paragraphs 1 to 3, wherein the fault selection unit determines, based on the output of the uncertainty estimation unit and the output of the embedding estimation unit, a fault and parameter pair that is estimated to have a large uncertainty in the estimation of the fault cause by the factor identification unit and a high diversity among data, as the fault and parameter pair to be inserted into the system.(Item 5) A data generation method in which a computer that optimizes learning of a first model that is a machine learning model or a statistical model that identifies a fault factor of a system executes: a factor identification process that identifies a fault factor of the system using the first model, an embedding process that embeds observed values ​​of the system into a low-dimensional space, an uncertainty estimation process that estimates uncertainty in the identification of the fault factor by the factor identification unit for a pair of a fault and a parameter, an embedding estimation process that estimates an embedded value by the embedding unit for the pair of the fault and the parameter, and a fault selection process that determines the validity of the pairs of fault and parameter ranked by the uncertainty estimation process and the embedding estimation process, and selects the pairs of fault and parameter to be inserted into the system. (Item 6) A program for causing a computer to function as each unit in the data generation device described in any one of Items 1 to 4, or a storage medium that stores a program.

[0062] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications or changes are possible within the scope of the gist of the present invention described in the claims.

[0063] REFERENCE SIGNS LIST 10 System 100 Data generation device 110 Factor identification unit 111 First model 120 Embedding unit 130 Uncertainty estimation unit 131 Second model 140 Embedding estimation unit 141 Third model 150 Fault selection unit 160 System control unit 700 Computer

Claims

1. A data generation device that optimizes the learning of a first model, which is a machine learning model or a statistical model that identifies a fault factor of a system, comprising: a factor identification unit that identifies a fault factor of the system using the first model; an embedding unit that embeds observed values of the system into a low-dimensional space; an uncertainty estimation unit that estimates the uncertainty of the factor identification unit's identification of the fault factor for a pair of a fault and a parameter; an embedding estimation unit that estimates an embedded value of the embedding unit for the pair of the fault and the parameter; and a fault selection unit that determines the validity of the pair of the fault and the parameter based on the estimation results of the uncertainty estimation unit and the embedding estimation unit, and selects the pair of the fault and the parameter to be inserted into the system.

2. The data generating device according to claim 1, wherein the embedding unit compresses the observed values of the system or values based on the observed values of the system into a low-dimensional vector.

3. The data generation device according to claim 1, wherein the factor identification unit identifies a fault factor of the system using the first model that has learned the relationship between an observed value of the system or a value based on the observed value of the system and the fault factor of the system; the uncertainty estimation unit estimates the uncertainty of identifying the fault factor using a second model that is a machine learning model or a statistical model that has learned the relationship between a pair of the fault and a parameter and the uncertainty of identifying the fault factor based on a probability vector that is an output value of the factor identification unit; and the embedding estimation unit estimates the embedded value of the embedding unit using a third model that is a machine learning model or a statistical model that has learned the relationship between the pair of the fault and a parameter and a low-dimensional vector that is an output value of the embedding unit.

4. A data generation device according to any one of claims 1 to 3, wherein the fault selection unit determines, based on the output of the uncertainty estimation unit and the output of the embedding estimation unit, a fault and parameter pair for which the factor identification unit's estimation of the fault factor is estimated to have a large uncertainty and high diversity among data, as the fault and parameter pair to be inserted into the system.

Images