Fault data generation device, fault data generation method, and program

Abstract

To discover critical fault with less experiment and shorter experiment time as a countermeasure for fault by machine learning and an AI (Artificial Intelligence) and to generate learning data for it.SOLUTION: The present invention is directed to a fault data generating apparatus having an inserted fault calculating unit 11 for determining an optimum fault inserted into a verification target system, a system control unit 12 for, based on instructions from the inserted fault calculating unit, executing insertion of the determined fault data into the verification target system, and an inference model unit 13 for inferring information relating to the fault by a machine learning model the observation data obtained from the verification target system by insertion of the fault data as an input.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a fault data generation device, a fault data generation method, and a program. [Background technology]

[0002] In recent years, attention has been focused on fault response based on machine learning and AI (Artificial Intelligence). In ICT (Information and Communication Technology) systems that continue to grow in scale and complexity, it is practically difficult to handle all fault response manually. From the perspective of reducing human costs and providing stable services, an approach that pre-learns system behavior and fault patterns based on huge amounts of data, and then uses the constructed machine learning model to quickly and accurately detect anomalies and identify their causes is becoming indispensable.

[0003] However, depending on the type of fault that occurs, machine learning models may not function as expected, resulting in erroneous inferences (predictions) or oversights. For example, for faults that occur infrequently or for different faults that output similar data, the performance of the machine learning model drops because learning is not performed sufficiently with limited data. Machine learning models are generally versatile, but the performance of the machine learning model may drop significantly, especially when the system exhibits qualitatively different behavior depending on the environmental state (traffic volume, etc.) or the type of fault (including location and scale). Therefore, in order to deal with such "critical faults," the problem arises of having to clarify what types of faults the machine learning model is vulnerable to and then preparing a sufficient amount of training data to deal with those faults.

[0004] In response to this, there is a method called "chaos engineering" that exposes critical failures that have not yet occurred (Non-Patent Document 1). This is an approach that observes the behavior of a system through experiments that artificially insert failures. It has been mainly introduced in microservice architectures, and a large amount of OSS (Open Source Software) and commercial tools have been developed. In recent years, chaos engineering has also been mentioned as a data generation method for machine learning models. It has also been proposed to respond to the above-mentioned failures by, for example, injecting (injecting) all kinds of failures to collect observation data, and pre-training a classifier to identify factors that associate the observation data with the type of failure (Patent Document 1). [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] C. Rosenthal, L. Hochstein, A. Blohowiak, N. Jones, and A. Basiri, Chaos Engineering: Building Confidence in System Behavior Through Experiments. O'Reilly Media, 2017.[3] Frazier, Peter I. "A tutorial on Bayesian optimization." arXiv preprint arXiv:1807.02811 (2018). [Patent documents]

[0006] [Patent Document 1] JP 2021-128538 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, since there are a huge number of variations in the environmental conditions and types of faults that can occur in reality, the above problems cannot be solved by simply applying the conventional chaos engineering scheme directly. That is, the current chaos engineering scheme performs "random fault occurrence" or "specific fault occurrence specified by the operator". However, when the number of possible environmental conditions and fault types is huge, random search is inefficient, and problems arise such as difficulty in obtaining data on critical faults and redundant data collection for faults for which sufficient learning data already exists. In addition, it is practically difficult for the operator to specify all critical faults in advance. The method of Patent Document 1 assumes comprehensive fault insertion, but this is also difficult.

[0008] The present invention has been made in consideration of the above points, and aims to discover critical faults and generate learning data for them at the lowest possible cost, i.e., with a small number of experiments or a short experiment time, by optimizing fault insertion experiments. [Means for solving the problem]

[0009] In order to solve the above problem, the invention according to claim 1 comprises: A mathematical algorithm is used to search preferentially for faults whose evaluation function has a value greater than a predetermined value based on an evaluation function that indicates the value of performing data generation by inserting fault data. an insertion fault calculation unit that calculates and determines an optimal fault to be inserted into a system to be verified; a system control unit that executes insertion of the determined fault data into the system to be verified based on an instruction from the insertion fault calculation unit; and an inference model unit that infers information about the fault using a machine learning model using as input observation data obtained from the system to be verified by inserting the fault data; The mathematical algorithm is a Bayesian optimization algorithm or a bandit algorithm that maximizes the evaluation function. A fault data generating device. Effect of the Invention

[0010] As described above, according to the present invention, by optimizing fault insertion experiments, it is possible to discover critical faults and generate learning data for them at the lowest possible cost, i.e., with a small number of experiments or a short experimental time. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating a relationship between a fault data generation device and a system under verification according to an embodiment, and a functional configuration of the fault data generation device. [Diagram 2] 1 is a hardware configuration diagram of a fault data generating device according to an embodiment. [Diagram 3] FIG. 1 illustrates a mathematical algorithm in an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0013] [Outline of the present embodiment] First, an overview of this embodiment will be described.

[0014] In this embodiment, an index for measuring the vulnerability of a machine learning model for faults, such as anomaly detection or factor identification, is introduced. In general, a machine learning model for anomaly detection or factor identification outputs an inference accuracy calculated using a probability distribution or the like in addition to an inference label. If the inference accuracy for the correct label is low, the machine learning model can be said to be vulnerable to a given fault. Therefore, a vulnerability index based on the inference accuracy can be defined for a machine learning model for anomaly detection or factor identification. Next, in this embodiment, under the assumption that similar faults should have similar vulnerability indices, an efficient search for highly vulnerable faults, i.e., critical faults, is performed by Bayesian optimization. Since a fault insertion experiment requires a certain amount of time to acquire data and the vulnerability function has an unknown shape, it is suitable as a use case of Bayesian optimization, which excels as an optimization method for black box functions in which sample acquisition is generally costly. In this way, data on critical faults can be efficiently collected with a small number of experiments, and the machine learning model can be sequentially updated to construct a machine learning model for anomaly detection or factor identification that can handle any environmental state or fault type.

[0015] [Functional configuration of the fault data generation device] Next, the functional configuration of the fault data generation device 1 will be described with reference to Fig. 1. Fig. 1 is a diagram showing the relationship between the fault data generation device and a system under verification according to an embodiment, and the functional configuration of the fault data generation device.

[0016] 1, the fault data generation device 1 is constructed by a computer, and is a device that, with regard to re-learning of the machine learning model m, does not finish updating the machine learning model m in one go, but uses the updated machine learning model m to repeat the processes of fault insertion, data collection, and re-learning to successively update the model, thereby being able to follow changes in the configuration of the system to be verified 2. The system to be verified is a physical system or a virtual system.

[0017] As shown in Fig. 1, the fault data generation device 1 has an insertion fault calculation unit 11, a system control unit 12, and an inference model unit 13 including a machine learning model m. Each of these units has a function realized by an instruction from a processor 101 in Fig. 2 described later based on a program. The fault data generation device 1 also has a learning database 10, which is constructed in a memory 102 or an auxiliary storage device 103 in Fig. 2 described later.

[0018] Among these, the insertion fault calculation unit 11 calculates and determines the optimal fault to be inserted as a Bayesian optimization engine. Specifically, the insertion fault calculation unit 11 determines the fault to be inserted by a mathematical algorithm that preferentially searches for faults whose evaluation function has a value larger than a predetermined value based on an evaluation function that indicates the value of performing data generation by fault insertion.

[0019] Here, the evaluation function is a vulnerability index to a fault defined based on the inference accuracy of the machine learning model m of the inference model unit 13. In addition, the mathematical algorithm is a Bayesian optimization algorithm or a bandit algorithm that maximizes the evaluation function.

[0020] The system control unit 12 also has a fault insertion engine 12a and a restoration engine 12b. The fault insertion engine 12a obtains a fault (c, x) insertion command from the insertion fault calculation unit 11, and inserts fault (c, x) data into the system 2 under verification. The restoration engine 12b obtains a fault propagation command from the insertion fault calculation unit 11, and outputs restoration data to the system 2 under verification.

[0021] The inference model unit 13 also calculates the observed data {z i (c,x)} i Using the input data, information about the fault is inferred by the machine learning model m, and the inference result h(c, x) is output as output data to the insertion fault calculation unit 11. As a result, the insertion fault calculation unit 11 calculates the vulnerability index V DCalculate (c,x) and the acquisition function, and determine the next fault (c,x) to be inserted.

[0022] The learning database 10 stores (accumulates) the observation data {zi(c,x)}i obtained from the system under verification 2 by the restoration of the system under verification 2 by the restoration engine 12b.

[0023] After the processes by the insertion fault calculation unit 11, the system control unit 12, and the inference model unit 13 are repeated a certain number of times, the inference model unit 13 receives data for model update from the learning database 10, and re-learns and updates the machine learning model m. The updated machine learning model m is appropriately imported into the inference model unit 13 that runs in the actual operation system.

[0024] [Hardware Configuration of the Fault Data Generation Device] Next, a hardware configuration diagram of the fault data generation device 1 will be described with reference to Fig. 2. Fig. 2 is a hardware configuration diagram of the fault data generation device according to the embodiment. Note that the hardware configuration when the verification target system 2 is a physical system is similar to that of the fault data generation device 1, and therefore a description thereof will be omitted.

[0025] 2, the fault data generation device 1 includes a processor 101, a memory 102, an auxiliary storage device 103, a connection device 104, a communication device 105, and a drive device 106. The hardware components constituting the fault data generation device 1 are connected to each other via a bus 107.

[0026] The processor 101 serves as a control unit that controls the entire fault data generation device 1, and includes various arithmetic devices such as a CPU (Central Processing Unit). The processor 101 reads various programs onto a memory 102 and executes them. The processor 101 may include a GPGPU (General-purpose computing on graphics processing units).

[0027] The memory 102 has a main storage device such as a Read Only Memory (ROM) and a Random Access Memory (RAM). The processor 101 and the memory 102 form a so-called computer, and the processor 101 executes various programs read onto the memory 102, causing the computer to realize various functions.

[0028] The auxiliary storage device 103 stores various programs and various information used when the processor 101 executes the various programs.

[0029] The connection unit 104 is a connection device that connects the fault data generating apparatus 1 to an external device (for example, a display device 110, an operation device 111).

[0030] The communication unit 105 is a communication device for transmitting and receiving various types of information to and from other units.

[0031] The drive device 106 is a device for setting a (non-transient) recording medium 130. The recording medium 130 here includes media that record information optically, electrically, or magnetically, such as a CD-ROM (Compact Disc Read-Only Memory), a flexible disk, a magneto-optical disk, etc. The recording medium 130 may also include semiconductor memories that record information electrically, such as a ROM (Read Only Memory), a flash memory, etc.

[0032] The various programs to be installed in the auxiliary storage device 103 are installed, for example, by setting the distributed recording medium 130 in the drive device 106 and reading out the various programs recorded on the recording medium 130 by the drive device 106. Alternatively, the various programs to be installed in the auxiliary storage device 103 may be installed by being downloaded from a network via the communication device 105.

[0033] [Processing or Operation of the Embodiment] Next, the process or operation of this embodiment will be described in detail. Before describing the process or operation, the problem setting will be described.

[0034] <Question setting> The problem considered in this embodiment will be formulated below. Here, formulation will be performed for a supervised class classification machine learning model used to identify the cause of a failure and an unsupervised binary classification machine learning model used to detect anomalies, but this is merely an example and can be applied even in situations where the model does not strictly follow the rules.

[0035] First, we present a formulation for a machine learning model for supervised classification (herein called the “cause specifier”). The fault space F = (C,X) ∋ (c,x) = (c,x1,…,x n ) is a set of all faults that are expected to occur in the system 2 under verification.

[0036] c∈C is the category of failures to be classified by the cause specifier, and represents the type of failure, the location of failure, etc. Compound failures that are a combination of multiple failures can also be handled by listing the combinations.

[0037] x∈X is another discrete or continuous multidimensional parameter that specifies the state of the target system 2, such as the traffic volume when a fault occurs or the scale of the fault (delay time if it is a delay fault, packet loss rate if it is a packet loss fault, CPU load rate if it is a CPU fault, etc.). Even if it is known that some fault has occurred during system operation, it is not possible to directly observe (c, x). To be precise, there may be a situation in which x can be partially observed, but it is not possible to directly observe c. Instead, the observed data z(c,x)=(z1(c,x),…,z n (c,x)) can be observed. Even if the fault is the same, the generated data may have fluctuations, so {z(c,x)|(c,x)∈F} can be considered as a multidimensional stochastic process parameterized by (c,x) (if (c,x) is fixed, z(c,x) can be considered as a random variable).i (c, x) can be any observable numerical quantity, such as various system metrics, the number of occurrences of a particular log, or an evaluation value of service quality.

[0038] During operation, a factor specifier is used to infer c, which cannot be directly observed, using the observed data z as input. In other words, the factor specifier D is

[0039]

number

[0040] For example, in a neural network model, the output value of the softmax function in the output layer corresponds to this. In general, the index argmax that gives the maximum value is used. c h c Let (z) be the inference result.

[0041] We introduce an index to represent the vulnerability of such a factor specifier D, that is, the possibility that inference may not be performed properly. The vulnerability VD(c,x) of the factor specifier D against a fault (c,x) is defined as

[0042]

number

[0043]

number

[0044]

number

[0045] Next, V. D Before explaining how to maximize (c,x), we will also explain an example of formulation for an unsupervised binary classification machine learning model (here called "anomaly detector"). A typical approach for anomaly detection is to learn only from training data corresponding to the normal state of the system under test 2, build a model, and judge whether test data is normal or abnormal by referring to the deviation from the normal state. Specifically, this is implemented using machine learning models such as one-class support vector machines and autoencoders. In this situation, the various concepts mentioned above can be defined as follows. The fault space is F=(C,X)∋(c,x)=(c,x1,…,x n ), where c∈{0, 1} is a binary variable that represents normal (0) or abnormal (1). (Note that the normal state (c=0) is also included in the fault space.) The observed data z(c,x) is defined in the same way as for the factor specifier. The anomaly detector D is

[0046]

number

[0047]

number

[0048]

number

[0049]

number

[0050]

number

[0051] In the following, we will explain the case of a factor identifier as an example, but a similar discussion can be applied to the case of an anomaly detector or another machine learning model.

[0052] The problem considered in this embodiment is as follows. Now, suppose that a factor specifier D is constructed based on prior data. Since it is impossible to prepare prior data comprehensively, D may be highly vulnerable to a specific fault. Therefore, suppose that an environment is available in which any fault (c, x) can be intentionally inserted and observation data for additional learning can be generated. This can be realized by using a chaos engineering tool or a self-made script. However, in general, the cycle of fault insertion, data acquisition, and restoration to the original state takes a certain amount of time, so that comprehensive data generation remains difficult. Then, the problem arises of how to search the fault space in order to efficiently find faults (c, x) that increase the vulnerability index V(c, x), generate additional learning data, and improve the accuracy of the factor specifier.

[0053] <Processing or Action> Next, the processing or operation of this embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing an algorithm in this embodiment.

[0054] To solve the above problems, the present embodiment takes a Bayesian optimization (Reference 1) approach. <Reference 1> Frazier, Peter I. "A tutorial on Bayesian optimization." arXiv preprint arXiv:1807.02811 (2018). Bayesian optimization is a general-purpose optimization method used to find the maximum value of black-box functions with high sample acquisition costs. In Bayesian optimization, the black-box objective function is modeled as a stochastic process such as a Gaussian process, and the balance between the magnitude of uncertainty and the magnitude of the expected value (the trade-off between exploration and exploitation) is considered. The utility obtained when a value is obtained at each point is expressed by an acquisition function, and optimal sampling is executed sequentially. In other words, the objective function is maximized by sampling (exploiting) the surrounding area where the maximum value is expected to be obtained at the current time, while simultaneously sampling (exploring) areas where the number of samplings is small and the uncertainty is high, in a balanced manner. In the case of this problem setting, the vulnerability index V D (c, x) is the objective function to be maximized. The objective of this embodiment is not necessarily to maximize V D The goal is to find faults with large (c,x) and generate data, but since Bayesian optimization attempts both exploration and exploitation as described above, it is possible to widely explore faults with high vulnerability and achieve the goal. If the variables representing faults (c,x) are only continuous variables, the usual Bayesian optimization method can be applied as is, but if there are only discrete variables, a multi-armed bandit algorithm is used instead of Bayesian optimization. If there are a mixture of discrete and continuous variables, it is necessary to continuously relax the discrete variables or use a method that combines Bayesian optimization and a multi-armed bandit algorithm, but in either case, it can be implemented using an open source Bayesian optimization tool (Reference 2) or existing algorithms (References 3 and 4). <Reference 2>Balandat, Maximilian, et al. "BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization." NeurIPS. 2020. <Reference 3> Nguyen, Dang, et al. "Bayesian optimization for categorical and category-specific continuous inputs." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020. <Reference 4>Ru, Binxin, et al. "Bayesian optimization over multiple continuous and categorical inputs." International Conference on Machine Learning. PMLR, 2020. As shown in Fig. 3, the algorithm basically consists of factor identification D and vulnerability index V D Bayesian optimization is applied with (c, x) as the objective function. First, the insertion fault calculation unit 11, the system control unit 12, and the inference model unit 13 perform random sampling N1 times (an appropriate natural number, which may be dynamically changed for each iteration) (S1).

[0055] A single sampling is the observation of a given fault (c,x) inserted into the target system.

[0056]

number

[0057]

number

[0058]

number

[0059] Here, the process (S1) will be described in detail with reference to FIG. S11: The insertion fault calculation unit 11 randomly selects a fault (c, x). S12: The fault injection engine 12a injects fault (c, x) data into the system 2 under verification. S13: The learning database 10 receives the observation data {z i (c,x)} i Get and store. S14: The inference model unit 13 extracts the observation data {z i (c,x)} i The insertion fault calculation unit 11 obtains the vulnerability index V(c,x), infers information about the fault using the machine learning model m, and outputs the inference result h(c,x) as output data to the insertion fault calculation unit 11. D Calculate (c,x) and the acquisition function, and determine the next fault (c,x) to be inserted. S15: The learning database 10 receives the observation data {z i (c,x)} i Get and store. S16: The restoration engine 12b performs restoration on the verification target system 2.

[0060] Next, the insertion fault calculation unit 11, the system control unit 12, and the inference model unit 13 repeat the cycle of calculating the acquisition function and sequentially determining the fault (c, x) to be sampled next N2 times (a suitable natural number, which may be dynamically changed for each iteration) according to the Bayesian optimization framework (S2). Various variations of the method of designing the acquisition function have been considered, but any of the methods can be applied to this embodiment.

[0061] Here, the process (S2) will be described in detail with reference to FIG. S21: The insertion fault calculation unit 11 randomly selects a fault (c, x). S22: The fault injection engine 12a injects fault (c, x) data into the system 2 under verification. S23: The learning database 10 receives the observation data {z i (c,x)} i Get and store. S24: The inference model unit 13 extracts the observation data {z i (c,x)} i The insertion fault calculation unit 11 obtains the vulnerability index V(c,x), infers information about the fault using the machine learning model m, and outputs the inference result h(c,x) as output data to the insertion fault calculation unit 11. D Calculate (c,x) and the acquisition function, and determine the next fault (c,x) to be inserted. S25: The learning database 10 receives the observation data {z i (c,x)} i Get and store. S26: The restoration engine 12b performs restoration on the verification target system 2. S27: The insertion fault calculation unit 11 calculates the vulnerability index V D Calculate (c,x) or the acquisition function. S28: The insertion fault calculation unit 11 determines the fault (c, x) to be inserted next by the calculation in S28.

[0062] Finally, the inference model unit 13 retrains the factor identifier using as training data all of the observation data obtained during the (N1+N2) samplings that were stored in A as an additional data set (S13). When retraining, online learning or sequential learning methods may be used, or batch learning may be performed by combining all of the pre-training data and the additional training data. The newly obtained factor identifier D' (D on the left side of S3 in Figure 3) is expected to be less vulnerable to specific faults than the previous factor identifier D. Furthermore, a vulnerability index V that is newly defined for D' is D' By repeating the above process using (c,x), the accuracy of the factor identification tool can be further improved. The repetition should be performed under some conditions (SomeConditions). A possible termination condition is that no faults that cause vulnerability to exceed a certain value are found for a sufficient number of trials.

[0063] [Application examples] The following are possible application examples of the embodiment.

[0064] Fault injection and data generation according to this embodiment are performed in the verification target system 2 to pre-construct a factor identifier, and then the factor identifier is introduced into the production environment.

[0065] The system under verification 2 performs fault insertion and data generation in this embodiment using a factor identifier similar to the one operating in the production environment. The factor identifier updated by re-learning is imported into the factor identifier in the production environment as appropriate. If the system under verification 2 is made to follow any changes in the configuration of the production environment, this embodiment will enable data on newly occurring faults to be generated in semi-real time.

[0066] A virtual environment such as a container environment that simulates a production environment is used as the above-mentioned verification target system 2. In a virtual environment, existing chaos engineering can be used, so implementation is easy, and data acquisition and restoration to the original state can be performed in a short time.

[0067] If a fault injection experiment is possible in a production environment, this embodiment may be implemented in the production environment. In other words, using a factor identifier operating in the production environment, the fault injection and data collection according to the present invention are actually repeated to update the factor identifier in semi-real time. Since fault injection in a production environment is common in chaos engineering schemes, this embodiment is also practical.

[0068] [Main Effects of the Embodiments] As described above, according to this embodiment, the difficulty of inference in a machine learning model is quantified by a vulnerability index, and critical faults that are highly vulnerable and difficult to infer are searched for based on Bayesian optimization. This makes it possible to efficiently discover critical faults with fewer experiments than the random search performed in conventional chaos engineering, and generate data for them. By repeatedly re-learning the machine learning model using these data, it becomes possible to improve the inference accuracy of the machine learning model for all faults. [Explanation of symbols]

[0069] 1. Fault data generator 2. System to be verified 10 Learning Database 11 Insertion Fault Calculation Unit 12 System control section 12a Fault Injection Engine 12b Restoration Engine 13 Inference Model Section m Machine Learning Model

Claims

1. An insertion fault calculation unit that calculates and determines optimal faults to be inserted into a system under verification using a mathematical algorithm that preferentially searches for faults whose evaluation function has a value greater than a predetermined value, based on an evaluation function that indicates the value of generating data by inserting fault data; a system control unit that executes insertion of the determined fault data into the system under verification based on a command from the insertion fault calculation unit; an inference model unit that infers information about the fault using a machine learning model with observation data obtained from the system under verification by inserting the fault data as an input; having The fault data generating device, wherein the mathematical algorithm is a Bayesian optimization algorithm or a bandit algorithm that maximizes the evaluation function.

2. The fault data generating device according to claim 1 , wherein the evaluation function uses a vulnerability index to faults that is defined based on an inference accuracy of a machine learning model in the inference model unit.

3. 2. The fault data generation device according to claim 1, wherein the system control unit performs restoration of the system under verification based on a command from the insertion fault calculation unit.

4. 4. The fault data generation device according to claim 1, wherein when the inference model unit re-learns a machine learning model using as input observation data obtained from the system to be verified by inserting the fault data, the inference model unit does not complete the update of the machine learning model in one go, but uses the updated machine learning model to repeat the processes of fault insertion, data collection, and re-learning to sequentially update the model.

5. A computer comprising: calculating and determining an optimal fault to be inserted into the system to be verified using a mathematical algorithm that preferentially searches for faults whose evaluation function has a value greater than a predetermined value based on an evaluation function that indicates the value of performing data generation by inserting fault data; inserting data of the determined fault into the system under test; Inferring information about the fault using a trained machine learning model using observation data obtained from the system under test by inserting the fault data as an input; The fault data generation method, wherein the mathematical algorithm is a Bayesian optimization algorithm or a bandit algorithm that maximizes the evaluation function.

6. A program for causing a computer to execute the method according to claim 5.

Images