Data generation device

Abstract

In order to improve the efficiency of learning of a machine learning model or a statistical model that identifies the cause of failure of a system, a data generation device improves the efficiency of learning of a first model which is the machine learning model or the statistical model that identifies the cause of failure of a system. The data generation device includes: an observation value estimation unit that estimates an observation value of the system with respect to a set of failure and parameter; a cause identification unit that identifies the cause of failure of the system by a first model; and a failure selection unit that determines the validity of the set of failure and parameter on the basis of an output result when the observation value of the system estimated by the observation value estimation unit is inputted to the cause specification unit, and selects the set of failure and parameter to be inserted into the system.

Description

Data Generator

[0001] The present invention relates to a data generating device.

[0002] A technique is known in which a fault (e.g., a fault type, a fault location, etc.) is inserted into a system, a factor identifier is machine-learned from the observed values ​​(fault data), and the observed data at the time of the fault occurrence is input to the machine-learned factor identifier to identify the fault factor (e.g., see Non-Patent Document 1).

[0003] J. Kawasaki, G. Moura, and Y. Suzuki, "Comparative analysis of network fault classification using machine learning," in NOMS 2020-2020 IEEE / IFIP Network Operations and Management Symposium. IEEE, 2020, pp. 1-6.

[0004] However, in order to construct a highly versatile factor identifier (such as a statistical model or a machine learning model), it is necessary to generate observation values ​​while changing the fault type and parameters. Therefore, conventional technologies have the problem of increasing time, cost, or the number of experiments, resulting in inefficiency.

[0005] The embodiments of the present invention have been made in consideration of the above-mentioned problems, and improve the efficiency of learning a machine learning model or a statistical model that identifies the cause of a system failure.

[0006] In order to solve the above problems, a data generation device according to an embodiment of the present invention is a data generation device that efficiently learns a first model, which is a machine learning model or a statistical model that identifies a fault cause of a system, and includes an observation value estimation unit that estimates an observation value of the system for a pair of a fault and a parameter, a factor identification unit that identifies a fault cause of the system using the first model, and a fault selection unit that determines the validity of the pair of the fault and the parameter based on an output result when the observation value of the system estimated by the observation value estimation unit is input to the factor identification unit, and selects the pair of the fault and the parameter to be inserted into the system.

[0007] According to an embodiment of the present invention, it is possible to efficiently train a machine learning model or a statistical model that identifies the cause of a system failure.

[0008] FIG. 1 is a diagram showing an example of the configuration of a data generation device according to the present embodiment; FIG. 2 is a diagram showing an overview of a framework according to the present embodiment; FIG. 3 is a diagram showing an example of an algorithm of the framework according to the present embodiment; FIG. 4 is a flowchart showing an example of a data generation process according to the present embodiment; FIG. 5 is a diagram showing an example of the hardware configuration of a computer; FIG. 6 is a diagram showing an example of an experimental result according to the present embodiment; FIG. 7 is a diagram showing an example of an experimental result according to the present embodiment;

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0010] <Outline of this embodiment> In the operation of ICT (Information and Communication Technology) systems, it is extremely important to accumulate knowledge about failures in advance. Conventional operations, in which analysis begins after a failure occurs, can result in service interruptions that affect customers and sometimes have a serious impact on business. Furthermore, even with an automatic recovery mechanism, it is practically difficult to prevent all failures in today's large-scale ICT systems, in which components interact with each other in complex ways. Therefore, it is necessary to assume that failures will inevitably occur and to investigate in advance how the system will behave when a failure occurs.

[0011] Fault injection is an approach used to accumulate knowledge about faults. While fault injection has traditionally been used as a means of abnormal system testing, in recent years the significance of fault injection has been rediscovered in the context of chaos engineering (a framework for fault injection experiments aimed at accumulating knowledge about dealing with problems in real environments and verifying the robustness of systems). In particular, artificial fault injection is relatively easy in today's virtualized and containerized ICT systems, and numerous tools for fault injection have been developed. The fault data obtained as a result of fault injection can be analyzed using machine learning and used to improve fault response.

[0012] For example, one example of combining fault injection and machine learning is the construction of a factor identifier that identifies fault causes (fault type, location, etc.). This involves generating all observed values ​​(fault data) based on fault injection, and then understanding their relationships in advance using supervised learning, with the aim of identifying fault causes from the observed values ​​as a classification problem. Creating a factor identifier that covers all fault causes has the advantage of being able to deal with faults that have not yet occurred during operation. The effectiveness of such efforts has already been confirmed using various machine learning models. For example, Non-Patent Document 1 describes the creation of factor identifiers using supervised learning using various types of machine learning models, and compares and verifies their performance.

[0013] (Problem) Conventional fault injection methods, such as those described in Non-Patent Document 1, are not necessarily effective in building machine learning models. First, as a prerequisite, the range of system states that we must explore in fault injection experiments is extremely vast, making comprehensive experiments practically impossible. For example, consider a task involving identifying which component is causing the CPU load in a central processing unit (CPU) load fault. In this case, fault injection should explore more than just the component space. Since the traffic volume between each component is an important feature in fault identification, the number of user requests, which has a significant impact on it, must also be controlled. In other words, to build a versatile fault identification tool, observations must be generated while varying not only the type of fault to be estimated but also all parameters that define the system state.

[0014] Conventional methods for exploring a vast parameter space include designing fault injections based on domain knowledge or randomly determining faults and parameters. However, the former method requires time and human resources, and is unsuitable for creating general-purpose machine learning models. The latter method also has the problem of poor training efficiency. For example, faults that are difficult to identify require more training data (especially near the decision boundary of the model) than faults that are easy to identify. However, random insertion treats all faults uniformly, requiring numerous experiments before a highly accurate factor identification tool can be built.

[0015] The present invention has been made in consideration of the above-mentioned problems, and provides a framework for improving the efficiency of fault insertion experiments in order to build a highly accurate factor identifier with a small number of fault insertion experiments. In particular, it provides a method for efficiently finding pairs of faults and parameters that are likely to be effective for learning a factor identifier and generating observed values. In other words, it eliminates redundant experiments that add little new information and prioritizes the generation of data for cases that are critical for learning. This enables a highly accurate factor identifier to be built in a short time or at low cost.

[0016] (Solution) In this embodiment, a framework is provided that assigns execution priorities while estimating the effectiveness of fault injection experiments, enabling rapid learning of a factor identification device. This framework introduces an observation value estimator that estimates observed values ​​for pairs of faults and parameters. This observation value estimator is a machine learning model (or statistical model) that estimates observed values ​​more quickly than in a real system. By inputting these estimated observed values ​​into a factor identification device undergoing learning, it is possible to obtain, in advance, information, albeit imprecise, on whether learning has been successful for a given pair of faults and parameters. This information is considered to represent the effectiveness of each pair of faults and parameters (pairs that have not been able to learn are highly worth executing), and therefore, the execution priorities of experiments are determined based on this information, and the fault injection experiments are actually performed.

[0017] <Functional Configuration> Fig. 1 is a diagram showing an example of the configuration of a data generation device according to this embodiment. The data generation device 100 is an information processing device having a computer configuration, or a system including multiple computers. The data generation device 100 realizes each of the functional configurations shown in Fig. 1 by executing a predetermined program on the computer included in the data generation device 100.

[0018] 1, the data generating device 100 realizes various functional components, such as an observed value estimating unit 110, a cause identifying unit 120, a fault selecting unit 130, a system control unit 140, and a learning database 150. Note that at least some of the above functional components may be realized by hardware.

[0019] The observed value estimation unit 110 executes an observed value estimation process for estimating an observed value of the system 10 to be verified for a pair of a fault and a parameter. For example, the observed value estimation unit 110 estimates the observed value of the system 10 using a second model 111, which is a machine learning model or a statistical model that has learned the relationship between the pair of a fault and a parameter and the observed value of the system 10 or a value based on the observed value of the system 10.

[0020] The pair of a fault and a parameter includes a fault type, a fault location, and parameters indicating the state of the system 10 (e.g., the number of users, a setting value of the system 10, etc.). The observed value of the system 10 is an observed value generated by the system 10 when a pair of a fault and a parameter is inserted into the system 10. The observed value estimator 110 estimates an observed value (estimated value) that is an estimate of the observed value of the system 10.

[0021] The factor identification unit 120 executes a factor identification process to identify a failure factor of the system 10. For example, the factor identification unit 120 identifies (estimates) a failure factor of the system 10 by using a first model 121, which is a machine learning model or a statistical model that has learned the relationship between the observed values ​​of the system 10 or values ​​based on the observed values ​​of the system, and the failure factor of the system 10.

[0022] The fault selection unit 130 determines the validity of the fault and parameter pair based on the output result when the observed value (estimated value) of the system 10 estimated by the observed value estimation unit 110 is input to the cause identification unit 120, and selects the fault and parameter pair to be inserted into the system 10.

[0023] For example, the fault selection unit 130 inputs pairs of faults and parameters to the observed value estimation unit 110, and the observed value estimation unit 110 inputs observed values ​​(estimated values) of the system 10 to the cause identification unit 120. The fault selection unit 130 also compares the estimated values ​​regarding the fault type and / or location output by the cause identification unit 120 with the actual fault type and / or location, and determines the validity of the faults and parameters input to the observed value estimation unit 110 based on the comparison result. For example, the fault selection unit 130 determines that the validity is high when the two are different or when the difference is large, based on the accuracy of the comparison result, the certainty of the estimation, the similarity, etc.

[0024] The system control unit 140 includes, for example, a fault insertion unit 141 and a current status restoration unit 142. The fault insertion unit 141 executes a fault insertion process to insert a pair of a fault and a parameter selected by the fault selection unit 130 into the system 10 to be verified. The current status restoration unit 142 executes a current status restoration process to restore the system 10 to its current status before the fault insertion unit 141 inserts a new pair of a fault and a parameter. The system control unit 140 inserts the pair of a fault and a parameter selected by the fault selection unit 130 into the system 10, and acquires observed values ​​of the system 10 as learning data for further learning the first model 121 and the second model 111.

[0025] The learning database 150 stores the observation data output by the system 10 when the system control unit 140 inputs a pair of a fault and a parameter into the system 10 as learning data for further learning the first model 121 and the second model 111.

[0026] <Problem Setting> The problem setting of this embodiment will now be described. Let y∈Y be the fault index that can be inserted into the system to be verified. The fault index is discrete and is the target that we estimate (classify) during testing. The fault index is defined based on the fault type, such as packet loss or CPU load, and the location where the fault occurs. In addition to the fault index, let u∈U be a background parameter (hereinafter simply referred to as a parameter) that defines the state of the target system. u is a vector including continuous and discrete values, and its value can be freely set during fault insertion experiments. The parameter u is, for example, the number of users or a service tuning parameter, and is not necessarily the target of estimation during testing. u may be extended to time-series data.

[0027] The state of the system to be verified is expressed by the observed value x∈X. x is a vector including continuous and discrete values, and may be the system's output value itself or something that can be calculated from the output value. The observed value x may be arbitrarily defined to be used as an input feature of the factor identifier described below. The observed value x may be, for example, the CPU usage rate or traffic volume of each component. In this formulation, the system to be verified can be regarded as a function f (or a stochastic process) that gives the observed value x from the fault and parameter pair (y, u).

[0028] In this embodiment, the aim is to construct a factor identifier h. This is a function that gives an estimated value ^y∈Y of a fault index from an observed value x, and corresponds to the first model 121 in FIG. 1. For example, when a neural network model is used as the factor identifier h, the estimated fault index ^y is obtained by taking the argmax of the output value of the softmax function. Note that the superscript "^" of ^y should actually be written directly above "y", but due to the notational constraints of the specification, it is written as "^y" here.

[0029] Building a fault identification tool based on fault injection consists of several phases. First, in the selection phase, pairs (y,u)∈Y×U that are expected to have a high injection effect are selected based on certain criteria from a large number of pairs. In the subsequent insertion phase, the selected {(y,u)} is injected into the target system to obtain {x}. Then, in the training phase, a fault identification tool h is trained based on the obtained data {(x,y)}. These phases are repeated to build the fault identification tool h. After building the fault identification tool h, we evaluate whether it can correctly estimate the fault index y when given x, which is the value when a fault actually occurs. This final evaluation is called the test phase. These stages can be performed in a verification environment or, similar to chaos engineering, in a commercial environment.

[0030] The task in this embodiment is as follows: Y, U, X, and f are given. Assume that obtaining observed values ​​in a fault injection experiment, i.e., sampling the value of f, is expensive or time-consuming. Therefore, how should each phase be designed to achieve a highly accurate factor identifier h with as few injections as possible? In particular, how should {(y, u)} be selected to be given to f in the selection phase?

[0031] (Basic Concept) In the selection phase, rather than uniform random selection or exhaustive selection, the learning efficiency of h is improved by preferentially selecting specific {(y,u)}. This embodiment focuses on the following two points. First, whether (y,u) is worth inserting can be measured by whether the cause identifier h can correctly estimate the fault at this point. For faults that are difficult to identify, it is necessary to intensively select a larger number of {u}. Second, there is the possibility of generating pseudo-data. For example, a neural network (NN) with high expressive power may be able to simulate fault data. In addition, data can be output in a much shorter time than injecting faults into a real system.

[0032] However, achieving the above two points is not easy. Labeled observation data is required to know whether the factor identification unit h can correctly estimate a fault. Furthermore, if the pseudo data generated by the NN is of low quality, it may adversely affect the learning of the factor identification unit h. In consideration of these problems, in this embodiment, pseudo data is generated by introducing an NN or any other machine learning model, but this pseudo data is used not as learning data but as labeled data for indirectly measuring the performance of the factor identification unit h.

[0033] (Overview of Framework) Fig. 2 is a diagram showing an overview of the framework according to this embodiment. The framework 200 according to this embodiment includes the above-mentioned selection phase 210, injection phase 220, training phase 230, and test phase 240.

[0034] A key feature of the framework 200 according to this embodiment is the introduction of an observation value estimator ~f 211 to the selection phase 210. Given a fault and parameter pair (u, y), the observation value estimator ~f 211 estimates ~x of the observation value x of the system under verification. Because observation values ​​generally include continuous values, ~f is a regression model, and x and ~x have different values. While the performance of the observation value estimator ~f 211 is not guaranteed, it has the advantage of being able to obtain ~x much faster than the system under verification f 221 when a fault and parameter pair (y, u) is input. When ~x is input to the factor identifier h 212, the estimated fault index is ~y. Again, there is no guarantee that this matches the true value y or ~y estimated using the output value x of the system f.

[0035] Note that the superscript "~" for ~f should actually be written directly above the "f", but due to limitations on the notation in the specification, it is written as "~f" here. Similarly, the superscript "~" for ~x should actually be written directly above the "x", but it is written as "~x" here. Similarly, the superscript "~" for ~y should actually be written directly above the "y", but it is written as "~y" here.

[0036] (Framework Algorithm) FIG. 3 is a diagram showing an example of a framework algorithm according to this embodiment.

[0037] In step S1, the framework 200 performs initial training on N init The fault and parameter pairs {(y,u)} are input to the system to be verified (System f221), and the initial data D init ={(y,u,x)} (step S11). init Using this, an observation estimator ~f211 and a factor specifier h212 are trained.

[0038] In step S2, the framework 200 inputs the input value candidates S={(y, u)} (all possible) to the observation value estimator ∼f 211 and the factor specifier h 212 in the selection phase 210, and generates the data D temp =(y, u, ~x, ~y) is obtained (step S21). The observation value estimator ~f211 corresponds to the second model 111 in Fig. 1. The factor identifier h212 corresponds to the first model 121 in Fig. 1.

[0039] 2, the framework 200 inputs a fault and a parameter pair (y, u) to an observation value estimator ~f211, which estimates an observation value (estimated value) ~x = ~f(y, u). The framework 200 also inputs the estimated observation value (estimated value) ~x to a factor identifier h212, which identifies a fault factor (estimated value) ~y = h(~x). The framework 200 can perform this process at high speed.

[0040] The framework 200 also determines whether y≠~y for each (y,u,~x,~y), and if y≠~y, stores the fault and parameter pair (y,u) as a tentative candidate in S' (step S22). Here, y≠~y means that at least one of ~f and f is significantly off the estimate. Therefore, this process makes it possible to exclude simple (y,u) that have already been trained, i.e., (y,u) with low practical value.

[0041] In step S3, the framework 200 inserts N add A fault and parameter pair (y, u) is arbitrarily selected and inserted into the system f221 to be verified, to obtain the observed value (y, u, x).

[0042] In step S4, in the learning phase 230, the framework 200 re-trains the observation value estimator ∼f 211 and the factor identifier h 212 using data D, which is a combination of the data up to that point and the additional data obtained in the insertion phase 220. The framework 200 repeatedly executes the processes of steps S1 to S3 a predetermined number of times.

[0043] Thus, an essential feature of the present algorithm 300 is that it effectively utilizes the high speed of ~f, even though the quality of ~f is not guaranteed. Furthermore, the present algorithm 300 avoids adverse effects on the learning of the factor identifier h212 by not directly using pseudo data based on ~f, whose accuracy is not guaranteed, as training data. On the other hand, the present algorithm 300 does not require exact values ​​when obtaining execution values, so it uses ~f instead of f to quickly obtain approximate values ​​over a wide space. Furthermore, the present algorithm 300 is expected to further enhance the above-mentioned effects by repeatedly re-learning using additional data, as the accuracy of ~f improves.

[0044] In addition to the above processing, the algorithm 300 may include the following modifications or improvements.

[0045] In the above explanation, N is add The factors are selected arbitrarily, and the criteria for this include the following: 1) Uniform random sampling 2) Sampling based on a probability distribution designed based on the error estimation rate (so that factors with a high error estimation rate are more likely to be selected preferentially) 3) When the factor specifier h212 outputs not only ~y but also a certainty (such as the output value of a softmax function), the priority is determined using a parameter based on the certainty, such as (certainty of ~y) - (certainty of y).

[0046] Furthermore, in the above explanation, only data for which ~y ≠ y was used as training data for the observation value estimator ~f211 and the factor identifier h212, but there is also an approach in which data for which ~y = y is also used for training. In this case, it is possible to assume that there is a quality bias between the training data for which ~y ≠ y and the training data for which ~y = y, and to prioritize one of the training data by weighting the loss function during training. In particular, it is considered that the quality of the training data for which ~y = y will improve once the training of the observation value estimator ~f211 and the factor identifier h212 has progressed to a certain extent, so the weight of the loss function may be gradually increased as training progresses.

[0047] Furthermore, the learning of the observation value estimator ~f211 and the factor identifier h212 may be performed by re-learning from the initial state in each iteration, or by additional learning based on a trained model.

[0048] <Processing Flow> Figure 4 is a flowchart showing an example of data generation processing according to this embodiment. This processing illustrates an example of data generation processing executed by the data generation device 100, which is an example of the framework 200 according to this embodiment. Here, we will mainly explain the operation of each functional component in the data generation processing that narrows down the fault and parameter pairs (y, u) to be inserted into the system 10 to be verified. Note that at the start of the processing in Figure 4, the first model 121, which is the factor specifier h212, and the second model 111, which is the observation value estimator ~f211, are assumed to have completed initial learning.

[0049] In step S401, the observation value estimation unit 110 estimates an observation value (estimated value) ~x of the system 10 to be verified for the pair (y, u) of fault and parameter input from the fault selection unit 130, and passes the estimated ~x to the cause identification unit. For example, the observation value estimation unit 110 estimates the observation value (estimated value) ~x of the system 10 using a second model 111, which is a machine learning model or a statistical model that has learned the relationship between the pair of fault and parameter and the observed value of the system 10 or a value based on the observed value of the system 10.

[0050] In step S402, the factor identification unit 120 identifies (estimates) a fault factor (estimated value) ~y of the system 10 from the observed value (estimated value) ~x of the system 10 received from the observed value estimation unit 110, and passes the identified fault factor (estimated value) ~y of the system 10 to the fault selection unit 130. For example, the factor identification unit 120 inputs the observed value (estimated value) ~x of the system 10 to the first model 121, which is a machine learning model or a statistical model that has learned the relationship between the observed value of the system 10 or a value based on the observed value of the system, and the fault factor of the system 10, to obtain the fault factor (estimated value) ~y of the system 10.

[0051] In step S403, the fault selection unit 130 determines the validity of the fault and parameter pair (y, u) based on the fault cause (estimated value) ~y of the system 10 received from the cause identification unit 120, and selects the fault and parameter pair (y, u) to be inserted into the system 10. For example, the fault selection unit 130 compares the estimated value of the fault type and / or location output by the cause identification unit 120 with the actual fault type and / or location, and if the two differ, determines that the validity is high and selects the pair as a candidate for the fault and parameter pair to be inserted into the system 10.

[0052] In step S404, the system control unit 140 inserts the fault and parameter pair (y, u) selected by the fault selection unit 130 into the system 10, and acquires the resulting observation value x of the system 10 as training data.

[0053] By the process of FIG. 4, the data generating device 100 can generate data such as pairs of faults and parameters to be inserted into the system 10, and training data, in order to build a highly accurate first model 121 with fewer insertions.

[0054] <Hardware Configuration> The data generating device 100 has the hardware configuration of a computer 500 as shown in Fig. 5. Alternatively, the data generating device 100 is configured by a plurality of computers 500. Note that the data generating device 100 may be realized by a program executed on a virtual machine on the cloud.

[0055] 5 is a diagram showing the hardware configuration of a computer 500. In the example of FIG. 5, a computer 500 includes a processor 501, a memory 502, a storage device 503, a communication device 504, an input device 505, an output device 506, a bus B, and the like.

[0056] The processor 501 is, for example, an arithmetic device such as a CPU (Central Processing Unit) that executes predetermined programs to realize various functions. The memory 502 is a storage medium readable by the computer 500, and includes, for example, a RAM (Random Access Memory) and a ROM (Read Only Memory). The storage device 503 is a computer-readable storage medium, and may include, for example, a HDD (Hard Disk Drive), an SSD (Solid State Drive), various optical disks, and magneto-optical disks.

[0057] The communication device 504 includes one or more pieces of hardware (communication devices) for communicating with other devices via a wireless or wired network. The input device 505 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The output device 506 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that outputs to the outside.

[0058] The bus B is commonly connected to the above components and transmits, for example, address signals, data signals, and various control signals. The processor 501 is not limited to a CPU, and may be, for example, a DSP (Digital Signal Processor), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array).

[0059] (Supplementary Note) The data generating device 100 in this embodiment is not limited to being realized by a dedicated device, but may also be realized by a general-purpose computer. In this case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed to realize the function. Note that the term "computer system" here includes hardware such as an OS and peripheral devices.

[0060] Furthermore, "computer-readable recording media" includes various storage devices such as portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines when transmitting programs via networks such as the Internet or communication lines such as telephone lines, and devices that store programs for a certain period of time, such as volatile memory within computer systems that serve as servers or clients in such cases.

[0061] Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in a computer system, or may be one that is realized using hardware such as a PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

[0062] <Effects of this embodiment> Using AWS (registered trademark) services, a Kubernetes (registered trademark) ver. 1.22 cluster was constructed on two virtual nodes running Amazon (registered trademark) Linux (registered trademark) 2. The system to be verified was the publicly available Sock-shop, a demo environment for Kubernetes-based microservices simulating an EC (Electric Commerce) site. This environment consisted of 14 containers. Various container metrics were collected using a separately created Prometheus (registered trademark) pod. Background traffic was generated using the load testing tool Locust.

[0063] This allows a pseudo-user to open a catalog page, log in, add products to a cart, and place an order. ChaosMesh was used for fault injection. In particular, the inventors considered a fault that would inject CPU load or packet loss into one of the containers. The scale of the fault to be inserted (the increase in CPU load, the loss rate in packet loss) can be specified as an integer value.

[0064] Considering the problem of identifying the container inserted by the fault, the number of fault indices was set to |Y| = 14. The background parameters were set to two dimensions u = (u_1, u_2), where u_1 is the number of Locust pseudo-users (an even value between 0 and 120), and u_2 is a parameter representing the scale of the insertion fault (an even value between 10 and 49). The observed value x has 92 dimensions and consists of metrics such as CPU, memory, and sent / received traffic obtained from Prometheus, as well as response time statistics, request processing rate, and request failure rate obtained from Locust.

[0065] The inventors created a data pool by collecting observed values ​​for all faults and background parameters in advance. When using the framework according to this embodiment in operation, faults are injected into the system sequentially and data is collected, so no data pool exists. However, in this experiment, in order to maintain fairness among the methods and to conduct a large number of experiments in a short period of time and perform statistical evaluation, data was accumulated in advance and accessed for the experiment.

[0066] For each fault and parameter pair (y,u), we waited two minutes after injecting the fault into the system, and then obtained the observations after the system state had stabilized. Since it is necessary to restore the system to its current state before injecting another fault, this may take even longer. Since each iteration requires at least a few minutes, conducting a fault injection experiment without using a data pool could take several days or more.

[0067] The baselines to be compared with the proposed framework are as follows: Random: Random insertion. In sequential fault insertion and training data generation, (y,u) is determined uniformly randomly. This is essentially equivalent to determining all training data samples uniformly randomly in advance. Uncertainty: Uncertainty-based insertion. After inputting all {(y,u)} into the observation value estimator, N values ​​of Pr(~y)-Pr(y) are selected in descending order. add It selects and inserts {(y,u)}. It differs from the proposed framework in that it does not involve a hard judgment based on whether y = ~y or not. UsePseudo: Uses pseudo data. In the proposed framework, {(y,u)} where y = ~y is included in the output data of the observation value estimator and used to train the fault detector. This is a simple method that uses pseudo data generated by a NN to solve the problem of limited training data.

[0068] The hyperparameters of the NN used for the factor specifier and observation value estimator are as follows (excluding the dimensions of the input and output layers): number of units (input, 50, 100, 50, output), activation function ReLu (the final layer of the factor specifier is softmax), dropout rate 0.5, mini-batch 32, learning rate 0.001.

[0069] The experimental results are shown in Figures 6 and 7. Graph 600 in Figure 6 plots the number of fault insertions required to first exceed the threshold accuracy of the fault identifier. Graph 600 shows that Proposal 601, the proposed method according to this embodiment, reached the given target accuracy value with 40 to 50 fewer insertions than the other baselines. This result can also be seen in Table 700 in Figure 7, which is an excerpt from the results in Figure 6. As mentioned above, fault insertion can take a considerable amount of time depending on the system, and this difference can result in a difference of days in real time.

[0070] Furthermore, if faults are injected in a commercial environment to train a model, as in chaos engineering, then the proposed method according to this embodiment is expected to enable the release of a fault detector at an earlier stage.

[0071] As described above, the data generating device 100 according to this embodiment can improve the efficiency of learning a statistical model or a machine learning model that identifies system failure causes. For example, the data generating device 100 can generate fault data with higher learning value compared to random fault injection or learning data generation, thereby reducing the number of fault injections, experiment time, experiment costs, and the like.

[0072] Summary of Embodiments This specification discloses at least the data generation device described in the following paragraphs. (Item 1) A data generation device for efficiently learning a first model that is a machine learning model or a statistical model that identifies a fault cause of a system, the data generation device comprising: an observation value estimation unit that estimates an observation value of the system for a pair of a fault and a parameter; a factor identification unit that identifies a fault cause of the system using the first model; and a fault selection unit that determines the validity of the pair of a fault and a parameter based on an output result when the observation value of the system estimated by the observation value estimation unit is input to the factor identification unit, and selects the pair of a fault and a parameter to be inserted into the system. (Item 2) The data generation device described in paragraph 1, wherein the factor identification unit identifies a fault cause of the system using the first model that has learned the relationship between the observation value of the system or a value based on the observation value of the system and the fault cause of the system, inserts the pair of a fault and a parameter selected by the fault selection unit into the system, and uses the observation data of the system as training data for further training the first model. (3) The data generation device according to paragraph 1 or 2, wherein the observation value estimation unit estimates the observation values ​​of the system using a second model that is a machine learning model or a statistical model that has learned the relationship between the pairs of faults and parameters and the observation values ​​of the system or values ​​based on the observation values ​​of the system, and inserts the pairs of faults and parameters selected by the fault selection unit into the system, and uses the observation data of the system as learning data for further training the second model. (4) The data generation device according to any of paragraphs 1 to 4, wherein the fault selection unit inputs the pairs of faults and parameters to the observation value estimation unit, compares estimated values ​​regarding the type and / or location of the fault obtained by inputting the observation values ​​of the system estimated by the observation value estimation unit into the cause identification unit with the actual type and / or location of the fault, and determines validity of the pairs of faults and parameters based on the comparison result.(5) A data generation method, in which a data generation device that improves the efficiency of learning of a first model that is a machine learning model or a statistical model that identifies a cause of a system failure executes: an observation value estimation process that estimates an observation value of the system for a pair of a failure and a parameter, a cause identification process that identifies a cause of a system failure using the first model, and a failure selection process that determines the validity of the pair of the failure and the parameter based on an output result when the observation value of the system estimated in the observation value estimation process is input to the cause identification process, and selects the pair of the failure and the parameter to be inserted into the system. (6) A program for causing a computer to function as each unit in the data generation device described in any one of paragraphs 1 to 4, or a storage medium that stores a program.

[0073] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications or changes are possible within the scope of the gist of the present invention described in the claims.

[0074] REFERENCE SIGNS LIST 10 System 100 Data generation device 110 Observation value estimation unit 111 Second model 120 Cause identification unit 121 First model 130 Fault selection unit 140 System control unit 141 Fault insertion unit

Claims

1. A data generation device that improves the efficiency of learning of a first model, which is a machine learning model or statistical model that identifies the causes of system failures, comprising: an observation value estimation unit that estimates an observation value of the system for a pair of a failure and a parameter; a factor identification unit that identifies the causes of system failures using the first model; and a failure selection unit that determines the validity of the pair of failure and parameter based on the output result when the observation value of the system estimated by the observation value estimation unit is input to the factor identification unit, and selects the pair of failure and parameter to be inserted into the system.

2. The data generation device according to claim 1, wherein the factor identification unit identifies the fault factor of the system using the first model that has learned the relationship between the observed values of the system or values based on the observed values of the system and the fault factors of the system, inserts the pair of the fault and parameter selected by the fault selection unit into the system, and uses the observed data of the system as learning data for further learning the first model.

3. The data generation device according to claim 1 or 2, wherein the observation value estimation unit estimates the observation value of the system using a second model, which is a machine learning model or a statistical model that has learned the relationship between the fault and parameter pair and the observation value of the system or a value based on the observation value of the system, inserts the fault and parameter pair selected by the fault selection unit into the system, and uses the observation data of the system as training data for further training the second model.

4. The data generation device according to claim 1, wherein the fault selection unit inputs the pair of fault and parameter to the observation value estimation unit, compares the estimated value regarding the type and / or location of the fault obtained by inputting the observed value of the system estimated by the observation value estimation unit to the cause identification unit with the actual type and / or location of the fault, and determines whether or not to insert the pair of fault and parameter into the system based on the accuracy of the comparison result or the certainty of the estimation.

Images