Causal model construction device, causal model construction method, and program

Abstract

A causal model construction device according to one aspect of the present disclosure comprises a collection unit configured to acquire network topology information representing the network topology of an ICT system for which an abnormal location is to be estimated, and a model construction unit configured to use the network topology information to construct a causal model for estimating the abnormal location from observation data at a time when an abnormality has occurred in the ICT system.

Description

[Technical Field]

[0001] The present disclosure relates to a causal model construction device , cause Method for constructing a model , and and programs. [Background technology]

[0002] For ICT (Information and Communication Technology) providers, it is important to understand the status of anomalies that occur within ICT systems and to respond quickly to them. In this context, research has been conducted on methods for early detection of anomalies that occur within ICT systems and methods for estimating the location of anomalies.

[0003] As a method for estimating the location of an anomaly, a method has been proposed in which the relationship between the anomaly location and the change in data (hereinafter also referred to as "observation data") in an ICT system caused by that anomaly is modeled as a causal model using a Bayesian network, and the anomaly location is estimated from the observation data at the time of the anomaly (Non-Patent Documents 1 to 3). These methods can be classified as either rule-based methods or data-driven methods.

[0004] The rule-based approach is a modeling method that follows predefined rules. In the rule-based approach, the relationship between anomaly locations and changes in observed data is modeled primarily using the knowledge of experts, such as ICT system operators. For example, in Non-Patent Document 1, a rule is created from expert knowledge that states that the normality or abnormality of a router only affects the observed data of adjacent links, and a causal model is constructed using this rule and the adjacent relationships in the ICT system's network topology. In addition, Non-Patent Document 2 proposes making it easier to construct a causal model by creating abstract rules called templates.

[0005] The data-driven method is a method of modeling from data. In the data-driven method, observed data from when an abnormality occurred in the past is used to model the relationship between the abnormal location and the change in the observed data at that time. For example, in Non-Patent Document 3, the relationship between a certain failure is modeled using data from multiple past cases.

[0006] Incidentally, methods for estimating abnormal locations use syslogs and traffic information from ICT systems, but in recent years, it has become easier to obtain various types of observation data other than syslogs and traffic information, such as flow data, telemetry data, and sensor data related to communication devices. Therefore, it is thought that by using these various types of observation data, it will be possible to estimate abnormal locations with finer granularity. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Srikanth Kandula, Dina Katabi, and Jean-philippe Vasseur. Shrink: A tool for failure diagnosis in IP networks. Proceedings of the 2005 ACM SIGCOMM workshop on Mining network data, pages 173-178, 2005. [Non-patent document 2] He Yan, Lee Breslau, Zihui Ge, Dan Massey, Dan Pei, and Jennifer Yates. G-RCA: A Generic Root Cause Analysis Platform for Service Quality Management in Large IP Networks. IEEE / ACM Transactions on Networking, 20(6):1734-1747, 2012. [Non-patent document 3] Kandula, Srikanth and Mahajan, Ratul and Verkaik, Patrick and Agarwal, Sharad and Padhye, Jitendra and Bahl, Paramvir. Detailed diagnosis in enterprise networks. ACM SIGCOMM Computer Communication Review, vol.39, num.4, pp.243-254, 2009. Summary of the Invention [Problem to be solved by the invention]

[0008] However, when constructing a causal model using various types of observational data, the following problems arise.

[0009] Issue 1: Rule-based methods require expert knowledge in advance for modeling, but the number of types of observation data used in conventional technologies is extremely small. In addition, abnormalities that occur in ICT systems have a ripple effect and affect various types of observation data, making it difficult to create rules for each and every relationship between abnormalities that occur in ICT systems and the various types of observation data.

[0010] Issue 2: When various types of observational data are input into a Bayesian network, it is necessary to determine whether the value of each piece of observational data obtained from an ICT system is normal or abnormal (this is also known as binarization). Conventional technology targets observational data that has an extremely small number of types and is easy to binarize (for example, alert information indicating whether an alert has occurred). However, when various types of observational data are input, the normal range for each piece of observational data has its own characteristics, making it difficult to binarize it while taking these into account.

[0011] Issue 3: Data-driven methods require observational data from when anomalies occurred in the past, but anomalies generally do not occur frequently in ICT systems, and as the types of observational data become more diverse, the number of patterns that the observational data can take for anomalies increases. For this reason, it is generally difficult to collect enough anomaly cases to compensate for this increase.

[0012] Issue 4: Furthermore, in recent years, ICT system virtualization technology has led to increasingly frequent changes in the network topology of ICT systems. Consequently, the observation data acquired from ICT systems also changes frequently. For this reason, it is difficult to create rules for the relationship between anomalies and observation data one by one using rule-based methods, and it is difficult to collect sufficient anomaly cases using data-driven methods.

[0013] The present disclosure has been made in consideration of the above points, and aims to provide a technology that can construct a causal model from network topology information when constructing a causal model for various types of observation data. [Means for solving the problem]

[0014] A causal model construction device according to one aspect of the present disclosure includes a collection unit configured to acquire network topology information representing the network topology of an ICT system in which an abnormality location is to be estimated, and a model construction unit configured to use the network topology information to construct a causal model for estimating the abnormality location from observation data when an abnormality occurs in the ICT system. [Effects of the Invention]

[0015] When constructing a causal model for various types of observation data, a technique is provided that enables the construction of the causal model from network topology information. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram illustrating an example of a hardware configuration of an abnormality location estimation device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of a functional configuration of an abnormality location estimation device according to the present embodiment. [Figure 3] 10 is a flowchart illustrating an example of a causal model construction process according to the present embodiment. [Figure 4] 10 is a flowchart illustrating an example of an abnormality location estimation process according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] An embodiment of the present invention will be described below. In the following embodiment, an anomaly location estimation device 10 will be described. The anomaly location estimation device 10 constructs a causal model from network topology information of an ICT system and uses this causal model to estimate an anomaly location in the ICT system from various types of observation data. The anomaly location estimation device 10 according to this embodiment has a "model construction phase" in which a causal model is constructed from network topology information of the ICT system, and an "anomaly location estimation phase" in which this causal model is used to estimate an anomaly location from observation data at the time of an anomaly occurrence. The anomaly location estimation device 10 in the model construction phase may be referred to as a "model construction device," for example. Network topology information is information representing the network topology of an ICT system. Network topology information is, for example, information representing a graph structure in which various devices (e.g., routers, servers, etc.) constituting the ICT system are represented as nodes and communication paths between the nodes are represented as links.

[0018] <Theoretical structure> First, the theoretical configuration of the causal model construction in the model construction phase and the abnormality location estimation in the abnormality location estimation phase will be described.

[0019] Let i∈{1, ,N} be the equipment that constitutes the ICT system for which the causal model is constructed and the abnormality location is identified, and let x i ∈{0,1}, where N represents the number of devices, and x i When it is 0, it indicates a normal state, and when it is 1, it indicates an abnormal state.

[0020] Also, let j∈{1, ,M} be the observation data, and let the state of observation data j be y j ∈{0,1}, where M represents the number of observations, and y j is assumed to represent a normal state when it is 0 and an abnormal state when it is 1. Note that examples of observed data j include various data that can be obtained from devices that make up the ICT system (e.g., syslog, traffic information, flow data, telemetry data, sensor data, etc.).

[0021] For each device i, a representative node k∈{1, ,N} is introduced, and the state of the representative node k is expressed as r k ∈{0,1}, where r k When it is 0, it indicates a normal state, and when it is 1, it indicates an abnormal state.

[0022] In addition, x i , y j and r k can be multi-valued, 3 or more values, instead of 2 values ​​of 0 or 1.

[0023] Each device i has one representative node k. The representative node k is a node that represents the state of the observation data that can be obtained from the corresponding device i. The state r of the representative node k k is determined based on the contribution of the observed data to the anomaly (described later). i r represents the state of observation data that can be obtained from the corresponding equipment i k causal model (i.e., the state r of representative node k) k construct a causal model for

[0024] Below, we will explain how to build a causal model and how to estimate the location of an anomaly, and then we will explain how to determine the contribution to an anomaly and the state of the representative node.

[0025] ·Causal model construction method and abnormality location estimation method The causal model is a prior probability P(X=x1, ,x N |α) and the conditional probability P(R=r1, ,r N The prior probability is the probability that each device will easily enter an abnormal state, and is defined as follows:

[0026]

number

[0027] Next, we define the conditional probability. The conditional probability represents the causal relationship between a device and a representative node and the degree of that relationship. In addition, the causal relationship is expressed as follows: if a device i has a causal relationship with a representative node k, then x i and r k Between the edges i,k Here, the causal relationship between device X and representative node R is defined as follows using network topology information:

[0028]

number

[0029] Then, let E be the set of all edges between device X and representative node R, and let edge e i,k The parameter φ represents the index of i,k This gives us the following:

[0030]

number

[0031]

number

[0032] Finally, when the state of the representative node is given, the anomaly location is estimated by solving the following using the prior probability and conditional probability:

[0033]

number

[0034] As described above, it is possible to construct causal models for various types of observed data using only network topology information.

[0035] -Determining the contribution to anomalies and the state of representative nodes Next, we will explain how to determine the contribution to an anomaly and the representative node. Let c be an M-dimensional vector, and the element c of each vector j Let x represent the contribution of observation data j to the anomaly. Here, the contribution to the anomaly is a value that represents the degree to which each observation data item affects the anomaly. Therefore, observation data acquired from equipment close to the abnormal equipment will have a higher contribution than observation data acquired from equipment far from the abnormal equipment. In this way, by using the contribution as an input rather than using the observation data value directly, it can be assumed that an abnormal equipment item only affects the observation data acquired from its nearby equipment. Furthermore, because the contribution to the anomaly represents the degree to which each observation data item affects the anomaly, there is no need to consider the diversity of the characteristics of each observation data item, and it is possible to set the binarization threshold based solely on the magnitude of the contribution.

[0036] The contribution to anomalies can be calculated by applying the methods described in, for example, References 3 and 4 to an AutoEncoder (Reference 2) trained using M-dimensional normal observation data.

[0037] For example, let the loss function used to train AutoEncoder be L(v) = ||v-^v||, where v is the input to AutoEncoder and ^v is the output from AutoEncoder. In this case, the contribution c is c=argmin γIt can be calculated as L(v + γ) + λ|γ|, where λ is a preset constant. This means finding a γ that reduces the value of L(v) (i.e., reduces the degree of anomaly). This is because the found γ is considered to reduce the degree of anomaly, that is, to contribute to the anomaly. Note that the second term in the above equation for calculating the contribution c is a penalty term to ensure that γ satisfies sparsity.

[0038] Next, we will explain how to determine the state of the representative node. The set of the top s values ​​of the contribution factor c to the anomaly is called D. s Here, the value of s can be determined arbitrarily, but it can be set to, for example, the integer part of 1% of the number of types of observation data M. Then, among the element numbers of each element of c, D s A set of element numbers corresponding to the values ​​contained in s That is, Ω s ={j||c j |∈D s}

[0039] Then, the state of the representative node r k is determined by the following formula:

[0040]

number

[0041] That is, for each k∈{1, ,N}, calculate f(k) and add Ω to the index set that represents the result of the calculation. s If there is at least one element of k = 1, otherwise r k =0.

[0042] <Example of Hardware Configuration of Anomaly Location Estimation Device 10> An example of the hardware configuration of an abnormality location estimating device 10 according to this embodiment is shown in Fig. 1. As shown in Fig. 1, the abnormality location estimating device 10 according to this embodiment includes an input device 101, a display device 102, an external I / F 103, a communication I / F 104, a RAM (Random Access Memory) 105, a ROM (Read Only Memory) 106, an auxiliary storage device 107, and a processor 108. These hardware components are connected to each other via a bus 109 so as to be able to communicate with each other.

[0043] The input device 101 is, for example, a keyboard, a mouse, a touch panel, a physical button, etc. The display device 102 is, for example, a display, a display panel, etc. Note that the abnormality location estimating device 10 does not necessarily have to include at least one of the input device 101 and the display device 102.

[0044] The external I / F 103 is an interface with an external device such as a recording medium 103a. The abnormality location estimating device 10 can read from and write to the recording medium 103a via the external I / F 103. Examples of the recording medium 103a include a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

[0045] The communication I / F 104 is an interface for connecting the anomaly location estimation device 10 to a communication network. The RAM 105 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 106 is a non-volatile semiconductor memory (storage device) that can store programs and data even when the power is turned off. The auxiliary storage device 107 is a storage device (storage device) such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a flash memory. The processor 108 is an arithmetic device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

[0046] The abnormality location estimation device 10 according to this embodiment has the hardware configuration shown in Fig. 1 and can thereby realize the causal model construction process and the abnormality location estimation process described below. Note that the hardware configuration shown in Fig. 1 is an example, and the hardware configuration of the abnormality location estimation device 10 is not limited to this. For example, the abnormality location estimation device 10 may have multiple auxiliary storage devices 107 and multiple processors 108, may not have some of the hardware shown in the figure, or may have various hardware other than the hardware shown in the figure.

[0047] <Example of functional configuration of abnormality location estimation device 10> Fig. 2 shows an example of the functional configuration of the anomaly location estimation device 10 according to this embodiment. As shown in Fig. 2, the anomaly location estimation device 10 according to this embodiment includes a collection unit 201, a causal model construction unit 202, a contribution degree calculation unit 203, an estimation unit 204, and a user interface unit 205. These units are realized, for example, by processing in which one or more programs installed in the anomaly location estimation device 10 are executed by the processor 108 or the like. The anomaly location estimation device 10 according to this embodiment also includes an ICT system data DB 301, a causal model DB 302, and a contribution degree DB 303. These DBs are realized, for example, by the auxiliary storage device 107 or the like.

[0048] The collection unit 201 collects network topology information and each piece of observation data j from the ICT system. The network topology information and each piece of observation data j collected by the collection unit 201 are stored in the ICT system data DB 301.

[0049] The causal model construction unit 202 uses the network topology information stored in the ICT system data DB 301 to construct a causal model (i.e., the prior probability P(X=x1, . . . , x N |α) and the conditional probability P(R=r1, ,r N |X, β, φ) is constructed. The causal model constructed by the causal model construction unit 202 is stored in the causal model DB 302.

[0050] When estimating an abnormality location, the contribution calculation unit 203 calculates the contribution c to the abnormality using each piece of observation data j stored in the ICT system data DB 301. The contribution c calculated by the contribution calculation unit 203 is stored in the contribution DB 303.

[0051] The estimation unit 204 estimates the abnormality location ^X using the causal model stored in the causal model DB 302 and the contribution degree c stored in the contribution degree DB 303. That is, the estimation unit 204 estimates the state r of the representative node k from the contribution degree c. k After determining the state r of these representative nodes k, k The abnormal location ^X is estimated using the above equation 5.

[0052] The user interface unit 205 presents the abnormal location ^X estimated by the estimation unit 204 to a user (for example, an operator of an ICT system, etc.).

[0053] <Causal model construction process> The causal model construction process according to this embodiment will be described below with reference to Fig. 3. The causal model construction process is a process executed in the model construction phase. In the following, it is assumed that the network topology information collected by the collection unit 201 is stored in the ICT system data DB 301.

[0054] The causal model construction unit 202 receives the network topology information stored in the ICT system data DB 301 (step S101).

[0055] Next, the causal model construction unit 202 uses the network topology information input in step S101 to construct a causal model (the prior probability P(X=x1, . . . , x N |α) and the conditional probability P(R=r1, ,r N |X, β, φ) is constructed (step S102).

[0056] Then, the causal model constructing unit 202 stores the causal model constructed in the above step S102 in the causal model DB 302 (step S103).

[0057] <Abnormal location estimation process> The abnormality location estimation process according to this embodiment will be described below with reference to Fig. 4. The abnormality location estimation process is a process executed in the abnormality location estimation phase. In the following, it is assumed that some abnormality has occurred in the ICT system, and that each piece of observation data j at that time has been collected by the collection unit 201 and stored in the ICT system data DB 301.

[0058] The contribution calculation unit 203 receives the observation data j at the time of the occurrence of the abnormality (step S201).

[0059] Next, the contribution calculation unit 203 calculates the contribution c to the anomaly using each observation data j input in step S201 (step S202). That is, the contribution calculation unit 203 calculates, for example, the loss function used in learning the AutoEncoder as L(v)=||v-^v||, and calculates c=argmin γ Calculate the contribution c by L(v+γ)+λ|γ|.

[0060] Next, the contribution calculation unit 203 stores the contribution c calculated in the above step S202 in the contribution DB 303 (step S203).

[0061] Next, the estimation unit 204 estimates the abnormality location ^X using the causal model stored in the causal model DB 302 and the contribution c stored in the contribution DB 303 (step S204). That is, the estimation unit 204 calculates the state r of the representative node k from the contribution c using the above equation (6). k After determining the state r of these representative nodes k, k The abnormal location ^X is estimated using the above equation 5.

[0062] Then, the user interface unit 205 outputs the abnormal location ^X estimated in step S204 to the display device 102 such as a display, and presents it to the user (step S205).

[0063] <Summary> As described above, in the model construction phase, the anomaly location estimation device 10 according to this embodiment uses only network topology information under the assumption that "when an anomaly occurs in a certain device, it will affect the observation data of the device and its neighboring devices," and calculates the state r of the representative node k. k Furthermore, in the anomaly location estimation phase, the anomaly location estimation device 10 according to this embodiment can estimate the anomaly location by the causal model (Bayesian network) using the contribution c calculated from each piece of observation data j at the time of an anomaly occurrence. This allows the anomaly location estimation device 10 according to this embodiment to solve the above-mentioned problems 1 to 4.

[0064] That is, the anomaly location estimation device 10 according to this embodiment solves the problem 1, that "anomalies spread and affect various types of observation data," by using data called "contribution to anomaly," and also solves the problem that "it is difficult to formulate rules for the relationships between anomalies and various types of observation data one by one" by introducing a node called a representative node into the Bayesian network.

[0065] In addition, by using data called "contribution to anomalies," it is not necessary to consider the normal state of each observation data j, and it is possible to binarize it based only on the magnitude of the contribution value, thereby resolving issue 2. Furthermore, since a causal model can be constructed from network topology information alone, issue 4 is resolved, and in addition, issue 3 does not become a problem because past anomaly data is not used.

[0066] As described above, the abnormality location estimation device 10 according to this embodiment solves the above-mentioned problems 1 to 4, and makes it possible to estimate the abnormality location of an ICT system using a causal model for various types of observation data that can be acquired from the ICT system.

[0067] The present invention is not limited to the above-described specifically disclosed embodiments, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims.

[0068] [References] Reference 1: Kazuyuki Tanaka, [Tutorial Lecture] Fundamentals of Probabilistic Information Processing and Belief Propagation Algorithms, IEICE Technical Report, 2004. Reference 2: M. Sakurada and T. Yairi, "Anomaly detection using autoencoders with nonlinear dimensionality reduction," in Proc. MLSDA, ser. MLSDA'14, 2014, p. 4-11. Reference 3: Y. Ikeda, K. Tajiri, Y. Nakano, K. Watanabe, and K. Ishibashi, "Estimation of contributing dimensions to detected anomalies with variational autoencoders," arXiv preprint arXiv:1811.04576, 2018. Reference 4: Scott Lundberg, Su-In Lee, "A Unified Approach to Interpreting Model Predictions," in Proc. NIPS 2017. [Explanation of symbols]

[0069] 10. Anomaly location estimation device 101 Input Device 102 Display device 103 External I / F 103a Recording media 104 Communication I / F 105 RAM 106 ROM 107 Auxiliary storage 108 processors 109 Bus 201 Collection Department 202 Causal Model Building Department 203 Contribution Calculation Unit 204 Estimation section 205 User Interface Section 301 ICT System Data DB 302 Causal Model DB 303 Contribution DB

Claims

1. a collection unit configured to acquire network topology information representing a network topology of an ICT system in which an abnormality location is to be estimated; a model construction unit configured to construct a causal model for estimating an abnormality location from observation data when an abnormality occurs in the ICT system, using the network topology information; and and The model construction unit A causal model construction device configured to construct a Bayesian network as the causal model, which is defined by a priori probability representing the likelihood of an equipment constituting the ICT system falling into an abnormal state, and a conditional probability representing the causal relationship between the equipment and the state of a representative node representing the state of the observation data obtained from the equipment, and the degree of the causal relationship.

2. a collection procedure for acquiring network topology information representing the network topology of an ICT system in which an anomaly location is to be estimated; a model construction step of constructing a causal model for estimating the location of an anomaly from observation data when an anomaly occurs in the ICT system, using the network topology information; The computer executes The model building procedure includes: A causal model construction method in which a Bayesian network is constructed as the causal model, which is defined by a priori probability representing the likelihood of an equipment constituting the ICT system falling into an abnormal state, and a conditional probability representing the causal relationship between the equipment and the state of a representative node representing the state of the observation data obtained from the equipment, and the degree of the causal relationship.

3. A program for causing a computer to execute the causal model building method according to claim 2.

Images