System failure monitoring device and system failure monitoring method

Abstract

To provide a system failure monitoring device capable of easily estimating a root cause of a system.SOLUTION: A data collection part 141 acquires Pod and calculation node information 132 being configuration information about a Pod 202 for executing processing corresponding to a request and tracing data 131 about the request to be processed by the Pod 202. A determination part including a processing time calculation part 142 and an abnormality degree calculation determination part 143 determines whether to be an abnormality request related to abnormality of a monitoring object system 1 in each request on the basis of the tracing data 131 and the Pod and calculation node information 132. A presentation part including an abnormality request distribution calculation part 144 and an abnormality request visualization part 145 generates and presents visualized data showing an abnormality request distribution obtained by plotting the abnormality request within a request space specified by coordinate axes about the request.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a system fault monitoring device and a system fault monitoring method. [Background technology]

[0002] With the spread of distributed systems such as microservice architectures, the challenges of their operation and management are increasing. For example, when a problem occurs in a distributed system, it is necessary to quickly identify whether the cause is in the application or the underlying infrastructure, and distributed system operation managers are calling for a reduction in the time required to identify the root cause.

[0003] To address the above issues, it is important to introduce technology that monitors system operation and detects early signs of problems. Such technology can significantly reduce the time required for root cause analysis and improve system stability.

[0004] In response to this, Patent Document 1 discloses a technology that calculates feature amounts from data collected using a monitoring tool and determines the cause of a microservice failure (the causal relationship between an infrastructure failure and an application failure) based on the feature amounts. This technology learns the relationship between feature amounts and teacher labels, and determines the cause of a microservice failure using feature amounts acquired in a production environment. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-144401 Summary of the Invention [Problem to be solved by the invention]

[0006] In order to meet the demands of diverse applications and services, distributed systems are generally designed to execute applications in a distributed manner using virtual processing units. This can be expected to improve resource utilization, scalability, and flexibility. However, such distributed systems involve a large number of virtual processing units, and the interactions and dependencies between different applications running in the virtual environment become complex, making it difficult to identify the cause of a failure and perform efficient failure analysis when a failure occurs.

[0007] When the technology described in Patent Document 1 is applied to the above-mentioned distributed system, it is necessary to acquire information indicating application failures and information indicating processing node failures, evaluate the hierarchical relationships and dependencies between applications executed on different processing nodes, and calculate the similarity of error messages for different failure information. Furthermore, these evaluation values ​​are stored as features, and training data is created by acquiring and associating teacher labels. This training data is used to generate a failure prediction model that determines whether two pieces of failure information are related. While this method can efficiently analyze the causes of failures in virtualized systems, it has the following problems.

[0008] Specifically, because fault data and training data must be accumulated, when a new application is introduced, fault data and training data related to that application must be collected and accumulated. If this data is insufficient, the accuracy and reliability of the fault prediction model will decrease. Furthermore, when an application is updated, the updated application may have specific fault causes and behaviors that are not covered by the fault prediction model of the existing application. Therefore, to improve accuracy and reliability, the fault prediction model must be updated. Therefore, in order to keep up with development styles such as agile development, which allow for rapid application updates, it is costly to update the fault prediction model.

[0009] The present disclosure has been made in consideration of the above-mentioned problems, and aims to provide a system failure monitoring device and a system failure monitoring method that can easily estimate the root cause of a system failure. [Means for solving the problem]

[0010] A system failure monitoring device according to one aspect of the present disclosure is a system failure monitoring device that monitors a monitored system having a plurality of components that perform processing in response to requests, the system failure monitoring device comprising: The monitoring system includes a collection unit that collects configuration information about each component and request information about each request processed by each component; a determination unit that determines, for each request, based on the configuration information and the request information, whether the request is an abnormal request related to an abnormality in the monitored system; and a presentation unit that generates and presents visualized data showing the request distribution in which the abnormal requests are plotted within a request space defined by coordinate axes related to the requests. [Effects of the Invention]

[0011] According to the present invention, it becomes possible to easily deduce the root cause of a system. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration according to an embodiment of the present disclosure. [Figure 2] FIG. 1 illustrates an example of the configuration of a monitored system. [Figure 3] This is a diagram showing an example of dependencies between microservices and their execution environments. [Figure 4] FIG. 2 is a functional block diagram illustrating an example of a functional configuration of a system failure monitoring device. [Figure 5] FIG. 10 is a diagram showing an example of tracing data. [Figure 6] FIG. 10 is a diagram illustrating an example of Pod / computing node information. [Figure 7] FIG. 10 is a diagram illustrating an example of request abnormality level information. [Figure 8] 10 is a flowchart illustrating an example of a system failure monitoring process. [Figure 9] 10 is a flowchart illustrating an example of a processing time calculation process. [Figure 10] 10 is a flowchart illustrating an example of an abnormality determination process. [Figure 11] 10 is a flowchart illustrating an example of a distribution calculation process. [Figure 12] FIG. 10 is a diagram illustrating an example of abnormal request distribution information. [Figure 13] FIG. 10 is a diagram illustrating an example of an output screen. [Figure 14] 10 is a flowchart illustrating another example of the system failure monitoring process. DETAILED DESCRIPTION OF THE INVENTION

[0013] Embodiments of the present disclosure will be described below with reference to the drawings. The following description and drawings are examples for explaining the present disclosure, and appropriate omissions and simplifications have been made for clarity of explanation. The present disclosure can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural. The position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the disclosure. Therefore, the present disclosure is not necessarily limited to the position, size, shape, range, etc., disclosed in the drawings. In the following description, various information may be described using expressions such as "table" and "list," but the various information may be expressed using data structures other than these. To indicate independence from these data structures, "XX table," "XX list," etc. may be referred to as "XX information." When describing identification information, expressions such as "identification information," "identifier," "name," "ID," and "number" are used, and these terms are interchangeable. [Example]

[0014] 1 is a diagram illustrating an example of a system configuration according to a first embodiment of the present disclosure. The system includes a monitored system 1, a client terminal 2, and a system fault monitoring device 100. The system fault monitoring device 100 is connected to the monitored system 1 and the client terminal 2 via a network 3 so as to be able to communicate with each other. The network 3 is, for example, a public network such as the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network).

[0015] The monitored system 1 is an information processing system for executing business operations, and is a distributed system that runs applications on a virtualization platform built on multiple physical machines 10. In this embodiment, the monitored system 1 is built with microservices deployed on the virtualization platform. Microservices are an architectural style that divides a large-scale application into multiple small, independent services. Each microservice can be developed, deployed, and operated independently, focusing on a specific business function or set of functions. Microservices generally communicate with each other via lightweight communication protocols (e.g., HTTP / REST or gRPC). In the monitored system 1, a single operation triggers multiple requests across multiple microservices, and these microservices work together to provide the overall application functionality. Hereinafter, microservices may also be simply referred to as services.

[0016] The client terminal 2 is, for example, a server computer, which is physical computer hardware owned by the operations management department of the monitored system 1. The client terminal 2 has a function of displaying information output from the system fault monitoring device 100 via the network 3.

[0017] The system fault monitoring device 100 monitors the monitored system 1, estimates the root cause of a fault that occurs in the monitored system 1, generates screen information relating to the root cause, and outputs it to the client terminal 2. Specifically, the root cause indicates whether the cause of the fault lies on the side of the application executed in the monitored system 1, or on the side of the execution platform that executes the application.

[0018] Fig. 2 is a diagram showing an example of the configuration of the monitored system 1. In this embodiment, the monitored system 1 is configured as a distributed system using microservices, and specifically, as shown in Fig. 2, includes a physical machine 205, a VM (Virtual Machine) 204, a virtualization platform 203, a computing node 202, a Pod 201, and an application container (App container) 200.

[0019] The physical machine 205 corresponds to the physical machine 10 in FIG. 1 and is a physical server that has hardware resources such as a CPU (Central Processing Unit), memory, and storage, and can run an operating system. There may be multiple physical machines 205, and they may be distributed across multiple data centers. The VM 204 (Virtual Machine) is a virtual computer created on the physical machine 205 using virtualization technology, and has its own OS, applications, and network settings. A virtualization infrastructure 203 for running applications is constructed by multiple VMs 204.

[0020] The compute node 202 is a worker node within the virtualization infrastructure 203 and is implemented as a physical machine 205 or a VM 204 for executing the application container 200. Each compute node 202 functions as a resource managed by the virtualization infrastructure 203. The pod 201 is the smallest deployment unit deployed on the virtualization infrastructure 203 and is a group of containers that share shared storage, networking, and other pod-specific settings. The pod 201 corresponds to a microservice, which is a component that performs processing in response to a request, and typically includes one or more application containers 200. The application container 200 is a lightweight and portable execution environment that includes an application and its dependencies. The application container 200 runs within the pod 201 and realizes the microservice. In this way, the monitored system 1 combines these elements to achieve efficient and flexible container orchestration.

[0021] Figure 3 shows an example of the dependencies between microservices and their execution environments.

[0022] Multiple services (services 300 to 303 in the example of Figure 3(a)) communicate with each other to provide functions in cooperation. Specifically, when a request is issued to service 1 in response to a user operation, service 1 issues separate requests to service 2 and service 3 in response to the request, processes the responses from service 2 and service 3, and responds to the user. Service 2 also issues a request to service 4, processes the response from service 3, and responds to service 1. In this way, there are microservices that communicate directly with each other (service 1 and service 2, service 1 and service 3, service 2 and service 4), and microservices that do not communicate directly with each other (service 2 and service 3, service 3 and service 4). By drawing an edge between two services that communicate directly, a service map showing the dependency relationships between the services, as shown in Figure 3(a), is created.

[0023] 3(b), application containers 200 providing the same service may exist on multiple computing nodes 202. In the example shown in the figure, Pods 306, 308, and 310 corresponding to Service 1, Service 2, and Service 3, respectively, exist on computing node 304, which is the first computing node 202. Furthermore, Pods 307, 309, 311, 312, and 313 corresponding to Service 1, Service 2, Service 3, and Service 4, respectively, exist on computing node 305, which is the second computing node 202. In this way, by having multiple application containers 200 for the same service exist on multiple computing nodes 202, processing capacity and fault tolerance can be improved.

[0024] 4 is a functional block diagram showing an example of the functional configuration of the system failure monitoring device 100. As shown in the figure, the system failure monitoring device 100 has an input unit 110, an output unit 120, a storage unit 130, a calculation unit 140, and a communication unit 150.

[0025] The input unit 110 is a functional unit that accepts input information. Specifically, the input unit 110 accepts input information entered by a user via input devices such as a keyboard and a mouse provided in the system fault monitoring device 100. The input unit 110 also outputs the accepted input information to the calculation unit 140.

[0026] The output unit 120 is a functional unit that displays various information on a display device such as a display provided in the system fault monitoring device 100 .

[0027] The storage unit 130 is a functional unit that stores various information. In this embodiment, the storage unit 130 stores tracing data 131, pod / computing node information 132, request abnormality degree information 133, threshold value information 134, and abnormal request distribution information 135.

[0028] 5 is a diagram showing an example of tracing data 131. The tracing data 131 is request information related to each request processed by a microservice, and includes the parent-child relationship of the request between microservices, the source, destination, response time, etc. Specifically, the tracing data 131 includes fields 131A to 131G.

[0029] Field 131A stores a trace ID (TraceID), which is an ID that identifies a series of requests resulting from a request sent from a user using the monitored system 1 to the monitored system 1 (more specifically, its API (Application Programming Interface) endpoint). Field 131B stores a processing ID (SpanID), which identifies the processing by Pod 201 of a target request included in the series of requests. Field 131C stores a parent ID (ParentID), which is an ID that identifies a parent request, which is the request immediately before the target request in the series of requests. Note that if the target request is a request sent from a user to the monitored system 1, the parent ID is left blank. Field 131D stores source information that indicates the source Pod (SourcePod), which is the Pod 201 that sent the target request. Field 131E stores destination information that indicates the destination Pod (DestinationPod), which is the Pod 201 to which the target request is sent. Note that processing according to the target request is executed in the destination Pod. Field 131F stores a start time, which is the time when the target request is sent. Field 131G stores the end time, which is the time when a response to the target request is sent back to the sender (client or sender Pod). The difference between the start time and the end time is the response time of the target request.

[0030] For example, the record in the second row of Figure 5 shows that a request sent from source Pod "service1-44f2ec26" to destination Pod "service2-096f9b6b" at start time "2023-11-04 09:31:08.5755022" was processed by destination Pod "service2-096f9b6b" and responded to by source Pod "service1-44f2ec26" at end time "2023-11-04 09:31:08.6955016." The trace ID of this process is "55c94b0d," the process ID is "91b8a50e," and the parent ID is "8b8d57f6."

[0031] 6 is a diagram showing an example of the pod / computing node information 132. The pod identifier / computing node information 132 is configuration information (system configuration information) related to components of the monitoring target system 1, and includes fields 132A to 132G.

[0032] Field 132A stores an ID, which is identification information that identifies each record of the pod / computing node information 132. Field 132B stores a pod name (Podname), which is the name of the pod 201 and is a unique identifier for the pod 201 in the same virtualization platform 203. Field 132C stores a service name (Servicename), which is information indicating the microservice to which the application container 200 running in the pod (the pod 201 identified in field 132B of the same record) belongs. Field 132D stores computing node information, which is information indicating the computing node 202 running the pod. If the pod is a deleted pod that existed in the past but has been deleted and no longer exists, the computing node information indicates "null." Field 132E stores zone information (Availablityzone), which indicates the data center to which the computing node (the computing node 202 running the pod) belongs. If the pod is a deleted pod, the zone information indicates "null." Field 132F stores the update time (update timestamp), which is the time (update timing) when Pod information (information in fields 132B to 132E) is acquired and updated using the API of the virtualization infrastructure 203. Field 132G stores the execution status (IsAvailable), which is information indicating whether the Pod exists at the update time. The execution status indicates "True" if the Pod exists, and indicates "False" if the Pod does not exist.

[0033] For example, the record with ID "10" in FIG. 6 indicates that Pod 201, whose Pod name is "service1-8e69e54d," was running the application container 200 for "Service1," but that it had already been deleted by the Pod information update time of "2023-11-01 01:00:05." The record with ID "50" indicates that Pod 201, whose Pod name is "service1-44f2ec26," is running the application container 200 for "Service1." The compute node information for Pod 201 is "i-4ece0a57," and the data center where the compute node 202 is located is "AZ1." The Pod 201 is running by the Pod information update time of "2023-11-04 09:00:05."

[0034] 7 is a diagram showing an example of the request abnormality degree information 133. The request abnormality degree information 133 is information indicating the relationship between each request and an abnormality in the monitoring target system 1, and includes fields 133A to 133H.

[0035] Field 133A stores an ID, which is identification information that identifies each record in the request abnormality level information 133. Field 133B stores source information that indicates the source Pod that sent the request. Field 133C stores destination information that is information that indicates the destination Pod that receives the request. Field 133D stores the processing time required for processing the request. The processing time is the response time from when the destination Pod receives a request to when it returns a response to the request, minus the waiting time from when the destination Pod sends a lower-level request, which is another request in response to the request, to another Pod until it receives a response from that other Pod.

[0036] Field 133D stores the degree of anomaly that evaluates the degree to which the request is related to an anomaly in the monitored system 1. Specifically, the degree of anomaly is a comparison value obtained by comparing the processing time of a request with the processing time of a comparison request, which is a past request to be compared. The comparison request is a request that has a predetermined homogeneous relationship with the request whose anomaly degree is being calculated, for example, a request that has the same source Pod and destination Pod as the source Pod and destination Pod of the request.

[0037] In this embodiment, the anomaly level is the percentile of the processing time of the request relative to the processing time of the comparison request. More specifically, the percentile is a value that indicates the position of the processing time of the comparison request when the processing times of the comparison requests are sorted in descending or ascending order. For example, the 90th percentile indicates that 90% of the processing times of the comparison requests are less than or equal to the processing time of the comparison request. However, the anomaly level is not limited to percentiles and can be changed as appropriate.

[0038] Field 133F stores an abnormality determination result, which is a determination result of whether or not an abnormality has occurred based on the abnormality level. Field 133G stores a process ID that identifies the process performed by the request. Field 133H stores an abnormality determination time (determination timestamp), which is the time (abnormality determination timing) when information related to the request (information in fields 133B to 133G) is acquired and stored.

[0039] For example, the record in the first row of Figure 7 shows that a request was issued from source pod "service1-44f2ec26" to source pod "service2-096f9b6b," the processing time of the request was 120 ms, the percentile anomaly score was 0.85, and it was determined to be abnormal based on the anomaly score. The processing ID of this request is "91b8a50e," and the anomaly determination time is "2023-11-04 10:20:00."

[0040] Returning to the explanation of Fig. 4, the calculation unit 140 is realized by a computer such as a CPU, and includes a data collection unit 141, a processing time calculation unit 142, an abnormality degree calculation determination unit 143, an abnormal request distribution calculation unit 144, and an abnormal request visualization unit 145.

[0041] The data collection unit 141 is a collection unit that collects tracing data 131 and pod / computing node information 132 from the monitored system 1. Specifically, the data collection unit 141 reads the identifier of each request of a process within the monitored system 1 via the network 3, acquires the necessary data items "trace ID," "processing ID," "parent ID," "source information," "destination information d," "start time," and "end time," and stores them in the storage unit 130 as tracing data 131. In addition, the data collection unit 141 reads current configuration information from an API provided by the virtualization platform 203 of the monitored system 1, acquires the necessary data items "pod name," "service name," "computing node information," "zone information," "update time," and execution status, and stores them in the storage unit 130 as pod / computing node information 132.

[0042] The method for acquiring the tracing data 131 is not limited to the above example, and the tracing data 131 may be acquired using an external distributed tracing tool (for example, Jaeger or Zipkin).

[0043] The processing time calculation unit 142 is a functional unit that calculates the processing time of each request based on the tracing data 131.

[0044] The abnormality degree calculation / determination unit 143 is a determination execution unit that calculates the abnormality degree corresponding to each request based on the processing time of each request and determines whether or not an abnormality exists. Specifically, the abnormality degree calculation / determination unit 143 determines whether or not an abnormality exists by determining whether or not the abnormality degree exceeds the threshold value indicated in the threshold value information 134.

[0045] The abnormal request distribution calculation unit 144 is a functional unit that calculates an abnormal request distribution by plotting abnormal requests, which are requests determined to be abnormal by the abnormality degree calculation determination unit 143, in a request space defined by coordinate axes related to the requests, and stores the calculated abnormal request distribution in the storage unit 130 as abnormal request distribution information 135. Details of the abnormal request distribution information 135 will be described later. The coordinate axes of the abnormal request distribution are set in advance or by the user of the monitored system 1 according to a hypothesis to be verified as the root cause of the abnormality. Multiple coordinate axes may be prepared. In this case, the abnormal request distribution calculation unit 144 calculates the abnormal request distribution for each coordinate axis. Furthermore, for each abnormal request distribution, the abnormal request distribution calculation unit 144 calculates a total distance, which is the sum of the distances between the center of gravity of the abnormal request distribution in the request space and the position of each abnormal request, as an evaluation value for evaluating the validity of the hypothesis.

[0046] The abnormal request visualization unit 145 is a functional unit that generates screen information including visualization data that visualizes the abnormal request distribution calculated by the abnormal request distribution calculation unit 144 and provides the screen information to the client terminal 2.

[0047] The communication unit 150 is a functional unit that transmits and receives information to and from external devices. For example, the communication unit 150 acquires necessary data from the monitored system 1. The communication unit 150 also transmits to the client terminal 2 screen information including visualization data and a UI (User Interface) for inputting information to the client terminal 2.

[0048] 8 is a flowchart for explaining an example of system failure monitoring processing by the system failure monitoring device 100. The system failure monitoring processing is started, for example, when the system failure monitoring device 100 receives an instruction to execute the system failure monitoring processing from the client terminal 2 via the communication unit 150.

[0049] When the fault monitoring process is started, the data collection unit 141 of the system fault monitoring device 100 collects the tracing data 131 and the pod / computing node information 132 from the monitored system 1 (step S101).

[0050] In the virtualization platform 203, Pods 201, which are execution units of microservices, are frequently started and deleted, so unless the Pod / computing node information 132 is updated in a timely manner, there is a risk that information about the source Pod and the destination Pod contained in the tracing data 131 may not exist in the Pod / computing node information 132. To avoid such a situation, in this embodiment, the information about the source Pod and the destination Pod acquired as the tracing data 131 is linked to Pod information on each computing node 202.

[0051] For example, in the examples of Figures 5 and 6, the source Pod of the request with the process ID "91b8a50e" is Pod 201 of "service1-44f2ec26", and the destination Pod is Pod 201 of "service2-096f9b6b". Pod 201 of "service1-44f2ec26" is deployed to the compute node 202 of "i-4ece0a57", and Pod 201 of "service2-096f9b6b" is deployed to the compute node 202 of "i-d39b2148". The update time of the Pod / Compute Node Information 132 is "2023-11-04 09:00:05", and the Pod of "service1-44f2ec26" and Pod 201 of "service2-096f9b6b" are running. These pieces of information are stored in the tracing data 131 and the pod / computing node information 132 in the storage unit 130 .

[0052] Next, the processing time calculation unit 142 executes a processing time calculation process to calculate the processing time of each request (step S102).

[0053] FIG. 9 is a flowchart illustrating an example of the processing time calculation process in step S102 of FIG.

[0054] In the processing time calculation process, the processing time calculation unit 142 first collects target tracing data, which is a record related to a request sent during a specific time period, which is a time period for which the processing time of the request is to be calculated, from the tracing data 131 in the storage unit 130 (step S201). The specific time period may be specified by the user, for example.

[0055] Next, the processing time calculation unit 142 executes a loop process A that repeats the processes of steps S203 to S208 for each request transmitted in the specific time period based on the target tracing data (step S202).

[0056] In loop processing A, the processing time calculation unit 142 first determines whether or not there is a lower-level request of the target request based on the target tracing data (step S203). A lower-level request is a request that is sent to another Pod 201 in processing according to the target request.

[0057] If a lower-level request does not exist ("No" in step S203), the processing time calculation unit 142 calculates the processing time of the target request by "end time of the target request - start time of the target request" (step S204). On the other hand, if a lower-level request exists ("Yes" in step S203), the processing time calculation unit 142 determines whether two or more lower-level requests exist (step S205).

[0058] If there is only one lower request (if "No" in step S205), the processing time calculation unit 142 calculates the processing time of the target request as "(end time of the target request - start time of the target request) - (end time of the lower request - start time of the lower request)" (step S206). On the other hand, if there are two or more lower requests (if "Yes" in step S205), the processing time calculation unit 142 calculates the processing time of the target request as "(end time of the target request - start time of the target request) - (latest end time of the lower request - earliest start time of the lower request)" (step S207).

[0059] In the example in Figure 5, the request with the processing ID "91b8a50e" was sent from Pod 201 with "service1-44f2ec26" to Pod 201 with "service2-096f9b6b," and the response time, which is the difference between the start time and end time, was 120 ms. The subordinate request of the request with the processing ID "91b8a50e" was sent from Pod 201 with "service2-096f9b6b" to Pod 201 with "service4-cdcfa4a6," and the response time was 60 ms. Therefore, the processing time of the request with the processing ID "91b8a50e" is 60 ms (120 ms - 60 ms), which is the response time of the request itself minus the wait time, which is the response time of the subordinate request.

[0060] The processing time calculation unit 142 stores the calculated processing time in the request abnormality degree information 133 (step S208).

[0061] Then, when the processing time calculation unit 142 has executed the processes of steps S203 to S208 for all of the requests transmitted in the specific time period, it ends the loop process A (step S209) and ends the processing time calculation process.

[0062] Note that the method for calculating the processing time of a request is not limited to the above example. For example, it is also possible to embed an SDK (Software Development Kit) of a monitoring tool capable of monitoring processing time in the source code of each microservice, obtain the response time and waiting time of each request from the monitoring tool, and calculate the processing time.

[0063] Returning to the explanation of the system failure monitoring process in Fig. 8, once the processing time of each request has been calculated, the abnormality degree calculation / determination unit 143 executes an abnormality determination process to compare the processing time of each request with the processing time of a past comparison request, and determine an abnormal request related to an abnormality in the monitored system 1 (step S103).

[0064] FIG. 10 is a flowchart for explaining an example of the abnormality determination process in step S103 of FIG.

[0065] In the abnormality determination process, the abnormality degree calculation determination unit 143 starts B, which executes loop process A that repeats the processes of steps S302 to S308 for each request sent during a specific time period, based on the target tracing data (step S301).

[0066] In loop processing B, first, the anomaly degree calculation determination unit 143 identifies the source Pod and destination Pod of the target request based on the tracing data 131, and determines whether or not there are a predetermined required number or more of comparison requests, which are past requests having the same source Pod and destination Pod as the target request (step S302). The required number is set to a value sufficient for calculating the anomaly degree by, for example, the operations manager of the system fault monitoring device 100.

[0067] If the number of requests to be compared is less than the required number ("No" in step S302), the abnormality degree calculation determination unit 143 identifies the source service to which the source Pod of the target request belongs and the destination service to which the destination Pod of the target request belongs, based on the tracing data 131 and the Pod / computing node information 132, and substitutes a request that has the same source service and destination service as the source service and destination service as the request to be compared (step S303).

[0068] If the number of comparison requests is equal to or greater than the required number (if "No" in step S302), or if a comparison request has been substituted (if step S303 is completed), the anomaly degree calculation determination unit 143 calculates the percentile of the processing time of the target request relative to the processing time of all comparison requests as the anomaly degree of the target request (step S304). Specifically, the anomaly degree (percentile) is calculated by dividing the number of comparison requests that have a processing time shorter than the processing time of the target request by the total number of comparison requests.

[0069] The abnormality degree calculation / determination unit 143 compares the abnormality degree with the threshold value indicated by the threshold value information 134, and determines whether the abnormality degree exceeds the threshold value (step S305).

[0070] If the abnormality degree does not exceed the threshold value ("No" in step S305), the abnormality degree calculation determination unit 143 determines that the target request is a normal request that is not an abnormal request related to an abnormality in the monitored system 1 (step S306). On the other hand, if the abnormality degree exceeds the threshold value ("Yes" in step S305), the abnormality degree calculation determination unit 143 determines that the target request is an abnormal request (step S307). Then, the abnormality degree calculation determination unit 143 stores the determination result of step S306 or S307 as an abnormality determination result in the request abnormality degree information 133 (step S308).

[0071] For example, in the example in Figure 5, the processing time for a request with the processing ID "91b8a50e" is 60 ms. The anomaly level (percentile) of the request, calculated from the history of comparison requests sent from Pod 201 in "service1-44f2ec26" to Pod 201 in "service2-096f9b6b," is 85% (0.85), and the threshold is 65%. In this case, because the anomaly level exceeds the threshold, the request with the processing ID "91b8a50e" is determined to be an anomalous request.

[0072] Then, when the abnormality degree calculation / determination unit 143 has executed the processes of steps S303 to S308 for all of the requests transmitted in the specific time period, it ends the loop process B (step S309) and terminates the abnormality determination process.

[0073] Returning to the explanation of the system failure monitoring process in Fig. 8, the abnormal request distribution calculation unit 144 executes a distribution calculation process to calculate an abnormal request distribution, which is a distribution of abnormal requests, based on the request abnormality degree information 133 (step S104).

[0074] Fig. 11 is a flowchart for explaining an example of the distribution calculation process in step S104 of Fig. 8. Note that a plurality of coordinate axes defining the abnormal request distribution are prepared in advance, and the following distribution calculation process is performed for each of the coordinate axes.

[0075] In the distribution calculation process, the abnormal request distribution calculation unit 144 first selects coordinate axes (X-axis, Y-axis) for calculating the abnormal request distribution from the coordinate axes prepared in advance (step S401).

[0076] In this embodiment, at least two coordinate axes are provided corresponding to two hypotheses to be verified as the root cause of an anomaly. The first hypothesis is for verifying application-related failures, and is the hypothesis that "the distribution of abnormal requests is biased toward a specific service." In this case, the corresponding coordinate axis is a coordinate axis (service-order coordinate axis) on which abnormal requests having the same source service and the same destination service are plotted at the same position. The second hypothesis is for verifying execution infrastructure-related failures, and is the hypothesis that "the distribution of abnormal requests is biased toward a specific computing node." In this case, the corresponding coordinate axis is a coordinate axis (computing node-order coordinate axis) on which abnormal requests having the same source computing node and the same destination computing node are plotted at the same position. Note that the coordinate axes are not limited to these, and additional axes may be added as appropriate depending on the hypotheses for the cause of the failure to be verified.

[0077] The abnormal request distribution calculation unit 144 acquires coordinate values ​​indicating the position of each abnormal request in the request space defined by the selected coordinate axes (step S402). The abnormal request distribution calculation unit 144 calculates the position of the center of gravity (x C , y C ) is calculated (step S403). C , y C ) is calculated as the average value of the X-axis and Y-axis of the abnormal requests, as shown in Equation 1. In Equation 1, N is the number of abnormal requests, (x i ,y i ) indicates the position of the i-th abnormal request.

number

[0078] The abnormal request distribution calculation unit 144 calculates the total distance, which is the sum of the distances from the position of each abnormal request to the center of gravity (step S404). For example, the distance Li is calculated using equation 2, and the total distance L total is calculated using Equation 3.

number

number

[0079] For example, suppose that the positions of four abnormal requests plotted in the request space on the service-order coordinate axis are (4, 1), (3, 1), (3, 2), and (4, 2). In this case, the center of gravity of the abnormal request distribution is (3.5, 1.5), and the total distance, which is the sum of the distances between the positions of each abnormal request and the center of gravity, is 2.828.

[0080] In Equation 2 and Equation 3, the distance is the Euclidean distance. In this case, the smaller the total distance, the more the abnormal requests are biased in the request space of the selected coordinate axis, that is, the more valid the hypothesis corresponding to the selected coordinate axis is. Note that the distance is not limited to the Euclidean distance, and any distance that can express the degree of dispersion of the abnormal requests, such as the Manhattan distance, may be used.

[0081] Next, the abnormal request distribution calculation unit 144 stores the positions and total distances of the abnormal requests in the storage unit 130 as the abnormal request distribution information 135 (step S405), and ends the process.

[0082] 12 is a diagram showing an example of the abnormal request distribution information 135. The abnormal request distribution information 135 shown in FIG.

[0083] The abnormal request information 1351 includes fields 135A to 135C. Field 135A stores an abnormal ID, which is an ID for identifying an abnormal request. Field 135B is provided for each coordinate axis and stores the coordinate value of the abnormal request on that coordinate axis. In the example of FIG. 12, there are two coordinate axes, and accordingly there are two fields 135B. Field 135C stores the coordinate value calculation time (calculation timestamp), which is the time when the coordinate value of the abnormal request is calculated and stored.

[0084] The distribution information 1352 includes fields 135D to 135H. Field 135D stores an ID that identifies the center of gravity of abnormal requests when a set of abnormal requests in a specific time period is plotted in a request space of specific coordinate axes. Field 135E stores the center of gravity calculation time (center of gravity timestamp), which is the time when the center of gravity of the abnormal requests is calculated and stored. Field 135F stores the coordinate axis name for identifying the coordinate axis. Field 135G stores the center of gravity coordinates, which are the coordinate values ​​of the center of gravity. Field 135H stores the total distance from the center of gravity.

[0085] For example, the record on the first line of the abnormal request information 1351 indicates that the coordinates of an abnormal request with an abnormality level ID of "1" in the space of the service-order coordinate axis are (4,1), the coordinates in the space of the compute node-order coordinate axis are (5,1), and the coordinate value calculation time is "2023-11-04 10:20:00." The record on the first line of the distribution information 1352 indicates that the centroid coordinates of the abnormal request when a set of abnormal requests in a specific time period are plotted in the request space of the service-order coordinate axis are (3.5,1.5), and the total distance is 2.828.

[0086] Returning to the explanation of the system fault monitoring process in Fig. 8, the abnormal request visualization unit 145 determines a display coordinate axis, which is a coordinate axis to be displayed, based on the abnormal request distribution information 135 (step S105). Specifically, the abnormal request visualization unit 145 determines, as the display coordinate axis, the coordinate axis with the shortest total distance among the coordinate axes included in the abnormal request distribution information 135. In the example of Fig. 12, the total distance on the service-order coordinate axis is 2.828, and the total distance on the compute node-order coordinate axis is 8.485. In this case, the abnormal request visualization unit 145 determines that the distribution of abnormal requests is more biased on the service-order coordinate axis with the shorter total distance, and therefore determines that the cause is more likely to be on the application side, and determines that the display coordinate axis to be used in the screen information is the service-order coordinate axis.

[0087] Next, the abnormal request visualization unit 145 outputs and displays screen information showing the abnormal request distribution using the determined display coordinate axes to the client terminal 2 (step S106), and ends the process.

[0088] Fig. 13 is a diagram showing an example of screen information displayed by the client terminal 2. Screen information 400 shown in Fig. 13 is a heat map screen that visualizes communication between Pods.

[0089] Specifically, screen information 400 is an example in which service-order coordinates are selected as the display coordinates. In this case, the X-axis 400A lists the source services of the anomalous requests in service order, and the Y-axis 400B lists the destination services of the anomalous requests in service order. Each display unit, which plots the anomalous requests, is assigned a color as visual information according to the anomaly level of the anomalous request. For example, display unit 400C plots an anomalous request sent from Pod 2 to Pod 4, and its anomaly level is 0.76. Furthermore, screen information 400 indicates that requests with a high degree of anomaly are concentrated in requests sent from Service 1 to Service 2, indicating that a failure is likely occurring in the application corresponding to Service 2.

[0090] Screen information 401 is a reference example in which the computation node order coordinate system is selected as the display coordinate system. Specifically, in this case, the X-axis 400A arranges the computation nodes that have sent the abnormal requests in the order of the computation nodes, and the Y-axis 400B arranges the computation nodes that have received the abnormal requests in the order of the computation nodes. In this example, requests with high abnormality levels are dispersed, indicating that there is a low possibility that a failure has occurred in the execution platform. Therefore, in this example, the abnormal request visualization unit 145 displays screen information 400 but does not display screen information 401.

[0091] As described above, in this embodiment, the data collection unit 141 acquires the pod / computing node information 132, which is configuration information about the pods 202 that execute processing in response to requests, and the tracing data 131 about the requests processed by the pods 202. The determination unit, including the processing time calculation unit 142 and the anomaly degree calculation / determination unit 143, determines, for each request, whether the request is an abnormal request related to an abnormality in the monitored system 1, based on the tracing data 131 and the pod / computing node information 132. The presentation unit, including the abnormal request distribution calculation unit 144 and the abnormal request visualization unit 145, generates and presents screen information showing the abnormal request distribution, in which the abnormal requests are plotted in a request space defined by coordinate axes related to the requests. Therefore, by checking the abnormal request distribution, it is possible to estimate the root cause of a system failure, making it easy to estimate the root cause of the system failure. As a result, it is possible to shorten the time required for initial response to failure cause investigations, thereby enabling more efficient and rapid operational management of the microservice architecture.

[0092] Furthermore, in this embodiment, the processing time of a request is compared with past comparison requests to determine whether the request is an abnormal request or not. This eliminates the need to accumulate failure data and build machine learning models, etc., making it possible to estimate the root cause easily, at low cost, and with high reliability.

[0093] Furthermore, in this embodiment, the processing time for a request is the response time from receiving the request to responding minus the waiting time, so it is possible to accurately evaluate the time required for processing the request in each Pod 201, thereby improving the accuracy of estimating the root cause.

[0094] In this embodiment, the comparison request is a request whose source Pod and destination Pod are the same as those of the target request, which makes it possible to improve the accuracy of estimating the root cause.

[0095] Furthermore, in this embodiment, if the number of requests identical to the target request from the source Pod and the destination Pod is less than the required number, the requests identical to the target request from the service provided by the source Pod and the service provided by the destination Pod are substituted for the comparison request. In this case, even if the number of requests identical to the target request from the source Pod and the destination Pod is small, it is possible to suppress a decrease in the accuracy of estimating the root cause.

[0096] In this embodiment, the presentation unit selects a display coordinate axis from the plurality of coordinate axes based on the total distance, which is the sum of the distances between the positions of the abnormal requests in the request space and the center of gravity of the abnormal requests, for each of the plurality of coordinate axes. This makes it possible to display screen information suitable for estimating the root cause, making it easier to estimate the root cause.

[0097] In addition, in this embodiment, the coordinate axes can be set by the user, so that appropriate coordinate axes can be set according to the hypothesis to be verified as the root cause of the abnormality, making it possible to estimate and verify various causes. [Example]

[0098] This embodiment corresponds to a modified example of the first embodiment. Therefore, the following mainly describes the differences from the first embodiment. This embodiment differs from the first embodiment mainly in that, instead of the tracing data 131, error log information indicating the history of abnormalities related to requests that have occurred in the system fault monitoring device 100 is used as the request information related to the requests.

[0099] FIG. 14 is a flowchart illustrating an example of a system failure monitoring process according to this embodiment.

[0100] 14, the data collection unit 141 of the system fault monitoring device 100 collects Pod / computing node information 132 and error log information from the monitored system 1 and stores them in the storage unit 130 (step S501). Specifically, the error log information is information about an abnormality that is recorded in the Pod 202 on the request receiving side when an abnormality occurs in processing according to a request.

[0101] Next, the anomaly degree calculation / determination unit 143 calculates the anomaly degree of each request based on the Pod / computing node information 132 and the error log information (step S502). For example, the anomaly degree calculation / determination unit 143 calculates the anomaly degree of each request based on the ratio of requests with an anomaly occurring among requests with the same source Pod and destination Pod during a predetermined period, or the importance of an anomaly occurring among requests with the same source Pod and destination Pod during a predetermined period.

[0102] Then, the abnormality degree calculation / determination unit 143 determines, for each request, whether the abnormality degree of the request exceeds the threshold value indicated in the threshold value information 134, and determines whether the request is an abnormal request (step S503).

[0103] Thereafter, the processes of steps S104 to S106 described with reference to FIG. 8 are executed in the same manner as in the first embodiment.

[0104] In this embodiment as well, by checking the abnormal request distribution, it becomes possible to estimate the root cause of a system failure, and therefore it becomes possible to easily estimate the root cause of a system failure.

[0105] The above-described embodiments of the present disclosure are merely illustrative examples of the present disclosure, and are not intended to limit the scope of the present disclosure to these embodiments alone. Those skilled in the art may implement the present disclosure in various other forms without departing from the scope of the present disclosure. [Explanation of symbols]

[0106] 1: Monitored system 2: Client terminal 10: Physical machine 100: System fault monitoring device 110: Input unit 120: Output unit 130: Memory unit 140: Calculation unit 141: Data collection unit 142: Processing time calculation unit 143: Anomaly degree calculation and determination unit 144: Anomaly request distribution calculation unit 145: Anomaly request visualization unit 150: Communication unit 200: Application container 202: Computation node 203: Virtualization platform 205: Physical machine

Claims

1. A system fault monitoring device for monitoring a monitored system having a plurality of components that execute processes in response to requests, comprising: a collection unit that collects configuration information about each component and request information about each request processed by each component; a determination unit that determines, for each request, whether the request is an abnormality request related to an abnormality in the monitored system based on the configuration information and the request information; a presentation unit that generates and presents visualized data showing an abnormal request distribution in which the abnormal requests are plotted in a request space defined by coordinate axes related to the requests.

2. The determination unit a processing time calculation unit that calculates, for each request, a processing time required for processing in response to the request based on the configuration information and the request information; 2. The system fault monitoring device according to claim 1, further comprising: a determination execution unit that determines, for each of the requests, whether the request is an abnormal request based on a comparison value obtained by comparing the processing time of the request with the processing time of a comparison request that is in a predetermined homogeneous relationship with the request.

3. 3. The system fault monitoring device according to claim 2, wherein the request processing time is the response time from when the component receives the request to when it responds to the request minus the waiting time from when the component sends a lower-level request in response to the request to another component to when it receives a response from the other component.

4. 4. The system fault monitoring device according to claim 3, wherein the comparison request is a request in which a source component that is the component from which the comparison request originates and a destination component that is the component to which the comparison request originates are the same as those of the current request.

5. The component provides a particular service, the monitored system has a plurality of the components that provide the same service, 5. The system fault monitoring device of claim 4, wherein, when the number of identical requests is less than a predetermined required number, the judgment execution unit substitutes a request in which the service provided by the source component and the service provided by the destination component are identical to the request in question as the comparison request.

6. the request information is error log information indicating a history of abnormalities related to the request acquired in the monitoring target system, 2. The system fault monitoring device according to claim 1, wherein the determining unit determines, for each of the requests, whether or not the request is the abnormal request based on the error log information.

7. 2. The system fault monitoring device according to claim 1, wherein the presentation unit selects a display coordinate axis to be used in the visualization data from the plurality of coordinate axes based on a total distance that is the sum of distances between the position of each abnormal request in the request space and the center of gravity of each abnormal request for each of the plurality of coordinate axes that are different from each other.

8. 8. The system fault monitoring device according to claim 7, wherein said coordinate axes can be set by a user.

9. the determination unit calculates, for each request, an abnormality degree that evaluates the degree to which the request is related to an abnormality in the monitored system based on the configuration information and the request information, and determines whether the request is an abnormal request based on the abnormality degree; 2. The system failure monitoring device according to claim 1, wherein the visualized data is a heat map that plots the abnormal requests with visual information according to the degree of abnormality of the abnormal requests.

10. A system fault monitoring method by a system fault monitoring device that monitors a monitored system having a plurality of components that execute processes in response to requests, comprising: collecting configuration information for each component and request information for each request processed by each component; determining, for each request, whether or not the request is an abnormal request related to an abnormality in the monitored system based on the configuration information and the request information; The system fault monitoring method generates and presents visualized data showing an abnormal request distribution in which the abnormal requests are plotted in a request space defined by coordinate axes related to the requests.

Images