Fault Localization for Cloud-Native Applications

Abstract

Provided herein are methods, systems, and computer program products for fault localization for cloud-native applications. The computer-implemented method includes classifying event-related alerts by processing one or more characteristics of the event-related alerts directed to at least one system, obtaining and processing a plurality of application logs based at least in part on the classification of the event-related alerts, identifying an error log from among the plurality of application logs based at least in part on processing the plurality of application logs, ordering the error logs using one or more prioritization techniques, and performing at least one automated action based at least in part on ordering the error logs.

Description

[Technical field]

[0001] This application relates generally to information technology, and more particularly to problem management technology. More specifically, anomaly detection relates to identifying potential problems and / or anomalous events in a set of events, and fault localization relates to narrowing down the set of potential causes of an already detected problem. Summary of the Invention

[0002] In one embodiment of the present invention, techniques for fault localization for cloud-native applications are provided. An exemplary computer-implemented method can include classifying event-related alerts directed to at least one system by processing one or more characteristics of the event-related alerts, and obtaining and processing a plurality of application logs based at least in part on the classification of the event-related alerts. The method also includes identifying an error log from among the plurality of application logs based at least in part on processing the plurality of application logs, ordering the error logs using one or more prioritization techniques, and performing at least one automated action based at least in part on ordering the error logs.

[0003] Another embodiment of the invention or elements thereof may be implemented in the form of a computer program product tangibly embodying computer-readable instructions that, when implemented, cause a computer to perform a number of method steps as described herein. Still further, another embodiment of the invention or elements thereof may be implemented in the form of a system including a memory and at least one processor coupled to the memory and configured to perform the method steps. Still further, another embodiment of the invention or elements thereof may be implemented in the form of a means or elements thereof for performing the method steps described herein. The means may include hardware modules or a combination of hardware modules and software modules, the software modules being stored in a tangible computer-readable storage medium (or multiple such media).

[0004] These and other objects, features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments, which is to be read in connection with the accompanying drawings. [Brief description of the drawings]

[0005] [Figure 1] FIG. 1 is a diagram of a system architecture according to an exemplary embodiment of the present invention. [Diagram 2] FIG. 1 is a diagram of a system architecture according to an exemplary embodiment of the present invention. [Diagram 3] 13 is an exemplary code snippet for initiating ingress logs as a field of interest, according to an exemplary embodiment of the present invention. [Figure 4] 11 is an exemplary code snippet for initiating application logs as a field of interest, according to an exemplary embodiment of the present invention. [Diagram 5] FIG. 1 illustrates inferring an execution behavior model from log data according to an exemplary embodiment of the present invention. [Figure 6]FIG. 13 is a workflow diagram for identifying identifiable error log templates for a given incident according to an exemplary embodiment of the present invention. [Figure 7] 1 is a flow diagram illustrating a technique according to an embodiment of the present invention. [Figure 8] FIG. 1 is a system diagram of an exemplary computer system capable of implementing at least one embodiment of the present invention. [Figure 9] FIG. 1 is a diagram of a cloud computing environment in accordance with an embodiment of the present invention. [Figure 10] FIG. 2 is a diagram of an abstract model layer according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] As described herein, embodiments of the present invention include fault localization for cloud-native applications. At least one embodiment includes providing fault localization for cloud-native and hybrid applications by using at least one topological sorting of a runtime service invocation relationship graph to identify potential failures. Such embodiments also include narrowing down potential causes of at least one already detected problem by monitoring logs in a given time window and performing log analysis to distinguish between error-related behavior and normal behavior within the same time interval by narrowing down error messages by using error distribution across the invocation relationship chain.

[0007] One or more embodiments also include inferring one or more components in the at least one microservice by extracting entities mentioned in the at least one log message and identifying a component in the at least one microservice that is generating an error. Additionally or alternatively, at least one embodiment includes analyzing various communications to determine an order in which requests arrive at the components, as well as processing the logs over a period of time to infer the request flow using graph isomorphism. Furthermore, one or more embodiments may include identifying services and / or users affected by the at least one failed component by identifying and / or analyzing an impact of the at least one failure.

[0008] FIG. 1 is a diagram of a system architecture according to an embodiment of the present invention. As depicted in FIG. 1, raw logs 102 are received from connectors, such as logDNA, Elasticsearch, etc. The logs 102 are then normalized via component 104 (e.g., via the Logstash component, or by identifying what the various fields in the corresponding Java(R) Script Object Notation (JSON) refer to (e.g., which fields indicate time, which fields refer to application name and / or server name, etc.) and modifying the data in such fields for formatting and / or scaling, so that the data can be used in a consistent manner in subsequent analysis) and sent to a stream processing platform 106. Each time an alert is raised, for example, via input 108, the duration of the alert and the exact time of the alert are extracted, and in step 110, these details are used to fetch the specific logs from the stream processing platform 106. In one or more embodiments, the alerts are set by application developers or site reliability engineers (SREs) to monitor the state of the system. If an error and / or problem occurs, an alarm is generated by the system. In addition, a log is generated at the same time as the alert is generated, describing information and / or related information about the error and / or problem corresponding to the alert.

[0009] Additionally, as also depicted in FIG. 1, the log configuration information 112 is used by the fault location and impact system 118 to determine fields representing timestamps and one or more services representing golden signal error and status codes. Application topology information 114 (which may include the order in which requests flow) is also sent as input to the fault location and impact system 118. In at least one embodiment, such inputs may also be inferred using the logs. Information regarding related entities 116 extracted from the description of the alert also serves as input to the fault location and impact system 118. Based at least in part on the inputs 112, 114, and 116, the fault location and impact system 118 determines at least one failed component 120, one or more affected services 122, and one or more affected customers 124, such determinations may be made, for example, by inferring one or more relationships over the input data. By way of example, in at least one embodiment, such inferred relationships may be determined by extracting entities mentioned in the processed log data that correspond to the error and / or problem, analyzing relevant application topology information to determine the order in which requests reach one or more components within an application and / or service that are affected or may be affected by the error and / or problem, and correlating the extracted and / or analyzed data across the above sources.

[0010] FIG. 2 is a diagram of a system architecture according to an exemplary embodiment of the invention. By way of example, FIG. 2 depicts monitoring of application logs 204 (e.g., as soon as the logs are generated in association with inputs 202). Using component 206, these application logs 204 are normalized via component 210 (similar to the normalization technique described in detail above in association with FIG. 1), and error logs 212 are extracted using application log configuration information 208 (e.g., information capturing and / or describing application logs) and converted into log templates 214. In at least one embodiment, ingress logs and / or golden signals 220 (e.g., gateway logs) and ingress gateway log configuration information 218 (e.g., information capturing and / or describing ingress gateway logs) are extracted via component 216.

[0011] As also depicted in FIG. 2, a first ranker 226 (out of the set of ranked templates 224) uses application topology information 222 as well as tracing information to determine at least one failed component, failed pod, and / or error log template 228. In at least one embodiment, tracing information may be present in one or more logs, and such embodiment may include developing and implementing one or more log parsers to extract such information and use it in one or more analyses (e.g., comparative analysis with historical data) to make the above determination. For the second ranker 230, tracing information is not used, but error distributions among various services are correlated in connection with determining at least one failed component, failed pod, and / or error log template 232. For the third ranker 234, golden signal errors are not used, but a list of failed components, failed pods, and / or error log templates 236 is inferred by finding deviations between runtime application logs and historical (e.g., healthy) logs.

[0012] Thus, at least one embodiment includes using a topological sort of the runtime service invocation relationship graph to identify common potential failures but with one or more differences. Such differences may include, for example, that the primary error signal is an alert based on a golden signal from a user facing service in the application. As used herein, a golden signal error refers to the primary error signal from which the alert is raised. Also, the golden signal error records the number of times requests have failed in the system. If there are more than a given number of failed requests within a certain time window, an alert is raised. A threshold of how many error signals are observed in the system may be configurable. Additionally, such differences may include, for example, that each error signal is a composite of multiple failed service request responses, that the alerting service may return successful responses to at least some user requests within the same time period, and that the upstream service may not necessarily have raised the alert.

[0013] As further described in detail herein, one or more embodiments include leveraging error log events from upstream microservices to surface faulty components and filtering out non-transaction related error signals. In such embodiments, the fault may be present in the log delta between failed and successful service requests, and a weighted topological sort to identify the faulty components may be performed, e.g.,<Microservice,Pod,Error Message> Techniques for grouping error log events from upstream services can include, for example, based on transaction traceability and based on temporal proximity to failed service requests. As described herein, a faulty component refers to a component that has a given number of transaction errors (e.g., above a given threshold).

[0014] 3 is an example code snippet for initiating an ingress log as a field of interest according to an example embodiment of the present invention. In this example embodiment, the code snippet 300 is executed by or under the control of at least one processing system and / or device. The example code snippet 300 also illustrates a sample of an ingress log and / or gateway log. From the timestamp, the request_id and _status information are extracted to locate the failure.

[0015] It should be understood that this particular example code snippet illustrates just one example implementation for initiating an ingress log as a field of interest, and that in other embodiments, one or more alternative implementations of such a technique may be used.

[0016] 4 is an example code snippet for initiating an application log as a field of interest according to an example embodiment of the present invention. In this example embodiment, the code snippet 400 is executed by or under the control of at least one processing system and / or device. The example code snippet 400 also illustrates a sample application log with a field container indicating which service emitted an error signal at time "@timestamp."

[0017] It should be understood that this particular example code snippet illustrates just one example implementation for initiating an application log as a field of interest, and that in other embodiments, one or more alternative implementations of such a technique may be used.

[0018] At least one embodiment includes ranking templates by error contribution, which can be performed, for example, using a frequency-based approach. Such an approach includes considering the runtime and historical contributions of templates, and templates with a large difference between runtime and offline contributions are ranked higher. For example, consider a case where template T1 occurs 5% of the time historically and 20% at runtime (i.e., the difference is 15%), while template T2 occurs 10% of the time historically and 12% at runtime (i.e., the difference is 2%). Thus, in such an example case, template T1 is ranked higher than T2.

[0019] FIG. 5 illustrates inferring an execution behavior model from log data, according to an exemplary embodiment of the invention. By way of example, FIG. 5 illustrates nodes 502, 504, 506, 508, 510, 512, 514, 516, and 518, where each node represents a template (e.g., a type of log message that corresponds to a print statement in the code), and each edge connects adjacent templates in a flow path. More specifically, FIG. 5 illustrates the execution behavior of a service in a healthy state. Each template (502, 504, 506, 508, 510, 512, 514, 516, and 518) represents a print statement in the log, and an edge between two templates, such as T3 and T4, indicates, for example, that T3 is executed first, then T4 is executed. This information is generated from the logs and used as an application topology by one or more embodiments to localize failures.

[0020] As further described herein, at least one embodiment includes inferring a service topology, where the service topology with the software stack can be obtained from one or more application performance management (APM) tools. Additionally or alternatively, at least one embodiment includes ranking the logs and / or templates with fault localization. Such an embodiment can include utilizing approximate service failure alert timespan information (e.g., to compare messages in error logs and normal logs) as well as leveraging available service knowledge (e.g., using service failure conditions to identify error-related behavior versus normal behavior, using log transaction identifiers (IDs), using component-level error log message conditions (including retrieving all error templates for an incident), and / or using at least one dynamic service component workflow).

[0021] Consider an exemplary case where an alert arrives. At least one embodiment can include extracting a time window and duration of the alert, and retrieving application logs for the given time window. Output of such an embodiment includes a list of faults and affected components, an error template for each component, and a list of pods. As used herein, a pod refers to an entity in which a service is hosted. In at least one embodiment, the same service can be hosted in multiple pods, and one or more embodiments include identifying which pod is faulty.

[0022] Consider another example case where an alert arrives. At least one embodiment can include extracting a time window and duration of the alert, and obtaining application logs and ingress logs for the given time window. An output of such an embodiment can include fault location information including an identification of at least one failed component, an error template, and a list of pods. Such output can also include an identification of the affected components, an error template for each of the affected components, and a list of pods in which the components are running.

[0023] Consider yet another exemplary case where an alert arrives. At least one embodiment can include extracting the time window and duration of the alert, as well as identities of entities present in the description of the alert (e.g., slot information, etc.), and retrieving application logs and ingress logs for the given time window. Output of such an embodiment includes fault location information including an identification of at least one failed component, an error template, and a list of pods. Such output can also include an identification of the affected components, an error template for each of the affected components, and a list of pods in which the components are running.

[0024] As further described herein, one or more embodiments include utilizing service failure condition information to identify error-related behavior versus normal behavior. Using ingress logs, such embodiments may include creating a list of two request_IDs, for example, one request_ID with a status of 200 and another request_ID with a status of 500 or greater. Also, using application logs, such embodiments may include obtaining an error template for an incident.

[0025] One or more embodiments also include log correlation, where the request_id in the ingress log is the same as the transaction_id in the application log. Such embodiments may further include using the above information to categorize error templates into multiple portions. For example, one portion is directed to error templates associated only with statuses 500 and above, another portion is directed to error templates associated with statuses 200 and 500 and above, and yet another portion is directed to error templates associated with statuses 200. In such an exemplary embodiment, templates associated only with statuses 500 (and above) are identifiable templates for a given incident and are therefore higher ranked.

[0026] One or more such embodiments may include using a component's call flow to rank the templates in each part, where the component that is hit last is the one that first emitted its error log and is the key error. A microservice may emit multiple error messages, and in such a scenario, at least one embodiment may include narrowing down the exact error message (also referred to herein as the key error message) that is the root cause of the failure. Such embodiments may also include using a combination of the component's rank and the template's runtime frequency (e.g., the number of times a log line occurs) to rank the errors at runtime.

[0027] FIG. 6 is a workflow diagram for identifying identifiable error log templates for a given incident, according to an exemplary embodiment of the present invention. FIG. 6 is similar to FIG. 1 and FIG. 2, but with additional details regarding how the log template 608 is generated. For example, the application log and / or incident service log 610 are reduced to the template 608 via component 612 (e.g., by using one or more templating-based techniques that facilitate clustering similar log lines) by learning and implementing a model using error message classifier information 602 and by analyzing historical logs 614 and performing offline template mining 616. Based on the template 608 and component flow 604, the distribution of errors across the calling relationship chain 606 is determined (e.g., by determining the error log frequency for each related microservice and correlating errors with approximately the same distribution) to narrow down to at least one error message. Based at least in part on this determination, a ranked list of identifiable error templates 618 is generated and used to determine at least one failed component 620, one or more affected services 622, and one or more affected customers 624.

[0028] 7 is a flow diagram illustrating a technique in accordance with an embodiment of the present invention. Step 702 includes classifying an event-related alert by processing one or more characteristics of the event-related alert directed to at least one system. In at least one embodiment, processing the one or more characteristics of the event-related alert includes processing temporal information regarding at least a duration of the event-related alert.

[0029] Step 704 includes retrieving and processing a plurality of application logs based at least in part on the classification of the event-related alert. In at least one embodiment, retrieving the plurality of application logs includes retrieving one or more error logs, or retrieving one or more ingress logs, or both. In such an embodiment, processing the plurality of application logs includes converting at least a portion of the one or more error logs into one or more templates.

[0030] Step 706 includes identifying an error log from among the plurality of application logs based at least in part on processing the plurality of application logs. In at least one embodiment, identifying the error log includes implementing one or more tracking techniques and / or extracting error messages from among the plurality of application logs based on one or more error frequencies.

[0031] Step 708 includes ordering the error logs using one or more prioritization techniques. In at least one embodiment, ordering the error logs using one or more prioritization techniques includes ordering error templates using application topology, ordering the error logs based on error distribution, and / or extracting one or more entities from an alert description and one or more log messages.

[0032] Step 710 includes performing at least one automated action based at least in part on ordering the error logs. In at least one embodiment, performing the at least one automated action includes determining one or more affected services based at least in part on ordering the error logs. Additionally or alternatively, performing the at least one automated action may include sequencing the error logs and restarting the at least one pod upon an indication that the at least one pod is not functioning based at least in part on pod identification information derived from at least a portion of the error logs.

[0033] One or more embodiments may also include implementing the techniques depicted in FIG. 7 offered as a service in a cloud environment.

[0034] The techniques depicted in FIG. 7 may also include providing a system as described herein, the system including separate software modules, each of the separate software modules embodied in a tangible computer-readable recordable storage medium. All of the modules (or any subset thereof) may be on the same medium, for example, or each may be on different media. The modules may include any or all of the components shown in the figures or described herein, or combinations thereof. In an embodiment of the invention, the modules may be executed, for example, on a hardware processor. Then, the method steps may be performed using the separate software modules of the system as described above, executed on the hardware processor. Furthermore, a computer program product may include a tangible computer-readable recordable storage medium having code adapted to be executed to perform at least one method step as described herein, including providing a system having separate software modules.

[0035] 7 may be implemented using a computer program product that may include computer usable program code stored on a computer readable storage medium in a data processing system, the computer usable program code being downloaded over a network from a remote data processing system. Also, in embodiments of the invention, the computer program product may include computer usable program code stored on a computer readable storage medium in a server data processing system, the computer usable program code being downloaded over a network to the remote data processing system for use in the computer readable storage medium of the remote system.

[0036] Embodiments of the present invention or elements thereof may be implemented in the form of an apparatus including a memory and at least one processor coupled to the memory and configured to perform the exemplary method steps.

[0037] In addition, embodiments of the present invention may utilize software running on a computer or workstation. With reference to FIG. 8, such implementations may utilize an input / output interface formed, for example, of a processor 802, a memory 804, and, for example, a display 806 and a keyboard 808. As used herein, the term "processor" is intended to include any processing device, such as a device that includes a CPU (Central Processing Unit) or other forms of processing circuitry or both. Furthermore, the term "processor" may refer to two or more individual processors. The term "memory" is intended to include memory associated with a processor or CPU, such as, for example, a RAM (random access memory), a ROM (read only memory), a fixed memory device (e.g., a hard drive), a removable memory device (e.g., a diskette), a flash memory, and the like. Additionally, as used herein, the phrase "input / output interface" is intended to include, for example, mechanisms for inputting data into a processing unit (e.g., a mouse) and mechanisms for providing results associated with a processing unit (e.g., a printer). The processor 802, memory 804, and input / output interfaces such as a display 806 and keyboard 808 may be interconnected, for example, via a bus 810 as part of a data processing unit 812. Suitable interconnections may also be provided, for example via the bus 810, to a network interface 814, such as a network card, which may be provided to interface with a computer network, and to a media interface 816, such as a diskette or CD-ROM drive, which may be provided to interface with media 818.

[0038] Thus, computer software containing instructions or code for carrying out the methods of the present invention, as described herein, can be stored in an associated memory drive (e.g., ROM, fixed or removable memory) and, when ready for use, can be loaded partially or entirely (e.g., into RAM) and implemented by a CPU. Such software can include, but is not limited to, firmware, resident software, microcode, etc.

[0039] A data processing system suitable for storing and / or executing program code includes at least one processor 802 coupled directly or indirectly to memory elements 804 via a system bus 810. The memory elements may include local memory utilized during the actual execution of the program code, bulk storage, and cache memory that provides temporary storage of at least some of the program code to reduce the number of times the code must be retrieved from bulk storage during execution.

[0040] Input / output devices, or I / O devices (including but not limited to keyboard 808, display 806, pointing device, etc.) can be coupled to the system either directly (such as via a bus 810) or through an intervening I / O controller (omitted for clarity).

[0041] Network adapters, such as network interface 814, may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Currently available types of network adapters include modems, cable modems, and Ethernet cards.

[0042] As used herein, including in the claims, a "server" includes a physical data processing system (e.g., system 812 as shown in FIG. 8) running a server program. It should be understood that such a physical server may or may not include a display and keyboard.

[0043] The present invention may be a system, method, and / or computer program product at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0044] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes: portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or grooved structures having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as being ephemeral signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over electrical wires.

[0045] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to an individual computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface of each computing / processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the individual computing / processing device.

[0046] The computer readable program instructions for carrying out the operations of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or object oriented programming languages ​​such as Smalltalk®, C++, and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer as a standalone software package, partly on the user's computer and partly on a remote computer, or all on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), can execute computer readable program instructions to individualize the electronic circuitry by utilizing state information of the computer readable program instructions to perform aspects of the invention.

[0047] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0048] These computer readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute on the processor of the computer or other programmable data processing apparatus, create means for performing the functions / acts specified in one or more blocks of the flow charts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer readable storage medium on which the instructions are stored comprises an article of manufacture including instructions that implement aspects of the functions / acts specified in one or more blocks of the flow charts and / or block diagrams.

[0049] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to produce a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / acts specified in one or more blocks of the flow charts and / or block diagrams.

[0050] The flow charts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block of the flow charts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for performing the specified logical function. In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown in succession may actually be performed as one step, or may be performed simultaneously, substantially simultaneously, partially or fully in a time-overlapping manner, or the blocks may sometimes be performed in reverse order depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flow chart diagrams, and combinations of blocks in the block diagrams and / or flow chart diagrams, may be implemented by a special-purpose hardware-based system that performs the specified functions or actions or executes a combination of special-purpose hardware and computer instructions.

[0051] It should be noted that any of the methods described herein may include the additional step of providing a system including separate software modules embodied in a computer-readable storage medium, where the modules may include, for example, any or all of the components described in detail herein, where the method steps may be performed using separate software modules and / or sub-modules of such a system executed on the hardware processor 802. Furthermore, a computer program product may include a computer-readable storage medium having code adapted to be implemented to perform at least one method step described herein, including providing a system having separate software modules.

[0052] In any event, it will be understood that the components illustrated herein may be implemented in various forms of hardware, software, or combinations thereof, such as application specific integrated circuits (ASICs), functional circuitry, a suitably programmed digital computer with associated memory, etc. Given the teachings of the invention provided herein, one of ordinary skill in the art will be able to contemplate other implementations of the components of the invention.

[0053] In addition, it should be appreciated at the outset that implementation of the teachings described herein is not limited to any particular computing environment: rather, embodiments of the present invention may be implemented in conjunction with any type of computing environment now known or later developed.

[0054] For example, cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0055] Its features are as follows:

[0056] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, automatically as needed without requiring human interaction with the service provider.

[0057] Broad network access: Functionality is available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin-client and thick-client platforms (e.g., cell phones, laptops, and PDAs).

[0058] Resource Pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated according to demand. There is a sense of location independence in that consumers typically have no control or information about the exact location of the resources provided, although the location may be specified at a high level of abstraction (e.g., country, state, or data center).

[0059] Rapid Scalability: Capabilities can be provisioned quickly and scalably, in some cases automatically, quickly scaled out, and quickly released and quickly scaled in. To the consumer, the capabilities available for provisioning often appear unlimited, and any amount can be purchased at any time.

[0060] Metering of Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at several levels of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services being utilized.

[0061] The service model is as follows:

[0062] Software as a Service (SaaS): The functionality offered to the consumer is the use of the provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application functions, with the exception of limited user-specific application configuration settings.

[0063] Platform as a Service (PaaS): The capability offered to the consumer is to deploy consumer-created or acquired applications, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the application hosting environment configuration.

[0064] Infrastructure as a Service (IaaS): The functionality offered to the consumer is the provisioning of processing, storage, network, and other basic computing resources on which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, deployed applications, and possibly limited control of select networking components (e.g., host firewalls).

[0065] The deployment model is as follows:

[0066] Private Cloud: Cloud infrastructure is operated exclusively for one organization, can be managed by that organization or a third party, and can exist on-premise or off-premise.

[0067] Community Cloud: Cloud infrastructure is shared by several organizations to support a particular community with a shared set of objectives (e.g., mission, security requirements, policies, and compliance concerns), may be managed by the organizations or a third party, and can exist on-premise or off-premise.

[0068] Public Cloud: The cloud infrastructure is made available to the general public or large industry organizations and is owned by an organization that sells cloud services.

[0069] Hybrid Cloud: Cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a unique entity, but are joined by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0070] Cloud computing environments are service-oriented with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0071] 9, an exemplary cloud computing environment 50 is depicted. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 that can communicate with local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or cell phone 54A, a desktop computer 54B, a laptop computer 54C, or an automobile computer system 54N, or combinations thereof. The nodes 10 can communicate with each other. They can be physically or virtually grouped in one or more networks, such as a private, community, public, or hybrid cloud, or combinations thereof, as described herein above (not shown). This allows the cloud computing environment 50 to provide infrastructure, platform, and / or software as a service without the cloud consumer having to maintain resources on a local computing device. It will be understood that the types of computing devices 54A-N shown in FIG. 9 are intended to be merely exemplary, and that the computing node 10 and cloud computing environment 50 can communicate with any type of computerized device over any type of network or over a network-addressable connection (e.g., using a web browser), or both.

[0072] Referring now to Figure 10, a set of functional abstraction layers provided by cloud computing environment 50 (Figure 9) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 10 are intended to be merely exemplary, and embodiments of the present invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0073] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: mainframes 61, RISC (reduced instruction set computer) architecture based servers 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, software components include network application server software 67, and database software 68.

[0074] The virtualization layer 70 provides an abstraction layer at which the following examples of virtual entities may be provided: virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75. In one example, the management layer 80 may provide the functions described below: Resource provisioning 81 provides dynamic procurement of computing resources and other resources utilized to perform tasks within the cloud computing environment. Metering and billing 82 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources.

[0075] In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, and protection for data and other resources. User portal 83 provides access to the cloud computing environment to consumers and system administrators. Service level management 84 provides allocation and management of cloud computing resources so that required service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides advance agreement on cloud computing resources in anticipation of future demand, and procurement of cloud computing resources according to SLAs.

[0076] The Workload Layer 90 provides examples of functionality for which a cloud computing environment may be utilized. Examples of workloads and functions that may come from this layer include: mapping and navigation 91, software development and lifecycle management 92, virtual classroom instruction delivery 93, data analytics processing 94, transaction processing 95, and, in accordance with one or more embodiments of the present invention, fault location 96.

[0077] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will be further understood that the terms "comprise" and / or "comprising", as used herein, specify the presence of stated features, steps, operations, elements, or components or combinations thereof, but do not exclude the presence or addition of other features, steps, operations, elements, components or groups or combinations thereof.

[0078] At least one embodiment of the present invention may provide beneficial effects such as providing fault localization for cloud-native and hybrid applications using topological sorting of a runtime service invocation relationship graph to identify potential failures.

[0079] The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used in this specification is chosen to best explain the principles of the embodiments, practical applications, or technical improvements over the art found in the market, or to allow others skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A computer-implemented method comprising: classifying the event-related alerts by processing one or more characteristics of the event-related alerts directed to at least one system; obtaining and processing a plurality of application logs based on the classification of the event-related alerts; identifying an error log from among the plurality of application logs based on the processing of the plurality of application logs; ordering the error log using one or more prioritization techniques; performing at least one automated action based on said sequencing of said error logs; wherein the method is performed by at least one computing device.

2. The method of claim 1 , wherein the processing one or more characteristics of the event-related alert comprises processing temporal information regarding at least a duration of the event-related alert.

3. The method of claim 1 or 2, wherein said obtaining said plurality of application logs comprises obtaining one or more error logs.

4. 4. The method of claim 3, wherein said processing said plurality of application logs comprises converting at least a portion of said one or more error logs into one or more templates.

5. The method of claim 1 , wherein said obtaining said plurality of application logs comprises obtaining one or more ingress logs.

6. 6. The method of claim 1, wherein ordering the error log using one or more prioritization techniques comprises ordering error templates using an application topology.

7. 7. The method of claim 1, wherein ordering the error log using the one or more prioritization techniques comprises ordering the error log based on an error distribution.

8. 8. The method of claim 1, wherein sequencing the error log using the one or more prioritization techniques comprises extracting one or more entities from the alert description and one or more log messages.

9. The method of claim 1 , wherein the identifying the error log comprises implementing one or more tracing techniques.

10. 10. The method of claim 1, wherein the identifying the error logs comprises extracting error messages from among the plurality of application logs based on one or more error frequencies.

11. 11. The method of claim 1, wherein the performing of the at least one automated action comprises determining one or more affected services based on the sequencing of the error logs.

12. 12. The method of claim 1, wherein performing the at least one automated action includes restarting at least one pod upon an indication that at least one pod is not functioning based on the sequencing of the error log and pod identification information derived from at least a portion of the error log.

13. A program executable by a computer, causing the computer to execute each step of the method described in any one of claims 1 to 12.

14. 1. A system comprising: Memory, at least one processor operatively coupled to the memory, classifying the event-related alerts by processing one or more characteristics of the event-related alerts directed to at least one system; obtaining and processing a plurality of application logs based on the classification of the event-related alerts; identifying an error log from among the plurality of application logs based on the processing of the plurality of application logs; ordering the error log using one or more prioritization techniques; performing at least one automated action based on said sequencing of said error logs; the at least one processor configured to A system comprising:

15. The system of claim 15 , wherein the processing one or more characteristics of the event-related alert comprises processing temporal information regarding at least a duration of the event-related alert.

16. 16. The system of claim 14 or 15, wherein ordering the error log using the one or more prioritization techniques comprises at least one of ordering error templates using an application topology and ordering the error log based on an error distribution.

Images