Computer system, computer-implemented method, program, and computer-implemented system (learning causality)

Abstract

To address a technical problem such that many cloud applications are bothered by limited observation possibilities that make localization of a position of a fault in one or a plurality of application micro-services difficult.SOLUTION: A computer implementation method includes: learning causal relations between two or more application micro-services; and applying the learned causal relations to dynamically localize an application fault. First micro-service error log data corresponding to selectively injected errors is collected. A learned causal graph is generated based on the collected first micro-service error log data. Second micro-service error log data corresponding to a detected application and an ancestral matrix is built using the learned causal graph and the second micro-service error log data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present embodiments relate to a system, computer program product, and computer-implemented method that leverages causal intervention to infer a causal graph between application microservices via active causal learning, and leverages the learned causal graph to perform fault localization. [Background technology]

[0002] In the art, a monolithic application is understood to be a self-contained application that is independent of other applications. Microservices or microservice architecture typically refers to a computing environment in which applications are structured as a suite of modular components or services based on functional definitions, each running in its own process and communicating through lightweight mechanisms. In certain microservice architectures, data is stored outside the services; therefore, the services are stateless; these services or components are commonly referred to as "atomic services." Each atomic service is a lightweight component that independently executes modular services; each atomic service supports a specific task and communicates with other services using a defined interface, such as an application programming interface (API). Microservice architectures support and enable scalability in hybrid networks.

[0003] Generally, microservices are an architectural approach, often cloud-native, in which a single application is composed of multiple smaller, loosely coupled, independently deployable components or services called microservices. Microservices typically (but not necessarily) have their own stacks, including databases and data models, communicate with each other through a combination of REST (representational state transfer) application program interfaces (APIs), and are organized by business entities. Industrial microservice applications have hundreds or more microservices, some of which have dependencies. As the number of application microservices scales, the complexity of the dependencies between microservices grows. The topology of an application's microservices can be fixed, but is often unknown.

[0004] The complexity of microservice dependencies, coupled with often unknown microservice topologies, leads to the complexity and inefficiency of fault localization. This is a significant advancement for developing systems, computer program products, and computer-implemented methods that can perform fault localization of application microservices. In certain exemplary embodiments, the systems, computer program products, and computer-implemented methods are operable with minimal observation data in a production environment. Summary of the Invention [Problem to be solved by the invention]

[0005] Many cloud applications suffer from limited observability that makes it difficult to localize failures in one or more application microservices. [Means for solving the problem]

[0006] Embodiments include systems, computer program products, and methods that learn causal relationships between application microservices and dynamically leverage the learned causal relationships for fault localization. This Summary is provided to introduce in a simplified form a selection of representative concepts that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used in any way to limit the scope of the claimed subject matter.

[0007] In one aspect, a computer system includes a processor operably coupled to a memory and an artificial intelligence (AI) platform in communication with the processor and the memory. The AI ​​platform includes a staging manager, a production manager, and a director operably coupled to the AI ​​platform. The staging manager is configured to learn causal relationships between two or more application microservices. First microservice error log data corresponding to one or more selectively injected errors is collected, and a learned causal graph is generated based on the collected first microservice error log data. The learned causal graph represents dependencies of the application microservices affected by the selective error injection. The production manager operably coupled to the staging manager is configured to dynamically locate sources of the application errors. Second microservice error log data corresponding to the application errors is collected, and an ancestor matrix is ​​constructed based on the learned causal graph and the collected second microservice error log data. The ancestor matrix is ​​utilized to identify sources of the errors. A director operably coupled to the production manager is configured to identify a microservice associated with the identified error source.

[0008] In another aspect, a computer-implemented method for learning causal relationships between two or more application microservices is provided. First microservice error log data corresponding to one or more selectively injected errors is collected, and a learned causal graph is generated based on the collected first microservice error log data. The learned causal graph represents dependencies of the application microservices affected by the selective error injection. A source of the application error is dynamically located, where the source appears in collecting second microservice error log data corresponding to the application error and constructing an ancestor matrix based on the learned causal graph and the collected second microservice error log data. The ancestor matrix is ​​utilized to identify the source of the error and the microservice associated with the identified error source.

[0009] In yet another aspect, a computer program product is provided. The computer program product comprises a computer-readable storage medium having program code embodied thereon. The program code is executable by a processor to learn causal relationships between two or more application microservices. Program code is provided for collecting first microservice error log data corresponding to one or more selectively injected errors and generating a learned causal graph based on the collected first microservice error log data. The learned causal graph represents dependencies of the application microservices affected by the selective error injection. Program code is further provided for dynamically locating sources of application errors. Second microservice error log data corresponding to the application errors is collected, and an ancestor matrix is ​​constructed based on the learned causal graph and the collected second microservice error log data. The ancestor matrix is ​​utilized to identify sources of the errors and microservices associated with the identified error sources.

[0010] In a further aspect, a computer-implemented method for training an artificial intelligence model is provided. First error log data corresponding to one or more selectively injected microservice failures is collected, and a causal graph is learned based on the collected error log data, referred to in one embodiment as the first error log data. The causal graph represents dependencies of the affected application microservices. The application failures are dynamically located, where the locating includes collecting second error log data corresponding to the detection of the application failures. The second error log data and the learned causal graph are utilized to identify a source of the application failures.

[0011] In yet a further aspect, a computer system includes a processor operably coupled to a memory and an artificial intelligence (AI) platform in communication with the processor and the memory. The AI ​​platform includes a staging manager. A production manager is provided and operably coupled to the AI ​​platform. The staging manager is configured to train an AI model. First error log data corresponding to one or more selectively injected microservice failures is collected, and a causal graph is learned based on the collected first error log data. The causal graph represents dependencies of the affected application microservices. The production manager operably coupled to the staging manager is configured to dynamically locate the application failure. Second error log data corresponding to the detection of the application failure is collected. The second error log data and the learned causal graph are utilized to identify a source of the application failure.

[0012] These and other features and advantages will become apparent from the following detailed description of exemplary embodiments, when read in conjunction with the accompanying drawings, which describe and illustrate various systems, subsystems, devices, apparatus, models, processes, and methods in additional aspects. [Brief explanation of the drawings]

[0013] The drawings referenced herein form part of the specification and are incorporated herein by reference. Features shown in the drawings are intended as illustrations of only some embodiments and not all embodiments, unless expressly indicated otherwise.

[0014] [Figure 1] FIG. 1 is a schematic diagram of a computer system that supports and enables active learning in a staging environment to learn a causal graph and leverages the learned causal graph in a production environment to locate detected application failures.

[0015] [Figure 2] FIG. 2 is a block diagram illustrating AI platform tools and their associated application program interfaces (APIs) as shown and described in FIG. 1.

[0016] [Figure 3] This is a flowchart for learning causal relationships between microservices.

[0017] [Figure 4] FIG. 1 is a block diagram illustrating an example intervention pattern.

[0018] [Figure 5] FIG. 1 is a block diagram illustrating an example intervention matrix.

[0019] [Figure 6] 4 is a flowchart of using the transitive reduced causal graph from the output of FIG. 3 for fault localization in a production environment.

[0020] [Figure 7]FIG. 7 is a block diagram illustrating an example of a computer system / server of a cloud-based support system for implementing the systems and processes described above with respect to FIGS. 1-6.

[0021] [Figure 8] FIG. 1 is a block diagram illustrating a cloud computing environment.

[0022] [Figure 9] FIG. 1 is a block diagram illustrating a set of function abstraction model layers provided by a cloud computing environment. DETAILED DESCRIPTION OF THE INVENTION

[0023] It will be readily understood that the components of the exemplary embodiments, as generally described and illustrated in the figures herein, could be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of embodiments of the systems, computer program products, and methods, and other aspects described herein, as set forth in this description and the accompanying figures, is not intended to limit the scope of the embodiments as claimed, but is merely representative of selected embodiments.

[0024] References throughout this specification to "a select embodiment," "one embodiment," or "one embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment. Thus, the appearances of the phrases "selected embodiment," "in one embodiment," or "in one embodiment" in various places throughout this specification are not necessarily referring to the same embodiment. It should be understood that various embodiments may be combined with each other and that embodiments may be used to modify each other.

[0025] The exemplary embodiments are best understood by referring to the drawings, in which like parts are designated by like numerals throughout. The following description is intended to be merely exemplary and merely illustrates certain selected embodiments of devices, systems and processes consistent with the embodiments as claimed herein.

[0026] Cloud computing is on-demand access via the Internet to computing resources such as applications, servers (including physical and virtual servers), data storage, development tools, and network capabilities hosted in remote data centers and managed by a cloud service provider. Software as a Service (SaaS), also known as cloud-based software or cloud applications, is an example of application software hosted in the cloud and accessible via a web browser, client machine, or application program interface (API). Cloud computing is further illustrated and described in FIG. 8. Microservices or microservice architecture is a cloud-native architectural approach in which a single application is composed of many loosely coupled, independently deployable components or services. However, many cloud applications suffer from limited observability, which makes it difficult to localize failures in one or more application microservices.

[0027] As shown and described herein, intrusive causal learning is applied to one or more cloud applications in a pre-deployment environment, also referred to herein as a staging environment, commonly used for software testing to assess the quality of applications before they are deployed. The staging environment provides a testing and evaluation arena to mitigate errors during production and is therefore referred to herein as a pre-deployment environment. The staging environment serves as a learning arena for causal models associated with application microservices. The production environment describes the configuration in which an application operates for its intended purpose. More specifically, the production environment is the real-time configuration in which application execution occurs. As shown and described below, the production environment monitors error log data and leverages the learned causal models from the staging environment to accurately and efficiently localize application failures using minimal observational data.

[0028] A causal model can be described as a graph, e.g., a causal graph consisting of nodes and edges that map cause and effect relationships. A causal graph is a directed acyclic graph (DAG) in which edges between two nodes encode causal relationships. In a directed graph, edges are only arrows, and an acyclic graph is a graph with no feedback loops. Thus, a DAG is a graph with only arrows for edges and no feedback loops, i.e., a node is neither its own ancestor nor its own descendant. For example, X is the direct cause of Y, e.g., X → Y, such that forcing X to take a particular value results in the realization of Y. In a causal diagram, an arrow on an edge represents the direct effect of a parent node on a child node. A node with no parent is called a root or source node. A node with no children is called a terminal. A path or chain is a sequence of adjacent edges. In a causal diagram, a directed path represents a causal path from a start node to an end node, e.g., from a parent node to a terminal node, and in one embodiment, one or more intermediate nodes between the root node and the terminal node. A DAG therefore represents a complete causal structure, in that all sources of dependency are explained by causal links.

[0029] As shown and described herein, computer systems, methods, and computer program products are provided that utilize fault injection to learn causal relationships between microservices and leverage the learned causal relationships in real time, along with application error log data, to identify and locate application error sources for one or more application microservices. Many cloud applications utilize multiple microservices. Industrial microservice applications have hundreds or more microservices and complex dependencies between them. The topology of application microservices is fixed but often unknown. These applications have limited observability, making it difficult to locate failures in a corresponding microservice or multiple microservices. The systems, methods, and computer program products shown and described herein use observational data in the form of error log data to identify hidden causal graphs or subsets of true causal edges between microservices. A causal model is a mathematical model that represents causal relationships within an individual system or a population. As shown and described herein, computer systems, computer program products, and computer-implemented methods are provided that use pre-deployment fault injection to learn accurate causal graphs through interventional causal learning, and use the learned graphs to perform effective and accurate fault localization.

[0030] Referring to FIG. 1, a schematic diagram of a platform computing system 100 is shown. In an exemplary embodiment, the system 100 includes or incorporates an artificial intelligence (AI) platform 150. As shown, a server 110 is provided that communicates with multiple computing devices 180, 182, 184, 186, 188, and 190 via a network connection 105. The server 110 is configured with a processing unit (also referred to herein as a processor) 112 that communicates with memory 116 via a bus 114. The server 110 is shown as having an AI platform 150 for cognitive computing, including natural language processing (NLP) and machine learning (ML), that communicates with one or more of the computing devices 180, 182, 184, 186, 188, and 190 via the network 105. More specifically, computing devices 180, 182, 184, 186, 188, and 190 communicate with each other and with other devices or components via one or more wired and / or wireless data communication links, where each communication link may include one or more of a wire, a router, a switch, a transmitter, a receiver, etc. In this networked configuration, server 110 and network connections 105 enable communication detection, recognition, and resolution. Other embodiments of server 110 may be used with components, systems, subsystems, or devices, or combinations thereof, other than those shown herein.

[0031] The AI ​​platform (150) is depicted herein as being configured with tools that support active learning in a staging environment to learn a causal graph and leverage the learned causal graph in a production environment to localize detected application failures. It is understood in the art that active learning is a form of machine learning. The tools include, but are not limited to, a staging manager (152), a production manager (154), and a director (156). While FIG. 1 depicts each of the tools (152), (154), and (156) as part of the AI ​​platform (150), it should be understood that in one embodiment, any one or combination of the tools (152), (154), and (156) is not necessarily part of the AI ​​platform (150) or an operational AI. In an exemplary embodiment, the staging manager (152) is part of the AI ​​platform (150), and the production manager (154) or the director (156), or both, are each non-AI, i.e., the production manager (154) or the director (156), or both, are operatively coupled to the processor (112) and the AI ​​platform (150), and the functions of the production manager (154) or the director (156), or both, are performed without the use of artificial intelligence.

[0032] Artificial intelligence (AI) relates to the field of computer science that focuses on computers and computer behavior as they relate to humans. AI refers to intelligence when a machine is able to make informed decisions, maximizing its chances of success in a given topic. More specifically, AI can learn from datasets to solve problems and provide relevant recommendations. For example, in the field of AI computer systems, natural language systems (such as the IBM Watson® artificial intelligence computer system or other natural language question-answering systems) process natural language based on knowledge acquired by the system. To process natural language, the system may be trained using data derived from a database or knowledge corpus.

[0033] Machine learning (ML), a subset of AI, utilizes algorithms that learn from data and make predictions based on this data. AI refers to intelligence when a machine is able to make informed decisions, maximizing the chances of success in a given topic. More specifically, AI can learn from datasets to solve problems and provide relevant recommendations. Cognitive computing is a blend of computer science and cognitive science. Cognitive computing utilizes self-learning algorithms that use minimal data, visual recognition, and natural language processing to solve problems and optimize human processes.

[0034] At the core of AI and associated reasoning lies the concept of similarity. The process of understanding natural language and objects requires reasoning in terms of relationships, which can be difficult. Structures, including static and dynamic structures, define determined outputs or actions for given deterministic inputs. More specifically, the determined outputs or actions are based on explicit or inherent relationships within the structures. They depend on sufficient data sets to build those structures.

[0035] Shown herein is an AI platform (150) configured to receive input (102) from one or more sources. For example, the AI ​​platform (150) may receive input (e.g., microservices-based applications) over a network (105) from one or more of a plurality of computing devices (180), (182), (184), (186), (188), and (190). Further, as shown herein, the AI ​​platform (150) is operatively coupled to a knowledge base (160). While one knowledge base (160) is shown in FIG. 1 , it should be understood that variations of the system (100) may be utilized to support two or more knowledge bases in communication with the AI ​​platform (150).

[0036] According to an exemplary embodiment, an AI platform (150) is configured to learn causal relationships of application microservices. A staging manager (152) is shown herein as being embedded within the AI ​​platform (150). The staging manager (152) is configured to selectively inject one or more errors into the application microservices, collect corresponding application log data, subject the error log data to a filter or filtering process to identify log data corresponding to the injected one or more errors, and utilize the error log data to generate a causal graph, which is stored in a corresponding knowledge base (160). In an exemplary embodiment, the causal graph is an AI model, also referred to herein as a trained AI model. The process of creating a causal graph to be stored in the knowledge base (160) is shown and described in FIG. 3. An initial aspect of causal learning focuses on error injection into application microservices. Errors are injected into the application microservices by the staging manager (152). Errors can be injected individually, e.g., one microservice at a time, or into a set of microservices, e.g., two or more microservices at a time. In one embodiment, errors can be injected randomly. Similarly, in one embodiment, the error injection can follow a pattern. The error injection is aimed at creating problems associated with the functionality of an application microservice. For example, the error injection can be in the form of blocking a particular microservice, slowing down the operation of the microservice, or otherwise making the microservice unavailable to the application.

[0037] An error log is a record of errors an application, operating system, or server encounters during operation. For example, some common entries in an error log include table corruption or configuration corruption. The error log may capture a wealth of information, which in one embodiment may include relevant or irrelevant data. The staging manager (152) addresses this aspect by preprocessing the log data to identify error logs corresponding to or associated with the injected error. In an exemplary embodiment, the staging manager (152) filters the log data to extract specific message text associated with the injected error. An example filter may be in the form of, but is not limited to, one or more keywords or combinations of keywords in the error log. Application of the filter provides focus on relevant log data, also referred to herein as error log data. The staging manager (152) collects or otherwise identifies or obtains the error logs remaining after preprocessing to learn causal relationships between application microservices, also referred to herein as causal learning. Details of causal learning are shown and described in detail in FIGS. 3-5. Causal learning effectively computes correspondences between the injected fault-stricken microservice and each related microservice to understand which microservices are affected or impacted by the fault injection. More specifically, causal learning identifies directional connections between microservices. In an exemplary embodiment, causal learning produces output in the form of a set of microservices represented in a DAG, where the represented set of microservices is associated with log data that emitted or otherwise captured or documented one or more errors. Thus, the staging manager generates a causal graph of application microservices from the error log data.

[0038] The staging manager (152) uses the output of the set of microservices to generate or otherwise construct a corresponding causal graph, e.g., a DAG. More specifically, directed edges between two microservices are selectively removed from the set of microservices. In an exemplary embodiment, selective removal filters out a selection of one or more edges through transitive reduction. Details of selective removal are shown and described in FIG. 3. A DAG is generated, or in one embodiment, regenerated, from the set of microservices remaining in the set of microservices. Thus, a causal graph of microservices is generated from the reduced set of affected microservices, which in one embodiment is a subset of the application microservices.

[0039] As illustrated herein, knowledge base (160) is shown as having a library (162) configured to receive and store the generated causal graphs. While only one library is shown, in one embodiment, knowledge base (160) may include one or more additional libraries. Illustratively, library (162) is shown as having multiple applications, each having a first error log and a corresponding causal graph. Illustratively, library (162) is shown as having multiple applications, each having a first error log and a corresponding causal graph. Illustratively, library (162) is shown as having multiple applications, each having a first error log and a corresponding causal graph. N (164 N ) are shown. Although only three applications are shown, this number is for illustrative purposes and should not be considered limiting. Each application is referred to herein as a log 0 (1660), a log 1 (1661), and a log N (166 N ), and the corresponding first error log shown as graph 0 (1680), graph 1 (1681), and graph N (168 N) and a corresponding causal graph shown as

[0040] A user flow refers to the path a prototypical user takes through an application to complete a task. A user flow takes a user from their entry point, through a set of steps, to a successful outcome, and to a final action, such as purchasing a product. Confounding is a causal concept defined in terms of a data-generating model. A confounder is a variable that influences both the dependent and independent variables. As shown and described herein, the staging manager 152 addresses unobserved confounds due to user flow by inferring a causal graph from error log data.

[0041] A staging manager (152), shown in FIG. 1 as part of the AI ​​platform (150) but not part of the AI-based or AI platform (150) in an alternative exemplary embodiment, is configured to generate a causal graph between application microservices. A causal effect means something has happened or is happening based on something else that has happened or is happening. In terms of microservices, an error on a first microservice A can cause an error in a second microservice B. This is represented by a directed edge from A to B, e.g.,

number

[0042] The staging manager (152) and its functionality for generating a causal graph from an error log associated with selective error injection function offline. In one embodiment, error log data associated with an application and generated by the staging manager (152) is referred to herein as first error log data. The production manager (154) is provided to support online processing, more specifically, to locate error sources. In one embodiment, the production manager (154) is operably coupled to the AI ​​platform (150). Similarly, in one embodiment, the production manager (154) and its functionality are performed in real time as a dynamic component. Error log data associated with application processing and execution, as well as the staging environment associated with the functionality of the staging manager (152), is collected by the production manager (154). In one embodiment, error log data associated with the production manager (154) is referred to herein as second error log data. As shown herein by way of example, the second error log data is stored in a knowledge base (160), and is referred to herein as (1700), (1701), and (170 N ), and each second error log data is stored in the corresponding application (1640), (1641), and (1642). N) processing. The difference between the first error log data and the second error log data lies in the way the error log data is generated. The staging manager (152) operates offline and intentionally injects one or more errors into application microservices, and the first error log data provides documentation of the impact of the error injection. Meanwhile, the production manager (154) operates online and generates second error log data that provides documentation of the impact of application processing errors. Thus, the staging manager (152) artificially creates microservice failures, and the production manager (154) responds to application errors detected during application processing and execution.

[0043] The error log data collected by the production manager 154 is generated in real time. The production manager 154 utilizes the collected second error log data to calculate correspondences between the microservice that is the subject of the failure and other application microservices, and generates an ancestor matrix using the corresponding causal graph associated with the first error log data and stored in the knowledge base 160. For application 0 1640, the calculated correspondences are shown herein as 1720. Details of the process for generating the ancestor matrix are shown and described in FIG. 3. By way of example, evaluation of application 0 1640 using the corresponding causal graph 1680 generates the ancestor matrix 1740. Using the ancestor matrix 1740 and the calculated correspondences 1720, the production manager 154 compares the ancestor matrix, e.g., 1740, against the correspondences, e.g., 1720, calculated by the production manager 154, using a metric function that evaluates the similarity between the sequences. In one embodiment, the metric function is Hamming distance or cosine similarity. Details of fault localization are shown and described in FIG. 6. In an exemplary embodiment, the metric function yields an estimated location of the fault and generates a top-k list of possible fault locations, where k is a configurable value. Thus, the production manager (154) applies fault localization to the learned causal graph and utilizes a threshold distance to estimate or otherwise identify the fault location.

[0044] The director 156 is shown herein as operatively coupled to the production manager 154. Based on the evaluation, the director 156 identifies or recommends one or more faulty microservices as the source of the detected errors. In an exemplary embodiment, the director 156 communicates the faulty one or more microservices to a subject matter expert (SME) for remediation.

[0045] As illustrated herein, the staging manager (152) learns the causal relationships and stores a representation of the learned causal relationships, referred to herein as a causal graph, in a knowledge base (160). A production manager (154), in communication with the knowledge base (160), uses the learned causal graph and the second log data to determine a top-k list of possible failure locations for a given application failure. In one embodiment, the director (156) stores possible application failure locations, e.g., microservices, in the knowledge base (160). As illustrated herein by way of example, an application (1640) may store a list of possible failure locations (176). 0,0 ), (176 0,1 ), ..., (176 0,k ) The failure location shown here is for Application 0 (1640). Although not shown, in one embodiment, Application 1 (1641) or Application N (164 N ) or both may contain lists or groups of possible fault locations. Alternatively, the director (156) may be configured not to further populate the knowledge base (160) with the top k list of possible fault locations.

[0046] In some exemplary embodiments, the server (110) may be an IBM Watson® system available from International Business Machines Corporation of Armonk, New York, extended with mechanisms of the exemplary embodiments described below. The tools, collectively referred to as a staging manager (152), a production manager (154), and a director (156), are shown as embodied in or integrated into an AI platform (150) of the server (110). In one embodiment, the staging manager (152) is embodied in the AI ​​platform (150), and the production manager (154) and the director (156) are operatively coupled to the AI ​​platform (150). In another embodiment, the tools may be implemented in a separate computing system (e.g., server 190) connected to the server (110) via the network (105). Regardless of where it is implemented, the tool functions to support the identification of causal pairs for application microservices and leverage the identified causal pairs to dynamically localize failures.

[0047] The types of information handling systems that can utilize the AI ​​platform (150) range from small handheld devices, such as handheld computers / cell phones (180), to large mainframe systems, such as mainframe computers (182). Examples of handheld computers (180) include personal digital assistants (PDAs), personal entertainment devices, such as MP4 players, portable televisions, and compact disc players. Other examples of information handling systems include pen or tablet computers (184), laptop or notebook computers (186), personal computer systems (188), and servers (190). As shown, the various information handling systems can be networked together using a computer network (105). Types of computer networks (105) that can be used to interconnect the various information handling systems include local area networks (LANs), wireless local area networks (WLANs), the Internet, public switched telephone networks (PSTNs), other wireless networks, and any other network topology that can be used to interconnect information handling systems. Many information handling systems include a non-volatile data store, such as a hard drive or non-volatile memory, or both. Some information handling systems may use a separate non-volatile data store (e.g., server 190 may use a non-volatile data store 190). A ) and the mainframe computer (182) uses a non-volatile data store (182a). A ) can be a component that is external to the various information handling systems, or it can be internal to one of the information handling systems.

[0048] The information handling system utilized to support the AI ​​platform (150) may take many forms, some of which are illustrated in FIG. 1. For example, the information handling system may take the form of a desktop, server, portable, laptop, notebook, or other form factor computer or data processing system. In addition, the information handling system may take other form factors, such as a personal digital assistant (PDA), gaming device, ATM machine, portable telephone device, communications device, or other device that includes a processor and memory. In addition, the information handling system may embody a northbridge / southbridge controller architecture, although it will be understood that other architectures may be utilized.

[0049] An application program interface (API) is understood in the art as a software intermediary between two or more applications. With respect to the AI ​​platform 150 shown and described in FIG. 1, one or more APIs may be utilized to support one or more of the tools 152, 154, and 156 and their associated functionality. Referring to FIG. 2, a block diagram 200 is provided illustrating the tools 152, 154, and 156 and their associated APIs. As shown, multiple tools are embedded within the AI ​​platform 205, including a staging manager 252 associated with API 0 212, a production manager 254 associated with API 1 222, and a director 256 associated with API 2 232. Each of the APIs may be implemented in one or more languages ​​and interface specifications.

[0050] As shown, API0 (212) is configured to support the offline task of selectively injecting errors into application microservices and processing the corresponding error logs, also referred to herein as first error log data, to generate or otherwise learn a causal graph. API1 (222) provides functional support for the online task of collecting all microservice error log data, also referred to herein as second error log data, that correspond to application errors and building an ancestor matrix based on the learned causal graph. API2 (232) provides functional support for fault localization, which in one embodiment includes applying a metric function to evaluate similarity between sequences and utilize the evaluation, along with the associated ancestor matrix, to identify a subset of microservices, e.g., a top-k subset, that are or may be the source of the detected error. As shown, each of APIs 212, 222, and 232 is operatively coupled to API orchestrator 260, otherwise known as an orchestration layer, which is understood in the art to function as an abstraction layer that threads transparently with the separate APIs. In one embodiment, the functionality of the separate APIs may be combined or combined. In another embodiment, the functionality of the separate APIs may be further divided into additional APIs. Thus, the organization of APIs shown herein should not be considered limiting. Accordingly, the functionality of tools may be embodied or supported by their respective APIs, as shown herein.

[0051] Referring to Figure 3, a flowchart (300) is provided illustrating a process for learning causal relationships between microservices. An initial aspect of learning causal relationships involves identifying application microservices through selective and controlled fault injection. As shown herein, the variable S Totalrepresents the quantity of application microservices (302). For each of the represented microservices, e.g., S = 1 to S Total Errors are selectively injected and corresponding log data is collected (304) for the microservices. s The selective error injection in step (304) may be to block, remove, or delay a microservice. The selective error injection in step (304) may be applied to microservices individually or in combination; for example, two or more microservices may be subjects of the fault injection. It is understood in the art that there are a variety of faults or errors that may be applied to a microservice. In one embodiment, the form or type of error injection in step (304) is randomly selected for application to one or more microservices. Similarly, in an exemplary embodiment, the fault injection in step (304) is controlled or supported by an error injection pattern. Thus, an initial aspect of learning causal relationships between application microservices involves selective error injection targeted to one or more microservices.

[0052] Error propagation is a term that refers to the manner in which, at a given computation stage, some of the errors arise from errors in the previous stage. In microservice architectures, and more specifically, in inter-microservice dependencies, an error introduced in one microservice can extend uncertainty to one or more related microservices. An error is injected, and corresponding application log data is collected. Log data is understood in the art to be automatically generated and time-stamped documentation of events. With respect to an application and its embedded microservices, and more specifically with respect to microservice error injection, the log data identifies the direct or indirect effects of the injected error on other application microservices that are not directly affected by the error injection. In one embodiment, the log data is a log file that records messages associated with the functionality of one or more microservices, including one or more microservices affected by the microservice into which the fault was injected and, in one embodiment, one or more microservices that are not affected by the injected fault. In an exemplary embodiment, the log file is utilized for error tracing associated with the injected fault. Thus, error injection artificially creates a problem in an application microservices architecture, and the log files document log data of one or more microservice considerations as related to the injected error.

[0053] It is understood in the art that a log file consists of multiple messages containing text and corresponding timestamps. Some of the messages or message content may contain irrelevant or extraneous information regarding the injected error. For example, the log data may include a message, e.g., an error message, that a particular microservice may not be able to process a request in response to a fault being injected into a different application microservice. To address the log files, and in one embodiment, to address the abundant amount of log data, the log files and corresponding log data collected in step (304) undergo processing or preprocessing to filter out, e.g., remove (306), log data that is irrelevant to the injected error. In one embodiment, one or more defined keywords are applied to the log files as a filter to extract relevant or useful log data, which in one embodiment returns all error logs. In an exemplary embodiment, a subset of the original log data remains after the filtering step, and the microservices associated with the subset of log data are the subjects of causal learning. Following step 306, causal learning through intervention patterns is utilized to identify directional connections between microservices that are actors for the log data that survived preprocessing (308). Details of causal learning are illustrated and described in FIGS. 4 and 5. In one embodiment, causal learning is a form of machine learning that utilizes causal inference. In step 308, causal learning includes learning correlation scores between microservices based on intervention patterns and corresponding intervention matrices, and a representation of a learned causal graph using transitive reduction. The correlation score evaluation in step 308 identifies the strength of the correspondence between the microservice that is the actor of the failure s' and the microservices identified from the subset of log data. As illustrated and described in FIG. 4, the correlation score is evaluated against a configurable threshold. The evaluation from step 308 generates an output in the form of a DAG consisting of a set of edges that exceed the correlation score evaluation, with each edge representing a microservice that is the actor of the failure and an affected microservice (310).In an exemplary embodiment, as illustrated herein, the graph generated in step 310 undergoes transitive reduction 312, which selectively removes one or more edges and generates a causal graph. Transitive reduction is an edge removal operation on a directed graph that preserves certain important properties and structure of the graph. Transitive reduction preserves important structural properties of the learned causal graph and is used to build an ancestry of the learned causal graph to locate the faulty service. Details of transitive reduction are shown and described in detail below. Thus, log data associated with fault injections is utilized as a source to generate the causal graph.

[0054] Referring to FIG. 4, a block diagram (400) illustrating an example intervention pattern is provided. The vector v(s') is the intervention pattern vector for microservice s', where s' is the microservice that is the subject of the injected fault. In one embodiment, as illustrated herein, the fault injection aspect may be in the form of blocking the microservice from performing its intended function. The vector v(s') trepresents how other microservices in the application are affected by the blocked microservice s' in time bin t. As shown in this example, the entries in the vector are in the form of bits of 0 and 1. In one embodiment, an entry of 0 in the vector indicates that the microservice is not affected by the blocked microservice, and an entry of 1 in the vector indicates that the microservice is affected, e.g., experiencing an error. Similarly, in one embodiment, the representation of the vector entries may be inverted, and therefore the entry representation should not be considered limiting. The vectors shown herein target the fault-injected microservice s' over time bin t and document the reaction of the application microservices to the fault injection. Multiple vectors are utilized to generate a corresponding intervention matrix C. An example intervention matrix is ​​shown and described in FIG. 5. In an example embodiment, the strength of correlation between microservice s' and all other microservices is evaluated as follows:

number

[0055] Referring to FIG. 5, a block diagram (500) illustrating an example intervention matrix (510) is provided. As shown, the intervention matrix C(s') is for microservice s' into which a fault has been injected. As shown in this example, there are five microservices. One of microservice s' has been injected with an error or fault, and the remaining four microservices s0, s1, s2, and s3 are either affected or unaffected by the injected error. As shown herein by way of example, at time period t=1, microservice s0 is shown as having two errors at (520), microservice s1 is shown as having one error at (522), and microservices s2 and s3 are each shown as having no errors at (524) and (526), ​​respectively. The intervention matrix C is shown to include multiple time periods (530), also referred to herein as time bins T. Therefore, C(s') is an intervention matrix formed from multiple intervention pattern vectors, which indicates the reactions of all microservices affected by the fault injection microservice s'.

[0056] For a DAG with individual nodes representing microservices and directed edges representing ancestral relationships between the nodes, causal learning in step (310) involves inferring ancestral edges for nodes in the DAG that have fault injections (312). As shown and described in Figure 1, correlation evaluation is performed in a production environment and managed by a production manager (154). The following pseudocode demonstrates inferring correlations of ancestral edges associated with microservice dependencies: [Table 1] Here, C is an intervention matrix indicating other microservices, e.g., s0, s1, s2, and s3, affected by the fault injected into microservice s', as shown in FIG. 5, and E is a set of tuples of directed edges between microservices that emit errors during application processing. The intervention matrix is ​​a compilation of the intervention pattern vector v(s'). As shown herein, the correlation score between microservice s' and microservice s is learned and evaluated against a threshold for the correlation score τ, which in one embodiment is an adjustable threshold. For example, if the correlation score corr(s', s) > τ, this is an indication that microservice s' and microservice s are highly correlated. Transitive reduction is an edge removal operation on a directed graph that preserves some important properties and structure of the graph. The output from ancestor edge inference in step (312) is a causal graph. The process of error injection into one or more selected microservices as shown herein is referred to as the pre-deployment fault injection stage. In one embodiment, the set of causal edges in a learned causal graph as presented herein is guaranteed to include only a set or subset of true causal edges that have theoretically high probability of causal relationships. Thus, the causal graph is generated based on log data information collected through the use of one or more fault injections.

[0057] The inferred ancestor edges from the various fault injections are combined into the representation by performing transitive reduction (314) to ensure that only the subset of true causal edges that preserves lineage is in the compact representation. The transitive reduction of a directed graph G is another directed graph G' with the same number of vertices and the fewest possible number of edges, such that for every pair of vertices, a path between vertices in G exists if and only if such a path exists in G'. The following pseudocode demonstrates transitive reduction as applied to a causal graph E: [Table 2] where G represents the regenerated causal graph of the microservices with (a, b) removed from the set of directed edges E. In one embodiment, the steps illustrated and described herein may be performed offline. Thus, transitive reduction is used to identify a compact representation of the learned causal graph that represents the dependencies of the subset of microservices related to the microservice s' into which the fault was injected.

[0058] Referring to Figure 6, a flowchart (600) is provided illustrating using the transitive reduced causal graph from the output of Figure 3 for fault localization in a production environment. In an exemplary embodiment, the fault localization described herein is performed in real time. An error in an unknown intervened microservice is detected (602), and all log data corresponding to the detected error, also referred to herein as second log data, is collected (604). In an exemplary embodiment, the collection of the second log data is performed in real time. Following the collection of the second error log, the learned transitive reduced causal graph G from the staging environment is leveraged to locate the fault (606). The following pseudocode demonstrates estimating the locality of a faulty microservice (608): [Table 3] where G T is the transitive reduction of the learned causal graph G. The correlation evaluation shown in the pseudocode uses the same function, e.g., 1{·}. For example, corr=[0.8 0.1 0.1 0.9 0.1 0.2...]∈Z N×1 and τ = 0.3, then 1{0.8>.3} = 1 and 1{0.1<.3} = 0. Based on this example, 1{corr} = [1 0 0 1 0 0...]∈Z N×1The distance estimate Dist(s) utilizes the correlation matrix A to measure the distance between rows. In one embodiment, each row of the correlation matrix A has entries in the form of a bit, where 1 represents a microservice that has an ancestor in the learned causal graph and 0 represents the reverse, e.g., a microservice that has no ancestor in the learned causal graph. The distance estimate represents the number of points at which two corresponding data differ. In one embodiment, the distance estimate may be utilized in the form of a Hamming distance or cosine similarity. In an exemplary embodiment, a metric function provides an estimated location of the fault and generates a top-k list of possible fault locations, where k is a configurable value. Thus, as presented herein, the correlation matrix A is constructed based on the learned causal graph G, and the location of the fault is estimated using the distance estimate.

[0059] The processes shown and described in Figures 3 and 6 illustrate scenarios in which a fault is injected into a single microservice, either planned or unplanned, respectively. In one embodiment, these processes may be extended to injecting faults into pairs or subsets of microservices. Similarly, in one embodiment, the process shown and described in Figure 6 may be extended to full causal graphs rather than transitive reduced graphs. As illustrated herein, fault localization involves building an ancestor matrix A based on the learned causal graph G and estimating the location of the fault using distance evaluation. In an exemplary embodiment, multiple estimated fault locations, e.g., the top k, may be generated from the fault localization process. Thus, using a pre-deployment fault injection system, log data is accumulated and processed as a source for learning the causal graph G, which is then used in real time to perform fault localization dynamically and efficiently.

[0060] Certain exemplary embodiments of the systems, methods, and computer program products described herein generate high-quality sets of cause-effect pairs in a substantially or completely unsupervised, automated manner. Exemplary embodiments further involve using the cause-effect pairs for further processing of the representation as a causal knowledge graph, and for use in decision support or predictive analytics.

[0061] Aspects of identifying and verifying causal pairs are illustrated and described in conjunction with the tools and APIs shown in Figures 1 and 2 and the processes illustrated in Figures 3 and 6, respectively. Aspects of functional tools 152, 154, and 156 and their associated functionality may be embodied in a computer system / server at a single location, or in one embodiment, configured in a cloud-based system that shares computing resources. Referring to Figure 7, a block diagram 700 is provided illustrating an example of a computer system / server 702, hereafter referred to as a host 702, in communication with a cloud-based support system to implement the processes described above with respect to Figures 3 and 6. The host 702 is operable with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, or configurations, or combinations thereof, that may be suitable for use with the host (702) include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics products, network PCs, minicomputer systems, mainframe computer systems, and file systems (e.g., distributed storage environments and distributed cloud computing environments) that include any of the above systems, devices, and their equivalents.

[0062] The host (702) may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The host (702) may be implemented in a distributed cloud computing environment (710) where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0063] As shown in FIG. 7 , the host (702) is depicted in the form of a general-purpose computing device. Components of the host (702) may include, but are not limited to, one or more processors or processing units (704), such as a hardware processor, a system memory (706), and a bus (708) coupling various system components, including the system memory (706), to the processing unit (704). The bus (708) represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus. The host (702) typically includes a variety of computer system-readable media. Such media can be any available media that is accessible by the host 702 and includes both volatile and non-volatile media, removable and non-removable media.

[0064] The system memory 706 may include computer-system-readable media in the form of volatile memory, such as random access memory (RAM) 730 or cache memory 732, or both. By way of example only, the storage system 734 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, typically referred to as a "hard drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from or writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical media. In such cases, each may be connected to the bus 708 by one or more data media interfaces.

[0065] A program / utility (740) having a set (at least one) of program modules (742) may be stored in system memory (706), as well as, by way of example and not limitation, an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or any combination thereof, may comprise an implementation of a networked environment. The program modules (742) generally perform the functionality and / or methodology of embodiments that support and enable active learning through selective fault injection for causal graph generation and leverage the output of active learning for dynamic fault localization. For example, the set of program modules (742) may include tools (152), (154), and (156) as illustrated in FIG. 1 .

[0066] The host (702) may communicate with one or more external devices (714), such as a keyboard, pointing device, etc., a display (724), one or more devices that allow a user to interact with the host (702), or any device that allows the host (702) to communicate with one or more other computing devices (e.g., a network card, a modem, etc.), or combinations thereof. Such communication may occur via an input / output (I / O) interface (722). Still further, the host (702) may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or combinations thereof, via a network adapter (720). As shown, the network adapter (720) communicates with the other components of the host (702) via a bus (708). In one embodiment, multiple nodes of a distributed file system (not shown) communicate with the host (702) via an I / O interface (722) or via a network adapter (720). Although not shown, it should be understood that other hardware and / or software components may be used in conjunction with the host (702). Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0067] In this document, the terms "computer program medium," "computer usable medium," and "computer readable medium" are used to generally refer to media such as system memory (706), including RAM (730), cache (732), and storage system (734), e.g., removable storage drives and hard disks installed in hard disk drives.

[0068] Computer programs (also called computer control logic) are stored in system memory 706. Computer programs may be received via a communications interface, such as a network adapter 720. When executed, such computer programs enable the computer system to perform the features of the present embodiments as discussed herein. In particular, when executed, the computer programs enable the processing unit 704 to perform the features of the computer system. Thus, such computer programs represent the controller of the computer system.

[0069] In one embodiment, the host (702) is a node in a cloud computing environment. As is known in the art, cloud computing is a model of service delivery that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with a service provider. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models. Examples of such characteristics are:

[0070] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, automatically as needed, without the need for human interaction with the service provider.

[0071] Wide network access: This capability is available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (eg, cell phones, laptops, and PDAs).

[0072] Resource Pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. There is location independence in that consumers generally have no control or knowledge over the exact location of the provided resources, although it may be possible to specify the location at a higher layer abstraction (e.g., country, state, or data center).

[0073] Rapid Elasticity: This capacity can be rapidly and elastically provisioned, in some cases automatically, to rapidly scale out, and rapidly released to rapidly scale in. To the consumer, the capacity available for provisioning often appears unlimited, and can be purchased in any quantity at any point in time.

[0074] Measured Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a layer of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services utilized.

[0075] The service model is as follows:

[0076] Software as a Service (SaaS): The consumer is offered the ability to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through a thin-client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0077] Platform as a Service (PaaS): The ability offered to consumers is to deploy applications they create or acquire, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does control the deployed applications and, in some cases, the application hosting environment configuration.

[0078] Infrastructure as a Service (IaaS): The ability offered to consumers is to provision processing, storage, network, and other basic computing resources, on which they can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does control the operating systems, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0079] The deployment model is as follows:

[0080] Private Cloud: This cloud infrastructure operates solely for an organization. It can be managed by the organization or a third party and can exist on-premise or off-premise.

[0081] Community Cloud: This cloud infrastructure is shared by several organizations and supports a specific community with shared concerns (e.g., mission, security requirements, policies, and compliance considerations). The community cloud may be managed by those organizations or a third party and may exist on-premises or off-premises.

[0082] Public Cloud: This cloud infrastructure is made available to the general public or large industry groups and is owned by an organization that sells cloud services.

[0083] Hybrid Cloud: This cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain distinct entities but are bound together by standard or proprietary technologies that allow for data and application portability (e.g., cloud bursting for load balancing between clouds).

[0084] Cloud computing environments are service-oriented with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0085] Referring now to FIG. 8, an exemplary cloud computing network 800 is shown. The cloud computing network 800 includes a cloud computing environment 850 having one or more cloud computing nodes 810 with which local computing devices used by cloud consumers may communicate. Examples of these local computing devices include, but are not limited to, personal digital assistants (PDAs) or mobile phones 854A, desktop computers 854B, laptop computers 854C, or automobile computer systems 854N, or combinations thereof. Individual nodes within the cloud computing node 810 may further communicate with each other. The nodes may be physically or virtually grouped (not shown) in one or more networks, such as a private cloud, community cloud, public cloud, or hybrid cloud, or combinations thereof, as described hereinabove. This enables the cloud computing environment 800 to provide infrastructure, platform, or software, or combinations thereof, as a service for which cloud consumers do not need to maintain resources on their local computing devices. It will be understood that the types of computing devices (854A-N) shown in FIG. 8 are intended to be illustrative only, and that the cloud computing environment (850) can communicate with any type of computerized device via any type of network, or network-addressable connection, or both (e.g., using a web browser).

[0086] Referring now to Figure 9, there is shown a set of functional abstraction layers (900) provided by the cloud computing network of Figure 8. It should be understood in advance that the components, layers, and functions shown in Figure 9 are intended to be merely exemplary, and that embodiments are not limited thereto. As shown, the following layers and corresponding functions are provided: a hardware and software layer (910), a virtualization layer (920), a management layer (930), and a workload layer (940).

[0087] The hardware and software layer (910) includes hardware and software components. Examples of hardware components include mainframes, such as IBM® zSeries® systems; RISC (reduced instruction set computer) architecture-based servers, such as IBM pSeries® systems, IBM xSeries® systems, and IBM BladeCenter® systems; storage devices; and networks and networking components. Examples of software components include network application server software, such as IBM WebSphere® application server software; and database software, such as IBM DB2® database software. (IBM, zSeries, pSeries, xSeries, BladeCenter, WebSphere, and DB2 are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide.)

[0088] The virtualization layer (920) provides an abstraction layer at which instances of virtual entities can be provided: virtual servers, virtual storage, virtual networks including virtual private networks, virtual applications and operating systems, and virtual clients.

[0089] In one example, the management layer (930) may provide the following functions: resource provisioning, metering and pricing, a user portal, service layer management, and SLA planning and fulfillment. Resource provisioning provides dynamic procurement of computing and other resources utilized to perform tasks within the cloud computing environment. Metering and pricing provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. The user portal provides access to the cloud computing environment for consumers and system administrators. Service layer management provides cloud computing resource allocation and management so that requested service layers are met. Service layer agreement (SLA) planning and fulfillment provides advance arrangements and procurement of cloud computing resources where future requirements are anticipated according to SLAs.

[0090] The Workloads layer (940) provides examples of functions for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this layer include, but are not limited to, mapping and navigation, software development and lifecycle management, virtual classroom instruction delivery, data analytics processing, transaction processing, and causal knowledge identification and extraction.

[0091] While specific embodiments of the present invention have been shown and described, it will be apparent to those skilled in the art that changes and modifications can be made based on the teachings herein without departing from the embodiments and their broader aspects. Accordingly, the appended claims are intended to encompass within their scope all such changes and modifications as fall within the true spirit and scope of the embodiments. It is to be understood, further, that the embodiments are defined solely by the appended claims. Where a specific number of introduced claim elements is intended, such intention will be expressly recited in the claims; in the absence of such recitation, it will be understood by those skilled in the art that no such limitation exists. As a non-limiting example, and to aid in understanding, the following appended claims include the use of the introductory phrases "at least one" and "one or more" to introduce claim elements. However, the use of such language should not be construed as implying that the introduction of a claim element by the indefinite article "a" or "an" limits any particular claim containing such introduced claim element to embodiments containing only one of such element, even if both the introductory phrase "one or more" or "at least one" and an indefinite article such as "a" or "an" are included in the same claim; the same is true for the use of definite articles in claims. As used herein, the term "and / or" means either or both (any combination or all of the referenced terms or expressions); for example, "A, B and / or C" includes A alone, B alone, C alone, A and B, A and C, B and C, and A, B and C.

[0092] The present embodiments may be a system, a method, or a computer program product, or a combination thereof. In addition, selected aspects of the present embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software or hardware aspects, or both, all of which may be generally referred to herein as a "circuit," "module," or "system." Furthermore, aspects of the present embodiments may take the form of a computer program product embodied in a computer-readable storage medium (or media) having computer-readable program instructions that cause a processor to perform aspects of the present embodiments. So embodied, the disclosed system, method, or computer program product, or combination thereof, operates to provide improved identification and verification of causal pairs.

[0093] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, dynamic or static random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), magnetic storage devices, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves that record instructions, and any suitable combination of the foregoing. Computer-readable storage medium, as used herein, should not be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0094] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to a computer-readable storage medium in the respective computing / processing device for storage.

[0095] The computer-readable program instructions for carrying out the operations of the present embodiments may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java®, Smalltalk®, C++, etc., and traditional procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server or server cluster. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions to personalize the electronic circuitry by utilizing state information of the computer readable program instructions to perform aspects of the present embodiments.

[0096] Aspects of the present embodiments are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to the embodiments. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0097] These computer-readable program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, whereby the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions can also be stored on a computer-readable storage medium, whereby the instructions can instruct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, whereby the computer-readable storage medium having the instructions stored thereon comprises an article of manufacture including instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0098] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be executed on the computer, other programmable apparatus, or other device to generate a computer-implemented process, whereby the instructions executing on the computer, other programmable apparatus, or other device implement the functions / operations specified in one or more blocks of the flowcharts or block diagrams, or both.

[0099] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions, that implement the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may be executed in the reverse order, depending on the functionality involved. Additional blocks not represented in the figures may be included, for example, before, after, or concurrently with one or more of the illustrated blocks. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

[0100] While specific embodiments have been described herein for purposes of illustration, it will be understood that various modifications may be made without departing from the spirit and scope of these embodiments. In particular, identifying and verifying causal pairs may be performed by different computing platforms or across multiple devices. Furthermore, data storage and / or corpus may be localized, remote, or spread across multiple systems. Accordingly, the scope of protection of the embodiments is limited only by the following claims and their equivalents.

Claims

1. a computer processor operably coupled to the memory; an artificial intelligence (AI) platform in communication with the computer processor and the memory, the AI ​​platform comprising:

1. A staging manager configured to learn causal relationships between two or more application microservices, the learning comprising: collecting first microservice error log data corresponding to the one or more selectively injected errors; generating a learned causal graph based on the collected first microservice error log data, the learned causal graph representing dependencies of application microservices affected by selective error injection; and a staging manager, including a production manager operably coupled to the staging manager, the production manager configured to dynamically locate a source of an application error, the locating comprising: collecting second microservice error log data corresponding to the application error; constructing an ancestor matrix based on the learned causal graph and the collected second microservice error log data; utilizing the ancestor matrix to identify the source of the error; Production Manager, including a director operably coupled to the production manager, the director configured to identify a microservice associated with the identified error source; AI platform with A computer system comprising:

2. 10. The computer system of claim 1, wherein the learning of causal relationships between two or more application microservices further comprises the staging manager filtering the collected first microservice error log data to selectively remove a subset of first error log data.

3. The computer system of claim 1 , wherein the learning of causal relationships between application microservices and the generating of a causal graph are performed offline.

4. 10. The computer system of claim 1, wherein the fault location is performed online in real time.

5. The computer system of claim 1 , further comprising the staging manager configured to apply transitive reduction to the learned causal graph.

6. 6. The computer system of claim 1, wherein utilizing the ancestry matrix includes the production manager identifying multiple potential sources of the error, the computer system further comprising the production manager configured to apply a distance metric to estimate the error sources, the distance metric comprising Hamming distance or cosine similarity.

7. A computer-implemented method comprising: Learning causal relationships between two or more application microservices, collecting first microservice error log data corresponding to the one or more selectively injected errors; generating a learned causal graph based on the collected first microservice error log data, the learned causal graph representing dependencies of microservices affected by selective error injection; a learning stage having dynamically locating a source of an application error, collecting second microservice error log data corresponding to the application error; constructing an ancestor matrix based on the learned causal graph and the collected second microservice error log data; utilizing the ancestor matrix to identify the source of the error; a locating step comprising: identifying a microservice associated with the identified error source; A method comprising:

8. 8. The method of claim 7, wherein learning causal relationships between two or more application microservices further comprises filtering the collected first microservice log data to selectively remove a subset of first error log data.

9. 10. The method of claim 7, wherein learning causal relationships between two or more application microservices and generating a causal graph are performed offline.

10. The method of claim 7, wherein the fault location is performed online in real time.

11. The method of claim 7 , further comprising applying transitive reduction to the learned causal graph.

12. 12. The method of claim 7, wherein utilizing the ancestry matrix comprises identifying multiple potential sources of the error, the method further comprising applying a distance metric to estimate the error sources, the distance metric comprising Hamming distance or cosine similarity.

13. The processor A method for learning causal relationships between two or more application microservices, comprising: collecting first microservice error log data corresponding to the one or more selectively injected errors; generating a learned causal graph based on the collected first microservice error log data, the learned causal graph representing dependencies of microservices affected by selective error injection; a learning procedure having 1. A procedure for dynamically locating the source of an application error, comprising: collecting second microservice error log data corresponding to the application error; constructing an ancestor matrix based on the learned causal graph and the collected second microservice error log data; utilizing the ancestor matrix to identify the source of the error; a location procedure comprising: identifying a microservice associated with the identified error source; A program that executes.

14. 14. The program of claim 13, wherein learning causal relationships between two or more application microservices further comprises filtering the collected first microservice log data to selectively remove a subset of first error log data.

15. The program of claim 13 , wherein the steps of learning causal relationships between application microservices and generating the causal graph are performed offline.

16. 14. The program of claim 13, wherein the fault location is performed online in real time.

17. The program product of claim 13 , further causing the processor to perform the step of applying transitive reduction to the learned causal graph.

18. 18. The program of claim 13, wherein the utilizing the ancestry matrix comprises identifying multiple potential sources of the error, and wherein the program further causes the processor to perform applying a distance metric to estimate the error sources, the distance metric comprising Hamming distance or cosine similarity.

19. A computer-implemented method comprising: Training an artificial intelligence (AI) model, collecting first error log data corresponding to one or more selectively injected microservice failures; learning a causal graph based on the collected first error log data, the causal graph representing dependencies of affected application microservices; a training step, dynamically locating an application failure, collecting second error log data corresponding to the detection of the application failure; utilizing the second error log data and the learned causal graph to identify a source of the application failure; a locating step, A method comprising:

20. 20. The method of claim 19, wherein training the AI ​​model occurs offline and locating the application failure occurs in real time.

21. 20. The method of claim 19, wherein dynamically locating the application failure further comprises applying distance-based thresholding to estimate the source of one or more possible application failures.

22. 22. The method of claim 19, wherein the training the AI ​​model further comprises controlling fault injection and inferring ancestor edges for a microservice in response to receiving the fault injection.

23. 23. The method of claim 22, wherein training the AI ​​model further comprises applying transitive reduction to the learned causal graph, the transitive reduction combining ancestor edges inferred from two or more controlled fault injections.

24. a computer processor operably coupled to the memory; an artificial intelligence (AI) platform in communication with the computer processor and the memory, the AI ​​platform comprising:

1. A staging manager configured to train an AI model, the training comprising: collecting first error log data corresponding to one or more selectively injected microservice failures; learning a causal graph based on the collected first error log data, the causal graph representing dependencies of affected application microservices; a staging manager, including a production manager operably coupled to the staging manager, the production manager configured to dynamically locate application failures, the locating comprising: collecting second error log data corresponding to the detection of the application failure; utilizing the second error log data and the learned causal graph to identify a source of the application failure; Production Manager and AI platform with 1. A computer-implemented system comprising:

25. 25. The computer-implemented system of claim 24, further comprising the production manager configured to apply distance-based thresholding to estimate the source of one or more possible application failures.

Images