Log clustering for guided root cause analysis

Abstract

An exemplary embodiment may include acquiring a plurality of incident logs — each incident log contains events of a respective sequence —; classifying each of the events of each sequence into a respective event class; determining cluster spaces associated with each event class; determining relationships between at least some clusters in the cluster spaces from the plurality of incident logs; and possibly proposing one or more investigation steps to determine the root cause of a symptom found in a subsequent incident log based on the clusters and relationships — the symptom describes a problem experienced by the user —.

Description

Technology Field

[0001] Root cause analysis and defect resolution in computing systems are critical tasks that are often performed insufficiently. As a result, due to defects in networks of computing devices, individual computing devices, and / or software applications running on them, these resources may remain non-functional or in a limited functional state for a period before the root cause of the defect is determined. These delays adversely affect the performance of devices, networks, and applications, slowing down, reducing the reliability of, or rendering the services they provide completely unavailable. Background Technology

[0002] These and other technical problems can be overcome through the use of the embodiments of the present invention. In particular, these embodiments may include preprocessing a corpus of logs to identify problems, investigation steps, root cause determinations, and ultimate problem-solving methods. Each log may include text indicating user-provided, agent-provided, and / or automatically generated descriptions of the problem, investigation steps taken to determine the nature of the problem, the root cause of the problem, and how the problem was resolved.

[0003] When thousands, hundreds of thousands, millions, or more of these logs are stored in a database, they can be processed and used to train machine learning models. These models may include classifiers that categorize events in the logs into one or more predefined classes, as well as clustering models that group similar symptoms, investigation steps, root cause determinations, and solutions found in the classified events into respective cluster spaces. Probabilistic relationships between these clusters can be identified. For example, given symptoms, different investigation steps may have different probabilities of leading to the successful identification of the root cause.

[0004] This enables automated, semi-automated, or agent-based investigations of similar problems in the future to proceed according to these possibilities. In other words, once a new problem is identified, a series of investigation steps can be determined. By performing these steps in a given order, the time required to resolve the problem can be minimized or at least reduced. As a result, computing devices, systems, networks, and applications experience less downtime, and their overall reliability is improved.

[0005] Accordingly, the first exemplary embodiment may include acquiring a plurality of incident logs — each incident log contains events of a respective sequence —; classifying each of the events of each sequence into a respective event class; determining cluster spaces associated with each event class; determining relationships between at least some clusters in the cluster spaces from the plurality of incident logs; and proposing one or more investigation steps to determine the root cause of a symptom found in a subsequent incident log based on the clusters and relationships — the symptom describes a problem experienced by the user —.

[0006] A second exemplary embodiment may include acquiring an incident log containing an event indicating a symptom — the symptom describes a problem experienced by a user —; performing a comparison between the event and a plurality of symptom clusters within a symptom cluster space — the plurality of symptom clusters represent symptoms associated with events in a plurality of previously acquired incident logs —; identifying a symptom cluster from the symptom cluster space based on the comparison; selecting an investigation step cluster from an investigation step cluster space based on the symptom cluster — the investigation step cluster is associated with one or more root cause clusters from a root cause cluster space, the investigation step cluster space is derived from events in a plurality of previously acquired incident logs, and the root cause cluster space is also associated with events in a plurality of previously acquired incident logs — and determining that an investigation step from the investigation step cluster led to the identification of the root cause of the symptom — the root cause is from one of the root cause clusters —.

[0007] A third exemplary embodiment may include a non-transient computer-readable medium storing program instructions that, when executed by a computing system, cause the computing system to perform operations according to the first and / or second exemplary embodiments.

[0008] In a fourth exemplary embodiment, the computing system may include at least one processor, as well as memory and program instructions. The program instructions may be stored in memory and, when executed by at least one processor, may cause the computing system to perform operations according to the first and / or second exemplary embodiments.

[0009] In the fifth exemplary embodiment, the system may include various means for performing each of the operations of the first and / or second exemplary embodiments.

[0010] These, as well as other embodiments, aspects, advantages, and alternatives, will become apparent to those skilled in the art by reading the following detailed description with due reference to the accompanying drawings. Furthermore, the descriptions and drawings provided herein are intended merely to illustrate embodiments by example, and accordingly, various modifications are possible. For example, structural elements and process steps may be rearranged, combined, dispersed, removed, or otherwise altered within the scope of the claimed embodiments. Brief explanation of the drawing

[0011] FIG. 1 illustrates a schematic diagram of a computing device according to exemplary embodiments. FIG. 2 illustrates a schematic diagram of a server device cluster according to exemplary embodiments. FIG. 3 shows a remote network management architecture according to exemplary embodiments. FIG. 4 illustrates a communication environment including a remote network management architecture according to exemplary embodiments. FIG. 5 illustrates another communication environment including a remote network management architecture according to exemplary embodiments. FIG. 6a shows a sequence of events from an incident log according to exemplary embodiments. FIG. 6b shows a sequence of events from other incident logs according to exemplary embodiments. FIG. 7a is a flowchart illustrating training steps according to exemplary embodiments. FIG. 7b is a flowchart illustrating the prediction steps according to exemplary embodiments. FIG. 8a shows a cluster space according to exemplary embodiments. FIG. 8b illustrates probabilistic relationships between clusters according to exemplary embodiments. FIG. 9a is a flowchart according to exemplary embodiments. FIG. 9b is another flowchart according to exemplary embodiments. Specific details for implementing the invention

[0012] Exemplary methods, devices, and systems are described herein. It should be understood that the words “example” and “exemplary” are used herein to mean “provided as an example, an example, or an example.” Any embodiment or feature described herein as “example” or “exemplary” should not be interpreted as necessarily being more desirable or advantageous than other embodiments or features unless otherwise stated. Accordingly, other embodiments may be utilized and other modifications made without departing from the scope of the essence presented herein.

[0013] Accordingly, the exemplary embodiments described herein are not intended to be limiting. It will be readily understood that the aspects of the disclosure generally described herein and illustrated in the drawings can be arranged, substituted, combined, separated, and designed into a wide variety of different configurations. For example, there may be various ways of separating features into "client" and "server" components.

[0014] Additionally, unless otherwise indicated in the context, the features shown in each of the drawings may be used in combination with one another. Accordingly, the drawings should generally be regarded as component aspects of one or more overall embodiments, and it should be understood that not all depicted features are essential to each embodiment.

[0015] Additionally, any enumeration of elements, blocks, or steps in this specification or claims is for clarity. Accordingly, such enumeration should not be interpreted as requiring or implying that these elements, blocks, or steps follow a specific arrangement or are performed in a specific order.

[0016] I. introduction

[0017] Large enterprises are complex entities with many interrelated operations. Some of these are found throughout the company, such as HR (human resources), supply chain, IT (information technology), and finance. However, each firm also possesses its own unique operations that provide essential capabilities and / or create a competitive advantage.

[0018] To support widely implemented operations, companies typically use off-the-shelf software applications such as customer relationship management (CRM) and human capital management (HCM) packages. However, they may also require custom software applications to meet their own unique requirements. Large enterprises often possess dozens or hundreds of these custom software applications. Nevertheless, the benefits provided by the embodiments of the present invention are not limited to large enterprises and may be applicable to enterprises of any size or any other type of organization.

[0019] Many of these software applications are developed by individual departments within an enterprise. They range from simple spreadsheets to custom-built software tools and databases. However, the proliferation of siloed custom software applications presents numerous disadvantages. This negatively impacts a company's ability to operate, grow operations, innovate, and meet regulatory requirements. Due to the lack of a single system to integrate subsystems and data, companies may struggle to integrate, streamline, and enhance operations.

[0020] To efficiently create custom applications, companies will benefit from remotely hosted application platforms that eliminate unnecessary development complexity. The goal of such platforms is to reduce time-consuming and repetitive application development tasks, allowing software engineers and individuals in other roles to focus on developing unique and high-value features.

[0021] To achieve this goal, the concept of Application Platform as a Service (aPaaS) is introduced to intelligently automate workflows across the enterprise. While aPaaS systems are hosted remotely from the enterprise, they can access data, applications, and services within the enterprise through secure connections. Such aPaaS systems may possess numerous advantageous capabilities and features. These benefits and features can improve enterprise operations and workflows regarding IT, HR, CRM, customer service, application development, and security. However, the embodiments of the present invention are not limited to enterprise applications or environments and can be applied more broadly.

[0022] aPaaS systems can support the development and execution of MVC (model-view-controller) applications. MVC applications divide functionality into three interconnected parts (model, view, and controller) to separate the representation of information from how it is displayed to the user, thereby enabling efficient code reuse and parallel development. These applications can be web-based and provide CRUD (create, read, update, and delete) capabilities. This allows new applications to be built on a common application infrastructure. In some cases, structured applications may be employed, unlike MVC, which uses unidirectional data flow.

[0023] aPaaS systems can support standardized application components, such as a set of standardized widgets for graphical user interface (GUI) development. In this way, applications built using aPaaS systems have a common look and feel. Other software components and modules can also be standardized. In some cases, this look and feel can be branded or skinned with the company's custom logos and / or color schemes.

[0024] aPaaS systems can support the ability to configure the behavior of applications using metadata. This allows application behaviors to adapt rapidly to satisfy specific requirements. This approach reduces development time and increases flexibility. Furthermore, aPaaS systems support GUI tools that facilitate the creation and management of metadata, thereby reducing errors in the metadata.

[0025] aPaaS systems can support clearly defined interfaces between applications, allowing software developers to avoid unwanted dependencies between applications. Therefore, aPaaS systems can implement a service layer where persistent state information and other data are stored.

[0026] aPaaS systems can support a rich set of integration features to enable applications on them to interact with legacy applications and third-party applications. For example, aPaaS systems can support custom employee onboarding systems that integrate with legacy HR, IT, and accounting systems.

[0027] aPaaS systems can support enterprise-grade security. Additionally, since aPaaS systems can be hosted remotely, security procedures must be utilized when interacting with systems within the enterprise or with third-party networks and services hosted outside the enterprise. For example, aPaaS systems can be configured to share data between the enterprise and other third parties to detect and identify common security threats.

[0028] Other features, functions, and benefits of aPaaS systems may exist. This description is for illustrative purposes only and is not intended to be limiting.

[0029] As an example of the aPaaS development process, a software developer may be tasked with creating a new application using an aPaaS system. First, the developer can define a data model that specifies the data types used by the application and the relationships between them. Next, through the aPaaS system's GUI, the developer inputs (e.g., uploads) the data model. The aPaaS system automatically generates all the corresponding database tables, fields, and relationships (which can be accessed through a layer of object-oriented services).

[0030] Additionally, aPaaS systems can also build fully functional applications with client-side interfaces and server-side CRUD logic. These applications can serve as a foundation for further development for users. Advantageously, developers do not need to spend a significant amount of time on the core application functions. Furthermore, since the applications can be web-based, any internet-enabled client device can access them. Alternatively, or additionally, a local copy of the application can be accessed, for example, in the event that internet services are unavailable.

[0031] aPaaS systems may also support a rich set of predefined features that can be added to applications. These features include support for search, email, template creation, workflow design, reporting, analytics, social media, scripting, mobile-friendly output, and customized GUIs.

[0032] These aPaaS systems can represent GUIs in various ways. For example, a server device of an aPaaS system can generate a GUI representation using a combination of HTML (HyperText Markup Language) and JAVASCRIPT®. JAVASCRIPT® can include client-side executable code, server-side executable code, or both. The server device can transmit or otherwise provide this representation to the client device so that the client device displays it on the screen according to a locally defined look and feel. Alternatively, the GUI representation can take other forms, such as an intermediate form (e.g., JAVA® bytecode) that the client device can use to generate graphic output directly from it. Other possibilities also exist.

[0033] Additionally, user interaction with GUI elements such as buttons, menus, tabs, sliders, checkboxes, toggles, etc., may be referred to as their "selection," "activation," or "operation." These terms may be used regardless of whether the GUI elements interact via a keyboard, pointing device, touchscreen, or other mechanism.

[0034] aPaaS architecture is particularly powerful when integrated with enterprise networks and used to manage such networks. The following embodiments describe the architectural and functional aspects of exemplary aPaaS systems, as well as their features and benefits.

[0035] II. Exemplary computing devices and cloud-based computing environments

[0036] FIG. 1 is a simplified block diagram illustrating a computing device (100) illustrating some of the components that may be included in a computing device arranged to operate according to embodiments of the present invention. The computing device (100) may be a client device (e.g., a device actively operated by a user), a server device (e.g., a device that provides computing services to client devices), or some other type of computing platform. Some server devices may occasionally operate as client devices to perform specific operations, and some client devices may incorporate server features.

[0037] In this example, the computing device (100) includes a processor (102), memory (104), a network interface (106), and an input / output unit (108), all of which may be connected by a system bus (110) or a similar mechanism. In some embodiments, the computing device (100) may include other components and / or peripheral devices (e.g., detachable storage, printers, etc.).

[0038] The processor (102) may be one or more types of computer processing elements, such as a central processing unit (CPU), an auxiliary processor (e.g., a mathematical, graphical, or cryptographic auxiliary processor), a digital signal processor (DSP), a network processor, and / or an integrated circuit or controller that performs processor operations. In some cases, the processor (102) may be one or more single-core processors. In other cases, the processor (102) may be one or more multi-core processors having a number of independent processing units. The processor (102) may also include a register memory for temporarily storing instructions to be executed and related data, as well as a cache memory for temporarily storing recently used instructions and data.

[0039] Memory (104) is not limited but may be any form of computer-usable memory including random access memory (RAM), read-only memory (ROM), and non-volatile memory (e.g., flash memory, hard disk drives, solid-state drives, compact discs (CDs), digital video discs (DVDs), and / or tape storage). Thus, memory (104) represents both main memory units and long-term storage. Other types of memory may include biological memory.

[0040] Memory (104) may store program instructions and / or data on which the program instructions can operate. For example, memory (104) may store these program instructions on a non-transient computer-readable medium so that the instructions can be executed by the processor (102) to perform any methods, processes, or operations disclosed in this specification or the accompanying drawings.

[0041] As shown in FIG. 1, memory (104) may include firmware (104A), a kernel (104B), and / or applications (104C). Firmware (104A) may be program code used to boot or otherwise start part or all of the computing device (100). The kernel (104B) may be an operating system including modules for memory management, scheduling and management of processes, input / output, and communication. The kernel (104B) may also include device drivers that enable the operating system to communicate with hardware modules of the computing device (100) (e.g., memory units, networking interfaces, ports, and buses). Applications (104C) may be one or more user-space software programs, such as web browsers or email clients, as well as any software libraries used by these programs. Memory (104) may also store data used by these programs and other programs and applications.

[0042] The network interface (106) may take the form of one or more wired interfaces, such as Ethernet (e.g., Fast Ethernet, Gigabit Ethernet, etc.). The network interface (106) may also support communication over one or more non-Ethernet media, such as coaxial cables or power lines, or communication over wide-area media, such as SONET (Synchronous Optical Networking) or DSL (Digital Subscriber Line) technology. The network interface (106) may additionally take the form of one or more wireless interfaces, such as IEEE 802.11 (Wifi), BLUETOOTH®, GPS (Global Positioning System), or wide-area wireless interfaces. However, other forms of physical layer interfaces and other types of standard or proprietary communication protocols may be used through the network interface (106). Additionally, the network interface (106) may include multiple physical interfaces. For example, some embodiments of the computing device (100) may include Ethernet, BLUETOOTH®, and Wi-Fi interfaces.

[0043] The input / output unit (108) can facilitate interaction between the user and peripheral devices and the computing device (100). The input / output unit (108) may include one or more types of input devices, such as a keyboard, mouse, touchscreen, etc. Likewise, the input / output unit (108) may include one or more types of output devices, such as a screen, monitor, printer, and / or one or more light emitting diodes (LEDs). Additionally or alternatively, the computing device (100) may communicate with other devices, for example, using a universal serial bus (USB) or high-definition multimedia interface (HDMI) port interface.

[0044] In some embodiments, one or more computing devices, such as a computing device (100), may be deployed to support the aPaaS architecture. The exact physical location, connection, and configuration of these computing devices may not be known to or important to the client devices. Accordingly, the computing devices may be referred to as "cloud-based" devices that can be accommodated in various remote data center locations.

[0045] FIG. 2 illustrates a cloud-based server cluster (200) according to exemplary embodiments. In FIG. 2, operations of computing devices (e.g., computing devices (100)) may be distributed among server devices (202), data storages (204), and routers (206), all of which may be connected by a local cluster network (208). The number of server devices (202), data storages (204), and routers (206) in the server cluster (200) may vary depending on the computing tasks and / or applications assigned to the server cluster (200).

[0046] For example, server devices (202) may be configured to perform various computing tasks of the computing device (100). Thus, computing tasks may be distributed among one or more server devices (202). As long as these computing tasks can be performed in parallel, the distribution of these tasks can reduce the total time required to complete these tasks and return results. For simplification, both the server cluster (200) and individual server devices (202) may be referred to as "server devices." This nomenclature should be understood to mean that one or more individual server devices, data storage devices, and cluster routers may be involved in server device operations.

[0047] Data storage (204) may be data storage arrays comprising drive array controllers configured to manage read and write access to groups of hard disk drives and / or solid-state drives. The drive array controllers may be configured to manage backup or redundant copies of data stored in data storage (204), either alone or in conjunction with server devices (202), to protect against drive failures or other types of failures that prevent one or more server devices (202) from accessing units of data storage (204). Other types of memory may be used in addition to drives.

[0048] Routers (206) may include networking equipment configured to provide internal and external communications to the server cluster (200). For example, the routers (206) may include one or more packet switching and / or routing devices (including switches and / or gateways) configured to provide (i) network communications between server devices (202) and data storage (204) through a local cluster network (208), and / or (ii) network communications between the server cluster (200) and other devices through a communication link (210) to the network (212).

[0049] Additionally, the configuration of the routers (206) may be based at least partially on the data communication requirements of the server devices (202) and data storages (204), the latency and throughput of the local cluster network (208), the latency, throughput, and cost of the communication link (210), and / or other factors that may contribute to the cost, speed, fault tolerance, resilience, efficiency, and / or other design goals of the system architecture.

[0050] As a possible example, the data storage (204) may include any type of database, such as a SQL (structured query language) database. Various types of data structures may store information in this database, including, but not limited to, tables, arrays, lists, trees, and tuples. Additionally, any databases in the data storage (204) may be monolithic or distributed across multiple physical devices.

[0051] Server devices (202) may be configured to transmit data to data storage (204) and receive data from data storage (204). This transmission and retrieval may each take the form of SQL queries or other types of database queries and the output of such queries, respectively. Additional text, images, videos, and / or audio may also be included. Additionally, server devices (202) may organize the received data into web pages or web application representations. These representations may take the form of markup languages ​​such as HTML, XML (eXtensible Markup Language), or some other standardized or proprietary format. Furthermore, server devices (202) may have the ability to execute various types of computer scripting languages, such as Perl, Python, PHP (PHP Hypertext Preprocessor), ASP (Active Server Pages), JAVASCRIPT®, etc., but are not limited to these languages. Computer program code written in these languages ​​can not only provide web pages to client devices but also facilitate interaction between client devices and web pages. Alternatively or additionally, JAVA® can be used to facilitate the creation of web pages and / or provide web application functions.

[0052] III. Exemplary remote network management architecture

[0053] FIG. 3 illustrates a remote network management architecture according to exemplary embodiments. This architecture includes three main components (a management network (300), a remote network management platform (320), and public cloud networks (340)), all of which are connected via the Internet (350).

[0054] A. Management networks

[0055] The management network (300) may be an enterprise network used by the entity for computing and communication tasks as well as for data storage, for example. Accordingly, the management network (300) may include client devices (302), server devices (304), routers (306), virtual machines (308), a firewall (310), and / or proxy servers (312). The client devices (302) may be implemented by a computing device (100), the server devices (304) may be implemented by a computing device (100) or a server cluster (200), and the routers (306) may be any type of router, switch, or gateway.

[0056] Virtual machines (308) may be implemented by one or more of a computing device (100) or a server cluster (200). Generally, a virtual machine is an emulation of a computing system and mimics the functions of a physical computer (e.g., processors, memory, and communication resources). A single physical computing system, such as a server cluster (200), can support up to several thousand individual virtual machines. In some embodiments, virtual machines (308) may be managed by a centralized server device or application that facilitates the allocation of physical computing resources to individual virtual machines, as well as performance and error reporting. Enterprises often employ virtual machines to efficiently allocate computing resources as needed. Providers of virtualized computing systems include VMware® and Microsoft®.

[0057] The firewall (310) may be one or more specialized routers or server devices that protect the management network (300) from unauthorized attempts to access internal devices, applications, and services while allowing authorized communication originating from the management network (300). The firewall (310) may also provide intrusion detection, web filtering, virus scanning, application layer gateways, and other applications or services. In some embodiments not illustrated in FIG. 3, the management network (300) may include one or more virtual private network (VPN) gateways for communicating with a remote network management platform (320) (see below).

[0058] The management network (300) may also include one or more proxy servers (312). An embodiment of the proxy servers (312) may be a server application that facilitates communication and the movement of data between the management network (300), the remote network management platform (320), and public cloud networks (340). In particular, the proxy servers (312) may establish and maintain secure communication sessions with one or more computing instances of the remote network management platform (320). Through these sessions, the remote network management platform (320) may discover and manage aspects of the architecture and configuration of the management network (300) and its components.

[0059] Possibly, with the help of proxy servers (312), the remote network management platform (320) may also discover and manage aspects of public cloud networks (340) used by the management network (300). Although not shown in FIG. 3, one or more proxy servers (312) may be located in any of the public cloud networks (340) to facilitate this discovery and management.

[0060] Firewalls such as firewall (310) typically reject all communication sessions coming through the Internet (350) unless the session is ultimately initiated behind the firewall (i.e., from a device on the management network (300)) or the firewall is not explicitly configured to support the session. By placing proxy servers (312) behind the firewall (310) (e.g., within the management network (300) and protected by the firewall (310)), the proxy servers (312) can initiate these communication sessions through the firewall (310). Thus, the firewall (310) may not need to be specifically configured to support sessions coming from the remote network management platform (320), thereby avoiding potential security risks to the management network (300).

[0061] In some cases, the management network (300) may consist of a small number of devices and a small number of networks. In other deployments, the management network (300) may span multiple physical locations and may include hundreds of networks and hundreds of thousands of devices. Thus, the architecture shown in FIG. 3 can be scaled up or down by tens of times.

[0062] Additionally, depending on the size, architecture, and connectivity of the management network (300), various numbers of proxy servers (312) may be deployed internally. For example, each of the proxy servers (312) may be responsible for communicating with a remote network management platform (320) in relation to a part of the management network (300). Alternatively, or additionally, sets of two or more proxy servers may be assigned to this part of the management network (300) for load balancing, redundancy, and / or high availability purposes.

[0063] B. Remote network management platforms

[0064] The remote network management platform (320) is a hosting environment that provides aPaaS services to users, particularly to operators of the management network (300). These services may take the form of web-based portals using, for example, the aforementioned web-based technologies. Thus, users can securely access the remote network management platform (320) from, for example, client devices (302) or potentially from client devices outside the management network (300). Through the web-based portals, users can design, test, and deploy applications, generate reports, view analyses, and perform other tasks. The remote network management platform (320) may also be referred to as a multi-application platform.

[0065] As shown in FIG. 3, the remote network management platform (320) includes four compute instances (322, 324, 326, and 328). Each of these compute instances may represent one or more server nodes and / or one or more database nodes running dedicated copies of the aPaaS software. The arrangement of server and database nodes on physical server devices and / or virtual machines may be flexible and may vary based on enterprise requirements. These nodes may be combined to provide a set of web portals, services, and applications (e.g., a fully functional aPaaS system) available to a specific enterprise. In some cases, a single enterprise may use multiple compute instances.

[0066] For example, the management network (300) may be an enterprise customer of the remote network management platform (320) and may use compute instances (322, 324, and 326). The reason for providing multiple compute instances to a single customer is that the customer may want to independently develop, test, and deploy their own applications and services. Thus, compute instance (322) may be dedicated to application development related to the management network (300), compute instance (324) may be dedicated to testing these applications, and compute instance (326) may be dedicated to the real-time operation of the applications and services being tested. Computation instances may be referred to as hosted instances, remote instances, customer instances, or some other designation. Any application deployed to a compute instance may be a scoped application, that is, access to databases within the compute instance may be restricted to specific elements within it (e.g., one or more specific database tables or specific rows within one or more database tables).

[0067] For clarity, the disclosure herein refers to an array of application nodes, database nodes, aPaaS software running thereon, and underlying hardware as a “computation instance.” Users may also refer to the graphical user interfaces provided by this as “instances” in colloquial speech. However, unless otherwise defined herein, a “computation instance” is a computing system deployed within a remote network management platform (320).

[0068] The multi-instance architecture of the remote network management platform (320) contrasts with existing multi-tenant architectures, and multi-instance architectures offer several advantages over them. In multi-tenant architectures, data from various customers (e.g., enterprises) is mixed in a single database. The data of these customers is separated from one another, and this separation is enforced by software operating the single database. Consequently, a security breach in this system can affect the data of all customers, potentially leading to additional risks, particularly for government, healthcare, and / or financial regulated entities. Furthermore, any database operation affecting one customer is likely to affect all customers sharing that database. Therefore, if there is an outage due to hardware or software errors, this outage affects all of these customers. Similarly, if a database is upgraded to meet the requirements of a single customer, the database will not be available to all customers during the upgrade process. Often, due to the size of the shared database, these management windows will be long.

[0069] In contrast, a multi-instance architecture provides each customer with its own database on a dedicated compute instance. This prevents the mixing of customer data and allows each instance to be managed independently. For example, if one customer's instance experiences an outage due to errors or an upgrade, the other compute instances remain unaffected. Since the database contains data for only one customer, management downtime is limited. Furthermore, the simple design of a multi-instance architecture allows redundant copies of each customer's database and instance to be distributed geographically in various ways. This facilitates high availability, as real-time versions of the customer's instance can be moved when faults are detected or management is performed.

[0070] In some embodiments, the remote network management platform (320) may include one or more central instances controlled by an entity operating the platform. Like compute instances, the central instance may include several application and database nodes deployed on several physical server devices or virtual machines. Such a central instance may function as a repository for data that can be shared among at least some of the compute instances, as well as specific configurations of the compute instances. For example, definitions of application stores regarding common security threats that may occur on the compute instances, software packages commonly found on the compute instances, and / or applications that may be deployed on the compute instances may be in the central instance. The compute instances may communicate with the central instances through well-defined interfaces to obtain this data.

[0071] To support multiple compute instances in an efficient manner, the remote network management platform (320) may implement these multiple instances on a single hardware platform. For example, if the aPaaS system is implemented on a server cluster such as a server cluster (200), it may operate virtual machines that dedicate varying amounts of compute, storage, and communication resources to instances. However, full virtualization of the server cluster (200) may not be necessary, and other mechanisms may be used to separate the instances. In some examples, each instance may have a dedicated account and one or more dedicated databases on the server cluster (200). Alternatively, a compute instance such as a compute instance (322) may be spread across multiple physical devices.

[0072] In some cases, a single server cluster of the remote network management platform (320) may support multiple independent enterprises. Additionally, as described below, the remote network management platform (320) may include multiple server clusters deployed in geographically diverse data centers to facilitate load balancing, redundancy, and / or high availability.

[0073] C. Public cloud networks

[0074] Public cloud networks (340) may be remote server devices (e.g., multiple server clusters such as a server cluster (200)) that can be used for outsourced computing, data storage, communications, and service hosting operations. These servers may be virtualized (i.e., the servers may be virtual machines). Examples of public cloud networks (340) may include Amazon AWS Cloud, Microsoft Azure Cloud (Azure), Google Cloud Platform (GCP), and IBM Cloud Platform. Similar to the remote network management platform (320), multiple server clusters supporting the public cloud networks (340) may be deployed in geographically diverse locations for load balancing, redundancy, and / or high availability purposes.

[0075] The management network (300) can deploy applications and services to clients and customers using one or more public cloud networks (340). For example, if the management network (300) provides online music streaming services, the public cloud networks (340) can store music files and provide web interfaces and streaming capabilities. In this way, the enterprise of the management network (300) does not need to build and maintain its own servers for these operations.

[0076] The remote network management platform (320) may include modules that integrate with public cloud networks (340) to expose virtual machines and management services within them to the management network (300). The modules may enable users to request virtual resources, discover allocated resources, and provide flexible reporting on the public cloud networks (340). To establish this functionality, a user from the management network (300) may first set up an account on the public cloud networks (340) and request a set of associated resources. Then, the user may enter account information into the appropriate modules of the remote network management platform (320). Then, these modules may automatically discover manageable resources in the account and provide reports related to usage, performance, and billing.

[0077] D. Communication support and other actions

[0078] The Internet (350) may represent a part of the global Internet. However, the Internet (350) may alternatively represent other types of networks, such as private wide-area or local area packet switching networks.

[0079] FIG. 4 further illustrates the communication environment between the management network (300) and the computing instance (322) and introduces additional features and alternative embodiments. In FIG. 4, the computing instance (322) is replicated wholly or partially across data centers (400A and 400B). These data centers may be geographically far apart from each other and may be located in different cities or different countries. Each data center includes support equipment that facilitates communication with the management network (300) as well as with remote users.

[0080] In the data center (400A), network traffic to and from external devices flows through a VPN gateway (402A) or a firewall (404A). The VPN gateway (402A) can be peered with the VPN gateway (412) of the management network (300) via a security protocol such as IPSEC (Internet Protocol Security) or TLS (Transport Layer Security). The firewall (404A) can be configured to allow access from authorized users, such as users (414) and remote users (416), and to deny access to unauthorized users. Through the firewall (404A), these users can access the compute instance (322) and possibly other compute instances. A load balancer (406A) can be used to distribute traffic among one or more physical or virtual server devices hosting the compute instance (322). A load balancer (406A) can simplify user access by hiding the internal configuration of the data center (400A) (e.g., compute instance (322)) from client devices. For example, if the compute instance (322) includes multiple physical or virtual computing devices that share access to multiple databases, the load balancer (406A) can distribute network traffic and processing tasks across these computing devices and databases so that no single computing device or database becomes significantly busier than the others. In some embodiments, the compute instance (322) may include a VPN gateway (402A), a firewall (404A), and a load balancer (406A).

[0081] The data center (400B) may contain its own versions of the components in the data center (400A). Thus, the VPN gateway (402B), firewall (404B), and load balancer (406B) may each perform the same or similar operations as the VPN gateway (402A), firewall (404A), and load balancer (406A). Additionally, through real-time or near-real-time database replication and / or other operations, the compute instance (322) may exist simultaneously in the data centers (400A and 400B).

[0082] The data centers (400A and 400B) shown in FIG. 4 can facilitate redundancy and high availability. In the configuration of FIG. 4, data center (400A) is active and data center (400B) is passive. Thus, data center (400A) serves all traffic to and from the management network (300), while the version of the compute instance (322) in data center (400B) is updated in near real-time. Other configurations, such as both data centers being active, may also be supported.

[0083] If data center (400A) fails in some way or is not available to users in another way, data center (400B) may take over as the active data center. For example, DNS (domain name system) servers that associate the domain name of compute instance (322) with one or more Internet Protocol (IP) addresses of data center (400A) may re-associate the domain name with one or more IP addresses of data center (400B). After this re-association is completed (which may take one second or less than a few seconds), users may access compute instance (322) through data center (400B).

[0084] FIG. 4 also illustrates a possible configuration of a management network (300). As described above, proxy servers (312) and a user (414) can access the compute instance (322) through a firewall (310). Proxy servers (312) can also access configuration items (410). In FIG. 4, configuration items (410) may mean some or all of client devices (302), server devices (304), routers (306), and virtual machines (308), any of their components, any applications or services running on them, as well as relationships between devices, components, applications, and services. Thus, the term “configurations” may be an abbreviation for any physical or virtual device, any application or service that is remotely discoverable or managed by the compute instance (322), or any part of the relationships between the discovered devices, applications, and services. Configuration items can be represented in the configuration management database (CMDB) of the compute instance (322).

[0085] A configuration item that is stored or transmitted may be a list of attributes that characterize the hardware or software represented by the configuration item. These attributes may include the manufacturer, vendor, location, owner, unique identifier, description, network address, operating status, serial number, time of last update, etc. The class of the configuration item can determine which subset of attributes exists for the configuration item (for example, software and hardware configuration items may have different lists of attributes).

[0086] As described above, the VPN gateway (412) may provide a dedicated VPN to the VPN gateway (402A). Such a VPN may be helpful when there is a significant amount of traffic between the management network (300) and the compute instance (322), or when security policies suggest or require the use of a VPN between these sites. In some embodiments, any device in the management network (300) and / or the compute instance (322) communicating directly through the VPN is assigned a public IP address. Other devices and / or compute instances (322) in the management network (300) may be assigned private IP addresses (e.g., IP addresses selected from the ranges 10.0.0.0 - 10.255.255.255 or 192.168.0.0 - 192.168.255.255, abbreviated as subnets 10.0 0.0 / 8 and 192.168.0.0 / 16, respectively). In various alternatives, devices in the management network (300), such as proxy servers (312), may communicate directly with one or more data centers using a security protocol (e.g., TLS).

[0087] IV. Exemplary discovery

[0088] In order for the remote network management platform (320) to manage the devices, applications, and services of the management network (300), the remote network management platform (320) can first determine which devices exist in the management network (300), the configurations, configuration components, and operating states of these devices, and the applications and services provided by the devices. The remote network management platform (320) can also determine the relationships between the discovered devices, their components, applications, and services. Representations of each device, component, application, and service may be referred to as configuration items. The process of determining configuration items and relationships within the management network (300) is referred to as discovery and can be facilitated at least partially by proxy servers (312). Representations of configuration items and relationships are stored in the CMDB.

[0089] This section describes discovery performed on a management network (300), but the same or similar discovery procedures may be used on public cloud networks (340). Thus, in some environments, “discovery” may mean discovering configuration items and relationships on a management network and / or one or more public cloud networks.

[0090] For the purposes of the embodiments herein, “application” may mean one or more processes, threads, programs, client software modules, server software modules, or any other software running on a device or group of devices. “Service” may mean a high-level capability provided by one or more applications running on one or more devices that work in conjunction with each other. For example, a web service may include multiple web application server threads running on one device and accessing information from a database application running on another device.

[0091] FIG. 5 provides a logical representation of a method for discovering configuration items and relationships and a method for storing related information. For simplification, a remote network management platform (320), public cloud networks (340), and the Internet (350) are not illustrated.

[0092] In FIG. 5, the CMDB (500), the task list (502), and the IRE (identification and reconciliation engine) (IRE) (514) are placed and / or operate within the compute instance (322). The task list (502) represents a connection point between the compute instance (322) and the proxy servers (312). The task list (502) may be referred to as a queue, more specifically as an ECC (external communication channel) queue. The task list (502) may represent the queue itself as well as any associated processing, such as adding, removing, and / or manipulating information in the queue.

[0093] As a discovery occurs, the compute instance (322) may store the discovery tasks (jobs) that the proxy servers (312) must perform in the task list (502) until the proxy servers (312) request these tasks in one or more batches. When tasks are placed in the task list (502), the proxy servers (312) may be triggered to start their discovery operations or otherwise start. For example, the proxy servers (312) may poll the task list (502) periodically or occasionally, or be notified of discovery commands in the task list (502) in some other way. Alternatively or additionally, the discovery may be triggered manually or automatically based on triggering events (for example, the discovery may be automatically started at a specific time once a day).

[0094] Regardless of this, the computation instance (322) may send these discovery commands to the proxy servers (312) upon request. For example, the proxy servers (312) may repeatedly query the task list (502), obtain the next task within it, and perform this task until the task list (502) is empty or another stopping condition is reached. In response to receiving discovery commands, the proxy servers (312) may query various devices, components, applications, and / or services in the management network (300) (represented as devices (504, 506, 508, 510, and 512) in FIG. 5 for simplification). These devices, components, applications, and / or services may provide responses to the proxy servers (312) regarding their configuration, operation, and / or status. Subsequently, proxy servers (312) can provide this discovered information to a task list (502) (i.e., the task list (502) may have an outgoing queue to hold discovery commands until requested by the proxy servers (312), as well as an incoming queue to hold discovery information until read).

[0095] IRE (514) may be a software module that removes discovery information from the task list (502) and forms this discovery information into configuration items (e.g., representing devices, components, applications, and / or services found on the management network (300)) as well as relationships between them. Then, IRE (514) may provide these configuration items and relationships to CMDB (500) for internal storage. The operation of IRE (514) is described in more detail below.

[0096] In this way, the configuration items stored in the CMDB (500) represent the environment of the management network (300). For example, these configuration items may represent physical and / or virtual devices (e.g., client devices, server devices, routers, or virtual machines), applications running on them (e.g., web servers, email servers, databases, or storage arrays), as well as sets of services including multiple individual configuration items. Relationships may be arrays between configuration items or pairwise definitions of dependencies.

[0097] To enable discovery to occur in the manner described above, proxy servers (312), CMDB (500), and / or one or more credential stores may be composed of credentials for the devices to be discovered. The credentials may contain any type of information required to access the devices. These may include user ID / password pairs, certificates, etc. In some embodiments, these credentials may be stored in encrypted fields of the CMDB (500). The proxy servers (312) may include decryption keys for the credentials so that the proxy servers (312) can use these credentials to log on to or otherwise access the devices to be discovered.

[0098] There are two general types of findings—horizontal and vertical (top-down). Each is discussed below.

[0099] A. Horizontal discovery

[0100] Horizontal discovery is used to scan the management network (300), find devices, components, and / or applications, and then populate the CMDB (500) with configuration items representing these devices, components, and / or applications. Horizontal discovery also creates relationships between configuration items. For example, this could be a “runs on” relationship between a configuration item representing a software application and a configuration item representing the server device on which it runs. Typically, horizontal discovery does not recognize services and does not create relationships between configuration items based on the services on which they run.

[0101] There are two versions of horizontal discovery. One relies on probes and sensors, while the other also employs patterns. The probes and sensors may be scripts (e.g., written in JAVASCRIPT®) that collect and process discovery information about the device and then update the CMDB (500) accordingly. More specifically, the probes explore or inspect devices on the management network (300), and the sensors parse the discovery information returned from the probes.

[0102] Patterns are also scripts that collect data about one or more devices, process it, and update the CMDB. Patterns differ from probes and sensors in that they are written in a specific discovery programming language and are used to perform detailed discovery procedures for specific devices, components, and / or applications that often cannot be reliably discovered (or cannot be discovered at all) by more general probes and sensors. In particular, patterns can specify a series of actions that define the method for discovering a specific array of devices, components, and / or applications, the credentials to use, and the CMDB tables to populate with the configuration items resulting from this discovery.

[0103] Both versions may proceed through four logical steps: scanning, classification, identification, and discovery. Additionally, both versions may require the designation of one or more ranges of IP addresses in the management network (300) where discovery occurs. Each step may include communication between devices on the management network (300) and proxy servers (312), as well as between proxy servers (312) and the task list (502). Some steps may include storing partial or preliminary configuration items in the CMDB (500), which may be updated in a later step.

[0104] In the scanning phase, proxy servers (312) may examine each IP address in the specified IP address range(s) for open TCP (Transmission Control Protocol) and / or UDP (User Datagram Protocol) ports to determine the general type of device and its operating system. The presence of these open ports in the IP address may indicate that a specific application is running on the device to which the IP address is assigned, which can ultimately identify the operating system used by the device. For example, if TCP port 135 is open, the device is likely running the WINDOWS® operating system. Similarly, if TCP port 22 is open, the device is likely running a UNIX® operating system such as LINUX®. If UDP port 161 is open, the device can be further identified via SNMP (Simple Network Management Protocol). Other possibilities exist.

[0105] In the classification phase, the proxy servers (312) may further examine each discovered device to determine the type of operating system. The probes used for a specific device are based on information collected about the devices during the scanning phase. For example, if a device is found to have TCP port 22 open, a set of UNIX®-specific probes may be used. Similarly, if a device is found to have TCP port 135 open, a set of WINDOWS®-specific probes may be used. In either case, a set of appropriate tasks may be placed in the task list (502) performed by the proxy servers (312). These tasks may cause the proxy servers (312) to log on to a specific device or access information from a specific device. For example, if TCP port 22 is open, the proxy servers (312) may be instructed to initiate a Secure Shell (SSH) connection to the specific device and obtain information about a specific type of operating system from specific locations in the file system. Based on this information, the operating system may be determined. For example, a UNIX® device with TCP port 22 open can be classified as AIX®, HPUX, LINUX®, MACOS®, or SOLARIS®. This classification information can be stored in the CMDB (500) as one or more configuration entries.

[0106] In the identification phase, proxy servers (312) can determine specific details regarding the classified device. The probes used during this phase may be based on information collected about the specific devices during the classification phase. For example, if the device is classified as LINUX®, a set of LINUX®-specific probes may be used. Similarly, if the device is classified as WINDOWS® 10, a set of WINDOWS®-10-specific probes may be used. As with the classification phase, a set of appropriate operations may be placed in the list of operations (502) performed by the proxy servers (312). Through these operations, the proxy servers (312) may read information from the specific device, such as BIOS (basic input / output system) information, serial numbers, network interface information, media access control address(s) assigned to the network interface(s), IP address(s) used by the specific device, etc. This identification information may be stored in the CMDB (500) as one or more configuration items along with any related relationships between them. Doing so may include passing the identification information through the IRE (514) to avoid creating duplicate configuration items for clarity and / or to determine the table(s) of the CMDB (500) where the discovery information should be recorded.

[0107] In the exploration phase, proxy servers (312) can determine additional details regarding the operational status of the classified device. The probes used during this phase may be based on information collected about specific devices during the classification phase and / or identification phase. Again, a set of appropriate tasks may be placed in the task list (502) performed by the proxy servers (312). Through these tasks, the proxy servers (312) can read additional information from the specific device, such as processor information, memory information, lists of running processes (software applications), etc. Once again, the discovered information may be stored in the CMDB (500) as one or more configuration items as well as relationships.

[0108] SNMP can be utilized to perform horizontal discovery on specific devices, such as switches and routers. Instead of determining a list of running processes or other application-related information, or in addition to that, discovery can determine additional subnets known to the router and the operational status of the router's network interfaces (e.g., active, inactive, queue length, number of dropped packets, etc.). The IP addresses of additional subnets can be candidates for additional discovery procedures. Therefore, horizontal discovery can be performed iteratively or recursively.

[0109] Patterns are used only in the identification and exploration phases—under pattern-based discovery, the scanning and classification phases operate as they do when probes and sensors are used. After the classification phase is completed, a pattern probe is designated as the probe to be used during identification. Then, the pattern probe and the pattern it designates are initiated.

[0110] Patterns, through discovery programming languages, support numerous features that are unavailable or difficult to achieve with discovery using probes and sensors. For example, the discovery of devices, components, and / or applications in public cloud networks, as well as configuration file tracing, are much simpler to achieve using pattern-based discovery. Furthermore, these patterns can be customized more easily by users than probes and sensors. Additionally, patterns are more focused on specific devices, components, and / or applications, and consequently can execute faster than the more general approaches used by probes and sensors.

[0111] Once horizontal discovery is complete, configuration item representations of each discovered device, component, and / or application are available in the CMDB (500). For example, after discovery, operating system versions, hardware configurations, and network configuration details for client devices, server devices, and routers in the management network (300), as well as applications running on them, may be stored as configuration items. This collected information may be presented to the user in various ways to allow the user to view the hardware configuration and operational status of the devices.

[0112] Additionally, the CMDB (500) may include entries regarding relationships between configuration items. More specifically, it is assumed that a server device includes a number of hardware components (e.g., processors, memory, network interfaces, storage, and file systems) and that a number of software applications are installed or run on them. Relationships between the components and the server device (e.g., "contained by" relationships) and relationships between software applications and the server device (e.g., "running on" relationships) may be represented as in the CMDB (500).

[0113] More generally, the relationship between a software component installed or running on a hardware component can take various forms, such as "hosted thereon," "running on thereon," or "dependent on thereon." Thus, a database application installed on a server device may have a "hosted thereon" relationship with the server device to indicate that the database application is hosted on the server device. In some embodiments, the server device may have a "used by thereon" relationship with the database application to indicate that the server device is used by the database application. These relationships may be found automatically using the discovery procedures described above, but the relationships may also be established manually.

[0114] In this way, the remote network management platform (320) can discover and record hardware and software that are deployed on the management network (300) and provided by the management network (300).

[0115] B. Vertical discovery

[0116] Vertical discovery is a technique used to locate and map components that are part of an entire service, such as a web service. For example, vertical discovery can map a web service by representing the relationships between a web server application, a LINUX® server device, and a database that stores data for the web service. Typically, horizontal discovery is executed first to locate components and the underlying relationships between them, and then vertical discovery is executed to establish the relationships between the components that make up the service.

[0117] Specific types of services can be discovered using patterns, as these patterns can be programmed to locate specific configurations of hardware and software that match the description of the service's deployment method. Alternatively, or additionally, traffic analysis (e.g., examining network traffic between devices) can be used to facilitate vertical discovery. In some cases, service parameters can be manually configured to support vertical discovery.

[0118] Generally, vertical discovery attempts to find specific types of relationships between devices, components, and / or applications. Some of these relationships can be inferred from configuration files. For example, a web server application's configuration file may reference the IP address and port number of the database it depends on. Vertical discovery patterns can be programmed to find these references and infer relationships from them. Relationships can also be inferred from traffic between devices; for instance, if there is a significant amount of web traffic (e.g., TCP ports 80 or 8080) moving between a load balancer and a device hosting a web server, the load balancer and the web server may have a relationship.

[0119] Relationships discovered through vertical discovery can take various forms. For example, an email service may include email server software components and database application software components, each installed on different hardware device components. While the email service may have a "depends on" relationship with both of these software components, the software components may have a "used by" relationship with the email service. These services may not be fully determined by horizontal discovery procedures and may instead rely on vertical discovery and, possibly to some extent, manual configuration.

[0120] C. Benefits of discovery

[0121] Regardless of how the discovery information is obtained, it can be valuable for the operation of the management network. In particular, IT personnel can quickly determine where specific software applications are deployed and which components make up the services. This allows for the rapid identification of the root causes of service outages or performance degradation. For example, if two different services are experiencing slow response times, querying the CMDB can determine (among other activities) that the root cause is the database application used by the two services with high processor utilization. Consequently, IT personnel can address the database application without wasting time by considering the health and performance of the other components comprising the services.

[0122] In another example, assume that a database application runs on a server device and is used by the employee onboarding service as well as the payroll service. Therefore, it is clear that if the operation of the server device is interrupted for management purposes, the employee onboarding service and the payroll service will be affected. Similarly, dependencies and relationships between configuration items can represent the services that are affected if a specific hardware device fails.

[0123] Generally, configuration items and / or relationships between configuration items can be displayed on a web-based interface and represented in a hierarchical manner. Through this interface, modifications to these configuration items and / or relationships in the CMDB can be achieved.

[0124] Additionally, users from the management network (300) can develop workflows that can cause specific coordination activities to occur across multiple discovered devices. For example, an IT workflow can allow a user to change a common administrator password for all discovered LINUX® devices in a single action.

[0125] V. CMDB Identification Rules and Adjustments

[0126] A CMDB, such as CMDB (500), provides a repository of configuration items and relationships. When properly provisioned, it can be placed within a compute instance or play a key role in higher-level applications that accompany the compute instance. These applications may be related to enterprise IT service management, behavior management, asset management, configuration management, compliance, etc.

[0127] For example, IT service management applications can use information in the CMDB to determine which applications and services may be affected by components (e.g., server devices) that are malfunctioning, crashing, or overloaded. Similarly, asset management applications can use information in the CMDB to determine which hardware and / or software components are being used to support specific enterprise applications. Due to the importance of the CMDB, it is desirable that the information stored within it be accurate, consistent, and up-to-date.

[0128] The CMDB can be populated in various ways. As described above, the discovery process can automatically store information, including configuration items and relationships, in the CMDB. However, the CMDB can also be populated wholly or partially through manual input, configuration files, and third-party data sources. Given that multiple data sources can update the CMDB at any time, one data source may overwrite entries from another. Additionally, two data sources may each generate slightly different entries for the same configuration item, causing the CMDB to contain duplicate data. If either of these situations occurs, it can reduce the health and usability of the CMDB.

[0129] To mitigate this situation, these data sources may not write configuration items directly to the CMDB. Instead, they may write to the identification and coordination API (application programming interface) of the IRE (514). Then, the IRE (514) can use a set of configurable identification rules to uniquely identify the configuration items and determine whether and how they will be written to the CMDB.

[0130] Generally, an identification rule specifies a set of configuration item attributes that can be used for this unique identification. Identification rules have priorities, so rules with higher priorities are considered before rules with lower priorities. Additionally, a rule can be independent in that it identifies configuration items independently of other configuration items. Alternatively, a rule can be dependent in that it first uses metadata rules to identify dependent configuration items.

[0131] Metadata rules describe which other configuration items are included within a specific configuration item, or the host on which a specific configuration item is placed. For example, a network directory service configuration item may include a domain controller configuration item, while a web server application configuration item may be hosted on a server device configuration item.

[0132] The goal of each identification rule is to use a combination of attributes that can clearly distinguish a configuration item from all other configuration items and are expected not to change during the configuration item's lifetime. Some possible attributes for an exemplary server device may include serial number, location, operating system, operating system version, memory capacity, etc. If a rule specifies attributes that do not uniquely identify a configuration item, multiple components may be represented as the same configuration item in the CMDB. Additionally, if a rule specifies attributes that change for a particular configuration item, duplicate configuration items may be created.

[0133] Therefore, when a data source provides information about a configuration item to the IRE (514), the IRE (514) may attempt to match the information with one or more rules. If a match is found, the configuration item is written to the CMDB or updated if it already exists in the CMDB. If no match is found, the configuration item may be retained for further analysis.

[0134] Configuration item reconciliation procedures can be used to allow only authorized data sources to overwrite configuration item data in the CMDB. This reconciliation may be rule-based. For example, a reconciliation rule may specify that a particular data source is authorized for a particular configuration item type and a set of attributes. Then, the IRE (514) may allow only this authorized data source to write to a particular configuration item and prevent writes from unauthorized data sources. Thus, an authorized data source becomes the single source of true information regarding a particular configuration item. In some cases, unauthorized data sources may be allowed to write to a configuration item if they are creating the configuration item or if the attributes to be written are empty.

[0135] Additionally, multiple data sources may be authorized for the same configuration item or its attributes. To avoid ambiguity, these data sources may be assigned priorities to be considered during the writing of the configuration items. For example, a secondary authorized data source may write to the attributes of a configuration item until a primary authorized data source writes to the attribute. After that, further writes to the attribute by the secondary authorized data source may be blocked.

[0136] In some cases, duplicate configuration items may be automatically detected by IRE (514) or by other means. These configuration items may be deleted or flagged for manual deduplication.

[0137] VI. Incident Reporting and Resolution

[0138] Given the large-scale networks described herein, it is inevitable that some components (e.g., computing devices, systems, applications, services, and / or networks) will not function properly at any given time. These failures may be caused by incorrect configurations, incompatibilities, software bugs, hardware errors, or a mismatch between resource supply and demand (e.g., a device exhausting RAM or disk storage). These failures result in slowdowns, unreliability, and / or interruptions of services that users depend on. Clearly, these are technical issues in that one or more hardware or software components do not operate as designed, thereby adversely affecting systems and users.

[0139] Such defects are referred to as "problems" and can be reported, investigated, and resolved through an incident log (sometimes called an "incident," "IT trouble ticket," or "ticket"). An incident log is a document or record used to report and track problems related to computer systems, software, and network infrastructure. These incident logs are typically generated by users experiencing technical problems and then assigned to virtual agents (e.g., chatbots) and / or human agents for resolution. An incident log will typically contain information such as the user's contact information (e.g., email address and phone number), a description of the problem written by the user, the date and time the incident occurred, and any relevant system details (e.g., the user's client device and / or local network). As agents work to resolve the problem, additional information, such as the investigation steps taken and the status of the problem, may be added to the incident log.

[0140] Although the embodiments herein are described in the context of incident logs, these embodiments may also be used with other types of logs. Accordingly, the examples discussed below are for illustrative purposes only and not for limitation.

[0141] Typically, an incident log takes the form of one or more associated text files or database entries containing records with timestamps of any interactions between the user and the agent, as well as each action taken by the user and the agent. This text content may include the user's description of the problem, conversations between the user and the agent, the agent's investigation notes (which may include the output of diagnostic steps taken by the agent), a description of the root cause of the problem (if applicable), and a description of the remediation steps taken to resolve (fix) the problem.

[0142] When a problem is reported, its root cause may be unclear. The initial description of the problem may be ambiguous when it appears in an incident log. For example, a user might generate an incident log with a problem description stating, "I cannot connect to the network." An agent investigating this problem is likely to determine what type of client device the user is using (e.g., a laptop, mobile phone, or tablet), what operating system is being used on the device, and what network the user is attempting to access.

[0143] Accordingly, the agent may ask the user a series of questions (e.g., via phone, email, and / or chat session) to identify the user's operating environment. Then, the agent may request the user to perform one or more steps on the device, such as resetting the interface, changing configuration settings, or rebooting. The agent may also perform multiple steps, such as remotely accessing a router or Wi-Fi access point near the user to check the status of those devices and the network to which those devices are part. Some of the activities of the user or the agent may include running diagnostic tools (e.g., apps or scripts) that may provide additional insight into the problem.

[0144] Ultimately, the agent can identify the root cause of the problem. For example, assume that a user is unable to use their local Wi-Fi access point. Possible root causes include wireless interference from other devices, distance from the access point, incorrect Wi-Fi settings on the user's client device, outdated firmware installed on the access point, hardware failure of the access point, an overloaded Wi-Fi network, or a downstream network. Determining which of these is the actual root cause can potentially force the agent to go through a large number of investigation steps.

[0145] As mentioned, the results of each investigation step may be recorded in the incident log regardless of whether the root cause is successfully determined. Outputs from any diagnostic tools may also be recorded in the same way.

[0146] Once the root cause is identified, the solution is generally clear. For example, if the root cause is that the user is attempting to connect to a Wi-Fi access point that is too far away (and thus resulting in low signal strength at the user's location), the solution might be for the user to move closer to the access point or connect to a closer one. If the root cause is insufficient memory on the access point (likely due to a memory leak defect in the firmware), the solution might be for the agent to reboot the access point. In any case, actions taken by any party to resolve the issue may also be recorded in the incident log.

[0147] If the user is satisfied with the solution or is no longer hindered by the problem, the incident log can be closed. However, closed incident logs remain in the database for audit purposes and to aid in resolving similar problems in the future. Therefore, a database for a moderately sized organization can grow over time to contain tens of thousands of incident logs.

[0148] While the general context of the embodiments herein is that of an enterprise network with potentially many users, the described technologies may also be applicable to small-scale environments with fewer users. For example, problems experienced by users in residential, home office, or small office settings may also be addressed. Therefore, the applicability of these embodiments is not limited to the environments described herein.

[0149] FIG. 6a shows an example extracted from an incident log containing a sequence of events. This sequence focuses on a problem reported by a user, investigation steps taken by an agent, determination of the root cause, and subsequent resolution of the problem. Each event has a timestamp, an actor (the person or agent who generated the event), and text describing the event.

[0150] Event 600 involves user John Doe opening the incident log (ticket 0000001). This event includes the user's contact information and a brief description of the problem being experienced. In this case, the symptom of the reported problem is that the user is unable to connect to the network.

[0151] Event 602 involves the incident log being assigned to Agent Jane Smith.

[0152] Event 604 includes an investigation step in which an agent consults with the user and determines the user's laptop brand and model, as well as its operating system.

[0153] Event 606 includes an investigation step in which the agent determines that the user's laptop is connected to a Wifi access point with good signal strength.

[0154] Event 608 includes an investigation step in which the agent determines that the user cannot access any website regardless of the web browser used.

[0155] Event 610 includes an investigation step in which the agent performs a "ping" transaction on the user's laptop and receives no response. Ping is an application that sends packets from one computer to another and receives corresponding responses for each, along with an estimate of the latency between the computers.

[0156] Event 612 includes an investigation step in which the agent performs a "ping" transaction to the Wifi access point and determines that the ping time is irregular and high and that there is packet loss between the agent's computer and the Wifi access point.

[0157] Event 614 involves the identification of the root cause. The agent determines that the Wi-Fi access point has very high memory usage (98%) and has not been rebooted for about 6 months.

[0158] Event 616 contains the solution to the problem. The agent reboots the Wi-Fi access point and determines that the user can now access the network and the Wi-Fi access point is functioning as expected.

[0159] Event 618 involves the agent marking the incident log as resolved.

[0160] In particular, Event 600 describes the symptoms of the problem experienced by the user. Events 604, 606, 608, 610, and 612 describe the investigation steps. Event 614 describes the root cause of the problem, and Event 616 describes the solution to the problem. The events in this sequence are typical in that there are multiple investigation steps before the root cause is determined. Events 602 and 618 are incident log management steps that are not strictly related to the problem, investigation, root cause, or solution.

[0161] Figure 6b shows another example extracted from the incident log.

[0162] Event 650 involves user Teri Dactyl opening the incident log (ticket 0000004). This event includes the user's contact information and a brief description of the problem being experienced. In this case, the symptom of the reported problem is that the user cannot access Wi-Fi.

[0163] Event 652 involves the incident log being assigned to Agent Bob Jones.

[0164] Event 654 involves the incident log being updated to add more details about the symptoms of the problem.

[0165] Event 656 includes an investigation step in which the agent provides the user with an updated password, but discovers that this new password also did not work.

[0166] Event 658 includes an investigation step in which the agent determines that the authentication server has not allowed the user to use Wi-Fi, and then approves this allowance.

[0167] Event 660 includes the identification of the root cause of the problem and the solution. The agent determines that the user's credentials at the authentication server were outdated and that, as a result of updating them to authorize access, the user is now able to use Wi-Fi. This event also includes the incident log being marked as resolved.

[0168] Event 662 involves the incident log being closed.

[0169] In particular, events 650 and 654 describe the symptoms of the problem experienced by the user. Events 656 and 658 describe the investigation steps. Event 660 describes the solution to the problem as well as the root cause of the problem. Events 652 and 662 are incident log management steps that are not strictly related to the problem, investigation, root cause, or solution. Unlike the events in the sequence in Fig. 6a, the events in this sequence have two distinct events describing the symptoms and one event describing both the root cause and the solution.

[0170] These two examples alone illustrate the various ways in which symptoms, investigation steps, root cause determinations, and solutions can be performed and expressed in incident logs. Nevertheless, other arrangements of this content may also occur within incident logs. Therefore, the examples in FIGS. 6a and 6b are for illustrative purposes only and not for limitation.

[0171] Incident logs contain a wealth of information. As mentioned above, many organizations generate numerous such logs (e.g., tens, hundreds, or thousands) on a weekly, monthly, or quarterly basis. When combined, incident logs can contain an organization's memory—a record of the problems experienced and how they were resolved—which can be invaluable when attempting to resolve similar issues in the future.

[0172] Therefore, organizations can gain the advantage of mining incident logs to determine the investigation steps most likely to resolve a given problem most quickly and efficiently. Agents can then use or be guided by this insight when attempting to determine the root cause and solve new problems. One potentially important observation is that not all investigation steps are equally likely to lead to a root cause determination.

[0173] For example, in the case of the incident log in Fig. 6a, obtaining the brand, model, and operating system of the users' laptops in Event 604 did not contribute to determining the root cause. On the other hand, pinging the Wi-Fi access point in Event 612 and remotely accessing the access point in Event 614 to determine its memory usage directly led to the identification of the root cause of the problem and a solution.

[0174] The embodiments herein provide techniques for processing incident logs using various types of machine learning-based natural language processing tools. Through these techniques, the summarization and characterization of past incident logs are performed, which can be used to guide agents on how to proceed while handling similar incidents they may encounter in the future. First, an incident log model is trained using a corpus of existing incident logs. Then, this model is used to predict the investigation steps most likely to lead to root cause determination for subsequent incidents.

[0175] Figure 7a shows the training steps. Each of these blocks is defined in more detail below.

[0176] Block 700 involves preprocessing incident logs to prepare them for use in training a model. Block 702 involves summarizing events from the incident logs to obtain shorter but relevant text representations. Block 704 involves classifying the summarized events into symptoms, investigation steps, root causes, and solutions. Block 706 involves determining cluster spaces for events of each class. Block 708 involves determining probabilistic relationships between clusters of events. These probabilistic relationships provide the probabilities that any particular investigation step will determine the root cause of a given problem.

[0177] Figure 7b shows the prediction steps. Each of these blocks is defined in more detail below.

[0178] Block 750 includes preprocessing, summarizing, and / or classifying the new incident log to prepare it for use with the model. Block 752 includes determining one or more symptoms similar to those of the new incident log from the symptom cluster space. Block 754 includes selecting the investigation step most likely to lead to a root cause determination from the investigation step cluster space. Since Block 754 may be performed for more than one investigation step, only investigation steps that have not yet been selected are considered. Block 756 includes determining whether the root cause has been determined based on the selected investigation step. If it has not been determined, control returns to Block 754. If it has been determined, control passes to Block 758. Block 758 may include applying a solution to the determined root cause.

[0179] These embodiments provide a technical solution to a technical problem. One technical problem to be solved is a method for rapidly resolving issues with computing devices, systems, and / or networks. In practice, it is important that these problems be resolved quickly, as interruptions adversely affect the performance of these devices, systems, and networks, causing performance below specifications or expectations. As a result, even if the provided services operate, they may be unreliable or slow.

[0180] In conventional technology, individual agents determined the sequence of investigation steps based on pre-established scripts or their own experience and knowledge. However, these script-based and ad hoc techniques do not always lead to a solution, and even fewer within a reasonable timeframe. Furthermore, prior art relies on the subjective judgments and experiences of individual agents, resulting in significant variations in outcomes from case to case. Consequently, conventional technology has rarely been effective in rapidly resolving device, system, or network performance degradation.

[0181] The embodiments herein overcome these limitations by utilizing incident logs already available within the organization to accelerate the resolution of new problems. In this way, problem resolution can be achieved in a faster, more accurate, and robust manner. Doing so offers several advantages. First, there is no need to develop dozens of scripts for agents when various types of symptoms are presented. Second, these techniques do not rely on the individual tendencies and biases of agents, but instead employ object processes for root cause determination. Third, the embodiments reduce downtime caused by performance degradation and interruptions of computing devices, systems, and networks, thereby enabling the provided services to be accessible with a much higher level of stability. Furthermore, other technical improvements may emerge from these embodiments, and other technical problems may be resolved.

[0182] A. Training Stage

[0183] This subsection provides a detailed disclosure of the blocks associated with the training phase. However, more or fewer blocks of the same or different nature may be used. For example, in some embodiments, preprocessing and summary blocks may be omitted.

[0184] 1. Preprocessing of incident logs

[0185] Prior to the start of the training itself, the corpus of incident logs may be preprocessed. For example, this may include any type of normalization or outlier removal. In some cases, this may include removing boilerplate or form text from the incident logs. Examples of boilerplate or form text may be content automatically added to some or all incident log events, such as "This incident has been queued for consideration by an agent. The agent will contact you if you have any questions during the investigation of your incident." Since this text is common across incidents and is not useful for determining the root cause, it can be safely removed to allow the model to focus on the remaining content that can better predict the root causes. Other types of preprocessing may include the removal of punctuation and / or stop words, stemming, and lemmatization. In some cases, preprocessing steps may replace the names of users and agents with generic terms such as "User" and "Agent," respectively.

[0186] For example, a preprocessed version of Event 616 might be "rebooted NetworkSouth user is using it to access the internet verified can ping his IP address and that of NetworkSouth with low latency and that the memory utilization of NetworkSouth is around 50%".

[0187] 2. Summary of Events

[0188] After arbitrary processing, each event in the corpus (preprocessed or unpreprocessed) can be transformed or summarized. Various techniques, such as extractive or abstract summarization, can be used to perform this block. Extractive summarization involves selecting the most relevant sentences or phrases from an event and assembling them into a shorter version. Abstract summarization involves generating a shorter version of an event by creating new sentences that capture the event's key semantic meaning.

[0189] For example, a summary version of preprocessed event 616 may be "rebooted NetworkSouth and verified internet access; IP address and that of NetworkSouth have low ping latency; memory utilization of NetworkSouth is 50%".

[0190] 3. Classification of Events

[0191] After summarization, each event is classified into one or more classes. These classes may be symptoms of the problem, investigation steps, determination of the root cause, and methods for resolving the problem. In many cases, each event will fall precisely into one of these classes. However, some events may be classified into two or more classes, including content related to more than one class.

[0192] Various types of pre-trained classifiers can be used to perform this classification. For example, Naive Bayes ( ), support vector machines, decision trees, random forests, or neural network classifiers can be trained on a set of events with labeled ground-truth (GT) classifications. This may involve manually labeling multiple events and then training these classification models to predict classes of new events based on their text content. For example, a summarized version of event 616 may be classified as an investigation step class based on the classifier being trained on similar types of events representing investigation steps.

[0193] In some cases, classifications may be reinforced or otherwise performed using additional heuristics, such as initially assuming that all events containing text from users are related to the symptoms of the problem and / or using terms such as "root cause" and "solution" to place events into their respective classes. Some classifiers may also have a fifth class, which is a default or inclusive class, where events that cannot be classified into any of the four classes above are placed. Events related to management steps, such as events 602 and 618, may be placed within this class.

[0194] 4. Determination of cluster spaces

[0195] Then, events within each class of events can be clustered. That is, for a class of events (e.g., symptoms, investigation steps, root causes, and solutions), there may be a set of clusters. This clustering may be performed independently or semi-independently for each class. The class-specific clustering process may include feature selection and then actual clustering. In the embodiments set forth herein, clustering is not supervised, but some form of supervision may be possible.

[0196] Feature selection involves determining measurable qualities within the text of each class to be used as the basis for clustering. These qualities may relate to word choice, word frequency, word length, word semantics, sentence sentiment, or some other features. Ultimately, each of these features can be mapped to one or more dimensions. Thus, a combination of features forms a multidimensional vector (typically consisting of numerical values) for each event in the class.

[0197] Actual clustering can utilize various techniques such as k-means (dividing events into a predefined number of clusters based on their vectors, where each event is assigned to the cluster with the closest mean value), hierarchical modeling (constructing a tree structure of clusters, where events are grouped together based on the similarity of their vectors), topic modeling (e.g., identifying underlying topics or themes in events based on word distribution and grouping events into clusters accordingly), and / or spectral modeling (projecting the text of events into a high-dimensional space and then identifying clusters based on the similarity between events in that space).

[0198] To explain clustering in more detail, word vector modeling is an example of spectral modeling in which a neural network is trained to assign a vector representation to each word in events based on contextual words that frequently occur together. Once vectors are established for each word, vector operations (e.g., averaging of a specific form) can be used on the words within an event to represent the contextual meaning of the overall event. Other techniques, such as paragraph vectors, BERT (bidirectional encoder representations from transformer), or various types of large-scale language models, can also be used to perform this function.

[0199] FIG. 8a illustrates an example of a clustering process. Here, each event is mapped to a two-dimensional cluster space (800) for the purpose of city. In various embodiments, larger cluster spaces (e.g., n-spaces) may be used. As mentioned in FIG. 8, the cluster space (800) is for symptom classes. Distinct cluster spaces (not shown) may be used for other classes (e.g., investigation steps, root causes, and / or solutions).

[0200] Vectors for events 1, 2, and 3 are provided. In particular, the feature selection process determined that event 1 has the vector (-0.7461, 0.8854), event 2 has the vector (-0.4653, 0.7222), and event 3 has the vector (-0.3815, 0.6837). Each of these events is marked as X in the cluster space (800). Likewise, other events for which vectors are not explicitly shown in FIG. 8 are also marked as X in the cluster space (800).

[0201] When all events within a class are projected as vectors and mapped into cluster space, clustering techniques can be used to associate groups of adjacent events into clusters. The goal of this n-dimensional clustering is to group events together based on their proximity to one another in cluster space. This technique can be operated by first initializing a set of centroids, which are points representing the centers of the clusters. Then, the technique can iteratively assign each data point to the nearest centroid and update the centroid to be the average of all data points assigned to it. This process continues until the centroids converge and the clusters stabilize. At the vertex, each centroid can be, for example, the point that minimizes the sum of the squared distances between all points in the cluster and the centroid itself. In some cases, clustering techniques may generate one or more residual points that do not belong to any cluster. Residual points can occur when there is noise in the dataset.

[0202] Figure 8a shows three clusters and one residual point. Cluster 802 contains Event 2 and Event 3, and Clusters 804 and 806 contain other events. Event 1 is a residual point that is not a member of any cluster. Since clustering occurs based on the semantic content of events, all events in each cluster are likely to be semantically similar to one another (e.g., identical or similar topics). Events in other clusters are likely to be semantically less similar to one another (or not semantically similar at all), and accordingly, are likely to be different topics.

[0203] In some cases, human-readable or human-understandable names or labels may be assigned to clusters. These labels may be single words or short sequences of words that semantically describe or summarize the types of events assigned to the clusters. In some embodiments, this may involve identifying the most frequent or semantically representative features in the clusters (e.g., through techniques such as feature importance or frequency analysis, topic modeling, or sentiment modeling to extract the most prevalent keywords or phrases). For example, a cluster of all symptoms related to a user being unable to access a Wi-Fi network may be labeled with the text "Wi-Fi not working".

[0204] 5. Determination of probabilistic relationships between clusters

[0205] Once cluster spaces and the clusters within them are established, probabilistic relationships between the clusters can be determined. This may involve identifying, for each cluster of events, which clusters of subsequent events are most likely to follow. For example, given a cluster of symptoms, the training data may contain a relatively limited set of investigation step clusters that lead to a root cause cluster. That is, given a specific symptom, there is one or more types of investigation steps most likely to lead to the identification of the root cause.

[0206] FIG. 8b illustrates this mapping for a small number of clusters, each being a set of clusters for symptoms, investigation steps, root causes, and solutions. The symptom clusters (850) are each associated with different types of symptoms determined through block 706. The investigation step clusters (852) are each associated with different types of investigation steps determined through block 706. The root cause clusters (854) are each associated with different types of root causes determined through block 706. The solution clusters (856) are each associated with different types of solutions determined through block 706.

[0207] Given these clusters, the probabilities of the paths can be determined via FIG. 8b by parsing the events within each. As shown, symptom cluster A (containing events associated with a specific type of symptom) is followed only by the types of investigation steps found in investigation step clusters B and C. Similarly, investigation step clusters B and C are followed only by the types of root causes found in root cause clusters D, E, F, and G. As mentioned, once a root cause is identified, the solution is generally clear. Thus, root cause clusters D, E, F, and G each lead to a relatively small number of possible solutions. Therefore, it is generally permissible to use the procedures herein to identify the root cause, as doing so will also almost always identify an appropriate solution.

[0208] With respect to probabilities, consider the relationships between investigation step clusters B and C and root cause clusters D, E, F, and G. Each of investigation step clusters B and C has a specific probability of identifying the root cause of the associated symptoms. For example, investigation steps in investigation step cluster B lead to a root cause in root cause cluster D in 30% of cases and to a root cause in root cause cluster F in 60% of cases. In the other 10% of cases, investigation steps in investigation step cluster B do not lead to a root cause. Similarly, investigation steps in investigation step cluster C lead to a root cause in root cause cluster E in 10% of cases, to a root cause in root cause cluster F in 20% of cases, and to a root cause in root cause cluster G in 10% of cases. In the other 60% of cases, investigation steps in investigation step cluster C do not lead to a root cause.

[0209] Given only this information, it is clear that an investigation step from investigation step cluster B is more likely to lead to the root cause than an investigation step from investigation step cluster C. Alternatively, these probabilities can be conditional based on the symptom clusters. That is, given a symptom in symptom cluster A, the probability that an investigation step from investigation step cluster B leads to the root cause would be 90%, and the probability that an investigation step from investigation step cluster C leads to the root cause would be 40%. If symptoms originate from different clusters, these probabilities may differ (not illustrated in Fig. 8b).

[0210] These probabilities can be represented in the form of a directed acyclic graph between various clusters, where the edges represent the probabilities of proceeding from one cluster to another. However, other graph representations are also possible.

[0211] In any case, FIG. 8b is merely a small example for illustrative purposes. The embodiments herein may include tens, hundreds, or more of clusters of each type interconnected in various and complex ways. How these probabilistic relationships are used to recommend investigation steps is explained below.

[0212] B. Prediction Step

[0213] This subsection provides a detailed disclosure of the blocks associated with the prediction step of FIG. 7b. However, more or fewer blocks of the same or different nature may be used. This subsection also assumes that events were identified from a corpus of incident logs and that these events were used to generate probabilistic relationships between clusters of events as illustrated in FIG. 8b.

[0214] The blocks of Fig. 7b are typically performed on data from new incident logs before symptoms are resolved. In particular, given symptoms, the blocks of Fig. 7b provide techniques for recommending specific investigation steps most likely to lead to the identification of the root cause. This recommendation may be provided to a person or a virtual agent.

[0215] 1. Preprocessing, summarizing, and classification of new incident logs

[0216] As mentioned above, preprocessing is not limited but may include applying any type of normalization or outlier removal to the new incident log, including the removal of punctuation and / or stop words, stemming, lemmatization, word or phrase replacement, word or phrase deletion, etc. The preprocessing that occurs as part of block 750 may be the same or similar to that of block 700.

[0217] Also, as mentioned above, summarization may include, for example, applying an extractive or abstract summary to a preprocessed incident log. The summary occurring as part of block 750 may be the same or similar to that of block 702.

[0218] As mentioned above, classification may involve applying a trained classifier to the new incident log that has been preprocessed and summarized. The trained classifier may be a Naive Bayes, Support Vector Machine, Decision Tree, Random Forest, or Neural Network classifier. The classification occurring as part of Block 750 may be the same or similar to that of Block 704. Since the goal of Block 750 is to identify one or more symptoms from the incident log, only events classified as symptoms may be subject to the next steps.

[0219] 2. Determination of similar symptoms

[0220] Determining similar symptoms may involve taking one or more events classified as symptoms and applying feature selection as described above. As previously mentioned, feature selection may involve projecting the text of the symptoms onto a multidimensional vector. This vector can then be compared with the characteristics of the symptom clusters established during the training phase. For example, the Euclidean distance (the straight-line distance between two points in n-space) or cosine similarity (the cosine of the angle between two vectors in n-space) between this vector and the centroid of each symptom cluster can be calculated. Then, a symptom from a new incident log can be considered likely to be a member of the closest symptom cluster (in terms of Euclidean distance) or the most similar symptom cluster (in terms of cosine similarity). Other possibilities exist. For example, in some cases, more than one closest symptom cluster may be identified.

[0221] 3. Selection of Investigation Step Clusters

[0222] When symptoms and clusters of similar symptoms are identified, an investigation step cluster is selected based on the likelihood that this investigation step cluster will lead to the root cause. Since multiple investigation step clusters may lead to the root cause of this symptom, this selection may be repetitive in nature, as shown in Blocks 754 and 756.

[0223] Consider FIG. 8b as an example, assume that a new incident log is determined to contain symptoms similar to those of symptom cluster A. In this case, there are two possible investigation step clusters to recommend, investigation step cluster B and investigation step cluster C. As mentioned above, one or both of these clusters may be recommended to a person or virtual agent based on probability relationships for one or more root cause clusters (854).

[0224] Various techniques may be used to determine the order in which these investigation step clusters are proposed. For example, the investigation step cluster with the highest probability of leading to the root cause (i.e., investigation step cluster B) may be proposed first. This cluster may include labels or associated text describing one or more investigation steps that the agent should take. If these steps identify the root cause of the symptom in block 756, control passes to block 758. If it does not identify it, control returns to 754, where the investigation step cluster with the next highest probability of leading to the root cause (i.e., investigation step cluster C) may be proposed. This process continues until the root cause is identified or there are no other investigation step clusters that have not been attempted. Another proposal criterion may be the investigation step cluster most likely to reduce the number of candidate root cause clusters.

[0225] In other words, the system (e.g., a remote network management platform) proposes a series of investigation steps to the agent based on trained models. The agent performs each proposed investigation step until the root cause is determined or the system exhausts all suggestions. In the latter case, the agent processes the incident logs relying on its subjective experience.

[0226] For example, if the symptom is “Wifi not working,” the first proposed investigation step may be identifying the Wi-Fi access point that the user is attempting to connect to, and the second proposed investigation step may be identifying whether the user is using the correct Wi-Fi password, etc. In some cases, 12 or more investigation steps may be available in the proposal.

[0227] 4. Application of the Solution

[0228] Once the root cause is identified, the solution is typically clear, or at least the number of possible solutions is likely small (e.g., at most two or three). If the agent is virtual, it can automatically perform the solution if possible (e.g., password reset, providing the user with necessary information, device reboot, etc.). If the agent is human, they can perform the solution.

[0229] Ⅶ. Exemplary movements

[0230] FIGS. 9a and 9b are flowcharts illustrating exemplary embodiments. The processes illustrated in FIGS. 9a and 9b may be performed by a computing device such as a computing device (100) and / or a cluster of computing devices such as a server cluster (200). However, the processes may be performed by other types of devices or device subsystems. For example, the processes may be performed by a computing instance of a remote network management platform or a portable computer, such as a laptop or tablet device.

[0231] The embodiments of FIGS. 9a and 9b may be simplified by removing one or more of the features shown therein. Additionally, these embodiments may be combined with each other and / or features, aspects, and / or implementations of any of the prior drawings or otherwise described herein.

[0232] In FIG. 9a, block (900) may include acquiring a plurality of incident logs, each incident log containing events of a respective sequence. Block (902) may include classifying each of each sequence of events into a respective event class. Block (904) may include determining cluster spaces associated with each event class. Block (906) may include determining relationships between at least some clusters in cluster spaces from the plurality of incident logs. Block (908) may include proposing one or more investigation steps to determine the root cause of a symptom found in a subsequent incident log based on the clusters and relationships, wherein the symptom describes a problem experienced by the user.

[0233] In some embodiments, each event class includes symptoms, investigation steps, and root causes, the symptoms describe problems experienced by users, the investigation steps describe measures taken to determine the root causes of the symptoms, and the root causes are the primary causes for the observation of the symptoms.

[0234] In some embodiments, each event class also includes solutions, and the solutions describe the measures taken to correct the corresponding root causes.

[0235] In some embodiments, at least some of the incident logs contain text content, and the embodiments further include performing one or more of stop word removal, form text removal, stemming, or lemming on the incident logs prior to classifying each of the events of each sequence.

[0236] In some embodiments, at least some of the incident logs contain text content, and the embodiments further include performing an extractive summary or an abstract summary on the incident logs prior to classifying each of the events of each sequence.

[0237] In some embodiments, classifying each of the events in each sequence into each event class comprises using a classifier pre-trained on a corpus of labeled events from incident logs, wherein the labels of the labeled events indicate each event class, and the classifier has learned the associations between the contents of the incident logs and each event class.

[0238] In some embodiments, determining the cluster spaces associated with each event class comprises, for each of each event class, projecting the events classified within it onto multidimensional representations; and forming clusters in the cluster spaces based on distances or angles between the multidimensional representations.

[0239] In some embodiments, determining the relationships between at least some clusters in cluster spaces includes determining the probabilistic possibilities of events progressing from the first cluster of two clusters to the second cluster of two clusters based on the events of each sequence.

[0240] In some embodiments, determining the probabilistic possibilities involves constructing a directed acyclic graph of clusters, and the edges of the directed acyclic graph represent the probabilistic possibilities.

[0241] Some embodiments further include labeling each of the clusters based on the semantic content of the events within them.

[0242] Some embodiments further include, after determining the root cause of the symptoms found in a subsequent incident log, causing the computing device to change its configuration, change one or more applications running, or reboot.

[0243] In FIG. 9b, block 950 may include acquiring an incident log containing an event indicating a symptom, wherein the symptom describes a problem experienced by the user. Block 952 may include performing a comparison between the event and a plurality of symptom clusters within a symptom clusters space, wherein the plurality of symptom clusters represent symptoms associated with events in a plurality of previously acquired incident logs. Block 954 may include identifying a symptom cluster from the symptom cluster space based on the comparison. Block 956 may include selecting an investigation step cluster from an investigation step cluster space based on the symptom cluster, wherein the investigation step cluster is associated with one or more root cause clusters from a root cause cluster space, the investigation step cluster space is derived from events in a plurality of previously acquired incident logs, and the root cause cluster space is also associated with events in a plurality of previously acquired incident logs. Block 958 may include determining that an investigation step from an investigation step cluster led to the identification of the root cause of the symptom, wherein the root cause is from one of the root cause clusters.

[0244] In some embodiments, performing a comparison between an event and a plurality of symptom clusters includes determining a similarity matrix between each of the event and the plurality of symptom clusters.

[0245] In some embodiments, identifying a symptom cluster from a symptom cluster space includes selecting a symptom cluster because it is most similar to an event with respect to a similarity matrix.

[0246] Some embodiments may further include classifying an event into a symptom event class using a classifier pre-trained on a corpus of labeled events from incident logs prior to performing a comparison between the event and a plurality of symptom clusters, wherein the labels of the labeled events indicate each event class, and the classifier has learned the associations between the content of the labeled events in the incident logs and each event class.

[0247] In some embodiments, an investigation step cluster is selected because it has the highest probability of leading to one of the root cause clusters within the investigation step cluster space.

[0248] In some embodiments, an investigation step cluster is selected because it has the highest probability of reducing the number of candidate root cause clusters within the investigation step cluster space.

[0249] Some embodiments may further include selecting a solution cluster from a solution cluster space based on a root cause, and the solution cluster includes a solution describing measures taken to correct the root cause.

[0250] In some embodiments, the solution includes causing the computing device to change its configuration, change one or more applications running, or reboot.

[0251] VII. finish

[0252] The present disclosure is not limited to the specific embodiments described in this application, which are intended as examples of various aspects. As will be apparent to those skilled in the art, many modifications and changes may be made without departing from the scope thereof. In addition to those described herein, functionally equivalent methods and apparatuses within the scope of the present disclosure will be apparent to those skilled in the art from the descriptions above. Such modifications and variations are intended to fall within the scope of the appended claims.

[0253] The above detailed description describes various features and operations of the disclosed systems, devices, and methods with reference to the attached drawings. The exemplary embodiments described herein and in the drawings are not intended to be limiting. Other embodiments may be utilized and other modifications made without departing from the scope of the gist presented herein. It will be readily understood that the aspects of the disclosure generally described herein and shown in the drawings may be arranged, substituted, combined, separated, and designed into a wide variety of different configurations.

[0254] In relation to some or all of the message flowcharts, scenarios, and flowcharts depicted in the drawings and described herein, each step, block, and / or communication may represent the processing of information and / or the transmission of information according to exemplary embodiments. Alternative embodiments are included within the scope of these exemplary embodiments. In these alternative embodiments, actions described, for example, as steps, blocks, transmissions, communications, requests, responses, and / or messages may be executed differently from the order shown or discussed, and this includes substantially simultaneously or in reverse order depending on the relevant function. Additionally, more or fewer blocks and / or actions may be used with any of the message flowcharts, scenarios, and flowcharts discussed herein, and these message flowcharts, scenarios, and flowcharts may be combined with one another in part or wholly.

[0255] A step or block representing the processing of information may correspond to a circuit that can be configured to perform specific logic functions of the method or technique described herein. Alternatively or additionally, a step or block representing the processing of information may correspond to a module, segment, or part of program code (including related data). The program code may include one or more instructions executable by a processor to implement specific logic operations or actions in the method or technique. The program code and / or related data may be stored on any type of computer-readable medium, such as a storage device including RAM, a disk drive, a solid-state drive, or other storage media.

[0256] Computer-readable media may also include non-transient computer-readable media that store data for a short period, such as register memory and processor cache. Non-transient computer-readable media may further include non-transient computer-readable media that store program code and / or data for a long period. Thus, non-transient computer-readable media may include secondary or permanent long-term storage such as, for example, ROM, optical or magnetic disks, solid-state drives, or CD-ROM (compact disc read-only memory). Non-transient computer-readable media may also be any other volatile or non-volatile storage systems. Non-transient computer-readable media may be considered as computer-readable storage media, for example, as tangible storage devices.

[0257] Additionally, a step or block representing one or more information transmissions may correspond to information transmissions between software and / or hardware modules on the same physical device. However, other information transmissions may be between software modules and / or hardware modules on different physical devices.

[0258] The specific arrangements shown in the drawings should not be considered as limitations. It should be understood that other embodiments may include more or fewer of each element shown in the given drawings. Additionally, some of the illustrated elements may be combined or omitted. Furthermore, exemplary embodiments may include elements not shown in the drawings.

[0259] Although various aspects and embodiments are disclosed herein, other aspects and embodiments will be obvious to those skilled in the art. The various aspects and embodiments disclosed herein are for illustrative purposes only and are not intended to be limiting, and their true scope is indicated by the following claims.

Claims

Claim 1 delete Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 delete Claim 6 delete Claim 7 delete Claim 8 delete Claim 9 delete Claim 10 delete Claim 11 delete Claim 12 A step of acquiring an incident log containing an event indicating a symptom — said symptom describes a problem experienced by a user —; a step of performing a comparison between said event and a plurality of symptom clusters within a symptom cluster space — said plurality of symptom clusters represent symptoms associated with events in a plurality of previously acquired incident logs —; a step of identifying a symptom cluster from the symptom cluster space based on said comparison; a step of selecting an investigation step cluster from an investigation step cluster space based on said symptom cluster — said investigation step cluster is associated with one or more root cause clusters from a root cause cluster space, said investigation step cluster space is derived from said events in the plurality of previously acquired incident logs, and said root cause cluster space is also associated with said events in the plurality of previously acquired incident logs —; a step of performing an investigation step associated with said investigation step cluster by a computing device in response to selecting said investigation step cluster; a step of determining that performing said investigation step from said investigation step cluster led to the identification of the root cause of said symptom — said root cause is from one of the root cause clusters — said A method comprising: a step of selecting a solution cluster from a solution cluster space based on a root cause, wherein the solution cluster includes a solution describing measures taken to correct the root cause; and a step of performing the measures associated with the solution by the computing device in response to the selection of the solution cluster. Claim 13 In claim 12, the step of performing the comparison between the event and the plurality of symptom clusters comprises determining a similarity matrix between each of the event and the plurality of symptom clusters. Claim 14 In claim 13, the step of identifying the symptom cluster from the symptom cluster space comprises selecting the symptom cluster because it is most similar to the event in relation to the similarity matrix. Claim 15 A method according to claim 12, further comprising, prior to performing the comparison between the event and the plurality of symptom clusters, the step of classifying the event into a symptom event class using a classifier pre-trained on a corpus of labeled events from the incident logs — wherein the labels of the labeled events indicate respective event classes, and the classifier has learned the associations between the content of the labeled events in the incident logs and the respective event classes. Claim 16 In paragraph 12, the investigation step cluster is selected because it has the highest probability of leading to one of the root cause clusters within the investigation step cluster space. Claim 17 In paragraph 12, the method wherein the investigation step cluster is selected because it has the highest probability of reducing the number of candidate root cause clusters within the investigation step cluster space. Claim 18 delete Claim 19 In claim 12, the solution comprises causing a computing device to change its configuration, change one or more applications it is running, or reboot. Claim 20 delete Claim 21 In claim 12, the method wherein the computing device comprises a virtual agent. Claim 22 A non-transient computer-readable medium storing program instructions that cause said computing system to perform operations when executed by said computing system, said operations comprising: an operation of acquiring an incident log containing an event indicating a symptom — said symptom describes a problem experienced by a user —; an operation of performing a comparison between said event and a plurality of symptom clusters within a symptom cluster space — said plurality of symptom clusters represent symptoms associated with events in a plurality of previously acquired incident logs —; an operation of identifying a symptom cluster from said symptom cluster space based on said comparison; and an operation of selecting an investigation step cluster from an investigation step cluster space based on said symptom cluster — said investigation step cluster is associated with one or more root cause clusters from a root cause cluster space, said investigation step cluster space is derived from said events in the plurality of previously acquired incident logs, and said root cause cluster space is also associated with said events in the plurality of previously acquired incident logs —; A non-transient computer-readable medium comprising: an action of performing an investigation step associated with said investigation step cluster by a computing device in response to selecting said investigation step cluster; an action of determining that performing said investigation step from said investigation step cluster led to the identification of a root cause of said symptom ― said root cause is from one of the root cause clusters ―; an action of selecting a solution cluster from a solution cluster space based on said root cause ― said solution cluster includes a solution describing measures taken to correct said root cause ―; and an action of performing said measures associated with said solution by the computing device in response to selecting said solution cluster. Claim 23 In paragraph 22, the operation of performing the comparison between the event and the plurality of symptom clusters comprises determining a similarity matrix between each of the event and the plurality of symptom clusters, in a non-transient computer-readable medium. Claim 24 In paragraph 23, the operation of identifying the symptom cluster from the symptom cluster space comprises selecting the symptom cluster because it is most similar to the event in relation to the similarity matrix, in a non-transient computer-readable medium. Claim 25 A non-transient computer-readable medium, further comprising program instructions that, when executed by a computing system, cause the computing system to perform operations including classifying the event into a symptom event class using a classifier pre-trained on a corpus of labeled events from the incident logs prior to performing the comparison between the event and the plurality of symptom clusters — wherein the labels of the labeled events indicate respective event classes, and the classifier has learned the associations between the contents of the labeled events in the incident logs and the respective event classes. Claim 26 In paragraph 22, the investigation step cluster is a non-transient computer-readable medium selected because it has the highest probability of leading to one of the root cause clusters within the investigation step cluster space. Claim 27 In paragraph 22, the investigation step cluster is a non-transient computer-readable medium selected because it has the highest probability of reducing the number of candidate root cause clusters within the investigation step cluster space. Claim 28 Acquire an incident log containing an event indicating a symptom — said symptom describes a problem experienced by a user —; perform a comparison between said event and a plurality of symptom clusters within a symptom cluster space — said plurality of symptom clusters represent symptoms associated with events in a plurality of previously acquired incident logs —; identify a symptom cluster from the symptom cluster space based on said comparison; select an investigation step cluster from the investigation step cluster space based on said symptom cluster — said investigation step cluster is associated with one or more root cause clusters from a root cause cluster space, said investigation step cluster space is derived from said events in the plurality of previously acquired incident logs, and said root cause cluster space is also associated with said events in the plurality of previously acquired incident logs —; A system comprising one or more processors configured to perform the investigation steps associated with the investigation step cluster by the computing device in response to selecting the investigation step cluster; to determine that performing the investigation step from the investigation step cluster led to the identification of the root cause of the symptom — said root cause is from one of the root cause clusters —; to select a solution cluster from a solution cluster space based on said root cause — said solution cluster includes a solution describing measures taken to correct said root cause —; and to perform said measures associated with said solution by the computing device in response to selecting the solution cluster. Claim 29 A system according to claim 28, wherein performing the comparison between the event and the plurality of symptom clusters includes determining a similarity matrix between each of the event and the plurality of symptom clusters. Claim 30 In claim 29, a system for identifying the symptom cluster from the symptom cluster space comprises selecting the symptom cluster because it is most similar to the event in relation to the similarity matrix. Claim 31 A system according to claim 28, wherein the one or more processors further configure the system to classify the event into a symptom event class using a classifier pre-trained on a corpus of labeled events from the incident logs prior to performing the comparison between the event and the plurality of symptom clusters — wherein the labels of the labeled events indicate respective event classes, and the classifier has learned the associations between the contents of the labeled events in the incident logs and the respective event classes. Claim 32 In paragraph 28, the above investigation step cluster is selected because it has the highest probability of leading to one of the root cause clusters within the above investigation step cluster space, in a system. Claim 33 In paragraph 28, the above investigation step cluster is selected because it has the highest probability of reducing the number of candidate root cause clusters within the above investigation step cluster space, in a system.

Images