Diagnostic Framework in Computing Systems

Abstract

Embodiments of the present disclosure relate to a diagnostic framework in a computing system. Various techniques for diagnosis, troubleshooting, and root cause analysis are disclosed herein. In one embodiment, a method includes receiving operational data from a computing system and identifying an operational mode based on the operational data. The operational mode includes a time sequence of operations, events, or system states associated with the computing system. The method also includes receiving annotations and associating the received annotations with the identified operational mode. The annotations identify a cause, reason, or corrective suggestion corresponding to a violation of the time sequence.

Description

[0001] This application is a divisional application of the invention patent application with international application number PCT / US2016 / 045445, international application date August 4, 2016, entering the Chinese national stage on February 13, 2018, Chinese national application number 201680048325.4, and invention name “Diagnostic Framework in Computing Systems”. Background Art

[0002] Cloud computing typically utilizes a group of virtual machines or other remote servers to process, manage, or store user data. A communication network can interconnect remote servers to form a computing structure. During operation, one or more remote servers of the computing structure can collaborate with each other to provide a distributed computing environment that supports the execution of various user software applications and provides other computing services. Summary of the invention

[0003] This Summary is provided to introduce in a simplified form some concepts that will be further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0004] In some cloud computing environments, a cloud service provider may provide computing facilities (e.g., tenant sites) to multiple users or tenants to execute user software applications. One difficulty with this arrangement is that the cloud service provider typically does not have access to the operating parameters, event logs, performance metrics, or other operational data of the individual tenant sites. Therefore, when a tenant experiences a service outage, performance degradation, or other operational problem, the cloud service provider may not have sufficient information to diagnose, troubleshoot, or perform root cause analysis of the operational problem.

[0005] Several embodiments of the disclosed technology provide a diagnostic system and framework that enables tenants or cloud service providers to effectively diagnose, troubleshoot, or perform root cause analysis of operational problems of tenant sites or computing structures supporting tenant sites. In certain embodiments, the diagnostic system may include an analysis component, an enhancement component, and an interpretation component that are operationally coupled to each other. The analysis component may be configured to examine historical and / or real-time operational data of at least a portion of a tenant site or computing structure. Through inspection, the diagnostic system may identify an operating mode (e.g., a sequence of states or events) or associate certain performance indicators (e.g., cache data size, network traffic, etc.) under normal operating conditions. The enhancement component may be configured to receive comments from, for example, a developer or administrator of a computing structure or tenant site. The comments may include possible causes, reasons, or predictions that lead to a violation of the correlation of an operating mode or performance indicator under normal operating conditions. The annotated operating mode and / or correlation of the performance indicator may then form an operational profile.

[0006] The explanation component can be configured to utilize the operation profile to determine whether any operation mode or performance indicator correlation is violated during operation. In response to determining that one of the operation modes or one of the performance indicator correlations is violated, the explanation component can be configured to determine the cause, possible reason, or prediction associated with the violation based on the annotations in the operation profile. The explanation component can also associate the period of violation with the period when the tenant experienced the operation problem. In this way, the diagnostic system can effectively and automatically identify the possible cause, reason, or prediction corresponding to the operation problem experienced by the tenant. In other embodiments, the explanation component can also be configured to provide mitigation suggestions, automatic operation adjustments, and / or perform other suitable operations based on the identified possible cause, reason, or prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a schematic diagram illustrating a computing framework having a diagnostic system configured according to an embodiment of the disclosed technology.

[0008] Figure 2 It is shown to be suitable for Figure 1 A block diagram of a diagnostic system and software components according to an embodiment of the disclosed technology.

[0009] Figure 3 and Figure 4 are respectively suitable for illustrating embodiments according to the disclosed technology Figure 2 Block diagram of example software modules of the analysis component and the interpretation component.

[0010] Figure 5 is a flow chart illustrating an embodiment of a process for performing operational diagnostics according to an embodiment of the disclosed technology.

[0011] Figure 6 is a flow chart illustrating an embodiment of a process for generating an operating mode according to an embodiment of the disclosed technology.

[0012] Figure 7 is a flow chart illustrating an embodiment of a process for performing interpretation of operational data according to an embodiment of the disclosed technology.

[0013] Figure 8 is a schematic diagram illustrating an example mode of operation according to an embodiment of the disclosed technology.

[0014] Fig. 9 is a schematic diagram illustrating an example graph of performance indicators according to an embodiment of the disclosed technology.

[0015] Fig.10 is a schematic diagram illustrating an example mode of operation with enhanced annotations according to an embodiment of the disclosed technology.

[0016] Figures 11A-11C is a diagram illustrating example correlations of operational issues to system health levels according to an embodiment of the disclosed technology.

[0017] Fig.12 Is suitable for Figure 1 A computing device that is a component of a computing framework. DETAILED DESCRIPTION

[0018] Certain embodiments of systems, devices, components, modules, routines, and processes for diagnosis, troubleshooting, or root cause analysis in computing systems are described below. In the following description, specific details of the components are included to provide a thorough understanding of certain embodiments of the disclosed technology. Those skilled in the relevant art will also understand that the disclosed technology may have additional embodiments or may be used without reference to the disclosed technology. Figure 1-12 The embodiments described are practiced without the need for several details.

[0019] As used herein, the term "computing fabric" generally refers to a computing system having multiple nodes interconnected in a "weave" or other suitable configuration via a communication network. A "node" in a computing fabric generally refers to a computing element. In some embodiments, a computing element may include a physical server having one or more computing processors, memory devices, and / or peripheral devices. In other embodiments, a computer element may include a virtual machine or other suitable virtualized entity having a virtualized processor, memory, and / or storage device. A physical computer network and / or one or more virtual computer networks may interconnect the nodes of the computing fabric.

[0020] The term "tenant site" as also used herein generally refers to a distributed computing environment provided by a computing structure that supports the execution of various user software applications in order to provide computing services to users or tenants. A tenant site may be hosted on one or more nodes of the computing structure and may be movable between nodes of the computing structure. In some embodiments, a tenant site may include one or more virtual machines. In other embodiments, a tenant site may include a virtual network controller and / or other suitable components.

[0021] As used herein, the term "operational mode" generally refers to a temporal sequence of operations observed, for example, at a tenant site. For example, an operation mode may include the following sequence: receiving a user query, searching a database based on the user query, and transmitting a response to the user query. Optionally, the operation mode may also include a time interval between each pair of operations in the operation sequence. In the foregoing example, the operation mode may also include a first time interval (e.g., 20 milliseconds) between receiving the user query and searching the database and a second time interval (e.g., 1 second) between searching the database and transmitting the response.

[0022] The term "performance indicator" as also used herein generally refers to one or more operating parameters each corresponding to a performance level of a computing system. Example performance indicators may include the size of available memory, the size of a data cache, a percentage of network utilization, the number of service requests over a period of time, and the like. A performance indicator may be related to one or more other performance indicators. In certain embodiments, a correlation coefficient may be calculated for a pair of performance indicators. The correlation coefficient may have a value (e.g., -1 to 1) for indicating whether a pair of performance indicators are related (e.g., having a value of approximately 1) or unrelated (e.g., having a value of approximately -1). In other embodiments, paired performance indicators may be related in other suitable ways.

[0023] As used herein, "annotations" to the relevance of an operating mode or performance indicator generally refer to input received from, for example, a developer or administrator of a computing structure or tenant site. In some embodiments, the annotations may include possible reasons, justifications, or predictions when the relevance of a particular operating mode or performance indicator is violated. In other embodiments, the annotations may also include suggested mitigation or corrective actions for the violation. For example, if no database search is performed within a period after receiving a user query, then in the above example, an annotation may be added to the operating mode that the database server is offline. The term "operational profile" also used herein generally refers to a data record of the relevance of operating modes and / or performance indicators to corresponding annotations.

[0024] In some cloud computing environments, a cloud service provider may provide distributed computing facilities or tenant sites to provide desired computing services. However, the cloud service provider typically does not have access to event logs, performance metrics, or other operational data at individual tenant sites because the tenant may control the corresponding tenant site. Therefore, when a tenant requests assistance from a cloud service provider for a service outage or other operational problem, the cloud service provider may not have sufficient or any information to diagnose, troubleshoot, or perform root cause analysis of the operational problem. Several embodiments of the disclosed technology may provide a diagnostic system and framework that enables a tenant or cloud service provider to effectively diagnose, troubleshoot, or perform root cause analysis of an operational problem, as described in more detail below.

[0025] Figure 1 is a schematic diagram showing a computing framework 100 having a diagnostic system configured according to an embodiment of the disclosed technology. Figure 1As shown, computing framework 100 may include client devices 102 (individually identified as first client device 102a and second client device 102b), computing structure 104, and diagnostic system 108 interconnected by computer network 103. Computer network 103 may include the Internet, a local area network, a metropolitan area network, a wide area network, and / or other suitable types of networks. For illustration purposes, Figure 1 Specific components of computing framework 100 are shown in FIG. In other embodiments, computing framework 100 may also include additional and / or different computing, networking, or other suitable types of components.

[0026] Client devices 102 may each include a desktop computer, a laptop computer, a tablet computer, a smart phone, and / or other suitable types of computing devices. Figure 1 As shown, the first client device 102a may include a desktop computer. The second client device 102b may include a laptop computer. In other embodiments, the client device 102 may also include other suitable hardware and / or software components. In any of the foregoing embodiments, the client device 102 may enable users or tenants 101 (individually identified as the first tenant 101a and the second tenant 101b) to access corresponding tenant sites 142 provided by the computing structure 104, as described in more detail below.

[0027] like Figure 1 As shown, the computing structure 104 may include a plurality of servers 106 (individually identified as a first server 106a and a second server 106b), each server 106 having a processor 132, a memory 134, and an input / output component 136 operatively coupled to each other. The processor 132 may include a microprocessor, a field programmable gate array, and / or other suitable logic device. The memory 134 may include volatile and / or non-volatile media (e.g., ROM; RAM, magnetic disk storage media; optical storage media; flash memory devices, and / or other suitable storage media) and / or other types of computer-readable storage media configured to store data received from the processor 132 and instructions for the processor 132. The input / output component 136 may be configured to accept input from an operator and / or an automated software controller (not shown) and provide output thereto.

[0028] The first server 106a and the second server 106b may individually contain instructions in the memory 134 that, when executed by the processor 132, cause the individual processors 132 to provide a hypervisor 140 (individually identified as a first hypervisor 140a and a second hypervisor 140b) and an operation monitor 141 (individually identified as a first operation monitor 141a and a second operation monitor 141b). The hypervisor 140 and the operation monitor 141 are Figure 1 140 and / or the operation monitor 141 may also include firmware and / or hardware components. Figure 1 106a or 106b, the operation monitor 141 may be a part of the hypervisor 140 or an operating system (not shown) executing on the corresponding first server 106a or second server 106b.

[0029] The hypervisor 140 may be individually configured to generate, monitor, terminate, and / or otherwise manage one or more virtual machines 144 organized as tenant sites 142. For example, Figure 1 As shown, the first server 106a may provide a first hypervisor 140a for managing a first tenant site 142a and a second tenant site 142b, respectively. The second server 106b may provide a second hypervisor 140b for managing a first tenant site 142a' and a second tenant site 142b', respectively. The tenant sites 142 may each include a plurality of virtual machines 144 associated with a particular tenant 101. For example, the first server 106a and the second server 106b may host tenant sites 142a and 142a' for the first tenant 101a, and may host tenant sites 142b and 142b' for the second tenant 101b. Each virtual machine 144 may execute a corresponding operating system, middleware, and / or application.

[0030] Still Figure 1 As shown in FIG. 1 , the computing structure 104 may also include one or more virtual networks 146 that interconnect tenant sites 142a and 142b across multiple hosts 106. For example, a first virtual network 142a interconnects first tenant sites 142a and 142a' at a first server 106a and a second server 106b. A second virtual network 146b interconnects second tenant sites 142b and 142b' at a first server 106a and a second server 106b. Although a single virtual network 146 is shown as corresponding to one tenant site 142, in other embodiments, multiple virtual networks 146 (not shown) may be configured to correspond to a single tenant site 146.

[0031] The operation monitor 141 can be configured to monitor the operation data 112 of the corresponding tenant site 142 and / or computing structure 104. In some embodiments, the operation data 112 can include the operation mode and / or performance indicators of the tenant site 142 or computing structure 104. For example, the operation data 112 can include warning logs, event logs, operation reports, and / or other suitable data from the tenant site 142 and / or computing structure 104. In some embodiments, the operation monitor 141, the hypervisor 140, or other suitable components of the server 106 can also store the previously acquired operation data 112 as historical data 116 in, for example, the memory 134 of the server 106. The historical data 116 can then be provided to the diagnostic system 108 for offline analysis. In other embodiments, the operation monitor 141 can be configured to transmit the operation data 112 to the diagnostic system 108 for online analysis and / or monitoring. Based on the operational data 112 and / or the historical data 116 , the diagnostic system 108 may be configured to generate an operational profile and monitor the operation of the tenant site 142 and / or the computing structure 104 , as described in greater detail below.

[0032] like Figure 1 As shown, the diagnostic system 108 may include a processor 142, a memory 144, and an input / output component 146 that are operatively coupled to each other. The processor 142 may include a microprocessor, a field programmable gate array, and / or other suitable logic devices. The memory 144 may include a memory configured to store data received from the processor 142 and instructions for the processor 142 (e.g., for executing the following reference Figure 5-7 The computer readable storage medium may include volatile and / or non-volatile media (e.g., ROM; RAM, magnetic disk storage media; optical storage media; flash memory devices, and / or other suitable storage media) and / or other types of computer readable storage media that may be used to store instructions for the methods discussed. The input / output components 136 may include a display, touch screen, keyboard, mouse, printer, and / or other suitable types of input / output devices configured to accept input from and provide output to an operator and / or an automated software controller (not shown).

[0033] The diagnostic system 108 may be configured to perform analysis of the operational data 112 and / or historical data 116 on an offline or online basis. In one embodiment, the diagnostic system 108 may parse the operational data 112 and identify operational modes of the tenant sites 142 and / or computing structures 104. Figure 8 An example operating mode is described in more detail. In another embodiment, the diagnostic system 108 can obtain performance indicators from the operational data 112 and calculate correlation coefficients for the performance indicators. Fig. 9One example correlation of performance indicators is described in further detail. In further embodiments, the diagnostic system 108 may also be configured to perform a combination of the foregoing analyses and / or other suitable analyses.

[0034] The diagnostic system 108 may also be configured to receive annotations for one or more identified operating modes or correlations of performance indicators. An annotation may be associated with each operation in an operating mode or a group of operations in an operating mode. Similarly, an annotation may be associated with the correlation of one or more performance indicators. Fig.10 An example of receiving annotations for operating modes is described in more detail. Then, the correlation between the operating mode and the performance indicator and the corresponding annotations can be stored together as a performance profile in, for example, the database 110 ( Figure 2 shown).

[0035] The diagnostic system 108 can also be configured to monitor the operation of the tenant site 142 and / or the computing structure 104 based on the generated performance profile. For example, in some embodiments, the diagnostic system 108 can receive the additional operating data 112 and perform analysis thereon to identify one or more additional operating modes and / or correlations of performance indicators. The diagnostic system 108 can then compare the additional operating modes and / or correlations of performance indicators with those in the performance profile.

[0036] Based on the comparison, the diagnostic system 108 can also determine and / or quantify the health level of the tenant site 142 or computing structure 104. For example, in one embodiment, a first number (e.g., 100) of user queries can be received in a period of time. If the received second number (e.g., 99) of user queries are processed according to an operating mode that is generally similar to those in the performance profile, the diagnostic system 108 can be configured to calculate a health level (e.g., 99%) based on the first number and the second number.

[0037] After determining the current health level, in some embodiments, the diagnostic system 108 can be configured to issue a warning or generate an event log if the current health level drops below a threshold. For example, the diagnostic system 108 can generate an email, text message, or other suitable message reporting the violation and / or health level to an administrator. In other embodiments, the diagnostic system can also be configured to provide recommendations for corrective actions to mitigate violations or improve health levels as results 114 based at least in part on annotations associated with the operating mode and / or correlation of the performance indicator. For example, the results 114 can include recommendations to increase the network bandwidth allocated to the tenant 101, allocate more memory space to a particular virtual machine 144 or tenant site 142, and / or other suitable actions.

[0038] In another embodiment, the diagnostic system 108 can be configured to perform root cause analysis based on reported operational issues and observed violations. In one embodiment, the diagnostic system 108 can be configured to aggregate operational issues (e.g., request timeouts, cache server unavailability, etc.) over a period of time to generate an operational problem timeline. The diagnostic system 108 can also be configured to aggregate observed violations over the same period of time to generate a system problem timeline. The diagnostic system 108 can then align, juxtapose, or compare the generated operational problem timeline with the system problem timeline to correlate the observed operational issues with the violations. After identifying the violations associated with the operational issues, the diagnostic system 108 can be configured to determine the root cause related to the observed operational issues and / or provide mitigation suggestions based on the annotations associated with the violations. Refer to the following Figures 11A-11C Describe an example of root cause analysis in more detail.

[0039] In yet other embodiments, the diagnostic system 108 may also be configured to predict a deterioration, recovery, repetition, or other future condition of the operation of the tenant site 142 and / or the computing structure 104. For example, in one embodiment, the diagnostic system 108 may predict an imminent network or server failure based on a violation of a correlation of an operating mode or performance metric. In another embodiment, the diagnostic system 108 may also predict that despite the detection of a violation, operation may return to normal. In yet other embodiments, the diagnostic system 108 may be configured to automatically adjust the operation of the tenant site 142 and / or the operation of the computing structure 104. For example, the diagnostic system 108 may be configured to request a fabric controller (not shown) of the computing structure 104 to allocate additional network bandwidth, memory space, or other resources to a particular tenant site 142.

[0040] In operation, tenant 101 can access computing facilities at corresponding tenant site 142 with client device 102 separately via computer network 103. During operation, operation monitor 141 or other suitable components of computing structure 104 can acquire and / or record operation data 112. Operation data 112 or any previously acquired historical data 116 can be provided to diagnostic system 108 for analysis. In some embodiments, diagnostic system 108 can perform offline analysis of historical data 116 to generate performance profile as described above. In other embodiments, diagnostic system 108 can perform online analysis of operation data 112 to generate performance profile. In other embodiments, diagnostic system 108 can also perform additional online or offline analysis of operation data 112 to update and / or otherwise modify performance profile on a continuous or periodic basis. As described above, diagnostic system 108 can then perform operation monitoring of tenant site 142 and / or computing structure 104 based on the generated performance profile.

[0041] Even though the diagnostic system 108 is shown as Figure 1 104 in the computing structure 104, but in other embodiments, the diagnostic system 108 can be part of the computing structure 104. For example, the diagnostic system 108 can be hosted on one or more servers 106 in the computing structure 104. In other embodiments, the diagnostic system 108 can also be part of the tenant site 142 or the hypervisor 140. In addition, the memory 144 of the diagnostic system 108 can contain instructions that, when executed by the processor 142, cause the processor 142 to provide various software components for performing one or more of the aforementioned functions of the diagnostic system 108. Reference Figure 2 One example of such a software component is described in more detail.

[0042] Figure 2 It is shown that it is suitable for Figure 1 A block diagram of a diagnostic system and software components according to an embodiment of the disclosed technology. Figure 2 In the drawings and other figures herein, individual software components, objects, classes, modules and routines can be computer programs, processes or procedures written as source code in C, C++, Java and / or other suitable programming languages. Components can include, but are not limited to, one or more modules, objects, classes, routines, attributes, processes, threads, executable files, libraries or other components. Components can be in source code or binary form. Components can include the following aspects: source code before compilation (e.g., classes, attributes, processes, routines), compiled binary units (e.g., libraries, executable files) or components (e.g., objects, processes, threads) that are instantiated and used at runtime.

[0043] Components within a system can take different forms within the system. As an example, a system including a first component, a second component, and a third component may include, but is not limited to, a system in which the first component is a property of source code, the second component is a binary compiled library, and the third component is a thread created at runtime. A computer program, process, or process may be compiled into an object, an intermediate, or a machine code, and is presented for execution by one or more processors of a personal computer, a network server, a laptop computer, a smart phone, and / or other suitable computing devices. Likewise, a component may include a hardware circuit. Those of ordinary skill in the art will recognize that hardware may be considered as solidified software, and software may be considered as liquefied hardware. To give just one example, software instructions in a component may be burned into a programmable logic array circuit, or may be designed as a hardware circuit with a suitable integrated circuit. Likewise, hardware may be simulated by software. Various implementations of source, intermediate, and / or target code and associated data may be stored in a computer memory, including a read-only memory, a random access memory, a disk storage medium, an optical storage medium, a flash memory device, and / or other suitable computer-readable storage media in addition to a propagation signal.

[0044] like Figure 2 As shown, the processor 142 of the diagnostic system 108 may include an analysis component 152, an enhancement component 154, and an interpretation component 156 that are operatively coupled to one another. Figure 2 Specific components of processor 142 are shown in FIG. 1 , but in other embodiments, processor 142 may also include input / output, networking, communication, and / or other suitable types of components. In other embodiments, at least one of analysis component 152, enhancement component 154, or interpretation component 156 may be provided by other processing components other than processor 142 of diagnostic system 108 (e.g., Figure 1 The virtual machine 144) is provided.

[0045] like Figure 2 As shown, the analysis component 152, the enhancement component 154, or the interpretation component 156 may also be operatively coupled to a database 110 containing records of the operational profiles 122. In one embodiment, the database 110 may reside in the memory 144 ( Figure 1 In other embodiments, database 110 may reside in computing structure 104 ( Figure 1 )'s network storage server (not shown) and / or other suitable storage components, or an external computing system.

[0046] like Figure 2 As shown, the analysis component 152 can be configured to receive and process information related to the tenant site 142 ( Figure 1) or at least a portion of the computing structure 104. In some embodiments, the analysis component 152 can be configured to analyze the operation data 112 and / or historical data 116 related to the operation type based on, for example, a user input 117 ( Figure 3 1 (shown) to identify certain computing, network, or other suitable types of operations contained in operational data 112 and / or historical data 116. Based on the identified operations, analysis component 152 can then determine one or more sequences of operations and time intervals therebetween.

[0047] In other embodiments, the analysis component 152 may also be configured to identify one or more performance indicators contained in the operational data 112 and / or the historical data 116. The analysis component 152 may then determine one or more correlation values ​​between pairs of performance indicators. Based on the correlation values, the analysis component 152 may indicate whether a first operational indicator (e.g., available memory size) is correlated with a second operational indicator (e.g., cache data size). The analysis component 152 may then provide the determined correlation of the one or more operational sequences and performance indicators as an operational mode 120 to the enhancement component 154 for further processing. Figure 3 One example of a software module suitable for analysis component 152 is described in more detail.

[0048] The enhancement component 154 may be configured to receive the annotation 118 and associate the received annotation 118 with the corresponding operation and / or dependency in the operation mode 120. In one embodiment, the annotation 118 may include possible causes that may lead to a violation of the sequence of operations in the operation mode 120. For example, the annotation 118 may identify that network traffic is the cause of the delay in the queuing of the received user request. In another embodiment, the annotation 118 may also include possible reasons that may lead to a violation of the identified dependency of the performance indicator in the operation mode 120. For example, the annotation 118 may identify that the possible reason for the continuous reduction of available memory in view of the usually constant cache data size is a memory leak. In another embodiment, the annotation 118 may also include possible mitigation, remediation, or other suitable data associated with the dependency of the sequence of operations or performance indicators in the violation operation mode 120. The enhancement component 154 may be configured to combine the operation mode 120 with the corresponding annotation 118 as an operation profile 122 to be stored in the database 110 and / or to be provided to the interpretation component 156.

[0049] In some embodiments, interpretation component 156 may be configured to receive operational data 112 in real time and monitor the operation of tenant site 142 or computing structure 104 in real time. In other embodiments, interpretation component 156 may also be configured to receive historical data 116 and analyze potential causes for operational problems. In other embodiments, interpretation component 156 may be configured to analyze operational data 112 and / or historical data 116 in other suitable ways.

[0050] In any of the foregoing embodiments, the explanation component 156 can be configured to determine one or more operating modes from the operating data 112 and / or the historical data 116. The explanation component 156 can then compare the determined operating modes to corresponding operating modes in the operating profile 122. In response to determining that an operating mode is violated, the explanation component 156 can be configured to look up the associated annotations 118 to provide justification, root causes, and / or mitigation suggestions. In other embodiments, the explanation component 156 can also be configured to calculate a health level of the tenant site 142 or computing structure 104 based on the number of violations of the detected operating mode, the total number of operations, and / or other suitable parameters. The explanation component 156 can also receive reported tenant issues 113, which include tenant 101 ( Figure 1 ) experienced. The explanation component 156 can then aggregate the tenant issues 113 over a period of time and compare the aggregated tenant issues 113 with violations of the operational mode detected over the same period of time. The explanation component 156 can provide reasons, root causes, and / or mitigation suggestions for the reported tenant issues 113. Figure 4 One example of a software module suitable for explaining component 156 is described in more detail.

[0051] Figure 3 is a diagram showing an embodiment of the disclosed technology suitable for Figure 2 A block diagram of example software modules of the analysis component 152 of FIG. Figure 3 As shown, the analysis component 152 may include an input module 160, a calculation module 166, a reporting module 162, and a pattern module 164 that are operatively coupled to one another. Figure 3 Specific modules are shown in FIG. 1 , but in other embodiments, analysis component 152 may also include output, network, or other suitable types of modules.

[0052] The input module 160 may be configured to receive the operational data 112, the historical data 116, and the user input 117. The input module 160 may also be configured to identify one or more events, problems, exceptions, states, performance indicators, or other suitable operational conditions contained in the operational data 112 and / or the historical data 116. For example, in one embodiment, the input module 160 may generate operational conditions based on a set of rules contained in the user input 117. The set of conditional rules may include one or more keywords, strings, or combinations thereof. For example, example keywords or strings include "cache service stopped", "unable to create a cache instance", "network timeout", etc. An example conditional rule associated with a cache operational event may be as follows:

[0053]

[0054] In the above example, when the state name "CacheInstanceIssue" is identified in the operational data 112 and / or the historical data 116, a pattern for failure to create a cache instance can be identified. Based on this set of conditional rules, the input module 160 can then scan the operational data 112 and / or the historical data 116 to generate a set of operational features with corresponding timestamps. In other embodiments, the above functions can also be accomplished via machine learning, regular expressions, or other suitable techniques.

[0055] The input module 160 can also be configured to identify a transition of system health from, for example, a healthy state to an unhealthy state, or vice versa, based on a set of health rules. The set of health rules can include one or more health conditions. For example, in one embodiment, the following health rule illustrates a health condition including an event that a cache service is started:

[0056]

[0057] If this health condition is met, the input module 160 can be configured to identify a transition of system health from an unhealthy state to a healthy state. In another example, the input module 160 can indicate a transition from a healthy state to an unhealthy state when a health condition of the CPU being under full load is met.

[0058] The input module 160 may also be configured to map certain performance indicators to system health states. For example, in one embodiment, the input module 160 may indicate a transition from a healthy state to an unhealthy state when the available memory size is less than a threshold (e.g., 50 MB); the number of memory page errors per second exceeds another threshold (e.g., 600 per second); etc. In other embodiments, the input module 160 may also map other performance indicators to other suitable system health states.

[0059] The calculation module 166 may include routines configured to perform certain types of calculations to support the operation of other modules. In one example, the calculation module 166 may include a counter that accumulates the number of transitions of operating conditions or health states. In another example, the calculation module 166 may include an accumulation routine that calculates the duration of a persistent unhealthy state. In another example, the calculation module 166 may be configured to calculate a correlation coefficient between a pair of performance indicators. In another example, the calculation module 166 may include linear regression, polynomial regression, interpolation, extrapolation, and / or other suitable subroutines.

[0060] The pattern module 164 may be configured to determine a temporal sequence of correlations of operating conditions and / or performance indicators. In one embodiment, the pattern module 164 may identify a sequence of events based on the processed operational data 112 and / or historical data 116 from the input module 160. An example routine (“TemporalRule”) for determining a temporal sequence may be as follows:

[0061]

[0062]

[0063] The above example routine can be run from a tenant site 142 ( Figure 1 ) or computing structure 104 ( Figure 1 ) is automatically derived from one or more successful stress, function or performance runs of the operation data 112. As shown above, the "RuleEvent" routine is configured to identify the sequence of events "Put", "ReplicationQueued", "Pending" and "ReplicationAcked" indicated in the operation data 112 and / or the historical data 116. The "TimeToNexEvent" routine is configured to determine the time interval between the above events in units of, for example, milliseconds or seconds. The above example specifies that the time interval between "Put" to "ReplicationQueued" should not exceed 5 milliseconds. The "ViolationClass" routine is configured to define certain violation classes when the above identified sequence is violated.

[0064] In another embodiment, the pattern module 164 may be configured to indicate the correlation of paired performance metrics based on, for example, the calculated correlation coefficients from the computing module 166. For example, in one embodiment, the correlation module 162 may indicate that the available memory size is inversely correlated with the data cache size based on a correlation coefficient of approximately -1. In another example, the pattern module 164 may indicate that the network utilization level is correlated with the number of service requests based on a correlation coefficient of approximately 1. An example routine for correlated performance metrics may be as follows:

[0065]

[0066] As shown above, the example routine defines that the performance metrics "network bandwidth usage", "cache bandwidth usage", and "average secondary response time" should be correlated with a correlation coefficient value of 0.998 in the range of (-1 to 1). If this correlation is violated, the violation is associated with the violation class "NetworkBottleNecks$ExternalProcessUsage".

[0067] The pattern module 164 may then combine the sequence of the operating conditions and / or the correlation of the performance metrics into a record of the operating mode 120 in an XML file, a spreadsheet, or other suitable type of data file. The pattern module 164 may also be configured to provide the operating mode 120 to the enhancement component 154.

[0068] Then, the enhancement component 154 may associate the annotation 118 with the operating mode 120 to generate the operation profile 122. In one embodiment, each health rule may be associated with an annotation 118 representing a violation class, thereby indicating the reason for the rule violation. The reason may be specific to the user software application or general for the distributed computing environment in the computing structure 104. For example, cache operations may have failure reasons such as excessive eviction, placement problems, memory cleaning problems, etc. Example general reasons may include trash, the system is paging, low RAM, network throttling, etc. The above example sequence may be associated with the annotation 118 as follows:

[0069] <a:string> ToomuchLoad$Contentions$GCIssues< / a:string>

[0070] (<a: string>Too much load$Competition$Garbage collection issues< / a: string>)

[0071] This previous annotation 118 indicates that if there is a violation of the above example sequence at "ReplicationQueued" (the second event in the time series), one of the reasons is too much load, competition, or garbage collection issues ("GCIssues").

[0072] When a health rule is violated, a health state may be indicated with a corresponding mitigation suggestion. For example, if the number of memory page errors exceeds a threshold, memory thrashing may be indicated, and the mitigation suggestion to avoid memory thrashing is to increase memory allocation. In other embodiments, each violation class may be associated with multiple mitigation suggestions, which may be provided to the detected health state based on a ranking of effectiveness, cost, or other suitable criteria. For example, each failure reason above may be associated with a corrective action, which may include a description of what a user is to do and / or an input to an automated system (e.g., a fabric controller associated with the computing structure 104) that takes corrective action (e.g., increase network bandwidth limits, terminate unneeded applications, etc.).

[0073] The reporting module 162 may be configured to summarize problems discovered when analyzing the operational data 112 and / or the historical data 116. The reporting module 162 may aggregate the identified operational conditions based on, for example, server-side problems, client-side problems, or based on other suitable criteria. For example, the reporting module 162 may be configured to show problems detected during a particular time period, such as, for example, the percentage of request timeouts, serialization exceptions, replication queue full, etc. In other embodiments, the reporting module 162 may also provide a summary of the system health during the same time period. For example, the reporting module 162 may identify that a crash has been identified, a network problem has been encountered, the CPU was under full load over a period of time, etc. The reporting module 162 may also identify when the system health changes from healthy to unhealthy, or vice versa. For example, the reporting module 162 may represent the system health as a state machine as follows:

[0074] High CPU usage from 10:10 PM to 10:15 PM; and

[0075] · Partition movement occurs from 10:13 PM to 10:20 PM.

[0076] Based on the output from the reporting module 162, an administrator or operator can discover potential performance issues without having to search through large amounts of data in the event log.

[0077] Figure 4 is a diagram showing an embodiment of the disclosed technology suitable for Figure 2 15 is a block diagram of example software modules of an interpretation component 156 of the present invention. The interpretation component 156 can be configured as an offline or online system. For example, in some embodiments, the interpretation component 156 can be configured as an offline system that receives and processes the operational data 112. In other embodiments, the interpretation component 156 can also be configured as an online system as part of a user software application or a separate service that runs in parallel with the user software application.

[0078] like Figure 4 As shown, the interpretation component 156 may include components that are functionally and / or structurally similar to Figure 3 The modules of the analysis component 152 in FIG. 1 are generally similar to certain modules. Therefore, Figure 4 Similar modules in the explanation component 156 are identified with similar reference numerals. In some embodiments, the explanation component 156 may include an input module 160, a calculation module 166, an analysis module 172, and a control module 174 that are operatively coupled to each other. In other embodiments, the explanation component 156 may also include other suitable modules and / or routines.

[0079] In certain embodiments, input module 160 and computation module 166 may cooperate to determine one or more operational modes and / or performance indicator correlations based on received operational data 112. Analysis module 172 may be configured to compare the determined operational mode and / or performance indicator correlations with corresponding operational mode and / or performance indicator correlations in operational profile 122. Analysis module 172 may then determine whether there are any violations of the operational modes and / or correlations. Based on the determined violations, if any, analysis module 172 may identify reasons for such violations and, based on annotations 118 ( Figure 3 ) to provide guidance or recommendations for corrective actions.

[0080] For example, in the above reference Figure 3 In the example sequence discussed, the example sequence (i.e., Put, ReplicationQueued, Pending, and ReplicationCompleted) is violated because ReplicationCompleted is not seen. In addition, a violation of the correlation between network bandwidth usage, cache bandwidth usage, and average secondary response time is detected. Based on the above violations, the analysis module 172 can determine that the reason for these violations may be due to a network bottleneck. The analysis module 172 can then provide the analysis results to the control module 174, which is configured to generate results 114, which provide, for example, identification of the violation and any suggestions for corrective action. For example, in the above example, the results 114 may recommend reducing the load or increasing the network capacity.

[0081] In other embodiments, the analysis module 172 may also be configured to aggregate tenant issues 113 (e.g., request timeouts, cache server unavailability, etc.) over a period of time to generate an operational problem timeline. The analysis module 172 may also be configured to aggregate observed violations over the same period of time to generate a system problem timeline. The analysis module 172 may then align, juxtapose, or compare the generated operational problem timeline with the system problem timeline to correlate the observed operational problem with the violation. After identifying the violation associated with the operational problem, the diagnostic system 108 may be configured to determine the root cause of the observed operational problem and / or provide mitigation recommendations based on the annotations associated with the violation.

[0082] Figure 5 is a flow chart illustrating an embodiment of a process 200 for performing operational diagnostics according to an embodiment of the disclosed technology. Figure 1 The computational framework 100 and Figure 2-4 Process 200 is described using software components of the present invention, but in other embodiments, process 200 can be implemented in other suitable computing systems, devices, or frameworks.

[0083] like Figure 5 As shown, process 200 may include, at stage 202, based on operational data 112 and / or historical data 116 ( Figure 1 ) generates an operating pattern. In some embodiments, the operating pattern may include a time sequence of events. In other embodiments, the operating pattern may also include the time intervals between consecutive events in the time sequence. In other embodiments, the operating pattern may also include the correlation between two or more performance indicators of the computing system. Figure 6 Example operations for generating an operating mode are described in more detail.

[0084] like Figure 5 As shown, process 200 may also include generating an operational profile at stage 204. The operational profile may include the operational mode generated from stage 202 and associated annotations received from a developer, administrator, or other suitable entity. Process 200 may include interpreting the operational data based on the operational profile at stage 206. Interpreting the operational data may be performed online or offline from a corresponding computing system. Figure 7 Example operations for interpreting operational data are described in greater detail.

[0085] Figure 6 2 is a flow chart illustrating an embodiment of a process 204 for generating an operating mode according to an embodiment of the disclosed technology. Figure 6As shown, process 204 may include receiving historical data at stage 210. Then, process 204 may include parsing the historical data at stage 212. Parsing the historical data may be performed, for example, by identifying events, conditions, performance indicators based on user-provided rules, as described above with reference to Figure 2 and 3 Then, process 204 may include identifying the time series at stage 214 and correlating the performance indicators at stage 216, as described above with reference to Figure 2 and 3 Process 204 may then include outputting an operating mode at stage 218. The operating mode may include correlations of the time series and / or performance indicators generated at stages 214 and 216, respectively.

[0086] Figure 7 2 is a flow chart illustrating an embodiment of a process 206 for performing interpretation of operational data according to an embodiment of the disclosed technology. Figure 7 As shown, process 206 may include receiving operational data at stage 220. Receiving operational data may be on an online or offline basis. Then, process 206 may include determining a system health level at stage 222. In one embodiment, determining the system health level includes determining one or more violations of the correlation of the operating mode or performance indicator in the operating profile. In other embodiments, determining the system health level may also include calculating a violation ratio based on the number of operations with violations and the total number of operations in a time period.

[0087] Then, process 206 may include correlating the system health level with the reported tenant problem at stage 224. In one embodiment, the system health level may be correlated with the tenant problem by comparing the timeline of the system health level with the tenant problem. In other embodiments, the system health level may be correlated with the tenant problem by linear regression, manual comparison, and / or other suitable techniques. Then, process 206 may include outputting a possible cause and / or corrective suggestion for the tenant problem at stage 226. As described above with reference to Figure 1-4 As described, possible causes and / or corrective suggestions may be determined by looking up annotations corresponding to detected violations.

[0088] Figure 8 is a schematic diagram illustrating an example operating mode 300 of a computing system under normal operating conditions according to an embodiment of the disclosed technology. Figure 8As shown, the operation mode 300 may include one or more sequences of operations, events, or states. In the illustrated embodiment, the operation mode 300 may include a first sequence having a put operation 302, a queue operation 304, and a pending state 306. The operation mode 300 may also include a second sequence having a Put operation 302, a queue operation 304, a replication request (Repl_Req) ​​310, a replication complete state 312, a commit state 308, and an acknowledgement (ACK) 316. The operation mode 300 may also include one or more time intervals between pairs of operations or states. For example, Figure 8 As shown, the time interval between the put operation 302 and the pending state 306 is 5 milliseconds. Another time interval between the put operation 302 and the committed state 308 is 50 milliseconds for payloads up to 100KB in size. Fig.10 As described in more detail, a violation of a sequence or one or more time intervals of a sequence may be associated with one or more annotations.

[0089] Fig. 9 4 is a schematic diagram showing a graph 400 showing example performance indicators according to an embodiment of the disclosed technology. Fig. 9 As shown, example chart 400 shows a chart of available memory, cache data size, network usage, and service request metrics over time. Figure 2 and 3 As discussed in more detail, two or more performance indicators may be correlated with each other. For example, available memory appears to be inversely correlated with cache data size. Service request metrics appear to be correlated with network usage.

[0090] Fig.10 is a schematic diagram illustrating an example mode of operation with enhanced annotation according to an embodiment of the disclosed technology. Fig.10 As shown, one or more potential reasons, causes, or corrective suggestions may be associated with a violation of the sequence. For example, if a placement operation 302 is not followed by a queue operation 304, a note may be associated with this condition to indicate that a possible cause may include too much load contention in the network. In another example, if a commit operation 308 is not followed by a confirmation 316, another note may be associated with this condition to indicate that a possible cause is a problem with the placement operation, and a corrective suggestion is to modify the configuration of the placement operation.

[0091] Figures 11A-11C is a schematic diagram illustrating an example interface for correlating operational issues with system health levels according to an embodiment of the disclosed technology. Fig.11A As shown, system health levels may be aggregated and plotted as blocks or other suitable representations on a timeline. Fig.11AExample health levels shown include network throttling and CPU crashing. The aggregated system health level can also be expanded to show the details of the system profile and the corresponding severity. Fig. 11B As shown, for example, tenant 101 ( Figure 1 ) can also be aggregated and plotted on a timeline. Similarly, operational issues can also be expanded to show more details and corresponding severity. Fig. 11C As shown, Fig.11A and Fig. 11B The two timelines in can be aligned, juxtaposed, or otherwise compared to correlate operational issues with system health levels. Figure 1-4 As described in more detail, based on such comparison, possible causes or reasons and corrective suggestions for operational problems may be provided.

[0092] Fig.12 Is suitable for Figure 1 The computing device 800 may be suitable for some components of the computing framework 100. For example, the computing device 800 may be suitable for Figure 1 The client device 102, server 106 or diagnostic system 108 of the present invention. In a very basic configuration 802, the computing device 800 typically includes one or more processors 804, and a system memory 806. A memory bus 808 can be used for communication between the processor 804 and the system memory 806.

[0093] Depending on the desired configuration, the processor 804 can be of any type, including but not limited to a microprocessor (μP), a microcontroller (μC), a digital signal processor (DSP), or any combination thereof. The processor 804 can include one or more levels of cache (such as a level 1 cache 810 and a level 2 cache 812), a processor core 814, and registers 816. An example processor core 814 can include an arithmetic logic unit (ALU), a floating point unit (FPU), a digital signal processing core (DSP core), or any combination thereof. An example memory controller 818 can also be used with the processor 804, or in some implementations, the memory controller 818 can be an internal part of the processor 804.

[0094] Depending on the desired configuration, system memory 806 may be of any type, including but not limited to volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. System memory 806 may include an operating system 820, one or more applications 822, and program data 824. Program data 824 may include, for example, operating profiles 122s. Figure 8 This depicted basic configuration 802 is shown in FIG. 8 by those components within the inner dashed line.

[0095] The computing device 800 may have additional features or functions and additional interfaces to support communication between the basic configuration 802 and any other devices and interfaces. For example, a bus / interface controller 830 may be used to support communication between the basic configuration 802 and one or more data storage devices 832 via a storage interface bus 834. The data storage device 832 may be a removable storage device 836, a non-removable storage device 838, or a combination thereof. Examples of removable storage devices and non-removable storage devices include magnetic disk devices such as floppy disk drives and hard disk drives (HDDs), optical disk drives such as compact disk (CD) drives or digital versatile disk (DVD) drives, solid state drives (SSDs), and tape drives, etc. Example computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.

[0096] System memory 806, removable storage device 836, and non-removable storage device 838 are examples of computer-readable storage media. Computer-readable storage media include one or more storage hardware or devices, examples of which include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage devices, magnetic cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by the computing device 800. Any such computer-readable storage media can be part of the computing device 800. The term "computer-readable storage medium" excludes propagating signals and communication media.

[0097] The computing device 800 may also include an interface bus 840 for supporting communications from various interface devices (e.g., output devices 842, peripheral device interfaces 844, and communication devices 846) to the basic configuration 802 via the bus / interface controller 830. Example output devices 842 include a graphics processing unit 848 and an audio processing unit 850 that may be configured to communicate with various external devices such as a display or speakers via one or more A / V ports 852. Example peripheral device interfaces 844 include a serial interface controller 854 or a parallel interface controller 856 that may be configured to communicate with external devices such as input devices (e.g., keyboards, mice, pens, voice input devices, touch input devices, etc.) or other peripheral devices (e.g., printers, scanners, etc.) via one or more I / O ports 858. Example communication devices 846 include a network controller 860 that may be arranged to support communication with one or more other computing devices 862 over a network communication link via one or more communication ports 864.

[0098] A network communication link can be an example of a communication medium. Communication media can generally be implemented by computer-readable instructions, data structures, program modules, or other data in a modulated data signal (such as a carrier wave or other transmission mechanism), and can include any information transfer medium. A "modulated data signal" can be a signal that has one or more characteristics set or changed in a way that information can be encoded in the signal. As an example and not limitation, communication media can include wired media such as a wired network or a direct wired connection, and wireless media such as acoustics, radio frequency (RF), microwaves, infrared, and other wireless media. The term "computer-readable medium" used herein can include storage media and communication media.

[0099] The computing device 800 may be implemented as part of a small portable (or mobile) electronic device such as a cellular phone, a personal data assistant (PDA), a personal media player device, a wireless web watch device, a personal headset device, a dedicated device, or a hybrid device including any of the above functionality. The computing device 800 may also be implemented as a personal computer, including both laptop and non-laptop computer configurations.

[0100] Specific embodiments of the present technology are described above for illustrative purposes. However, various modifications may be made without departing from the foregoing disclosure. In addition, many elements of one embodiment may be combined with other embodiments to supplement or replace the elements of other embodiments. Therefore, the present technology is not limited except as set forth in the appended claims.

Claims

1. A diagnostic system for diagnosing performance of a monitored computing system, comprising: processor; as well as a memory operably coupled to the processor, the memory storing computer executable instructions executable by the processor to cause the remote diagnostic system to: receiving operational data from the analyzed computing system, the operational data representing one or more of operations, events, and system states associated with the computing system; identifying an operating pattern based on the operating data, the operating pattern comprising a temporal sequence of the operations, events, or system states associated with the analyzed computing system; as well as Data representing an operational profile of the computing system is generated by associating annotations with the identified operational mode, the annotations identifying causes, justifications, or corrective suggestions corresponding to violations of the operational mode, wherein the operational profile includes the identified operational mode and the associated annotations, and wherein the operational profile allows the annotations to be provided to identify causes, justifications, or corrective suggestions corresponding to violations of the time series in the associated operational mode. 2 . The diagnostic system of claim 1 , wherein the received operational data from the analyzed computing system comprises operational data from the analyzed computing system operating under normal conditions. 3 . The diagnostic system of claim 1 , wherein the received operational data from the analyzed computing system comprises an event log from the analyzed computing system operating under normal conditions.

4. The diagnostic system of claim 1, wherein: The received operational data from the analyzed computing system includes an event log from the analyzed computing system operating under normal conditions; and The memory includes additional instructions executable by the processor to cause the diagnostic system to: parsing the event log; and A time sequence of events is identified based on the parsed event log.

5. The diagnostic system of claim 1, wherein: The received operational data from the analyzed computing system includes an event log from the analyzed computing system operating under normal conditions; and The memory includes additional instructions executable by the processor to cause the diagnostic system to: Parsing the event log; identifying a time sequence of events based on the parsed event log; as well as A time interval between a pair of events in the time series is identified.

6. The diagnostic system of claim 1, wherein: The received operational data from the analyzed computing system includes an event log from the analyzed computing system operating under normal conditions; and The memory includes additional instructions executable by the processor to cause the diagnostic system to: Parsing the event log; identifying a time sequence of events based on the parsed event log; identifying a time interval between a pair of events in the time series; as well as The received annotation is associated with a condition that the time interval between the pair of events in the time series is exceeded.

7. The diagnostic system of claim 1, wherein: The annotation is the first annotation; and The memory includes additional instructions executable by the processor to cause the diagnostic system to: identifying correlations of performance indicators of the computing system, the performance indicators each comprising an operating parameter corresponding to a performance level of the analyzed computing system; as well as A second annotation is received and associated with the dependency of a performance indicator, the second annotation identifying a reason, a justification, or a corrective suggestion corresponding to a violation of the dependency of the performance indicator.

8. A method of diagnosing performance of a computing system, comprising: receiving operational data from the computing system being analyzed; identifying an operating pattern based on the operating data from the analyzed computing system, the operating pattern comprising a temporal sequence of operations, events, or system states associated with the analyzed computing system; generating data representing an operational profile of the computing system by associating an annotation with the identified operational mode, the annotation identifying a cause, justification, or corrective suggestion corresponding to a violation of the time sequence; as well as Based on additional operational data from the computing system, providing the generated operational profile for diagnosing performance of the analyzed computing system to a diagnostic computing system, wherein the provided operational profile allows the diagnostic computing system to provide the annotation to identify a cause, justification, or corrective suggestion corresponding to a violation of the temporal sequence in an associated operating mode.

9. The method of claim 8, wherein receiving the operating data from the analyzed computing system comprises receiving operating data from the analyzed computing system operating under normal conditions.

10. The method of claim 8, wherein receiving the operational data from the analyzed computing system comprises receiving an event log from the analyzed computing system operating under normal conditions.

11. The method according to claim 8, wherein: Receiving the operational data from the analyzed computing system includes receiving an event log from the analyzed computing system operating under normal conditions; and Identifying the operating mode includes: Parsing the event log based on keywords; as well as A time sequence of events is identified based on the parsed event log.

12. The method according to claim 8, wherein: Receiving the operational data from the analyzed computing system includes receiving an event log from the analyzed computing system operating under normal conditions; and Identifying the operating mode includes: Parsing the event log based on keywords; identifying a time sequence of events based on the parsed event log; and A time interval between a pair of events in the time series is identified.

13. The method according to claim 8, wherein: Receiving the operational data from the analyzed computing system includes receiving an event log from the analyzed computing system operating under normal conditions; Identifying the operating mode includes: Parsing the event log based on keywords; identifying a time sequence of events based on the parsed event log; and determining a time interval between a pair of events in the time series; and Associating the received annotation includes associating the received annotation with a condition that the time interval between the pair of events in the time series is exceeded.

14. The method of claim 8, wherein: The annotation is the first annotation; and The method further comprises: identifying correlations of performance indicators of the analyzed computing systems, the performance indicators each comprising an operating parameter corresponding to a performance level of the computing systems; and A second annotation is received and associated with the dependency of a performance indicator, the second annotation identifying a reason, a justification, or a corrective suggestion corresponding to a violation of the dependency of the performance indicator.

15. The method according to claim 8, further comprising: receiving additional operational data from the analyzed computing system; identifying another operating mode based on the additional operating data; as well as The association of the received annotation with the identified another operating mode is updated.

16. A method of diagnosing performance of a computing system, comprising: receiving operational data from a computing system; identifying an operating mode based on the received operating data, the operating mode comprising a temporal sequence of operations, events, or system states of the computing system; determining whether the identified operating mode violates the operating profile by comparing the identified operating mode to a reference operating mode of the operating profile; as well as In response to determining that the identified operating mode violates the reference operating mode of the operating profile, a reason, justification or corrective suggestion contained in an annotation associated with the reference operating sequence in the operating profile is output via a device remote from the computing system for use in diagnosing the performance of the computing system.

17. The method of claim 16, wherein: Receiving the operational data includes receiving an event log from the computing system; and Identifying the operating mode includes: Parsing the event log based on keywords; as well as A time sequence of events is identified based on the parsed event log.

18. The method of claim 16, wherein: Receiving the operational data includes receiving an event log from the computing system being operated; and Identifying the operating mode includes: Parsing the event log based on keywords; identifying a time sequence of events based on the parsed event log; and A time interval between a pair of events in the time series is identified.

19. The method of claim 16, wherein: The annotation is the first annotation; and The method further comprises: identifying correlations of performance indicators of the computing system, the performance indicators each comprising an operating parameter corresponding to a performance level of the computing system; determining whether the identified dependency of the performance indicator violates a reference dependency of the performance indicator associated with a second annotation, the second annotation identifying a reason, a justification, or a correction suggestion corresponding to the violation of the dependency of the performance indicator; and In response to determining that the identified dependency of a performance metric violates the reference dependency of a performance metric, outputting the cause, justification, or correction suggestion contained in the second annotation for use in diagnosing performance of the computing system.

20. The method of claim 16, further comprising: receiving an operational problem of the computing system from a tenant of the computing system; determining whether a timeline of the operational issue is related to a timeline of the violation of the operational mode; as well as In response to determining that the timeline of the operational issue is related to the timeline of the violation of the operational mode, the reason, justification, or corrective suggestion associated with the received operational issue contained in the annotation is output.

Images