Predictive system repair

Abstract

Techniques for predictive system remediation are disclosed. Based on attributes associated with the application of one or more system-selected remediation actions to one or more problematic system behaviors in a system (e.g., a database system), the system determines the predicted effectiveness of one or more future applications of the remediation actions to the specific problematic system behaviors as of one or more future times. The system determines that the predicted effectiveness of the one or more future applications of the remediation actions is positive but does not meet a performance criterion. In response to determining that the predicted effectiveness is positive but does not meet the performance criterion, the system generates a notification corresponding to the predicted effectiveness not meeting the performance criterion. The system applies the remediation action to the specific problematic system behavior despite having determined that the predicted effectiveness does not meet the one or more performance criteria.

Description

Technical Field

[0001] The present disclosure relates to systems for managing computer hardware and / or software running on the hardware. Background Art

[0002] Systems of computer hardware and / or software running on the hardware (hereinafter referred to as "computer systems," e.g., database systems, virtual machines, clusters, distributed systems, and / or other types of computer systems, or combinations thereof) are subject to various problematic system behaviors. Generally speaking, problematic system behaviors are any type of behavior that affects access to the computer system and / or services supported by the computer system, the data integrity of data stored by the computer system, the responsiveness of the computer system, and / or any other performance characteristic of the computer system, or combinations thereof. Non-problematic system behaviors correspond to a system state in which the component(s) in question are functioning as expected. Problematic system behaviors indicate that one or more components of the computer system are in a degraded and / or non-operating state, e.g., due to changing or unexpected demands on the computer system or due to changing or unexpected interactions between components of the computer system. For example, problematic system behaviors may indicate that one or more components of the computer system have failed, are exhibiting symptoms of impending failure, and / or are not performing as expected. Examples of problematic system behavior may include, but are not limited to: bandwidth saturation; slow response times; high processor utilization; low storage space (e.g., disk space) availability; abnormally high request and / or transaction rates per time unit (e.g., per second); slow processing time per transaction; abnormally long time spent in wait states (e.g., input / output wait time, processor wait time, etc.); and / or any other type of system behavior that can be qualitatively or quantified as problematic, or any combination thereof.

[0003] Various remedial actions can be used to resolve problematic system behavior. Examples of remedial actions include allocating more resources, applying software patches, clearing caches, or restarting the computer system. In some cases, applying a remedial action may only temporarily resolve problematic system behavior. The problematic system behavior may recur in the future. If the problematic system behavior worsens over time, the applicable remedial actions may become correspondingly less effective over time.

[0004] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] In the figures of the accompanying drawings, embodiments are shown by way of example and not limitation. It should be noted that references to "one" or "an" embodiment in this disclosure do not necessarily refer to the same embodiment, and they mean at least one embodiment. In the drawings:

[0006] Figure 1 illustrates a system according to one or more embodiments;

[0007] Figure 2A-2B illustrates a set of operations for predictive system repair according to one or more embodiments;

[0008] Figure 3 illustrates an example according to one or more embodiments; and

[0009] Figure 4 A block diagram illustrating a computer system in accordance with one or more embodiments is shown. DETAILED DESCRIPTION

[0010] In the following description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding. One or more embodiments may be practiced without these specific details. Features described in one embodiment may be combined with features described in different embodiments. In some examples, well-known structures and devices are described with reference to block diagrams to avoid unnecessarily obscuring the present invention.

[0011] 1. General Overview

[0012] 2. System Architecture Overview

[0013] 3. Predictive system repair

[0014] 4. Illustrative Examples

[0015] 5. Other Matters; Extension

[0016] 6. Hardware Overview

[0017] 7. Computer Networks and Cloud Networks

[0018] 8. Microservice Applications

[0019] 1. General Overview

[0020] One or more embodiments include techniques for predictive system remediation. Based on attributes associated with the application of one or more system-selected remediation actions to one or more problematic system behaviors in a system (e.g., a database system), the system determines the predicted effectiveness of one or more future applications of the remediation actions to the specific problematic system behaviors as of one or more future times. The system determines that the predicted effectiveness of the one or more future applications of the remediation actions is positive but does not meet one or more performance criteria. In response to determining that the predicted effectiveness is positive but does not meet one or more performance criteria, the system generates a notification corresponding to the predicted effectiveness not meeting the one or more performance criteria. The system applies the remediation action to the specific problematic system behavior even though it has been determined that the predicted effectiveness of the one or more future applications of the remediation action does not meet the one or more performance criteria.

[0021] One or more embodiments described in this specification and / or recited in the claims may not be included in this general summary.

[0022] 2. System Architecture Overview

[0023] Figure 1 1 illustrates a system 100 according to one or more embodiments. Figure 1 As shown in FIG, system 100 includes a management interface 104, a self-healing system 102, a data repository 116, and various components thereof. In one or more embodiments, system 100 may include Figure 1 More or fewer components may be used than shown. Figure 1 The components shown in FIG can be local to each other or remote from each other. Figure 1 The components shown in FIG can be implemented in software and / or hardware. Each component can be distributed across multiple applications and / or machines. Multiple components can be combined into a single application and / or machine. Operations described with respect to one component can alternatively be performed by another component. Additional embodiments and / or examples related to computer networks are described below.

[0024] In embodiments, the management interface 104 refers to hardware and / or software configured to facilitate communication between a user (e.g., a user in an administrative role and / or performing administrative functions) and the self-healing system 102. The management interface 104 renders user interface elements and receives input via the user interface elements. Examples of interfaces include graphical user interfaces (GUIs), command line interfaces (CLIs), tactile interfaces, and voice command interfaces. Examples of user interface elements include check boxes, radio buttons, drop-down lists, list boxes, buttons, toggles, text fields, date and time pickers, command lines, sliders, pages, and forms.

[0025] In an embodiment, different components of the management interface 104 are specified in different languages. The behavior of user interface elements is specified in a dynamic programming language, such as JavaScript. The content of user interface elements is specified in a markup language, such as Hypertext Markup Language (HTML) or XML User Interface Language (XUL). The layout of user interface elements is specified in a style sheet language, such as Cascading Style Sheets (CSS). Alternatively, the management interface 104 is specified in one or more other languages, such as Java, C, or C++.

[0026] In an embodiment, the self-healing system 102 refers to hardware and / or software configured to perform the operations described herein for predictive system repair. Examples of operations for predictive system repair are described below. The self-healing system 102 is configured to "self-heal" by detecting one or more problematic system behaviors and applying one or more system-selected repair actions to the (one or more) problematic system behaviors, without requiring intervening user input to select and / or initiate the (one or more) repair actions. The self-healing system 102 can be a self-healing database system that includes a database 106. The self-healing system 102 can be configured to detect problematic system behaviors affecting the database 106 and apply one or more system-selected repair actions to "heal" the database.

[0027] In embodiments, problematic system behavior can be any type of behavior that affects access to the self-healing system 102, the data integrity of data stored by the self-healing system 102, the responsiveness of the self-healing system, and / or any other performance characteristic of the self-healing system 102, or a combination thereof. Problematic system behavior can be associated with one or more system metrics, where different values ​​of the metric(s) correspond to different system states, ranging from non-problematic system behavior to problematic system behavior. For example, the self-healing system 102 can track metrics corresponding to bandwidth usage, response time, processor utilization, storage availability, transaction rate, processing time, latency, and / or any other type of metric that quantifies system behavior, or a combination thereof. Non-problematic system behavior corresponds to a system state in which the component(s) in question are functioning as expected. Problematic system behavior indicates that one or more components of the self-healing system 102 are in a degraded and / or non-functional state. For example, problematic system behavior can indicate that one or more components of the self-healing system have failed, are exhibiting symptoms of impending failure, and / or are not performing as expected. Examples of problematic system behavior may include, but are not limited to: bandwidth saturation; slow response times; high processor utilization; low storage space (e.g., disk space) availability; abnormally high request and / or transaction rates per time unit (e.g., per second); slow processing time per transaction; abnormally long time spent in wait states (e.g., input / output wait time, processor wait time, etc.); and / or any other type of problematic system behavior or combination thereof.

[0028] In embodiments, a repair action can be any type of action or combination thereof designed to repair one or more problematic system behaviors. A repair action can restart or reset a component of the self-healing system 102 (e.g., a database, a service, a virtual machine, an operating system, and / or any other component of the self-healing system 102, or a combination thereof). Alternatively or additionally, a repair action can provision additional resources (e.g., network bandwidth, processor cycles, memory, storage, and / or any other type of resource, or a combination thereof) to the self-healing system 102. For example, the self-healing system 102 can operate in a data center, a virtual machine environment, and / or any other type of operating environment in which available resources are distributed across multiple physical and / or virtual systems. A repair action can allocate idle resources and / or reallocate resources from another system to the self-healing system 102. Alternatively or additionally, a repair action can reconfigure one or more components of the self-healing system. For example, if a network interface is saturated, a repair action can impose a data rate limit on transactions conducted through that network interface. Many different types of repair actions and / or combinations thereof can be applied to many different types of problematic system behaviors.

[0029] In an embodiment, the repair engine 108 refers to hardware and / or software configured to perform the operations described herein for detecting problematic system behavior, selecting repair actions to be applied to the problematic system behavior, and / or applying repair actions to the problematic system behavior. The repair engine 108 can be configured to monitor the components of the self-healing system 102 (e.g., using polling, a log agent, a heartbeat system in which components periodically report their health status, and / or any other type of monitoring or combination thereof). Based on the monitoring, the repair engine 108 can detect problematic system behavior. In response to detecting problematic system behavior, the repair engine 108 can select from a set of available repair actions, which may have been specified as applicable to one or more specific problematic system behaviors. The repair engine 108 can apply the system-selected repair actions to the problematic system behavior.

[0030] In an embodiment, the repair engine 108 is configured to predict the future effectiveness of one or more repair actions for resolving one or more problematic system behaviors. Specifically, the repair engine 108 may use information about previous applications of the repair actions to the problematic system behaviors to predict the future effectiveness of the repair actions. The repair engine 108 may use information stored in the data repository 116, as described below.

[0031] In an embodiment, the self-healing system 102 includes a machine learning engine 109. Machine learning encompasses various techniques in the field of artificial intelligence that address computer-implemented, user-independent processing to solve problems with variable inputs. The self-healing system 102 can be configured to use the machine learning engine 109 to perform one or more of the operations described herein to predict the future effectiveness of one or more repair actions.

[0032] In an embodiment, the machine learning engine 109 trains the machine learning model 110 to perform one or more operations. The trained machine learning model 110 uses training data to generate a function that, given one or more inputs to the machine learning model, calculates a corresponding output. The output may correspond to a prediction based on previous machine learning. In an embodiment, the output includes a label, classification, and / or categorization assigned to the provided input(s). The machine learning model 110 corresponds to a learning model for performing the desired operation(s) (e.g., labeling, classifying, and / or categorizing inputs).

[0033] In an embodiment, the machine learning engine 109 may use supervised learning, semi-supervised learning, unsupervised learning, reinforcement learning, and / or another training method or a combination thereof. In supervised learning, labeled training data includes input / output pairs, where each input is labeled with a desired output (e.g., a label, classification, and / or categorization), also known as a supervisory signal. In semi-supervised learning, some inputs are associated with supervisory signals and other inputs are not associated with supervisory signals. In unsupervised learning, the training data does not include a supervisory signal. Reinforcement learning uses a feedback system in which the machine learning engine 109 receives positive and / or negative reinforcement in the process of attempting to solve a specific problem (e.g., optimizing performance in a specific scenario according to one or more predefined performance criteria). In an embodiment, the machine learning engine 109 initially trains the machine learning model 110 using supervised learning and then uses unsupervised learning to continuously update the machine learning model 110.

[0034] In an embodiment, the machine learning engine 109 may use many different techniques to label, classify, and / or categorize the input. The machine learning engine 109 may transform the input into a feature vector that describes one or more properties ("features") of the input. The machine learning engine 109 may label, classify, and / or categorize the input based on the feature vector. Alternatively or additionally, the machine learning engine 109 may use clustering (also known as cluster analysis) to identify commonalities in the input. The machine learning engine 109 may group (i.e., cluster) the input based on those commonalities. The machine learning engine 109 may use hierarchical clustering, k-means clustering, and / or another clustering method, or a combination thereof. In an embodiment, the machine learning engine 109 includes an artificial neural network. An artificial neural network includes a plurality of nodes (also known as artificial neurons) and edges between the nodes. The edges may be associated with corresponding weights that represent the strength of the connection between the nodes, and the machine learning engine 109 adjusts these weights as the machine learning proceeds. Alternatively or additionally, the machine learning engine 109 may include a support vector machine. A support vector machine represents the input as a vector. The machine learning engine 109 can label, classify and / or categorize the input based on the vector. Alternatively or additionally, the machine learning engine 109 can use a naive Bayes classifier to label, classify and / or categorize the input. Alternatively or additionally, given a specific input, the machine learning model can apply a decision tree to predict the output of the given input. Alternatively or additionally, the machine learning engine 109 can apply fuzzy logic in situations where labeling, classifying and / or categorizing the input in a fixed set of mutually exclusive options is impossible or impractical. The foregoing machine learning model 110 and techniques are discussed for exemplary purposes only and should not be construed as limiting one or more embodiments.

[0035] In an embodiment, when the machine learning engine 109 applies different inputs to the machine learning model 110, the corresponding output is not always accurate. As an example, the machine learning engine 109 can use supervised learning to train the machine learning model 110. After training the machine learning model 110, if the subsequent input is the same as the input included in the labeled training data and the output is the same as the supervisory signal in the training data, then the output is definitely accurate. If the input is different from the input included in the labeled training data, then the machine learning engine 109 may generate a corresponding output that is inaccurate or of uncertain accuracy. In addition to generating a specific output for a given input, the machine learning engine 109 can be configured to generate an indicator representing the confidence (or lack of confidence) in the accuracy of the output. The confidence indicator can include a numerical score, a Boolean value, and / or any other type of indicator corresponding to the confidence (or lack of confidence) in the accuracy of the output.

[0036] In an embodiment, given problematic system behavior and candidate remedial actions, the machine learning engine 109 may be configured to predict the future effectiveness of the candidate remedial actions to resolve the problematic system behavior. The machine learning engine 109 may be configured to detect and store patterns in system behavior and / or previous applications of remedial actions to problematic system behavior. The machine learning engine 109 may be configured to predict the effectiveness of remedial actions and / or future system behavior based on these patterns.

[0037] In an embodiment, the machine learning engine 109 is configured to detect and store seasonal patterns 112. Seasonal patterns 112 are patterns of system behavior associated with particular seasons (i.e., time periods during which system behavior may vary in a relatively predictable manner due to seasonal factors). For example, holidays and sales events are often associated with seasonal system behavior. Seasonality is discussed in further detail in U.S. patent application Ser. No. 15 / 186,938, which is incorporated herein by reference in its entirety.

[0038] In an embodiment, the machine learning engine 109 is configured to detect and store non-seasonal patterns 114. Non-seasonal patterns 114 are patterns of system behavior that are not associated with a particular season. Non-seasonal patterns 114 may correspond to trends in system behavior over time (e.g., increasing wait times). Alternatively or additionally, non-seasonal patterns 114 may correspond to cyclical patterns in system behavior over time (e.g., moving between long wait times and short wait times according to a discernible pattern). Alternatively or additionally, seasonal patterns 112 and / or non-seasonal patterns 114 may reflect system behavior patterns when applying remediation actions to address problematic system behavior. Seasonal patterns 112 and / or non-seasonal patterns 114 may be based on information stored in a data repository 116, as described below. Generally speaking, in an embodiment, the machine learning engine 109 uses seasonal patterns 112 and / or non-seasonal patterns 114 to predict future system behavior.

[0039] In an embodiment, machine learning engine 109 is configured to consider seasonal patterns 112 and / or non-seasonal patterns 114 when predicting future system behavior and / or the future effectiveness of applying remedial actions to problematic system behavior. Predicting future system behavior and / or the future effectiveness of applying remedial actions to problematic system behavior is discussed in further detail below.

[0040] In an embodiment, the system 100 includes a data repository 116. The data repository 116 can be configured to store one or more system behavior definitions 120. The system behavior definitions 120 correspond to system behaviors that would be considered problematic or non-problematic. The system behavior definitions 120 can include data indicating a type of system behavior (e.g., processor utilization) and / or a specific system resource (e.g., a specific network interface). Alternatively or additionally, the system behavior definitions 120 can include a threshold metric (e.g., maximum processor utilization, minimum amount of free memory space, etc.) that, when met or exceeded, results in the corresponding system behavior being considered problematic. In an embodiment, whether a particular system behavior is considered "problematic" is based at least in part on seasonal patterns 112 of system behavior. For example, system behavior that might be considered problematic during non-peak season (e.g., high processor utilization, high request rate, high bandwidth saturation, etc.) might be considered non-problematic during peak season (e.g., during a promotion or holiday). One or more system behavior definitions 120 can include different criteria for different seasons and / or types of seasons.

[0041] Alternatively or additionally, the data repository 116 can be configured to store one or more remediation action definitions 122. A remediation action definition 122 corresponds to a behavior that can be selected by the self-healing system to attempt to remediate problematic system behavior (e.g., as defined by the system behavior definition 120). A remediation action definition 122 can include data indicating a type of system behavior (e.g., high processor utilization) to which the remediation action corresponding to the remediation action definition 122 is applicable. A remediation action definition 122 can indicate the applicable system behavior by mapping the remediation action definition 122 to a corresponding system behavior definition 120 (e.g., by including a unique identifier of the system behavior definition 120 in the remediation action definition 122, or by some other technique for mapping the remediation action definition 122 to the system behavior definition 120). Alternatively or additionally, a remediation action definition 122 can indicate the remediation action(s) to be applied. Examples of remediation actions are described above.

[0042] Alternatively or additionally, the data repository 116 can be configured to store a repair history 124. The repair history 124 includes data corresponding to one or more previous applications of one or more repair actions to one or more problematic system behaviors. The data can include attributes associated with the application(s). For example, the attributes can indicate: the time at which the repair action was applied; a metric associated with the problematic system behavior before the repair action was applied; a metric associated with the problematic system behavior after the repair action was applied; a data value indicating whether the application of the repair action successfully repaired the problematic system behavior; and / or any other type of information associated with one or more applications of one or more repair actions to one or more problematic system behaviors.

[0043] Alternatively or additionally, the data repository 116 can be configured to store repair configurations 126. The repair configurations 126 indicate one or more criteria for applying repair actions to problematic system behavior. The criteria for applying repair actions to problematic system behavior may be different from a threshold metric that indicates that the corresponding system behavior is considered problematic. For example, the criteria may indicate that a repair action should be applied only when the system behavior has been problematic for at least a specific amount of time or a specific amount (e.g., a specific amount or percentage above or below a threshold metric). Alternatively or additionally, the repair configurations 126 can indicate a preferred order of different repair actions, the time of day at which repair actions can be applied (e.g., restarting the self-healing system 102 only during off-peak hours), a maximum number of times to attempt a particular repair action, and / or any other criteria or combination thereof for applying repair actions to problematic system behavior.

[0044] In an embodiment, the system behavior definition 120, the repair action definition 122, the repair configuration 126 and / or other storage items or a combination thereof may include one or more performance criteria for the repair action corresponding to whether the repair action is considered successful. The one or more performance criteria for the repair action may correspond to a relative improvement in system behavior or an expected value of the system behavior at a future time or within a future time window. For example, one or more performance criteria may indicate that (optionally for a future time window) if a repair action for processor utilization is expected to reduce processor utilization by at least a certain percentage from a current processor utilization level, then the repair action is "successful". As another example, one or more performance criteria may indicate that (optionally for a future time window) if a repair action for storage space is expected to free up at least a certain percentage of currently utilized storage space, then the repair action is "successful". As yet another example, one or more performance criteria may indicate that (optionally for a future time window (e.g., the next thirty days)) the remedial action is "successful" if the system is not predicted to crash or enter a user-defined critical operating state (e.g., based on experience, wisdom, or risk aversion of the system administrator) and / or is associated with a load that has increased to a level that severely degrades performance (e.g., to a point that prevents actual or virtual users of the computer system from obtaining the benefits provided or even guaranteed by the computer system, or to a point that causes the owner of the computer system to expend an unacceptably large amount of resources compared to the purpose and acceptable cost of the computer system). In this example, as long as the system is not expected to enter the critical operating state within the specified time window in the future, the remedial action is successful in delaying any actual need to provide a more tailored solution to the potential problem. Alternatively or additionally, one or more performance criteria for the remedial action may correspond to an absolute metric. For example, one or more performance criteria may indicate that (optionally for a future time window) the remedial action for processor utilization is expected to reduce the processor utilization to below a certain amount (e.g., fifty percent utilization), then the remedial action is "successful." As another example, one or more performance criteria may indicate that (optionally for a future time window) a repair action on storage space is "successful" if it is expected to yield at least a certain amount of storage space (e.g., one terabyte). Many different types of performance criteria and / or combinations thereof may be used to define the "success" of a repair action.

[0045] In an embodiment, the data repository 116 is any type of storage unit and / or device for storing data (e.g., a file system, a database, a collection of tables, or any other storage mechanism). In addition, the data repository 116 may include multiple different storage units and / or devices. The multiple different storage units and / or devices may or may not be of the same type or located at the same physical site. In addition, the data repository 116 may be implemented or executed on the same computing system as one or more other components of the system 100. Alternatively or additionally, the data repository 116 may be implemented or executed on a computing system separate from one or more other components of the system 100. The data repository 116 may be communicatively coupled to one or more other components of the system 100 via a direct connection or via a network.

[0046] The information describing system behavior definitions 120, repair action definitions 122, repair history 124, and / or repair configuration 126 may be implemented across any component within system 100. However, for purposes of clarity and explanation, this information is illustrated within data repository 116.

[0047] In an embodiment, one or more components of system 100 are implemented on one or more digital devices. The term "digital device" generally refers to any hardware device that includes a processor. A digital device can refer to a physical device that executes an application or a virtual machine. Examples of digital devices include computers, tablet computers, laptop computers, desktop computers, netbooks, servers, web servers, network policy servers, proxy servers, general-purpose machines, special-purpose hardware devices, hardware routers, hardware switches, hardware firewalls, hardware firewalls, hardware network address translators (NATs), hardware load balancers, mainframes, televisions, content receivers, set-top boxes, printers, mobile phones, smartphones, personal digital assistants ("PDAs"), wireless receivers and / or transmitters, base stations, communication management equipment, routers, switches, controllers, access points, and / or client devices.

[0048] 3. Predictive system repair

[0049] Figure 2A-2B A set of example operations for predictive system repair according to one or more embodiments is illustrated. Figure 2A-2B Therefore, Figure 2A-2B The particular order of operations shown in should not be construed as limiting the scope of one or more embodiments.

[0050] In an embodiment, a system (e.g., Figure 1The self-healing system 102 of claim 1 obtains attributes associated with the application of one or more repair actions to the system (operation 202). As discussed above, the attributes may include any type of information associated with the previous application of the repair action(s), including, but not limited to: the time at which the repair action was applied; a metric associated with the problematic system behavior before the repair action was applied; a metric associated with the problematic system behavior after the repair action was applied; a data value indicating whether the application of the repair action successfully repaired the problematic system behavior; and / or any other type of information associated with one or more applications of one or more repair actions to one or more problematic system behaviors. The attributes may include labeled training data specifically designed and loaded into the system to train a machine learning model. Alternatively or additionally, the attributes may include unlabeled data obtained during operation of the system.

[0051] In an embodiment, the system trains a machine learning model to predict the effectiveness of applying a remediation action (operation 204). As discussed above, the system can train the machine learning model using supervised learning, unsupervised learning, reinforcement learning, and / or another training method, or a combination thereof. The system can train the machine learning model using labeled and / or unlabeled data (e.g., attributes associated with the application of one or more remediation actions to the system).

[0052] In an embodiment, the system receives user input to configure one or more performance criteria for the repair action (operation 206). As discussed above, the one or more performance criteria for the repair action indicate whether the repair action is considered successful when compared to a specific instance in which the repair action was applied. In an embodiment, the one or more performance criteria include the system configuring one or more performance criteria for the repair action via a user interface (e.g., Figure 1 One or more user-defined criteria received by the management interface 104).

[0053] In an embodiment, the system monitors for problematic system behavior (operation 208). The system may monitor for problematic system behavior by obtaining data associated with the current system behavior (e.g., using one or more of the monitoring techniques described above). Based on the data associated with the current system behavior, the system determines whether problematic system behavior is detected (operation 210). In an embodiment, whether a particular system behavior is considered "problematic" is based at least in part on seasonal patterns of system behavior. System behavior that might be considered problematic during non-peak seasons (e.g., high processor usage, high request rates, high bandwidth saturation, etc.) might be considered non-problematic during peak seasons (e.g., during promotions or holidays). The system may predict that such behavior that occurs during peak seasons will subside and / or self-correct by the end of the season, so that no remedial action is required. Whether system behavior is considered problematic may be based on one or more system behavior definitions (e.g., Figure 1 Different criteria may be applicable to different seasons and / or season types. Generally speaking, in an embodiment, the system monitors system behavior that is problematic because it deviates sufficiently (e.g., by a defined amount or degree, such as a certain number of standard deviations) and persists from expected system behavior.

[0054] In an embodiment, in response to determining that problematic system behavior is detected, the system determines whether any repair actions applicable to the detected problematic system behavior are available (operation 212). Specifically, the system may determine whether any repair action definitions exist (e.g., Figure 1 The system may then map a remediation action definition 122 to a specific problematic system behavior and / or type of problematic system behavior that is detected. If no remediation action applicable to the detected problematic system behavior is available, the system may generate a notification indicating that the system cannot "self-heal" (operation 214). The system may log the notification and / or send the notification to a user (e.g., a system administrator).

[0055] In an embodiment, in response to determining that a remedial action applicable to the detected problematic system behavior is available, the system determines the predicted effectiveness of applying the remedial action to the problematic system behavior (operation 216). The system may determine the predicted effectiveness of applying the remedial action to an instance of the current manifestation of the problematic system behavior. Alternatively, the system may determine the predicted effectiveness of applying the remedial action to one or more future instances of the problematic system behavior. In some cases, applying the remedial action may temporarily resolve the problematic system behavior. However, the problematic system behavior may reappear, and the remedial action may be less successful in subsequent applications. For example, the effectiveness of the remedial action may become increasingly less effective over time.

[0056] In an embodiment, the system uses a machine learning model (e.g., Figure 1 The system may determine the predictive effectiveness of applying a remedial action to the problematic system behavior using a machine learning model 110. The predictive effectiveness may include information about when a remedial action is predicted to no longer be effective (e.g., after a certain amount of time and / or a certain number of instances of the problematic system behavior). The predictive effectiveness may correspond to a metric that indicates the amount or degree to which the remedial action was successful and / or unsuccessful. The system may determine the predictive effectiveness before applying the remedial action to the current instance of the problematic system behavior. Alternatively or additionally, the system may pre-determine the predictive effectiveness of the remedial action to determine whether the system has any vulnerabilities to problematic system behaviors that are not currently manifested in the system.

[0057] In embodiments, the system determines whether the predicted effectiveness of applying the remedial action satisfies one or more performance criteria (operation 218), which indicate whether the remedial action is considered successful when compared to a specific instance of applying the remedial action. If the predicted effectiveness does not satisfy one or more performance criteria or otherwise indicates that the remedial action is predicted to be unsuccessful, the system may generate a notification (operation 220). The notification may include any type of information associated with the system's prediction, such as: the problematic system behavior; the remedial action; the predicted effectiveness; the timeframe within which the remedial action is predicted to fail to meet the one or more performance criteria; and / or any other type of information associated with the system's prediction, or a combination thereof. The system may log the notification and / or send the notification to a user (e.g., a system administrator). In embodiments, the system is configured to send the notification within a specified amount of time (e.g., within a week, a month, or any other user-configurable time period) before the remedial action is predicted to fail to meet the one or more performance criteria, to allow the user sufficient time to intervene and prevent the predicted failure. The amount of time may be based at least in part on the user's expected time to troubleshoot and / or resolve the problematic system behavior. The amount of time can be user-configurable, for example, via a user interface including controls for managing the self-healing behavior of the system as described herein. In an embodiment, the notification includes a link (e.g., a hyperlink, an application launcher, and / or another link) that, when selected by the user, directs the user to a graphical user interface including controls for managing the self-healing behavior of the system. Alternatively or additionally, the notification itself can include a graphical user interface with such controls. Some examples of user input corresponding to instructions to manage self-healing behavior are discussed below.

[0058] In an embodiment, the system determines whether another remedial action is applicable to the problematic system behavior (operation 222). The system can predict the effectiveness of applying each remedial action applicable to the problematic system behavior.

[0059] In an embodiment, even if the predicted effectiveness of applying the remedial action (or multiple remedial actions, if applicable) does not meet one or more performance criteria, or the remedial action is otherwise predicted to be unsuccessful, the system still applies the remedial action to the current instance of the problematic system behavior (operation 224). As discussed above, applying the remedial action can temporarily repair and / or improve the problematic system behavior. The system can continue to apply the remedial action to the instance of the problematic system behavior, performing "self-healing" to the best of the system's ability, until further action is taken to resolve the problematic system behavior. Although the predicted effectiveness does not meet one or more performance criteria, applying the remedial action can still allow the system to continue applying the remedial action for a period of time and / or for instances of the problematic system behavior for which the remedial action still meets one or more performance criteria. The system can continue to apply the remedial action until a point in time and / or instance of the problematic system behavior is reached at which the effectiveness of the remedial action fails to meet one or more performance criteria.

[0060] In an embodiment, the system applies multiple remediation actions to problematic system behavior. The system may apply the remediation actions in a specific order. For example, the system may apply the remediation actions in an order based on predicted effectiveness, predicted cost to the entity of the operating system, complexity (e.g., attempting to allocate more resources before applying a software patch, or cleaning up the cache before allocating more resources), or any other ranking criteria or combination thereof. Alternatively or additionally, the system may apply the remediation actions according to a defined order (e.g., a specific order specified by an administrator in the system configuration via a management interface). In an embodiment, the system stores a record of which remediation actions were most effective when applied, and prefers the most effective remediation actions in future instances of problematic behavior. Alternatively or additionally, the system may track problematic side effects of remediation actions (e.g., system downtime when applying a software patch), and in future instances of problematic system behavior, disfavor remediation actions with the most problematic side effects. The system may prompt the user for confirmation before applying remediation actions known to have problematic side effects.

[0061] In an embodiment, predicting the effectiveness of applying a remedial action before reaching a point in time and / or an instance of problematic system behavior where the remedial action does not meet one or more performance criteria allows a system administrator to take further action (e.g., in response to a notification from the system) to prevent the system from reaching that point in time and / or an instance of problematic system behavior. Alternatively or additionally, continuing to apply the remedial action to the problematic system behavior may continue to mitigate the problematic system behavior to some extent even if applying the remedial action does not meet one or more performance criteria.

[0062] In an embodiment, the system updates the machine learning model based on one or more applications of the remediation action(s) to the problematic system behavior (operation 226). The system may use unsupervised learning to continuously update the machine learning model based on problematic system behaviors detected during system operation and / or the results of the remediation actions applied to the problematic system behaviors. Attributes associated with the application of the remediation actions to the problematic system behaviors may be stored as part of the system's remediation history, which the system may use to update the machine learning model. In an embodiment, continuously updating the machine learning model improves the system's ability to predict the effectiveness of applying the remediation actions to the problematic system behaviors.

[0063] In an embodiment, the system adjusts self-healing based on user input (operation 228). As described above, the user can provide input in a graphical user interface reached via a link in the notification. The user can provide input to change the system behavior definition, the remediation action definition, the remediation configuration, and / or any other type of data used by the system for self-healing, or a combination thereof. For example, the system can notify the user that the predicted effectiveness of applying a remediation action does not meet one or more performance criteria. In response to the notification, the user can instruct the system to avoid applying the remediation action, apply the remediation action more frequently, apply a different remediation action (e.g., a remediation action selected by the user to address the problematic system behavior that is different from the remediation action selected by the system to address the problematic system behavior), adjust one or more performance criteria, adjust the thresholds defining the problematic system behavior, and / or make any other type of change to the system's self-healing behavior, or a combination thereof. The remediation action selected by the user can be selected from a set of remediation actions already defined for the system. Alternatively or additionally, if / when the problematic system behavior recurs, the user can define and select a new remediation action not previously defined for the system for the system to apply. As another example, the system can associate different weights with different remediation actions, where the weights help the system select which remediation action to apply in a particular situation. The user can provide user input to increase or decrease the weight(s) of one or more specific remedial actions, such that if / when the problematic system behavior recurs, the system uses the adjusted weight(s). Alternatively or additionally, the user can provide input notifying the system of problematic side effects of a remedial action (e.g., degrading the performance of a system component and / or a component of another system), such that the system considers the problematic side effects for future instances of the problematic system behavior. Alternatively or additionally, the user can store information about user-initiated remedial actions (other than system-selected remedial actions) that may have affected system performance. For example, the user can input data corresponding to system maintenance, such as replacing a network cable, upgrading a processor, moving the system to a different geographic location, and / or any other type of user-initiated remedial action, or a combination thereof, that may have increased or alleviated the problematic system behavior. Alternatively or additionally, the user can provide input indicating a solution to one or more problematic system behaviors. The solution may be of a type that the system has not yet detected and / or is not configured to detect. Alternatively or additionally, the user can instruct the system to clear cache, history, and / or other types of memory associated with the machine learning engine. For example, a user can instruct the system to clean up the entire memory of the machine learning engine or the memory of the machine learning engine before a specific date. Cleaning up the memory of the machine learning engine before a specific date can improve the ability of the machine learning engine to make predictions by eliminating outdated historical data. Generally speaking, in embodiments, the system adjusts future predictions based on additional information provided by the user.Tuning self-healing based on user input can allow the system to make a best-effort effort to self-heal while still allowing the user to control the operation of the system and make manual decisions about how the system should respond to problematic system behavior.

[0064] 4. Illustrative Examples

[0065] For the sake of clarity, detailed examples are described below. The components and / or operations described below should be understood as a specific example that may not be applicable to certain embodiments. Accordingly, the components and / or operations described below should not be interpreted as limiting the scope of any claim.

[0066] Figure 3 Graph 300 illustrates system performance over time. In this example, the predicted effectiveness of applying a remedial action is defined with reference to system performance (e.g., available processor cycles per unit time, available bandwidth, and / or any other type of system performance metric, or a combination thereof). Specifically, one or more performance criteria for successfully applying the remedial action correspond to restoring system performance to a threshold level. At the beginning of the graph, system performance decreases over time. At time T1, the system applies a remedial action, which results in an improvement in system performance. After time T1, system performance is above the threshold level but has not reached its previous high. System performance then begins to decrease again, and at time T2, the system applies the remedial action again. At time T3, based on the results of applying the remedial action at times T1 and T2, the system predicts that the effectiveness of applying the remedial action will not meet one or more performance criteria (i.e., restore system performance to at least the threshold level) after time T5. Despite this prediction, the system applies the remedial action again at time T4. Applying the remedial action at time T4 allows system performance to remain above the threshold level for an additional period of time. At time T5, system performance drops below the threshold level. At time T6, the system again applies the remedial action, but as predicted at time T3, the remedial action does not satisfy one or more performance criteria, i.e., fails to restore system performance to at least the threshold level. In this example, no further action is taken to prevent system performance from falling below the threshold level at time T5, or to allow the remedial action to be applied at time T6 to satisfy the one or more performance criteria. However, the system notification generated at time T3 may allow the administrator to intervene and take some further action ( Figure 3 (not shown) to ensure that system performance remains above a threshold level and / or that subsequent application of remedial actions (if necessary) meets one or more performance criteria.

[0067] 5. Other Matters; Extension

[0068] Embodiments are directed to a system having one or more devices including a hardware processor and configured to perform any of the operations described herein and / or recited in any of the following claims.

[0069] In an embodiment, a non-transitory computer-readable storage medium comprises instructions that, when executed by one or more hardware processors, cause performance of any of the operations described herein and / or recited in any claims.

[0070] According to one or more embodiments, any combination of the features and functions described herein may be used. In the foregoing description, embodiments of the present invention have been described with reference to many specific details, which may vary from embodiment to embodiment. Therefore, the description and drawings should be viewed in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the present invention and what the applicant intends to be the scope of the present invention is the literal and equivalent scope of the resulting claims in the specific form of the set of claims resulting from this application, including any subsequent corrections.

[0071] 6. Hardware Overview

[0072] According to one embodiment, the technology described herein is implemented by one or more special-purpose computing devices (i.e., computing devices specifically configured to perform certain functions). The special-purpose computing device can be hard-wired to perform the present technology, or can include a digital electronic device that is persistently programmed to perform the present technology, such as one or more application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or network processing units (NPUs), or can include one or more general-purpose hardware processors that are programmed to perform the present technology according to program instructions in firmware, memory, other storage devices, or a combination. Such a special-purpose computing device can also combine customized hard-wired logic, ASICs, FPGAs, or NPUs with customized programming to implement the present technology. The special-purpose computing device can be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device that combines hard-wiring and / or program logic to implement the present technology.

[0073] For example, Figure 4 4 is a block diagram illustrating a computer system 400 upon which embodiments of the present invention may be implemented. Computer system 400 includes a bus 402 or other communication mechanism for communicating information, and a hardware processor 404 coupled with bus 402 for processing information. Hardware processor 404 may be, for example, a general-purpose microprocessor.

[0074] Computer system 400 also includes a main memory 406, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 402 for storing information and instructions to be executed by processor 404. Main memory 406 may also be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 404. Such instructions, when stored in a non-transitory storage medium accessible to processor 404, render computer system 400 a special-purpose machine customized to perform the operations specified in the instructions.

[0075] Computer system 400 also includes a read only memory (ROM) 408 or other static storage device coupled to bus 402 for storing static information and instructions for processor 404. A storage device 410, such as a magnetic or optical disk, is provided and coupled to bus 402 for storing information and instructions.

[0076] Computer system 400 can be coupled to a display 412 via bus 402, such as a liquid crystal display (LCD), a plasma display, an electronic ink display, a cathode ray tube (CRT) monitor, or any other type of device for displaying information to a computer user. An input device 414, including alphanumeric and other keys, can be coupled to bus 402 for communicating information and command selections to processor 404. Alternatively or additionally, computer system 400 can receive user input via a cursor control 416, such as a mouse, trackball, touchpad, touch screen, or cursor direction keys, for communicating direction information and command selections to processor 404 and for controlling movement of a cursor on display 412. Such input devices typically have two degrees of freedom along two axes (a first axis (e.g., x) and a second axis (e.g., y)) to allow the device to specify a position in a plane. Display 412 can be configured to receive user input via one or more pressure-sensitive sensors, multi-touch sensors, and / or gesture sensors. Alternatively or additionally, computer system 400 may receive user input via a microphone, a video camera, and / or some other kind of user input device (not shown).

[0077] Computer system 400 can implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that, in combination with the computer system, makes computer system 400 a special-purpose machine or programs computer system 400 to be a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system 400 in response to processor 404 executing one or more sequences of one or more instructions contained in main memory 406. These instructions can be read into main memory 406 from another storage medium, such as storage device 410. Execution of the sequences of instructions contained in main memory 406 causes processor 404 to perform the process steps described herein. In alternative embodiments, hardwired circuitry can be used in place of or in combination with software instructions.

[0078] As used herein, the term "storage medium" refers to any non-transient medium that stores data and / or instructions that cause a machine to operate in a specific manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 410. Volatile media include dynamic memory, such as main memory 406. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, programmable read-only memory (PROM) and erasable PROM (EPROM), FLASH-EPROM, non-volatile random access memory (NVRAM), any other memory chip or cartridge, content addressable memory (CAM), and ternary content addressable memory (TCAM).

[0079] Storage media are distinct from, but may be used in conjunction with, transmission media. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wire, and optical fiber, including the wires comprising bus 402. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0080] Carrying one or more sequences of one or more instructions to the processor 404 for execution can involve various forms of media. For example, the instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions over the network via a network interface controller (NIC) (such as an Ethernet controller or a Wi-Fi controller). The NIC local to the computer system 400 may receive data from the network and place the data on the bus 402. The bus 402 carries the data to the main memory 406, from which the processor 404 retrieves and executes the instructions. The instructions received by the main memory 406 may optionally be stored on the storage device 410 before or after being executed by the processor 404.

[0081] Computer system 400 also includes a communication interface 418 coupled to bus 402. Communication interface 418 provides two-way data communication coupled to network link 420, wherein network link 420 is connected to local network 422. For example, communication interface 418 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem that provides data communication connections to a telephone line of a corresponding type. As another example, communication interface 418 can be a LAN card that provides data communication connections to a compatible local area network (LAN). Wireless links can also be implemented. In any such implementation, communication interface 418 sends and receives electrical signals, electromagnetic signals, or optical signals that carry digital data streams representing various types of information.

[0082] The network link 420 typically provides data communication to other data devices through one or more networks. For example, the network link 420 can provide a connection to a host computer 424 or to data equipment operated by an Internet service provider (ISP) 426 through a local network 422. The ISP 426, in turn, provides data communication services through the global packet data communication network now commonly referred to as the "Internet" 428. Both the local network 422 and the Internet 428 use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link 420 and through the communication interface 418 are example forms of transmission media that carry the digital data to and from the computer system 400.

[0083] Computer system 400 can send messages and receive data, including program code, through the network, network link 420, and communication interface 418. In the Internet example, server 430 can transmit the requested code for an application through Internet 428, ISP 426, local network 422, and communication interface 418.

[0084] The received code may be executed by processor 404 as it is received and / or stored in storage device 410 or other non-volatile storage for later execution.

[0085] 7. Computer Networks and Cloud Networks

[0086] In one or more embodiments, a computer network provides connections between a collection of nodes running software utilizing the techniques described herein. The nodes can be local to each other and / or remote from each other. The nodes are connected via a set of links. Examples of links include coaxial cables, unshielded twisted pairs, copper cables, optical fibers, and virtual links.

[0087] A subset of nodes implements a computer network. Examples of such nodes include switches, routers, firewalls, and network address translators (NATs). Another subset of nodes uses a computer network. Such nodes (also referred to as "hosts") can execute client processes and / or server processes. A client process requests a computing service (such as the execution of a specific application and / or the storage of a specific amount of data). The server process responds by performing the requested service and / or returning the corresponding data.

[0088] A computer network can be a physical network, comprising physical nodes connected by physical links. A physical node is any digital device. A physical node can be a function-specific hardware device, such as a hardware switch, a hardware router, a hardware firewall, and a hardware NAT. Additionally or alternatively, a physical node can be any physical resource that provides computing power to perform tasks, such as physical resources configured to execute various virtual machines and / or applications that perform corresponding functions. A physical link is the physical medium that connects two or more physical nodes. Examples of links include coaxial cables, unshielded twisted pairs, copper cables, and optical fibers.

[0089] A computer network can be an overlay network. An overlay network is a logical network implemented on another network (such as a physical network). Each node in the overlay network corresponds to a corresponding node in the underlying network. Therefore, each node in the overlay network is associated with both an overlay address (addressing the overlay node) and an underlying address (addressing the underlying node that implements the overlay node). Overlay nodes can be digital devices and / or software processes (such as virtual machines, application instances, or threads). The links connecting overlay nodes are implemented as tunnels through the underlying network. The overlay nodes at either end of the tunnel treat the underlying multi-hop path between them as a single logical link. Tunnel processing is performed by encapsulation and decapsulation.

[0090] In an embodiment, the client can be local to the computer network and / or remote from the computer network. The client can access the computer network through other computer networks (such as a private network or the Internet). The client can transmit the request to the computer network using a communication protocol (such as the Hypertext Transfer Protocol (HTTP)). The request is transmitted through an interface such as a client interface (such as a web browser), a program interface, or an application programming interface (API).

[0091] In an embodiment, a computer network provides a connection between a client and network resources. Network resources include hardware and / or software configured to execute server processes. Examples of network resources include processors, data storage devices, virtual machines, containers, and / or software applications. Network resources are shared among multiple clients. Clients independently request computing services from the computer network. Network resources are dynamically allocated to requests and / or clients on demand. The network resources allocated to each request and / or client can be expanded or reduced based on, for example, (a) computing services requested by a specific client, (b) aggregated computing services requested by a specific tenant, and / or (c) aggregated computing services requested by the computer network. Such a computer network may be referred to as a "cloud network."

[0092] In an embodiment, a service provider provides a cloud network to one or more end users. The cloud network can implement various service models, including but not limited to Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). In SaaS, the service provider provides the end user with the ability to use the service provider's applications that are being executed on network resources. In PaaS, the service provider provides the end user with the ability to deploy customized applications on network resources. Custom applications can be created using programming languages, libraries, services, and tools supported by the service provider. In IaaS, the service provider provides the end user with the ability to supply processing, storage, network, and other basic computing resources provided by the network resources. Any application, including an operating system, can be deployed on the network resources.

[0093] In embodiments, a computer network can implement various deployment models, including but not limited to private clouds, public clouds, and hybrid clouds. In a private cloud, network resources are provisioned for exclusive use to a specific group of one or more entities (as used herein, the term "entity" refers to an enterprise, organization, individual, or other entity). Network resources can be local to the specific entity group's premises and / or remote from the specific entity group's premises. In a public cloud, cloud resources are provisioned to multiple independent entities (also referred to as "tenants" or "clients"). A computer network and its network resources are accessed by clients corresponding to different tenants. Such a computer network may be referred to as a "multi-tenant computer network." Several tenants may use the same specific network resources at different times and / or simultaneously. Network resources can be local to the tenant's premises and / or remote from the tenant's premises. In a hybrid cloud, the computer network includes both private and public clouds. The interface between the private and public clouds allows for data and application portability. Data stored in the private cloud and data stored in the public cloud can be exchanged via the interface. Applications implemented in the private cloud and applications implemented in the public cloud may have dependencies on each other. Calls from applications in the private cloud to applications in the public cloud (and vice versa) can be executed via the interface.

[0094] In an embodiment, the tenants of a multi-tenant computer network are independent of each other. For example, one tenant can be separated from another tenant (by operation, tenant-specific practices, employees, and / or identification to the outside world). Different tenants may have different network requirements for the computer network. Examples of network requirements include processing speed, data storage capacity, security requirements, performance requirements, throughput requirements, latency requirements, resilience requirements, quality of service (QoS) requirements, tenant isolation, and / or consistency. The same computer network may need to implement different network requirements required by different tenants.

[0095] In one or more embodiments, in a multi-tenant computer network, tenant isolation is implemented to ensure that applications and / or data of different tenants are not shared with each other. Various tenant isolation methods can be used.

[0096] In an embodiment, each tenant is associated with a tenant ID. Each network resource of the multi-tenant computer network is labeled with a tenant ID. A tenant is allowed to access a specific network resource only if the tenant and the specific network resource are associated with the same tenant ID.

[0097] In an embodiment, each tenant is associated with a tenant ID. Each application implemented by the computer network is tagged with a tenant ID. Additionally or alternatively, each data structure and / or dataset stored by the computer network is tagged with a tenant ID. A tenant is only allowed to access a particular application, data structure, and / or dataset if the tenant and the particular application, data structure, and / or dataset are associated with the same tenant ID.

[0098] As an example, each database implemented by a multi-tenant computer network can be labeled with a tenant ID. Only the tenant associated with the corresponding tenant ID can access the data in a particular database. As another example, each entry in a database implemented by a multi-tenant computer network can be labeled with a tenant ID. Only the tenant associated with the corresponding tenant ID can access the data in a particular entry. However, a database can be shared by multiple tenants.

[0099] In one embodiment, a subscription list indicates which tenants have access to which applications. For each application, a list of tenant IDs of tenants authorized to access the application is stored. A tenant is allowed to access a specific application only if its tenant ID is included in the subscription list corresponding to the specific application.

[0100] In an embodiment, network resources corresponding to different tenants (such as digital devices, virtual machines, application instances, and threads) are isolated to tenant-specific overlay networks maintained by a multi-tenant computer network. As an example, packets from any source device in a tenant overlay network can only be sent to other devices within the same tenant overlay network. The encapsulation tunnel is used to prohibit any transmission from a source device on a tenant overlay network to devices in other tenant overlay networks. Specifically, a packet received from a source device is encapsulated in an outer packet. The outer packet is sent from a first encapsulation tunnel endpoint (communicating with a source device in the tenant overlay network) to a second encapsulation tunnel endpoint (communicating with a destination device in the tenant overlay network). The second encapsulation tunnel endpoint decapsulates the outer packet to obtain the original packet sent by the source device. The original packet is sent from the second encapsulation tunnel endpoint to a destination device in the same specific overlay network.

[0101] 8. Microservice Applications

[0102] According to one or more embodiments, the technology described herein is implemented using a microservices architecture. A microservice in this context refers to software logic designed to be independently deployable, having endpoints that can be logically coupled to other microservices to build various applications. Applications built using microservices are different from monolithic applications, which are designed as a single fixed unit and typically include a single logical executable program. With microservices applications, different microservices can be independently deployed as separate executable programs. Microservices can communicate via API endpoints according to other communication protocols and / or using Hypertext Transfer Protocol (HTTP) messages. Microservices can be managed and updated independently, written in different languages, and executed independently of other microservices.

[0103] Microservices provide flexibility in managing and building applications. Different applications can be built by connecting different sets of microservices without changing the source code of the microservices. Therefore, microservices act as logical building blocks that can be arranged in various ways to build different applications. Microservices can provide monitoring services that notify the microservice manager (such as If-This-Then-That (IFTTT), Zapier or Oracle Self-Service Automation (OSSA)) when a trigger event in the trigger event set exposed to the microservice manager occurs. Microservices exposed to applications can alternatively or additionally provide action services that perform actions in the application based on data received from the microservice manager (by passing values, connecting actions to other triggers and / or data passed from other actions in the microservice manager, controllable and configurable via the microservice manager). Microservice triggers and / or actions can be linked together to form recipes for actions that occur in optional different applications that are otherwise unaware of each other or do not have control or dependency on each other. These managed applications can be authenticated or plugged into the microservice manager, for example, using application credentials supplied by a user to the manager, without requiring re-authentication each time the managed application is used alone or in combination with other applications.

[0104] In one or more embodiments, microservices can be connected via a GUI. For example, microservices can be displayed as logical blocks within a window, frame, or other element of the GUI. A user can drag and drop microservices into an area of ​​the GUI used to build an application. A user can use a directed arrow or any other GUI element to connect the output of one microservice to the input of another microservice. The application builder can run validation tests (e.g., by checking data types, size limits, etc.) to confirm that the output and input are compatible.

[0105] trigger

[0106] According to one or more embodiments, the above-described techniques can be encapsulated into microservices. In other words, a microservice can trigger a notification based on the above-described techniques (which goes to the microservice manager for optional use by other inserted applications, referred to herein as "target" microservices) and / or can be represented as a GUI tile and connected to one or more other microservices. Trigger conditions can include absolute or relative thresholds for values ​​and / or absolute or relative thresholds for the amount of data to be analyzed or the duration of the data, such that a trigger to the microservice manager occurs whenever an inserted microservice application detects that a threshold has been crossed. For example, a user can request a trigger to the microservice manager when a microservice application detects that a value has crossed a trigger threshold.

[0107] In one embodiment, a trigger, when satisfied, can output data for consumption by a target microservice. In another embodiment, a trigger, when satisfied, outputs a binary value indicating that the trigger has been satisfied, or outputs the name of the field that met the trigger condition or other contextual information. Additionally or alternatively, the target microservice can be connected to one or more other microservices so that alerts are input to the other microservices. The other microservices can perform responsive actions based on the above techniques, including but not limited to deploying additional resources, adjusting system configuration, and / or generating a GUI.

[0108] action

[0109] In one or more embodiments, the inserted microservice application can expose an action to the microservice manager. The exposed action can receive the identification of the data or data object or the location of the data as input, which causes the data to be moved to the data cloud.

[0110] In one or more embodiments, an exposed action can receive as input a request to increase or decrease an existing alert threshold. The input can identify an existing in-application alert threshold and whether to increase, decrease, or delete the threshold. Additionally or alternatively, the input can request that the microservice application create a new in-application alert threshold. In-application alerts can trigger an alert to the user when the user logs into the application, or can trigger an alert to the user using a default or user-selected alert mechanism available within the microservice application itself, rather than triggering the alert to the user through another application plugged into the microservice manager.

[0111] In one or more embodiments, a microservice application can generate and provide output based on inputs that identify, locate, or provide historical data and define the extent or scope of the requested output. When triggered, the action causes the microservice application to provide, store, or display the output, for example, as a data model or as aggregated data describing the data model.

[0112] In the foregoing description, embodiments of the present invention have been described with reference to numerous specific details that may vary from embodiment to embodiment. The description and drawings should therefore be viewed in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the invention, and what applicants intend as the scope of the invention, is the literal and equivalent scope of the claims issued from this application in their specific form, including any subsequent amendments.

Claims

1. One or more non-transitory machine-readable media storing instructions that, when executed by one or more processors, cause: Based at least on a plurality of attributes associated with a plurality of applications of one or more system-selected remedial actions to one or more problematic system behaviors in the database system: determining a first predicted effectiveness of one or more future applications of a first remedial action to a particular problematic system behavior in the database system as of one or more future times; determining that a first predicted effectiveness of the one or more future applications of a first remedial action is positive but does not meet one or more performance criteria; In response to determining that the first predicted effectiveness of the one or more future applications of the first remedial action is positive but does not satisfy the one or more performance criteria: generating a notification corresponding to a first predicted effectiveness of the one or more future applications of the first remedial action not meeting the one or more performance criteria; A first remedial action is applied to the particular problematic system behavior even though a first predicted effectiveness of the one or more future applications of the first remedial action has been determined not to satisfy the one or more performance criteria.

2. The one or more media of claim 1 further storing instructions that, when executed by one or more processors, cause: monitoring the database system for one or more candidate problematic system behaviors; automatically detecting instances of the particular problematic system behavior while monitoring the database system without receiving any user input to identify instances of the particular problematic system behavior; in, Applying the first remedial action to the specific problematic system behavior is automatically performed in response to detecting an instance of the specific problematic system behavior while monitoring the database system, without requiring any user input regarding the first remedial action to be received after automatically detecting the instance of the specific problematic system behavior.

3. One or more media as described in claim 2, wherein detecting the particular problematic system behavior includes determining that the system behavior is problematic for a portion of a season in which the system behavior is detected, wherein the system behavior is determined to vary over time within a seasonal pattern that is determined to repeat over time.

4. The one or more media of claim 1 further storing instructions that, when executed by one or more processors, cause: The plurality of applications of the one or more system-selected remedial actions to the one or more problematic system behaviors in the database system are monitored to obtain the plurality of attributes associated with the plurality of applications of the one or more system-selected remedial actions.

5. The one or more media of claim 1 further storing instructions that, when executed by one or more processors, cause: determining a second predicted effectiveness of one or more future applications of a second remedial action to the particular problematic system behavior in the database system; Based on a comparison of the first predictive effectiveness and the second predictive effectiveness, it is determined that the first remedial action is a better candidate than the second remedial action for remediating the particular problematic system behavior.

6. The one or more media of claim 5 further storing instructions that, when executed by one or more processors, cause: A second repair action is determined to be compatible with the first repair action, and both the first repair action and the second repair action are applied to the particular problematic system behavior.

7. The one or more media of claim 1 further storing instructions that, when executed by one or more processors, cause: receiving user input via an administrative interface of the database system to configure the one or more performance criteria by adjusting an expected amount of time required to troubleshoot unresolved problematic system behavior; The one or more performance criteria are configured based at least on the user input.

8. The one or more media of claim 1 , wherein the plurality of attributes associated with the plurality of applications of the one or more system-selected repair actions include at least: a first subset of the plurality of attributes associated with a first application of a first remedial action to the particular problematic system behavior; as well as A second subset of the plurality of properties is associated with a second application of the first remedial action to the particular problematic system behavior.

9. The one or more media of claim 1 , wherein determining that the predicted effectiveness of the one or more future applications of the first remedial action is positive but does not meet the one or more performance criteria comprises one or more of: predicting that a series of applications of the first remedial action to the particular problematic system behavior are effective but with decreasing effectiveness, or It is predicted that the problematic system behavior will recur within a threshold amount of time after the one or more future applications of a first remedial action.

10. One or more media as described in claim 1, wherein determining the first predicted effectiveness of the one or more future applications of the first remedial action to the specific problematic system behavior is further based on metadata describing one or more manual actions performed on the database system.

11. The one or more media of claim 1 , wherein determining a first predicted effectiveness of one or more future applications of a first repair action to the particular problematic system behavior in the database system comprises: applying current system state data to a machine learning model configured to predict future effectiveness of remedial actions; as well as The machine learning model is trained using the plurality of properties associated with the plurality of applications of the one or more system-selected remediation actions to the one or more problematic system behaviors in the database system.

12. The one or more media of claim 1, wherein the particular problematic system behavior is associated with one or more of increased database response time, increased processor utilization, decreased available database storage, or increased database request rate.

13. The one or more media of claim 1, wherein the notification includes one or more of: a first link to a graphical user interface for enabling different user-selected remedial actions to resolve the one or more problematic system behaviors, a second link to the graphical user interface for increasing the weight of different user-selected remedial actions for the one or more problematic system behaviors when the one or more problematic system behaviors reoccur, a third link to the graphical user interface for defining new remedial actions for the one or more problematic system behaviors when the one or more problematic system behaviors reoccur, a fourth link to the graphical user interface for marking the one or more problematic system behaviors as resolved, or A fifth link to the graphical user interface is for cleaning the memory of the machine learning engine with respect to remedial actions attempted for the one or more problematic system behaviors before a specific date.

14. A method for predictive system repair, comprising: Based at least on a plurality of attributes associated with a plurality of applications of one or more system-selected remedial actions to one or more problematic system behaviors in the database system: determining a first predicted effectiveness of one or more future applications of a first remedial action to a particular problematic system behavior in the database system as of one or more future times; determining that a first predicted effectiveness of the one or more future applications of a first remedial action is positive but does not meet one or more performance criteria; In response to determining that the first predicted effectiveness of the one or more future applications of the first remedial action is positive but does not satisfy the one or more performance criteria: generating a notification corresponding to a first predicted effectiveness of the one or more future applications of the first remedial action not meeting the one or more performance criteria; A first remedial action is applied to the particular problematic system behavior even though a first predicted effectiveness of the one or more future applications of the first remedial action has been determined not to satisfy the one or more performance criteria.

15. The method of claim 14, further comprising: monitoring the database system for one or more candidate problematic system behaviors; automatically detecting instances of the particular problematic system behavior while monitoring the database system without receiving any user input to identify instances of the particular problematic system behavior; Wherein, applying the first remedial action to the specific problematic system behavior is automatically performed in response to detecting an instance of the specific problematic system behavior while monitoring the database system, without requiring any user input regarding the first remedial action to be received after automatically detecting the instance of the specific problematic system behavior.

16. The method of claim 15, wherein detecting the particular problematic system behavior comprises determining that the system behavior is problematic for a portion of a season in which the system behavior is detected, wherein the system behavior is determined to vary over time within a seasonal pattern that is determined to repeat over time.

17. The method of claim 14, further comprising: The plurality of applications of the one or more system-selected remedial actions to the one or more problematic system behaviors in the database system are monitored to obtain the plurality of attributes associated with the plurality of applications of the one or more system-selected remedial actions.

18. The method of claim 14, further comprising: determining a second predicted effectiveness of one or more future applications of a second remedial action to the particular problematic system behavior in the database system; Based on a comparison of the first predictive effectiveness and the second predictive effectiveness, it is determined that the first remedial action is a better candidate than the second remedial action for remediating the particular problematic system behavior.

19. The method of claim 18, further comprising: A second repair action is determined to be compatible with the first repair action, and both the first repair action and the second repair action are applied to the particular problematic system behavior.

20. The method of claim 14, further comprising: receiving user input via an administrative interface of the database system to configure the one or more performance criteria by adjusting an expected amount of time required to troubleshoot unresolved problematic system behavior; The one or more performance criteria are configured based at least on the user input.

21. The method of claim 14, wherein the plurality of attributes associated with the plurality of applications of the one or more system-selected remedial actions include at least: a first subset of the plurality of attributes associated with a first application of a first remedial action to the particular problematic system behavior; as well as A second subset of the plurality of properties is associated with a second application of the first remedial action to the particular problematic system behavior.

22. The method of claim 14, wherein determining that the predicted effectiveness of the one or more future applications of the first remedial action is positive but does not meet the one or more performance criteria comprises one or more of: predicting that a series of applications of the first remedial action to the particular problematic system behavior are effective but with decreasing effectiveness, or It is predicted that the problematic system behavior will recur within a threshold amount of time after the one or more future applications of a first remedial action.

23. The method of claim 14, wherein determining a first predicted effectiveness of the one or more future applications of a first remedial action to the particular problematic system behavior is further based on metadata describing one or more manual actions performed on the database system.

24. The method of claim 14, wherein determining a first predicted effectiveness of one or more future applications of a first remedial action to the particular problematic system behavior in the database system comprises: applying current system state data to a machine learning model configured to predict future effectiveness of remedial actions; as well as The machine learning model is trained using the plurality of properties associated with the plurality of applications of the one or more system-selected remediation actions to the one or more problematic system behaviors in the database system.

25. The method of claim 14, wherein the particular problematic system behavior is associated with one or more of increased database response time, increased processor utilization, decreased available database storage, or increased database request rate.

26. The method of claim 14, wherein the notification includes one or more of: a first link to a graphical user interface for enabling different user-selected remedial actions to resolve the one or more problematic system behaviors, a second link to the graphical user interface for increasing the weight of different user-selected remedial actions for the one or more problematic system behaviors when the one or more problematic system behaviors reoccur, a third link to the graphical user interface for defining new remedial actions for the one or more problematic system behaviors when the one or more problematic system behaviors reoccur, a fourth link to the graphical user interface for marking the one or more problematic system behaviors as resolved, or A fifth link to the graphical user interface is for cleaning the memory of the machine learning engine with respect to remedial actions attempted for the one or more problematic system behaviors before a specific date.

27. A system for repairing problematic system behavior, comprising: at least one device comprising a hardware processor; The system is configured to perform the method according to any one of claims 14-26.

28. A computer program product comprising instructions which, when executed by one or more processors of a computer, cause the computer to perform the method of any one of claims 14 to 26.

Images