How to proactively manage provisioning workflows for efficient cloud operations

Abstract

A method for determining intervention response categories for a provisioning workflow. The method determines a provisioning function for a provisioning step. The method performs outlier detection to identify and remove outliers from unintervention data, generating an unintervention normal data set. The method performs iterative grouping on the unintervention normal data set to determine significant variables for the provisioning function. The method performs response mapping of the provision using results of the iterative grouping, including the significance of errors and the presence or absence of errors in the unintervention normal data and the partial intervention data, to classify the provision into a response category.

Description

[Background technology]

[0001] Provisioning a virtual machine or bare metal server in the cloud involves several provisioning steps during which unstructured, time-stamped logs are generated as the workflow progresses through the provisioning steps and reflects the provisioning state. Of the tens of thousands of virtual server infrastructure (VSI) provisions that occur daily, hundreds may become stuck during a given provisioning step. Proactive detection and repair of out-of-date provisions typically requires manual identification of the errors that cause the provision to become stuck, which can become a significant performance bottleneck. Furthermore, not all out-of-date transactions require manual intervention as transactions have the ability to self-resolve. Current anomaly detection methods do not provide guidance on when to intervene. Summary of the Invention

[0002] Disclosed embodiments include systems, methods, and computer program products configured to determine intervention response categories for a provisioning workflow. In one embodiment, the systems, methods, and computer program products are configured to determine a provisioning function for a provisioning step, perform outlier detection to identify and remove outliers from the unintervention data to generate an unintervention normal data set, perform iterative grouping on the unintervention normal data set to determine significant variables in the provisioning function, and perform response mapping of the provision using the results of the iterative grouping, including the significance of errors and the presence or absence of errors in the unintervention normal data and the partial intervention data, to classify the provision into a response category.

[0003] Optionally, in some embodiments, the response category is one of immediate attention, delayed attention, and no attention.

[0004] Optionally, in some embodiments, the systems, methods, and computer program products use a Local Outlier Factor (LOF) method to identify outliers in the unattended data.

[0005] Optionally, in some embodiments, the system, method, and computer program product are configured to calculate a step duration for the provisioning step for every provision using an event log associated with the provisioning step, identify provisioning parameters for the provisioning step, and identify error messages in the event log that occur as part of the provisioning in the provisioning step.

[0006] Optionally, in some embodiments, the system, method, and computer program product are configured to calculate an F-statistic based on the provision parameters, the error message, and the step duration, determine important variables based on the F-statistic, and cluster the non-intervention normal data set into a plurality of clusters based on a combination of values ​​of the important variables.

[0007] Optionally, in some embodiments, the systems, methods, and computer program products are configured to compare the percentage of data points that exceed n x interquartile range for each cluster, where n is a variable, to a threshold to determine if the cluster is a candidate for further division.

[0008] Optionally, in some embodiments, the systems, methods and computer program products are configured to calculate the duration threshold for each cluster by calculating n×interquartile range for each cluster, where n is a variable.

[0009] Optionally, in some embodiments, the systems, methods, and computer program products are configured to classify the response category as delayed response if an error is present in the partial intervention data and in the no-intervention normal data set and the error is a significant variable as a result of the iterative grouping.

[0010] Optionally, in some embodiments, the system, method, and computer program product are configured to calculate a time to alarm of the error, where the time to alarm of the error is the maximum of all duration thresholds in all clusters with non-zero and non-NULL duration thresholds.

[0011] Optionally, in some embodiments, the systems, methods, and computer program products are configured to classify the response category as immediate response if the error is present in the partial intervention data and is not present in the no-intervention normal data set.

[0012] Optionally, in some embodiments, the systems, methods, and computer program products are configured to classify the response category as no action required if the error is present in the no-intervention normal data set and is absent in the partial intervention data.

[0013] Other embodiments and advantages of the disclosed embodiments are further described in the detailed description.

[0014] For a more complete understanding of the present disclosure, reference should now be made to the following brief description, taken in conjunction with the accompanying drawings and detailed description, in which like reference numerals represent like parts. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a schematic diagram illustrating a system for determining an intervention response category for a provisioning workflow according to one embodiment of the present disclosure. [Diagram 2] 1 is a flowchart illustrating a method for determining an intervention response category for a provisioning workflow according to one embodiment of the present disclosure. [Diagram 3] 1 is a chart illustrating a provisioning function according to one embodiment of the present disclosure. [Figure 4] 1 is a graph plotting skewness versus the number of nearest neighbors (k) in accordance with one embodiment of the present disclosure. [Diagram 5] 1 is a graph illustrating a subset of a set of skewnesses according to an embodiment of the present disclosure. [Figure 6] 1 is a graph illustrating a first maximum / minima according to an embodiment of the present disclosure. [Figure 7] 1 is a flowchart illustrating a method for performing iterative grouping according to one embodiment of the present disclosure. [Figure 8] 1 is a chart illustrating time to alarm threshold for key variables according to one embodiment of the present disclosure. [Figure 9] 1 is a chart showing results of training and testing data from distinct time periods, according to one embodiment of the present disclosure. [Figure 10] FIG. 2 is a block diagram illustrating the hardware architecture of a system according to one embodiment of the present disclosure in which aspects of the exemplary embodiments may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] The depicted diagrams are only exemplary and are not intended to assert or imply any limitation with respect to the environment, architecture, design, or process in which different embodiments may be implemented.

[0017] Although exemplary implementations of one or more embodiments are provided below, it should be understood at the outset that the disclosed systems, computer program products, and / or methods may be implemented using any number of technologies, whether currently known or in existence. The present disclosure should in no way be limited to the illustrative implementations, drawings, and technologies illustrated below, including the exemplary designs and implementations shown and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.

[0018] As used in this disclosure and in the claims, the terms "including" and "comprising" are used in an open-ended manner and should therefore be interpreted to mean "including, but not limited to." Unless otherwise indicated, as used throughout this document, "or" does not require mutual exclusivity, and the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0019] A module or unit as referred to herein may comprise one or more hardware or electrical components, such as electrical circuits, processors, and memories, that may be specifically configured to perform a particular function. Memory may be a volatile or non-volatile memory that stores data, such as, but not limited to, computer-executable instructions, machine code, and various other forms of data. A module or unit may be configured to execute one or more instructions using data to perform one or more tasks. In some cases, a module may refer to a particular collection of functions, software instructions, or circuits configured to perform a unique task. For example, a module may comprise software components, such as, but not limited to, data access objects, service components, user interface components, application programming interface (API) components, and the like; hardware components, such as electrical circuits, processors, memories; and / or combinations thereof. As referred to herein, computer-executable instructions may be in any form, including, but not limited to, machine code, assembly code, and high-level programming code written in any programming language.

[0020] FIG. 1 is a schematic diagram illustrating a system 100 for determining an intervention response category for a provisioning workflow, according to one embodiment of the present disclosure. The system 100 includes a provisioning system 110 for communicating with one or more clients 102 for provisioning of one or more Virtual Server Instances (VSIs) 104. Alternatively, the disclosed embodiments may provision any of a variety of computing resources, including but not limited to bare metal servers or VSIs. For example, a client 102 may request the provisioning system 110 to provision a VSI or bare metal server in the cloud. In one embodiment, the provisioning system 110 may provision a VSI or bare metal server in the cloud thousands of times a day. Each provisioning may include hundreds of steps, during which unstructured, time-stamped logs are generated as the workflow progresses through the provisioning steps and reflects the provisioning status. During the provisioning process, a provisioning may become stuck (i.e., unable to progress or complete) at one or more provisioning steps. The log data 112 may include various types of error messages indicating potential causes of stuck provisioning, some of which may be fatal errors.

[0021] Successful provisioning behavior is observed when the provisioning completes without being stuck at any provisioning step and therefore without requiring any form of intervention. In practice, there is a lack of labels that can be associated with provisions to characterize normal behavior. Human effort towards labeling such provisions is not trivial due to the volume of provisions (e.g., 50,000 provisions per day). Meanwhile, existing anomaly detection methods, such as those based on event log frequency counts, do not provide guidance on when or if to intervene in case of errors. Provisions may take longer than normal to complete, but they have self-resolving capabilities, so not all overdue transactions require manual intervention.

[0022] To improve upon existing methods, the present disclosure uses historical information of provisions that have and have not resulted in intervention to classify provisions into response categories that indicate if and when the provision should be intervened based on their errors. The disclosed embodiments provide high business value through reduced costs, human effort, and mean time to detection, resulting in improved mean time to resolution and customer satisfaction.

[0023] Specifically, and referring to FIG. 1, the provisioning monitoring system 120 retrieves log data 112. The term log data may include data provided in various ways, for example, from a database store or from an event log. Using historical information of the non-intervention data and the intervention data, the provisioning monitoring system 120 provides response categories that indicate if and when to intervene in the provisioning. In one embodiment, the provisioning monitoring system 120 includes an outlier detection module 130, an iterative grouping module 132, and a response mapping module 134. The outlier detection module 130 receives as input the non-intervention data 122. The non-intervention data 122 is training data with no records of human or system intervention. A provision may not have an intervention record for several reasons, including that it was not stuck at any provisioning step (considered as normal behavior), or that it is a stuck provision but there are insufficient resources to respond in a timely manner (considered as an outlier), or that the underlying issue is related but addressed in a different provision (also considered as an outlier). As will be further explained below, the outlier detection module 130 divides the unintervention provision data 122 into two groups: unintervention normal operating provision and outlier provision. In one embodiment, the outlier detection module 130 uses a local outlier factor method (LOF) conditional on skewness to generate the unintervention outlier data set 124 and the unintervention normal data set 126. LOF is based on the concept of local density where the locality is given by the nearest neighbor. The unintervention normal data set 126 is the unintervention data 122 with the outliers removed. The iterative grouping module 132 receives the unintervention normal data set 126 as input. The iterative grouping module 132 performs grouping of the unintervention normal data set 126 to determine the significance of the error and to calculate the duration threshold for various combinations of values ​​of the provision's important variables.In the present description, the terms "cluster" and "group", "clustering" and "grouping", "sub-cluster" and "sub-group" are used interchangeably. The response mapping module 134 classifies the provisions into a response category 136 using the error severity determined in the iterative grouping module 132, the optional duration threshold result of the iterative grouping, and the presence or absence of errors in the no-intervention normal data 126 and the partial intervention data 128. The partial intervention data 128 is training data with information of human or system intervention. The term "partial" is used to indicate that the partial intervention data 128 may not include all provisions that would have benefited from intervention. For example, due to the large number of provisions that occur, not all stuck provisions can be addressed by a limited number of cloud operators. The method would work as intended with the complete intervention data, if present. In one embodiment, the response category 136 is one of an immediate response, a delayed response (i.e., giving the stuck provision time to resolve itself), or no response required.

[0024] FIG. 2 is a flow chart diagram illustrating a method 200 for determining an intervention response category for a provisioning workflow according to one embodiment of the present disclosure. The method 200 may be performed by a system such as, but not limited to, the provisioning monitoring system 120 of FIG. 1. Provisioning is typically divided into individual steps, e.g., first available hosts are located, then the network is configured, then storage is configured. The described method may be applied to one step at a time or to groups of steps at a time. The largest group of steps may be the entire provisioning.

[0025] For all provisions that occurred during a given time period, the method 200 begins at step 202 by extracting provision-specific features from a collection of event logs, such as log data 112 of FIG. 1. The term "log data" is used to refer to data from a variety of sources, potentially including data stored in a database. As shown in FIG. 3, one embodiment of a provisioning function includes provisioning parameters 306. An additional provisioning-specific function in a provision step is the calculated step duration (st) associated with the provisioning step. i ) 304 and templated errors 308. In one embodiment, the method 200 includes a unique identifier PID i For every provision with a message or event timestamp in the event log that indicates the start of the provision step (

number

number

number

[0026] The method 200 performs outlier detection in step 204 to identify and remove outliers from the unintervention data, generating two disjoint sets: an unintervention normal data set and an unintervention outlier data set. The method 200 improves on the Local Outlier Factor (LOF) technique for outlier detection. LOF is based on the concept of local density. By comparing the local density of a point with the local densities of its k nearest neighbors, the method 200 can identify regions of similar / higher / lower density than their neighbors. A distance metric such as Manhattan distance can be used to indicate the distance between two points. In one embodiment, the LOF is calculated for a given provision step, subject to skewness, over the step durations corresponding to the provision in the unintervention set. The LOF splits the data into two sets for a pre-determined value of k, where k is the number of nearest neighbors. Various values ​​of k are considered in the method 200. In one embodiment, k can be selected as a % of the data set. The unintervention data set is a set of d i The provision step duration is st i As the provision data points given in 304, g = {d i The method 200 can calculate the value of k as K={(j / 100)×N}, where j is a percentage of the data set size and N is the number of data points in g. j For ∈K, repeat the following steps: (1) g and k j Given all data points d i Calculate the LOF for, where distance metric = Manhattan; (2) the outlier dataset

number

number

number

number

[0027] The method 200 performs iterative grouping in step 206 using the unintervention normal data set. Unsupervised learning methods look for patterns in the data and use features to cluster the data. This class of learning methods does not require labels, and therefore no user assistance is required for clustering purposes. In one embodiment, as described below, the method 200 performs iterative grouping in step 206 using the unintervention normal data set according to a method 700 shown in the flowchart of FIG. 7.

[0028] After the iterative grouping process, the method 200 performs response mapping of the provisions in step 208 using the severity of the errors determined by the iterative grouping, optionally the result of the duration threshold of the iterative grouping, and the presence or absence of errors in the unintervention normal data and the partial intervention data to classify the provisions into response categories. The main idea behind response mapping is to find a mapping function that maps input features to a set of output labels, where the relevant set of input features are the error messages. The set of output labels are the response categories, namely no action required, immediate action, and delayed action.

[0029] Each error that is present in the unintervention normal data set and is determined to be an important variable in any iteration of step 704 described below, and that is also present in the partial intervention data set i For the time to alarm, the maximum duration threshold (TimeToAlarm(error i ) = maximum(duration threshold j ), where maximum(duration threshold j ) is the maximum of all duration thresholds in all clusters identified by the iterative grouping process. Note that according to the iterative grouping, each cluster has one error encoding associated with the iterative grouping, and the encoding may represent "no error".

[0030] In one embodiment, if an error present in the normal, no-intervention data is not determined to be a critical variable, it gets a null time-to-alarm and a "no action required" label, regardless of whether it is present in the partial intervention data. This is particularly appropriate when minimizing false positives and reducing the burden on operations staff is a priority. For all errors found in the partial intervention data and not in the no-intervention normal data set, the time-to-alarm is 0 (TimeToAlarm(error k ) = 0), which is an error k is in the intervention data and error k If TimeToAlarm(error k ) = Null), meaning that the error does not require attention. Note that it is assumed that the data set is large enough that all errors are seen in at least one of the normal unintervention and / or partially intervened data.

[0031] In one embodiment, the response mapping label determined using the time to alarm threshold is calculated as follows: For each error assigned a time to alarm of 0, the label is "immediate action". For each error assigned a time to alarm of NULL, the label is "no action required". For each error assigned a time to alarm >0 but not NULL, the label is "delayed action". An alternative embodiment is where the label is assigned directly based on the characteristics of the error present during the provision step, without necessarily calculating the time to alarm. Specifically, for each error that is determined to be a critical variable in any iteration of step 704 and that is present in both the normal unintervention data and the partial intervention data, the label is a "delayed action" label. In one embodiment, if an error present in the normal unintervention data is not determined to be a critical variable, it gets a "no action required" label regardless of whether it is present in the partial intervention data. For an error that is present in the partial intervention data but not in the unintervention normal data, there is an "immediate action" label associated with it. For any error found in the unintervention normal data set that is determined to be a significant variable in any iteration of step 704, as described below with reference to Figure 7, and that is not found in the partial intervention data, there is a "no action required" label associated with it. Thus, individual provisions (for a given provision step) are assigned labels based on their errors, as shown and described below in Figure 8.

[0032] 7 is a flow chart illustrating a method 700 for performing iterative grouping using a non-intervention normal data set, according to one embodiment of the present disclosure. The inputs to the method 700 are (a) non-intervention normal data and (b) a temporary list of unprocessed clusters, e.g., tmpList, which is initialized to an empty list. The temporary list of unprocessed clusters includes, for each cluster, a list of important variables associated with it. The list of important variables associated with each cluster is also initialized to an empty list.

[0033] The method 700 begins in step 701 by aggregating a list of unprocessed clusters. At the start of the method, tmpList is populated with the group corresponding to unintervention normal data, which is the only group present in the list. This group is marked as "unprocessed". At this point, there are no significant variables associated with the cluster. In step 702, the method 700 determines whether all clusters in tmpList are marked as processed. If yes, the method 700 ends in step 713. If not all clusters are marked as processed, in step 703, the method 700 selects an unprocessed cluster and, for the current cluster, computes the provisioning functions of the provisioning step other than the step duration as factor variables and the step duration (st i) 304 and calculate the analysis of variance F-statistic, ignoring interaction effects. Non-limiting examples of provisioning functions used as factor variables are provisioning parameters 306, which may be coded, and coded error messages 310, as shown in FIG. 3. At the beginning of the process, the current cluster / group is the entire uninterventioned normal dataset. The F-statistic is used by the method 700 in step 704 to determine important variables that have an impact on step duration, for example by comparing the F-statistic to a threshold. In step 705, the method 700 determines whether there are new important variables for the cluster currently being processed. At the beginning of the process, the answer is "Yes" unless no variables are determined to be important during step 704.

[0034] If in step 705, there are no additional factor variables remaining or no new significant variables identified for the cluster, then in step 714, the method 700 marks the cluster as processed, assigns the value n×IQR as the duration threshold for the cluster, where IQR is the interquartile range and n is a number such as 1, 1.5, 2 or 3, and returns to step 701.

[0035] If there are new significant variables in step 705, the method 700 clusters the current cluster into multiple clusters in step 706. Each cluster corresponds to a unique combination of values ​​of the significant variables. The total number of unique combinations of values ​​of the significant variables (say M) is also determined in step 706. For each cluster in the multiple M clusters, the method 700 calculates in step 707 the n×interquartile range (IQR), (e.g., 6×IQR), and the percentage of data points whose step duration is greater than n×IQR, say P_o, where n is a variable. The IQR is calculated as the difference between the 75th and 25th percentiles of the set of step duration values ​​in the identified cluster. Additionally, since each of the multiple clusters needs to be examined to determine if its corresponding P_o is less than a preset threshold, a count of the number of subclusters processed so far (given by subclusterProcessedCnt) is maintained and initialized to 0. In step 707A, the method 700 compares the percentage of data points whose step duration is greater than n×IQR(P_o) for the sub-clusters obtained using step 706 with a preset threshold. If P_o is less than the preset threshold, the method 700 assigns the n×IQR calculated in step 707 as the duration threshold for the particular combination of significant variables associated with the cluster and marks the cluster as processed in step 708. Note that the set of significant variables associated with the cluster and their values ​​are known based on the results of step 704. A list of those significant variables is maintained with each cluster. In step 711, subclusterProcessedCnt is incremented by 1 to indicate that the cluster has been processed.

[0036] The method 700 repeats steps 707A and 708 or steps 710 and 711 and 712 until all M sub-clusters (generated in step 706) have been processed. Step 712 checks whether all sub-clusters generated by the method 700 in step 706 have been processed. For any sub-cluster generated using step 706, if P_o is greater than a preset threshold as determined in step 707A, then in step 710 the cluster is added to a temporary list of unprocessed clusters, tmpList, along with the important variables used to obtain this cluster. Note that an aggregated set of all such clusters is maintained in step 701. The result of such an iterative grouping approach is a set of clusters, each with a set of important variables and their values, along with its calculated duration threshold.

[0037] FIG. 8 shows a chart 800 in which "error" and "image template type" are identified according to an embodiment of the present disclosure. In an embodiment, "coded error" and "coded image template type" are identified as important variables based on the F-statistic determined in the first visit to step 704 of FIG. 7. The F-statistic is used to identify variables that have an effect on step duration. In the chart 800, the first seven of the eight rows correspond to the clusters determined in the iterative grouping step 706. For each cluster, the disclosed embodiment calculates the n×IQR and assigns that value as the cluster duration threshold. For each of the four coded error values ​​(0, 1, 2, and 3), the text on top gives the status of the coded error according to the coded error value. Specifically, errors coded 0 and 3 are present in the partial intervention data and the non-intervention normal data and are determined to be important variables in the iterative grouping procedure. Provisions with such errors will be assigned the label "delayed response". Errors coded 1 are present in the no intervention normal data but not in the partial intervention data. Provisions with such errors will be labeled "no action required". Errors coded 2 are only present in the partial intervention data. Provisions with such errors will be assigned the label "immediate action". Chart 800 also shows the duration threshold for each cluster, calculated as n x IQR. In chart 800, during the first visit to step 704, the important variables are identified as "coded error" and "coded image template type", resulting in parameter value combinations (0,0), (0,1), (1,0), (1,1) (3,0), (3,1), where the first value of each of these combinations is "coded error" and the second value in parentheses is "coded image template type".For the (encoded_error, encoded_template_type) value corresponding to (3,0), in step 707A, it is determined that the percentage of outliers is greater than the preset threshold, and therefore a new significant variable for this cluster (i.e., encoded OS type) is identified using steps 703, 704. This cluster is further subdivided into groups based on the unique combinations of the three significant variables, i.e., encoded error, encoded_template_type, and encoded OS type, resulting in the clusters shown in the third and fourth rows. Also in chart 800, the clusters where encoded OS type was not determined to be a significant variable have NA values ​​in the corresponding cells. The specific rows where encoded OS type is marked as NA are 1, 2, 5, 6, and 7. The first cluster corresponds to the combination of encoded error and encoded image template type with values ​​of 0, 0. The duration threshold for this cluster is 123.53. The second cluster corresponds to the combination of coded error and coded image template type with values ​​0, 1. The duration threshold of this cluster is 363.5. The third cluster corresponds to the combination of coded error, coded image template type, and coded OS type with values ​​3, 0, 0. The duration threshold of this cluster is 200.0. The fourth cluster corresponds to the combination of coded error, coded image template type, and coded OS type with values ​​3, 0, 1. The duration threshold of this cluster is 230.0. The fifth cluster corresponds to the combination of coded error and coded image template type with values ​​3, 1. The duration threshold of this cluster is 185.0. The five clusters generated by the iterative grouping process have their time to alarm set to 363.5, which is the maximum duration threshold among all clusters associated with errors that appeared in both the unintervention normal data and the partially intervened data and were determined to be significant variables in the iterative grouping procedure.These clusters correspond to coded errors 0 and 3 present in both the partial intervention data and the non-intervention normal data. This maximum threshold is set as the time to alarm of the delayed response category for errors determined to be significant variables present in both the partial intervention data and the non-intervention normal data. Thus, if a provision with an error coded 0 or 3 stalls for a period longer than the maximum threshold, the user or the system should intervene to address the stalled provision. The last row of the chart 800 corresponds to coded error 2 present only in the partial intervention data. Thus, the provision with coded error 2 would be labeled as "immediate response". The last row of the chart 800 has empty cells corresponding to coded error, coded image template type, and coded OS type, because this error 2 was not present in the non-intervention data and was not considered as part of the iterative grouping process. As mentioned above, in one embodiment, a response label is assigned based on the significant variable determination of the error and its occurrence in one or both of the partial intervention data and the non-intervention normal data, without calculating the time to alarm. In that case, the user may be warned of pending provisions potentially requiring a delayed response, but no specific time may be given.

[0038] FIG. 9 is a chart 900 showing results of training and testing data from distinct time periods, according to one embodiment of the present disclosure. Column 902 of chart 900 lists various provisioning steps, such as STEP1, STEP2, STEP3, and STEP4. Column 904 gives the number of provisions that include each provisioning step of column 902. Column 906 gives the percentage of true positive rate (i.e., recall). Column 908 gives the percentage of positive predictive value (i.e., precision). The true positive rate is measured in terms of the number of true positives and the number of false positives. A provision is a true positive when there is evidence of human intervention, as in the response classification is "immediate action" or "delayed action". A provision is a false positive when there is no evidence of human intervention, but the response label is assigned as "immediate action" or "delayed action". Similarly, a provision is a true negative when there is no evidence of human intervention and the response label is assigned as "no action required". A false negative is when human intervention occurs due to a stuck provisioning, but the assigned response label is "no action required." As shown in chart 900, using the disclosed embodiments, there is high precision and high recall with very few false positives. Thus, the disclosed embodiments provide an effective method for predicting intervention response provisioning workflows using data logs, guided by knowledge intervention data.

[0039] FIG. 10 is a block diagram illustrating a hardware architecture of a system 1000 according to an embodiment of the present disclosure in which aspects of an exemplary embodiment may be implemented. For example, the data processing system 1000 may be configured to store and execute instructions for performing the processes described in FIGS. 1, 2, and 7. In the illustrated example, the data processing system 1000 employs a hub architecture including a north bridge and memory controller hub (NB / MCH) 1006 and a south bridge and input / output (I / O) controller hub (SB / ICH) 1010. The processor 1002, main memory 1004, and graphics processor 1008 are connected to the NB / MCH 1006. The graphics processor 1008 may be connected to the NB / MCH 1006 via an accelerated graphics port (AGP®). A computer bus, such as bus 1032 or bus 1034, may be implemented using any type of communication fabric or architecture that provides for the transfer of data between different components or devices attached to the fabric or architecture.

[0040] In the illustrated example, a network adapter 1016 connects to the SB / ICH 1010. An audio adapter 1030, a keyboard and mouse adapter 1022, a modem 1024, a read-only memory (ROM) 1026, a hard disk drive (HDD) 1012, a compact disk read-only memory (CD-ROM®) drive 1014, a universal serial bus (USB®) port and other communication ports 1018, and a peripheral component interconnect / peripheral component interconnect express (PCI® / PCIe®) device 1020 connect to the SB / ICH 1010 via a bus 1032 and a bus 1034. The PCI® / PCIe® devices may include, for example, an Ethernet® adapter, an add-in card, and a personal computing (PC) card for a notebook computer. PCI® uses a card bus controller, PCIe® does not. ROM 1026 may be, for example, a flash basic input / output system (BIOS). Modem 1024 or network adapter 1016 may be used to send and receive data over a network.

[0041] The HDD 1012 and CD-ROM® drive 1014 connect to the SB / ICH 1010 via a bus 1034. The HDD 1012 and CD-ROM® drive 1014 may use, for example, an Integrated Drive Electronics (IDE®) or Serial Advanced Technology Attachment (SATA®) interface. In some embodiments, the HDD 1012 may be replaced by other forms of data storage devices, including, but not limited to, a solid state drive (SSD). A super I / O (SIO) device 1028 may be connected to the SB / ICH 1010. The SIO device 1028 may be a chip on the motherboard configured to help perform less demanding controller functions for the SB / ICH 1010, such as controlling a printer port, controlling fans, or controlling small light emitting diodes (LEDS) of the data processing system 1000, or a combination thereof.

[0042] The data processing system 1000 may include a single processor 1002 or multiple processors 1002. Additionally, the processor 1002 may have multiple cores. For example, in one embodiment, the data processing system 1000 may employ multiple processors 1002, including hundreds or thousands of processor cores. In some embodiments, the processors 1002 may be configured to perform a set of coordinated calculations in parallel.

[0043] An operating system executes on the data processing system 1000 using the processor 1002. The operating system coordinates and provides control for various components within the data processing system 1000 of FIG. 10. Various applications and services may operate in conjunction with the operating system. Instructions for the operating system, applications, and other data may be located on a storage device, such as one or more HDDs 1012, and loaded into the main memory 1004 for execution by the processor 1002. In some embodiments, additional instructions or data may be stored in one or more external devices. The processes described herein for the exemplary embodiments may be executed by the processor 1002 using computer usable program code, which may be located in a memory, such as the main memory 1004, the ROM 1026, or in one or more peripheral devices.

[0044] The present invention may be a system, method, or computer program product, or combination thereof, at any possible level of technical detail integration. The computer program product may include a computer readable storage medium having computer readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0045] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes the following: portable computer diskettes®, hard disks, random access memories (RAM), read only memories (ROM), erasable programmable read only memories (EPROM or flash memory), static random access memories (SRAM), portable compact disk read only memories (CD-ROM®), digital versatile disks (DVD®), memory sticks®, floppy disks®, mechanically encoded devices such as punch cards or ridge structures in grooves having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not intended to be construed as being a transient signal itself, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or an electrical signal transmitted over a wire.

[0046] The computer readable program instructions described herein may be downloaded from the computer readable storage medium to the respective computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may comprise copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions for storage in the computer readable storage medium in the respective computing / processing device.

[0047] The computer readable program instructions for carrying out the operations of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or object oriented programming languages ​​such as Smalltalk®, C++®, and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer readable program instructions to personalize the electronic circuitry by utilizing state information of the computer readable program instructions to perform aspects of the invention.

[0048] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0049] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to manufacture a machine such that the instructions, executed via the processor of the computer or other programmable data processing apparatus, generate means for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions, which may direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, may also be stored on a computer readable storage medium such that a computer readable storage medium having instructions stored therein includes an article of manufacture including instructions for performing aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0050] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, and cause the computer, other programmable apparatus, or other device to perform a series of operational steps to produce a computer-implemented method, such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / acts specified in one or more blocks in the flowcharts and / or block diagrams.

[0051] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions that includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur in a different order than that noted in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously or the blocks may sometimes be executed in reverse order depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, as well as combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a special purpose hardware-based system that performs the specified functions or acts or executes a combination of special purpose hardware and computer instructions.

[0052] The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. Furthermore, the steps of the methods described herein may be performed in any suitable order, or simultaneously where appropriate. The terms used herein are selected to best explain the principles of the embodiments, practical applications or technical improvements to the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0053] FIG. 10 is a block diagram illustrating a hardware architecture of a system 1000 according to an embodiment of the present disclosure in which aspects of an exemplary embodiment may be implemented. For example, the data processing system 1000 may be configured to store and execute instructions for performing the processes described in FIG. 2 and FIG. 3. In the illustrated example, the data processing system 1000 employs a hub architecture including a north bridge and memory controller hub (NB / MCH) 1006 and a south bridge and input / output (I / O) controller hub (SB / ICH) 1010. The processor 1002, main memory 1004, and graphics processor 1008 are connected to the NB / MCH 1006. The graphics processor 1008 may be connected to the NB / MCH 1006 via an accelerated graphics port (AGP®). A computer bus, such as bus 1032 or bus 1034, may be implemented using any type of communication fabric or architecture that provides for the transfer of data between different components or devices attached to the fabric or architecture.

[0054] In the illustrated example, a network adapter 1016 connects to the SB / ICH 1010. An audio adapter 1030, a keyboard and mouse adapter 1022, a modem 1024, a read-only memory (ROM) 1026, a hard disk drive (HDD) 1012, a compact disk read-only memory (CD-ROM®) drive 1014, a universal serial bus (USB®) port and other communication ports 1018, and a peripheral component interconnect / peripheral component interconnect express (PCI® / PCIe®) device 1020 connect to the SB / ICH 1010 via a bus 1032 and a bus 1034. The PCI® / PCIe® devices may include, for example, an Ethernet® adapter, an add-in card, and a personal computing (PC) card for a notebook computer. PCI® uses a card bus controller, PCIe® does not. ROM 1026 may be, for example, a flash basic input / output system (BIOS). Modem 1024 or network adapter 1016 may be used to send and receive data over a network.

[0055] The HDD 1012 and CD-ROM® drive 1014 connect to the SB / ICH 1010 via a bus 1034. The HDD 1012 and CD-ROM® drive 1014 may use, for example, an Integrated Drive Electronics (IDE®) or Serial Advanced Technology Attachment (SATA) interface. In some embodiments, the HDD 1012 may be replaced by other forms of data storage devices, including, but not limited to, a solid state drive (SSD). A super I / O (SIO) device 1028 may be connected to the SB / ICH 1010. The SIO device 1028 may be a chip on the motherboard configured to help perform less demanding controller functions for the SB / ICH 1010, such as controlling a printer port, controlling fans, or controlling small light emitting diodes (LEDs) of the data processing system 1000, or a combination thereof.

[0056] The data processing system 1000 may include a single processor 1002 or multiple processors 1002. Additionally, the processor 1002 may have multiple cores. For example, in one embodiment, the data processing system 1000 may employ multiple processors 1002, including hundreds or thousands of processor cores. In some embodiments, the processors 1002 may be configured to perform a set of coordinated calculations in parallel.

[0057] An operating system executes on the data processing system 1000 using the processor 1002. The operating system coordinates and provides control for various components within the data processing system 1000 of FIG. 10. Various applications and services may operate in conjunction with the operating system. Instructions for the operating system, applications, and other data may be located on a storage device, such as one or more HDDs 1012, and loaded into the main memory 1004 for execution by the processor 1002. In some embodiments, additional instructions or data may be stored in one or more external devices. The processes described herein for the exemplary embodiments may be executed by the processor 1002 using computer usable program code, which may be located in a memory, such as the main memory 1004, the ROM 1026, or in one or more peripheral devices.

Claims

1. 1. A system comprising: a memory for storing instructions for performing a method for determining an intervention response category for a provisioning workflow; and a processor configured to execute said instructions, the method comprising: Determining a provisioning function of a provisioning step; performing outlier detection to identify and remove outliers from the unattended data to generate an unattended normal data set; performing iterative grouping on the non-intervention normal data set to determine significant variables in the provisioning function; performing response mapping of provisions using the results of said iterative grouping, including the significance of errors and the presence or absence of errors in said unintervention normal data and said partial intervention data, to classify said provisions into response categories; A method comprising:

2. The method of claim 1 , wherein the response category is one of an immediate response, a delayed response, and no response required.

3. The method of claim 1 , wherein the outlier detection uses a local outlier factor (LOF) method to identify the outliers in the unattended data.

4. The processor determining a provisioning function of a provisioning step, calculating step durations of said provisioning steps for all said provisions using an event log associated with said provisioning steps; identifying provisioning parameters for said provisioning step; identifying error messages in the event log that occur as part of the provisioning step; The method of claim 1 , comprising:

5. The processor performing the iterative grouping using the unintervention normal data set, calculating an F statistic based on the provisioning parameters, the error message, and the step duration; determining important variables based on the F statistic; clustering the non-intervention normal data set into a plurality of clusters based on combinations of values ​​of the important variables; The method of claim 4 , comprising:

6. 6. The method of claim 5, wherein the percentage of data points that exceed n times the interquartile range, where n is a variable, is compared to a threshold for each cluster to determine whether the cluster is a candidate for further division.

7. The method of claim 5 , wherein the duration threshold is calculated for each cluster by calculating n×interquartile range for each cluster, where n is a variable.

8. 6. The method of claim 5, wherein if there is an error in the partial intervention data and in the no-intervention normal data set, and the error is a significant variable as a result of repeated grouping, then the response category is delayed response.

9. 9. The method of claim 8, wherein the time to alarm of the error is the maximum of all duration thresholds in all clusters that have non-zero and non-NULL duration thresholds.

10. 2. The method of claim 1, wherein if an error is present in the partial intervention data and not present in the no-intervention normal data set, the response category is immediate action.

11. 2. The method of claim 1, wherein if an error is present in the no-intervention normal data set and is absent in the partial intervention data set, then the response category is no action required.

12. 1. A system comprising: a memory for storing instructions; and a processor configured to execute the instructions, the instructions comprising: Determining a provisioning function of a provisioning step; performing outlier detection to identify and remove outliers from the unattended data to generate an unattended normal data set; performing iterative grouping on the non-intervention normal data set to determine significant variables in the provisioning function; performing response mapping of provisions using the results of said iterative grouping, including the significance of errors and the presence or absence of errors in said unintervention normal data and said partial intervention data, to classify said provisions into response categories; This is an instruction to the system.

13. The system of claim 12 , wherein the response category is one of an immediate response, a delayed response, and no response required.

14. 13. The system of claim 12, wherein the processor is further configured to execute the instructions to identify the outliers in the unintervention data using a local outlier factor (LOF) method.

15. The processor, calculating step durations of said provisioning steps for all said provisions using an event log associated with said provisioning steps; identifying provisioning parameters for said provisioning step; identifying error messages in the event log that occur as part of the provisioning step; The system of claim 12 , further configured to execute the instructions to:

16. The processor, calculating an F statistic based on the provisioning parameters, the error message, and the step duration; determining important variables based on the F statistic; clustering the non-intervention normal data set into a plurality of clusters based on combinations of values ​​of the important variables; The system of claim 15 , further configured to execute the instructions to:

17. 17. The system of claim 16, wherein the percentage of data points that exceed n times the interquartile range, where n is a variable, is compared to a threshold for each cluster to determine whether the cluster is a candidate for further division.

18. 16. The system of claim 15, wherein the duration threshold is calculated for each cluster by calculating n times the interquartile range for each cluster, where n is a variable.

19. 16. The system of claim 15, wherein the response category is a delayed response if an error is a significant variable as a result of repeated grouping and is present in the partial intervention data and is present in the no-intervention normal data set, and the time to alarm for the delayed response is the maximum of all duration thresholds in all clusters with non-zero and non-NULL duration thresholds, the response category is an immediate response if an error is present in the partial intervention data and not present in the no-intervention normal data set, and the response category is no response required if an error is present in the no-intervention normal data set and not present in the partial intervention data.

20. A computer program comprising the method according to any one of claims 1 to 11. A computer program for causing a computer to execute the above.

Images