Correlation of stack segment strengths in emerged relationships

Abstract

To provide a method for sequentially analyzing a series of thread damp samples to estimate intensity statistical data of a stack segment of a newly classified stack frame.SOLUTION: A method detects branch points along one or more stack frames in linear connection with stack segments (A, B, D) associated with a filter state, and classifies the stack segments into a plurality of new stack segments each of which includes a subset of a stack frame. The plurality of new stack segments are referred to by the stack segments. Next, the method initializes each filter state of the new stack segments on the basis of at least a part of the filter state of the stack segments.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application was filed on May 5, 2017, and is related to the application "Correlation of Stack Segment Intensity No. 15 / 588,521, entitled “Correlation of Stack Segment Strength in Emergent Relationships,” filed May 9, 2016, No. 62 / 333,786, filed May 9, 2016, entitled “Memory Usage Determination Techniques,” and U.S. Provisional Application No. 62 / 333,786, filed May 9, 2016, entitled “Memory Usage Determination Techniques,” No. 62 / 333,798, filed May 9, 2016, entitled “Compression Techniques for Encoding Stack Traces Information,” U.S. Provisional Application No. 62 / 333,804, filed May 9, 2016, entitled “Correlation of Stack Segment Intensity in Emergent Relationships,” U.S. Provisional Application No. 62 / 333,811, filed May 9, 2016, entitled “Systems and Methods of Stack Trace Analysis,” U.S. Provisional Application No. 62 / 333,821, filed May 9, 2016, entitled “Systems and Methods of Stack Trace Analysis,” Provisional Application No. 62 / 333,809, filed May 23, 2016, entitled “Characterization of Segments of Time-Series,” Application No. 62 / 340,256, filed May 5, 2017, entitled “Correlation of Thread Intensity and Heap Usage to Identify Heap-Hoarding Stack Traces” No. 15 / 588,531, filed May 5, 2017, entitled “Correlating Thread Intensity with Heap Usage to Identify Stack Traces Infested with Threads”; US Patent Application No. 15 / 588,526, filed on May 5, 2017, entitled "Compression Techniques for Encoding Stack Trace Information" This application is an international application of U.S. utility application Ser. No. 15 / 588,523, entitled "Compression Techniques for Encoding Video Signals," which claims the benefit of priority under 35 U.S.C. §119(e) to these applications, all of which are incorporated herein by reference for all purposes.

[0002] This application is a related application to the following applications filed concurrently herewith, all of which are incorporated herein by reference for all purposes:

[0003] (1) A patent application filed on May 8, 2017, entitled "CORRELATION OF THREAD INTENSITY AND HEAP USAGE TO IDENTIFY HEAP-HOARDING STACK TRACES No. PCT utility application PCT / US2017 / 031586 (Attorney Docket No. 088325-1047566 (175200PC)), entitled "Correlation of Thread Intensity and Heap Usage to Identify Thread Races."

[0004] (2) A patent application filed on May 8, 2017, entitled "MEMORY USAGE DETERMINATION TECHNIQUES ( and PCT utility patent application PCT / US2017 / 031589 (Attorney Docket No. 088325-1047576 (175210PC)), entitled "Memory Usage Determination Technique."

[0005] (3) Filed on May 8, 2017, entitled "COMPRESSION TECHNIQUES FOR ENCODING STACK TRACE INFORMATION" PCT utility patent application no. PCT / US2017 / 031593 (Attorney Docket No. 088325-1047578 (175220PC)). [Background technology]

[0006] background Typically, cloud service providers maintain operational resources to meet Service Level Agreements (SLAs) with their customers. Providers constantly monitor the performance metrics of the services they offer to ensure that cloud services conform to the SLAs. However, the operational resources will not be able to avoid impending SLA violations, as the available tools may not be capable of predicting or detecting such violations. In addition, the tools may not be capable of diagnosing the root causes of SLA violations, so operations will take more time to resolve such violations when they occur, which can negatively impact customer experience.

[0007] Moreover, such SLAs may require systematic analysis of the data to proactively act on actionable information contained therein to avoid SLA violations and determine whether contractual commitments have been met. Complying with service level agreements and other requirements can be very burdensome and become even more burdensome over time. Summary of the Invention [Problem to be solved by the invention]

[0008] To obtain the above capabilities, an approach is needed to represent the system using a high-level state model that can be easily updated based on the system's low-level events and system measurements. With respect to obtaining metrics for low-level events, the application programs underlying the system can be instrumented to collect accurate measurements of events. However, in such an approach, the instrumentation itself can affect the measurements. This problem can be more pronounced when the execution time of the instrumentation code before and after a method dominates the execution time of the method itself (e.g., when the number of method invocations is high).

[0009] overview A particular method is disclosed for sequentially analyzing a set of thread dump samples to estimate stack segment intensity statistics of newly classified stack frames. [Means for solving the problem]

[0010] One embodiment is directed to a method that may include determining, by a computer, a filter state of a stack segment referencing a set of linearly connected stack frames based on at least a portion of one or more stack traces, receiving the stack traces, and upon receiving the stack traces, detecting a branch point along the set of linearly connected stack frames of the stack segment based on at least a portion of the stack traces, and upon detecting a branch point along the set of linearly connected stack frames of the stack segment, splitting the stack segment into a plurality of new stack segments, each new stack segment including a subset of the stack frames referenced by the stack segment, the plurality of new stack segments, the method may further include initializing a plurality of filter states corresponding to the plurality of new stack segments based at least in part on the filter state of the stack segment, and separately tracking each of the filter states of the stack segment and the plurality of filter states corresponding to the plurality of new stack segments based at least in part on one or more subsequent stack traces.

[0011] Exemplary embodiments are described in detail below with reference to the following drawings: [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 illustrates an example of profiling the runtime of a single thread over a period of time with a relatively high frequency sampling rate. [Diagram 2] FIG. 2 illustrates an example invocation context tree. [Diagram 3] FIG. 2 illustrates an example of a thread dump of a virtual machine over a period of time according to some embodiments. [Figure 4] FIG. 2 illustrates an example of a thread classification signature according to some embodiments. [Diagram 5] FIG. 2 illustrates an example of a thread classification signature according to some embodiments. [Figure 6] FIG. 2 illustrates an example of a thread classification signature according to some embodiments. [Figure 7] 1 is a simplified flowchart illustrating generating and / or modifying one or more thread classification signatures in response to a thread dump, according to some embodiments. [Figure 8] 1 is a simplified flowchart illustrating the generation or modification of a thread classification signature in response to detecting a branch point. [Figure 9] 1 is a simplified flowchart illustrating identifying code corresponding to high heap usage according to some embodiments. [Figure 10] 1 is a simplified flowchart illustrating a calculation of the correlation between various classes of threads and high heap usage according to some embodiments. [Figure 11] 1 is an exemplary graph in which weights assigned to sample measurements are plotted against the sampling time interval associated with the sample measurements over the time domain of an exemplary data set. [Figure 12] 1 is an exemplary chart showing trend graphs derived by different linear regression methods for determining heap usage in a production environment. [Figure 13] 1 is an exemplary chart showing another trend graph illustrating erroneous results obtained by standard robust regression methods. [Figure 14] 4 is a simplified flowchart illustrating generation of a prediction of a signal, according to some embodiments. [Figure 15] 1 is a schematic diagram of a distributed system for implementing certain embodiments. [Figure 16] FIG. 1 is a simplified block diagram of one or more components of a system environment in which services may be provided as a cloud service, according to some embodiments. [Figure 17] FIG. 1 illustrates an example computer system that can be used to implement certain embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The patent or application file contains at least one drawing executed in color. Copies of such patent or patent application publications with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0014] Detailed Description I. Overview In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. However, it will be apparent that various embodiments may be practiced without these specific details. The drawings and description are not intended to be limiting.

[0015] The present disclosure generally relates to utilizing heap usage statistics and thread intensity statistics to identify code blocks within a multi-threaded process (e.g., an application program) for potential optimization and to predict future heap usage and / or thread intensity. Thread intensity statistics may be utilized to track the response, load, and resource usage of a process without instrumenting the underlying code of the process or using code injection. Specifically, the intensity of a thread type or stack segment type may be determined by the number of threads that the thread is executing or the stack segment is referencing. It may also refer to a statistical measure of the "hotness" of a code block. The hotness of a block may be quantified by the amount of execution (e.g., the number of times the code block is called times the execution time of the code block). Hotter code blocks have more calls and / or longer response times.

[0016] By analyzing a series of thread dumps output from a process at regular or irregular time intervals, some embodiments may provide a statistical sampling solution that is (1) low overhead, (2) non-disruptive, (3) always-on monitoring, and (4) avoids the problem of instrumented code dominating the execution time of the code being instrumented (i.e., the Heisenberg problem).

[0017] Some embodiments may classify threads and stack segments based on intensity statistics. By monitoring stack traces of individual threads contained in a thread dump received from a software execution environment (e.g., a virtual machine), the monitoring process may classify the threads into one or more thread classes based on the contents of the thread's stack trace. As more stack traces are analyzed, some embodiments may observe a branching of thread classes into subclasses, eventually creating a hierarchy of thread classes. For example, if stack segment (A) is observed to be a component of stack segment (A, B, D), thread type (A, B, D) may be said to be a subclass of thread type (A). Also, thread type (A, C) may be said to be a subclass of thread type (A). Thread type (A) includes subclasses (A, B, D) and (A, C) in the sense that the ensemble of intensity statistics corresponding to (A, B, D) and (A, C) may be represented by the intensity statistics corresponding to (A). In addition, some embodiments may walk down the thread class hierarchy (e.g., traverse a tree or graph) and observe how the intensity of a particular thread class may be proportional to the intensity of one or more subclasses of this thread class. For example, the thread intensity of (A) may be proportional to the thread intensity of (A,B,D) and (A,C). In other embodiments, each stack trace may be represented as a binary tree.

[0018] Some embodiments may provide one or more sequential filters for estimating measurements, rates of change, accelerations, seasonal factors, and residuals. Techniques for representing separate seasonal indices for multiple time periods (e.g., weekday and weekend periods) and normalizing the seasonal factors for the multiple time periods may be performed by such embodiments. Specifically, some embodiments may represent separate sequences of seasonal indices for each of the multiple time periods. For example, the multiple time periods may include weekday periods, weekend periods, end-of-quarter periods, or individual holiday periods. Also, in estimating seasonal indices for the multiple time periods, some embodiments may (1) renormalize the seasonal indices to a common sequence across all time periods; (2) fit smoothing splines across adjacent time periods to provide a smooth transition between cycles in one time period or between cycles in two adjacent time periods. By renormalization, seasonal factors between multiple time periods may have a common scale.

[0019] Some embodiments may correlate trends between the heap usage statistics and the intensity statistics of various thread classes to identify classes of threads that have intensity statistics highly correlated with high heap usage. In a software execution environment, there is a high probability that inefficient heap memory usage will be found among classes of threads whose intensity statistics are highly correlated with high heap usage. Once the classes of threads are identified, code associated with the classes of threads may be investigated and / or optimized.

[0020] Some embodiments may involve the use of a multi-threaded environment (e.g., a virtual machine) that executes processes. A system may build and maintain models (e.g., univariate models, multivariate models) of the thread classes (e.g., seasonal trends, linear trends, and first-order nonlinear trends of the intensity of each thread class). Such models may be used to obtain seasonally adjusted long-term forecasts of trends in the system's performance.

[0021] By (1) dynamically classifying threads to observe how the intensity of subclasses of a thread class contributes to the total intensity of a thread class, and (2) observing how closely various classes of threads correlate with detected periods of high heap usage, some embodiments may facilitate detection and observation of performance deficiencies within a cloud service provisioning system. Because even minor performance deficiencies often reveal problems in a process that may result in SLA violations, enabling service providers to detect and address performance deficiencies may significantly reduce the risk of such violations.

[0022] II. Thread runtime profiling 1-2 are diagrams illustrating an approach to profiling a running thread to determine how long various stack segments reside on the thread's call stack relative to one another. FIG. 1 illustrates an example of profiling the runtime of a single thread 100 over a period of time at a relatively high frequency sampling rate. In some cases, a particular approach may utilize a runtime profiler to output multiple stack trace samples of the thread and construct a tree of calling contexts 200 as shown in FIG. 2. If the runtime profiler employs a sampling interval that is short compared to the execution time of the thread, statistics on the number of observations (i.e., number of calls) per calling context of the thread can be used to accurately estimate and / or represent the execution time of the calling context relative to the sampling interval.

[0023] For example, as shown in Figure 1, the total execution time of thread 100 may be between 100 milliseconds and 1 second, while the sampling interval is between 10 milliseconds and 100 milliseconds. Depending on which methods the thread calls, different invocation contexts may exist in the thread's stack during the thread's execution. The thread may begin its execution by invoking a set of methods corresponding to stack segment A.

[0024] Note that a stack segment corresponds to a set of one or more linearly connected stack frames. Linearly connected stack frames are always observed together in a stack trace, so their intensity statistics are the same. Thus, stack segment A may correspond to multiple stack frames, such as stack frames a1, a2, and a3. Sampling a thread may result in a stack trace that describes the entire calling context of the sampled thread with a list of stack frames. If some of the stack frames in the list are linearly connected, these stack frames may be conceptually grouped together into one stack segment. As a result, the stack trace will contain one or more stack segments, each containing one or more stack frames.

[0025] As the thread continues its execution, code associated with stack segment A may cause the thread to invoke a set of methods that correspond to stack segment B. In turn, code associated with stack segment B may cause the thread to invoke yet another set of methods that correspond to stack segment D. After a short period of time, the runtime profiler may take sample 1 of thread 100, resulting in a first stack trace. From the first stack trace, the runtime profiler can determine that stack segments A, B, and D were on the stack at the time of the sample. One sample interval The runtime profiler may then take another sample 2 of the thread, resulting in a second stack trace. From the second stack trace, the runtime profiler can determine that stack segments A, B, and D were on the stack. As the thread continues execution, the method associated with stack segment D may return, resulting in a stack frame corresponding to stack segment D being popped off the stack. The runtime profiler may then take another sample 3 of the thread, resulting in a third stack trace. From the third stack trace, the runtime profiler can determine that stack segments A and B were on the stack.

[0026] As the thread executes, stack segment B calls stack segment E, which calls stack segment F. Then, sample 4 is taken, resulting in the fourth stack trace, which shows that stack segments A, B, E, and F were on the stack. Stack segments F, E, and B return each other. Then, sample 5 is taken, resulting in the fifth stack trace, which shows that only stack segment A is on the stack. Stack segment A pushes stack segment C onto the stack. Before stack segment C returns, samples 6 and 7 are taken, resulting in the sixth and seventh stack traces, which show that stack segments A and C are on the stack. Finally, stack segment C returns, leaving only stack segment A on the stack. When the method associated with stack segment A returns, the thread finishes execution.

[0027] As shown in FIG. 2, a tree 200 of invocation contexts represents the execution times of stack segments A-F over a sampling interval. Node 202 indicates that stack segment A was observed in all seven samples. Node 204 indicates that stack segment B was observed in four of the seven samples. Node 206 indicates that stack segment C was observed in two of the seven samples. Node 208 indicates that stack segment D was observed in two of the seven samples. Node 210 indicates that stack segment E was observed in one of the seven samples. Node 212 indicates that stack segment F was observed in one of the seven samples. Since the total execution time of thread 100 is approximately ten times the duration of the sampling interval, the number of observations of each stack segment will correlate closely with the execution time of the stack segment. For example, since stack segment B was observed four times, it can be inferred that the relative execution time of stack segment B is at least four times the sampling interval.

[0028] In some cases, the environment in which thread 100 runs (i.e., the software execution environment) may correspond to a virtual machine (e.g., Hotspot Java Virtual Machine (JVM)) in which one thread dump is output once per sampling interval. Before outputting the thread dump, the virtual machine may signal all of the running threads (e.g., thread 100) to pause at a safepoint. This safepoint mechanism may be similar to that used by a garbage collector to pause threads before performing a full garbage collection. Note that a thread running in kernel mode (e.g., running / blocking on an I / O operation) may not be paused at a safepoint until the thread leaves kernel mode and returns (e.g., back to JVM mode).

[0029] However, invoking the safepoint mechanism at a high frequency rate leads to significant overhead, so runtime profiling techniques that rely on high sampling rates may be more appropriate for development or testing environments than for production environments.

[0030] To reduce overhead, some embodiments reduce the sampling rate. Employ a system model to compensate. For example, some embodiments may track the intensity of threads of a multithreaded process and sample only those threads with an intensity above a threshold that determines the wait time. One advantage of embodiments employing a low or adaptive sampling rate is that threads running in kernel mode are less likely to be paused at a safe point. Other methods of reducing overhead may require increasing the sampling interval commensurate with the intensity of the threads being sampled. For example, a one-minute sampling interval may result in negligible overhead in a production environment, but may be too short to derive the relative execution times of a thread and its component stack segments in a production environment. Thus, some embodiments may provide an always-on performance monitoring solution for a production system that exhibits stationary mean ergodicity or cyclostationary mean ergodicity to match the assumptions of Little's formula. In such an embodiment, the always-on performance monitoring solution may be included in a monitoring process (i.e., a control system) that periodically samples threads running in one or more virtual machines of the production system.

[0031] III. Thread Classification Various embodiments provide techniques for sequentially analyzing a series of thread dump samples output from one or more virtual machines (e.g., JVMs) to identify thread classes and track intensity statistics related to those thread classes. For example, during execution of one or more multi-threaded processes within a virtual machine, a control system may periodically output thread dumps of the virtual machine. The thread dumps may provide a stack trace for each thread running in the virtual machine. For each stack trace received, the control system may analyze the text contained in the stack trace to classify the associated thread and update the intensity statistics tracked for all thread classes based on the stack trace.

[0032] In addition to classifying threads, embodiments may classify new stack segments as they appear at branching points along previously classified stack segments. If the control system observed a first stack trace before the thread class was discovered, the control system may consider the entire sequence of stack frames in the stack trace to be linearly connected because so far the entire sequence of stack frames have only appeared together. In response, the control system may initialize a thread class to classify the entire stack trace (i.e., the entire sequence of stack frames). As the control system observes subsequent stack traces that include different sequences of stack frames, the control system may initialize additional thread classes to classify each unique permutation of stack frames. In some cases, the control system may observe a stack trace that does not share a stack frame (i.e., has no stack frames in common) with the previously observed stack trace. In response, the control system may initialize another thread class to classify the entire new stack trace.

[0033] More generally, however, a control system may observe stack traces that share one or more stack frames with a previously observed stack trace. Returning to Figure 1, for example, suppose the first stack trace observed by the control system is {(A,B,D)} (i.e., the stack trace in specimen 1 or specimen 2). This stack trace contains stack frames contained in stack segments A, B, and D. The control system may initialize a thread class {(A,B,D)} to classify all threads observed as containing stack frames contained in stack segments A, B, and D. Then, suppose the second stack trace observed by the control system is {(A,C)} (i.e., the stack trace in specimen 6 or specimen 7). In this regard, the control system may determine that the first stack trace and the second stack trace are distinct from one another, but the first stack trace and the second stack trace share all of the stack frames contained in stack segment A. This results in a branch point in stack segment A. In response, the control system may initialize a thread class {(A,C)} to classify all threads that include stack segments A and C on their call stacks.

[0034] Note that because the stack frames contained in stack segment A were observed separately from the stack frames contained in stack segment (B,D), stack segments A and (B,D) are no longer considered by the control system to be linearly connected. However, the control system still considers the stack frames contained in stack segment A to be linearly connected, and the stack frames contained in stack segment (B,D) to be linearly connected. In this regard, the control system may initialize some thread segment components of thread class {(A,B,D)} and thread class {(A,C)} to classify the new stack segment formed by the newly discovered branch point. Specifically, the control system may initialize thread segment (A), thread segment (B,D), and thread segment (C). Thread segments (A) and (B,D) are components of thread class {(A,B,D)}, and thread segments (A) and (C) are components of thread class {(A,C)}.

[0035] Some embodiments may represent stack traces and stack segments using classification signatures. Specifically, a trace signature can be used to represent stack traces of a particular thread class, and a segment signature can be used to represent stack segments of a particular thread segment. Each trace signature is a tuple of labeled binary trees created by the composition analysis process. On the one hand, each segment signature of a thread segment may correspond to a node of a tuple corresponding to a thread class, with the thread segments of that thread class being members of the tuple. Later in the parsing process, the tuple may be used like a parse tree (e.g., as part of a production grammar) to recognize incoming stack traces.

[0036] Returning to the above example, after observing the first stack trace but before observing the second stack trace, the thread class {(A,B,D)} may correspond to a set of one binary tree. The one binary tree may include one root node representing stack segment (A,B,D) since the entire sequence of frames in the first stack trace is considered as one stack segment. After observing the second stack trace, the set would still include only one binary tree. However, now the binary tree may include three separate nodes: a root node representing stack segment (A,B,D), a first child node of the root node representing stack segment (A), and a second child node of the root node representing stack segment (B,D). The process of combining trace signatures and segment signatures is described in more detail below with reference to Figures 4-6.

[0037] Each node in the binary tree may be uniquely identified by a label or identifier, which may be referred to as a compact code. In some embodiments, threads of a particular thread class may be represented by one or more compact codes that identify each top node of the set corresponding to the thread class. In a manner similar to Huffman coding or other entropy coding schemes, some embodiments may associate shorter sets with thread classes that are more popular (i.e., of higher thread intensity) and / or that are discovered first. As a result, more common types of threads can be compactly represented by sequences of shorter compact codes. In some embodiments, this can be achieved by performing offline analysis ( This may be ensured by first analysing the probability distribution of stack traces (i.e., offline processing) and then sending the stack traces in decreasing order of frequency to the control system.

[0038] In embodiments that do not rely on offline analysis, the control system may receive stack traces in sequence along with thread dumps that are periodically output from one or more virtual machines (i.e., online processes).

[0039] The order in which stack traces of different types are observed can be affected by the intensity of each type of stack trace; that is, stack traces of higher intensity tend to be statistically observed earlier in the sequence. Thus, such embodiments may assume that (1) the thread intensity of a particular thread class represents the probability of occurrence of the associated stack trace, and (2) stack traces associated with higher intensity thread classes are more likely to be observed before stack traces associated with lower intensity thread classes. In this regard, the control system naturally derives the most compact representation for the highest intensity threads. Thus, by relying on thread intensity statistics rather than offline processing, some embodiments can provide a compression algorithm that is optimal for stack traces observed in response to a series of thread dumps.

[0040] A. Seasonal variation in thread strength

[0041]

number

[0042] In some embodiments, the seasonal trending process may use variable filter parameters to account for irregular sampling intervals (e.g., heap usage sampling and / or thread dump output) and to overcome the Cauchy distribution problem. The process may also support sequential filtering of multiple types of time periods of various lengths (e.g., 1 day, 2 days) (e.g., weekday time periods, weekend time periods, and holiday time periods). The process may also adjust the thread dump output rate to reduce overhead in response to seasonal variations while maintaining a particular confidence level of the thread intensity statistics determined based on the thread dumps. In some cases, adjusting the thread dump rate may also minimize the amount of thread dump data that needs to be transported over a network (e.g., LAN, Internet) to another machine (e.g., big data repository) for offline processing.

[0043] In some embodiments, the seasonal trending process may divide a weekday period (i.e., 24 hours) into 96 15-minute intervals, resulting in 96 seasonal indices (i.e., seasons) for each weekday period. The process may divide a weekend period (i.e., 48 hours) into 192 15-minute intervals, resulting in 192 seasonal indices for each weekend period. Upon receiving a dataset of a certain length (e.g., a time series recording thread dumps or heap usage for a 10-day period that includes one or two weekends), the process may apply a multi-period trending filter separately to the weekday and weekend periods to separate seasonal patterns observed over a weekday from those observed over an entire weekend, resulting in a set of 96 seasonal factors for the 96 seasonal indices included in each weekday and a set of 192 seasonal factors for the 192 seasonal indices included in each weekend. The process may then renormalize the weekday and weekend seasonal factors such that a seasonal factor of "1" represents a common baseline level for weekday and weekend periods.

[0044] It should be noted that if a seasonal index is assigned a seasonal factor greater than 1, the seasonal index will have a higher value than the average value compared to the rest of the time period. Conversely, if a seasonal index is assigned a seasonal factor less than 1, the seasonal index will have a lower value than the average value compared to the rest of the time period. For example, if the seasonal factor of thread intensity for a particular thread class for a seasonal index corresponding to the interval 9:00AM-9:15:00AM is 1.3, then the average thread intensity for that particular thread class during the interval 9:00AM-9:15:00AM is 30% higher than the average thread intensity for the entire weekday for that particular thread class.

[0045] In some embodiments, weekday periods repeat every 24 hours and weekend periods repeat every 5 or 7 days, but the seasonal trending process separates holidays (e.g., Labor Day, Christmas) into separate periods that repeat once every 12 months. Such a set of seasonal factors for holiday periods may be renormalized along with the sets of seasonal factors for weekday and weekend periods, such that a seasonal factor of 1 represents a common base level for all periods. Other frequencies may be used for each period as appropriate, as needed. For example, weekdays may be periods that repeat every 12 hours, while holidays may be separated by a frequency such as every 6 months.

[0046] In some embodiments, determining and tracking the intensity statistics may further include predicting future values ​​and rates of change. However, sampling intervals may be irregular or even approach zero randomly. If the sampling intervals approach zero randomly, the rates of change may become Cauchy-distributed random variables with undefined mean and standard deviation. To overcome the Cauchy distribution problem of determining seasonal trends using adaptive sampling intervals, some embodiments may employ various adaptations such as Holt's double exponential filter, Winter's triple exponential filter, Wright's extension for irregular intervals, Hanzak's time-proximate interval adjustment factor, outlier detection, and clipping with adaptive scaling of outlier cutoffs. Five sets of exponential filters may be sequentially adapted to the dataset to estimate sets of seasonal factors for weekday and weekend periods.

[0047] B. Classification Signatures and Compression Methods Particular embodiments may assign a variable length sequence of compact codes to a thread's stack trace, where the length of the sequence varies depending on the thread's strength. An example stack trace is shown below:

[0048] [Table 1]

[0049] In the example stack trace, you can see the stack frame "oracle.jdbc.driver..." under the stack segment for the Java Database Connectivity (JDBC) driver (i.e., the two stack frames containing "oracle.jdbc.driver..."). The stack frame "oracle adf model servlet ADFBindingFilter doFilter" under the stack segment of the MDS library (i.e., the three stack frames that contain "oracle.mds...") indicates that the MDS operation is invoked by an Application Development Framework (ADF) operation. As shown in the lower WebLogic stack segment in this stack trace (i.e., the four stack frames that contain "weblogic..."), the ADF operation is invoked by a Hypertext Transfer Protocol (HTTP) Servlet request.

[0050] As an example, the above stack trace can be encoded and compressed using a two-level Huffman coding scheme, resulting in a sequence of compact codes representing the example stack trace. At the first level, a compression tool (e.g., gzip) can detect substrings in the stack trace, such as "ServletRequestImpl.java" and "weblogic.servlet.internal.ServletRequestImpl.run", and derive Huffman codes for the substrings depending on how frequently they occur in the stack trace. To increase the compression ratio, shorter Huffman codes can be assigned to more frequently occurring substrings. After the first level of compression, the compressed stack trace may contain as metadata an encoding dictionary that can be used to recover the substrings from their Huffman codes.

[0051] The second level places the stack segments of the stack trace into segment signatures. By replacing the stack trace with another level of compression, it may be necessary to add another level of compression to the compressed stack trace. The step of adding a second level of compression is described in further detail below with reference to Figures 4-6.

[0052] C. Data Structure Examples A classification signature may be represented in memory via one or more object types. Specifically, some embodiments may use a ThreadClassificationInfo object to represent a classification signature (i.e., a trace signature) of a thread class, a SegmentInfo object to represent a classification signature (i.e., a segment signature) of a thread segment, a StackFrameInfo object to represent each element contained in a linearly connected stack frame in a stack segment, and a SeasonalTrendInfo object to encapsulate and track intensity statistics of a thread class or thread segment.

[0053] Below are example class / interface definitions that define a ThreadClassificationInfo object, a SegmentInfo object, a StackFrameInfo object, and a SeasonalTrendInfo object.

[0054] [Table 2] TIFF0007610657000004.tif174114 TIFF0007610657000005.tif42114

[0055] As can be seen from the above definitions, each ThreadClassificationInfo object, SegmentInfo object, and StackFrameInfo object has a unique identifier (i.e., ID), a name, and the same type (e.g., It includes a counter that tracks the number of times an object of the same type (same thread class, same thread segment, same type of stack frame) was observed in the most recent thread dump (i.e., numOfOccur), and another counter that tracks the number of times an object of the same type was observed in all thread dumps.

[0056] A ThreadClassificationInfo object may contain a list of SegmentInfo objects and a SeasonalTrendInfo object. In this regard, a ThreadClassificationInfo may correspond to a set of binary trees, while a list of SegmentInfo objects correspond to the nodes that make up the binary tree. A SeasonalTrendInfo object may record intensity statistics (e.g., filter states) for the thread class that the ThreadClassificationInfo object represents.

[0057] A SegmentInfo object may include a list of StackFrameInfo objects, a first child SegmentInfo object (i.e., firstSegment), a second child SegmentInfo object (i.e., secondSegment), a coalescing (i.e., parent) SegmentInfo object (i.e., coalescingSegment), a list of previous sibling SegmentInfo objects (i.e., previous nodes), a list of next sibling SegmentInfo objects (i.e., next nodes), and a SeasonalTrendInfo object. In this regard, a SegmentInfo object may correspond to a stack segment. If the SegmentInfo object corresponds to a leaf node, the list of StackFrameInfo objects may correspond to the linearly connected stack frames contained in the stack segment. If the SegmentInfo object is adjacent to a branch point, the sibling SegmentInfo object may correspond to the stack segment on the other side of the branch point, while the coalescing SegmentInfo object may correspond to a parent stack segment that includes the stack segment and the sibling stack segments. If the SegmentInfo object does not correspond to a leaf node, the child SegmentInfo object may correspond to a subsegment of the stack segment that was created when a branch point was discovered in the stack segment. A SeasonalTrendInfo object may record intensity statistics for the thread segment that the SegmentInfo object represents.

[0058] Some embodiments may classify stack segments of a stack trace by associating a list of StackFrameInfo objects that are observed together with one SegmentInfo node. That is, the SegmentInfo node is the union node of each StackFrameInfo object of a stack segment. Each StackFrameInfo object may have one union SegmentInfo node. If a branch point is detected anywhere along the linearly connected StackFrameInfo objects of a SegmentInfo node, some embodiments may create two new SegmentInfo nodes and split the linearly connected StackFrameInfo objects into a set of two linearly connected StackFrameInfo objects of the new SegmentInfo node. The two StackFrameInfo objects may then be connected via the branch point.

[0059] Each new SegmentInfo node becomes a coalescing node of a StackFrameInfo object in a portion of the segment. The FrameInfo object can be associated and updated to reference the appropriate combined SegmentInfo node. The two new SegmentInfo nodes are represented as left and right sibling nodes. The two new SegmentInfo nodes also become child nodes of the original SegmentInfo node, which becomes the parent of the two new SegmentInfo nodes. The parent SegmentInfo node can become the combined node of the two new SegmentInfo nodes.

[0060] The process of splitting stack segments in response to discovered branch points may result in a binary tree structure composed of SegmentInfo nodes. This splitting process may be viewed as a branching of thread classes (i.e., classes of stack traces) into thread subclasses. Some embodiments may continue to split stack segments into smaller stack segments as the intensity of the individual stack frames contained in the stack segments diverges over time. This allows drilling down into the thread class hierarchy to observe how the intensity of a thread class may be proportional to the intensity of a thread subclass.

[0061] In some embodiments, SegmentInfo nodes inside a binary tree are parent nodes to which StackFrameInfo objects are not all linearly connected because some stack frames are connected via branch points. In contrast, the StackFrameInfo objects of leaf SegmentInfo nodes may be linearly connected. Within a SegmentInfo node, StackFrameInfo objects connected via linear connections or branch points may be oriented as a stack with a StackFrameInfo below and a StackFrameInfo above. By convention, the upper StackFrameInfo object contained in the left sibling SegmentInfo node may be connected to the lower StackFrameInfo object contained in the right sibling SegmentInfo node via a branch point.

[0062] Each SegmentInfo node may contain a SeasonalTrendInfo object to track the intensity statistics of the thread (sub)class that the SegmentInfo node represents. When splitting a SegmentInfo node into two new child SegmentInfo nodes, some embodiments may clone the SeasonalTrendInfo object of the SegmentInfo node to create two new SeasonalTrendInfo objects and set one SeasonalTrendInfo object for each of the child SegmentInfo nodes.

[0063] Some embodiments provide the ability to replicate the filter state of the parent SegmentInfo node to the new child SegmentInfo node throughout the split process. In doing so, some embodiments can keep track of the ratio of the intensity statistics between the parent SegmentInfo node and the sibling SegmentInfo nodes. Specifically, the intensity statistics of the child SegmentInfo node are initially the same as the intensity statistics of the parent SegmentInfo node. However, as new samples are acquired, the intensity statistics of the child SegmentInfo node will begin to diverge from the intensity statistics of the parent node and from the intensity statistics of the other child SegmentInfo nodes. Because the filter states of the new stack segments are updated separately, the filter states of the new stack segments will begin to diverge from each other and from the filter states of the original stack segments.

[0064] In some cases, the intensity statistics between parent and sibling SegmentInfo nodes may converge to a constant ratio over time. Some embodiments may fit parent-child and sibling relationships between SegmentInfo nodes to define a correlation model for multivariate state estimation techniques. Specifically, if the process is stationary, the ratio of the intensity statistics between related SegmentInfo nodes will converge to a stationary state. Specifically, if the process is stationary in the strict sense or broad sense, the first and second moments of the joint probability distribution of the intensity statistics between related SegmentInfo nodes (which may include the mean, variance, autocovariance, and cross-covariance of the related SegmentInfo nodes) will not change over time. Therefore, the ratio of the intensity statistics between parent and sibling SegmentInfo nodes may be expected to converge over time. Thus, by keeping track of the intensity statistics of sibling SegmentInfo nodes through branching points and determining that the ratio of the intensity statistics between the parent SegmentInfo node and the sibling SegmentInfo node converges over time, some embodiments can use the ratio to define a correlation model for multivariate state estimation techniques. The resulting model can be used to generate anomaly detection methods and predictions.

[0065] A StackFrameInfo object may contain one or more previous and / or next StackFrameInfo objects (i.e., previous and next nodes), a connecting SegmentInfo object (i.e., coalescingSegment), and information identifying the code to which the StackFrameInfo object refers (i.e., classMethodLineNumber). If a StackFrameInfo object is not adjacent to a branch point, the StackFrameInfo object may be linearly connected to one previous stack frame and one next stack frame. A StackFrameInfo object may reference its contained SegmentInfo objects through the member variable coalescingSegment.

[0066] When it is time to process the latest thread dump, the numOfOccur member variables for each ThreadClassificationInfo object, SegmentInfo object, and StackFrameInfo object may be reset to 0. Each stack trace obtained from a thread dump may be parsed from the bottom up of the stack trace. After applying a first level Huffman encoding to compress the stack trace, each line of the stack trace may be parsed into a StackFrameInfo object. After parsing the list of StackFrameInfo objects into a list of SegmentInfo objects, some embodiments may attempt to match the list of SegmentInfo objects to a ThreadClassificationInfo object that contains a matching list of SegmentInfo objects. If no such ThreadClassificationInfo object exists, some embodiments may register a new ThreadClassificationInfo object to represent the list of SegmentInfo objects. Some embodiments may then update the numOfOccur and totalNumOfOccur member variables of the matching / new ThreadClassificationInfo object, as well as each SegmentInfo object and StackFrameInfo object contained in the matching / new ThreadClassificationInfo object. If the SegmentInfo node is a leaf-level node, the numOfOccur member variable of the node is It is equal to the numOfOccur member variable of each StackFrameInfo element contained in the node.

[0067] Some embodiments may then update the statistical measure of intensity encapsulated in the associated SeasonalTrendInfo object. Specifically, some embodiments may update the rawMeasure member variable contained in each SeasonalTrendInfo object by setting the numOfOccur member variable of the containing ThreadClassificationInfo object or SegmentInfo object to rawMeasure. Note that in some embodiments, rawMeasure may be updated every N thread dumps, in which case the SeasonalTrendInfo object's rawMeasure is set to the corresponding numOfOccur divided by N. In some embodiments, such embodiments may update the SeasonalTrendInfo object's rawMeasure member variable only if the numOfOccur member variable of the associated ThreadClassificationInfo object or the associated SegmentInfo object is non-zero. If the numOfOccur member variable is non-zero, the SeasonalTrendInfo object's rawMeasure is set to the value of numOfOccur divided by N, where N is the number of thread dumps since rawMeasure was last updated. In such an embodiment, the method treats the case when numOfOccur is zero as if no measurements are available. In this regard, rawMeasure is not updated if no measurements are available. In other words, such an embodiment tracks the number of thread dumps since rawMeasure "N" was last updated. The measurements of thread intensity may correspond to irregular time series. Note that exponential filters for irregular intervals (e.g., Holt's double exponential and Winter's triple exponential filters described above) can effectively filter rawMeasure to obtain de-seasonalized measurements and seasonal factors from a series of measurements made at irregular intervals. can be done.

[0068] Additionally, each SeasonalTrendInfo object may contain time series data generated by a set of five exponential filters being applied to each of the following statistical measurements: the raw measurement of thread intensity, the rate at which thread intensity is increasing or decreasing, the acceleration or deceleration of this rate, the seasonal factor of thread intensity, and the residual component. Within a SeasonalTrendInfo object, the state of the set of five exponential filters for the variables, the filter constants, the filter parameter adjustment weighting factors (which adjust for irregular intervals between samples), and the filter parameters may be represented by that time series data.

[0069] D. Example of generating classification signatures FIG. 3 illustrates an example of a thread dump of a virtual machine 300 over a period of time, according to some embodiments. In contrast to the runtime profiling of FIG. 1, where the sampling interval is 100 ms to 1 s, the sampling interval employed by the control system of FIG. 3 may be longer (e.g., between 20 s and 1 min) to reduce sampling overhead. As shown in FIG. 3, within two to three sampling intervals, a process running in the virtual machine 300 generates threads 302, 304, 306, 308, 310, and 312. Each of the threads 302-312 is associated with a different call stack during operation, so that a stack trace can be output when the thread dump is output. FIG. 3 illustrates a total of three output thread dumps: thread dump N, thread dump N+1, and thread dump N+2.

[0070] Figure 3 shows three consecutive thread dumps (A,B,D), (A,B,D), ( 3 shows three different types of stack traces observed in the order (A,C), (A,B,E). Stack trace (A,B,D) is observed twice. Before thread dump N is generated, thread 302 is spawned and begins running. When thread dump N is generated, stack trace (A,B,D) is observed for thread 302. Note that stack segment A, stack segment B, and stack segment D have not yet been identified, but for ease of explanation, the names of the stack segments are used throughout the example shown in FIG. 3. One sampling interval after thread dump N is generated, thread 302 terminates, and thread 304 is spawned and terminates without being sampled while threads 306 and 308 are spawned. When thread dump N+1 is generated, thread 310 emits stack trace (A,C), while thread 308 emits stack trace (A,B,D). Another sampling interval after thread dump N+1 is printed, threads 306 and 308 terminate, thread 310 is spawned and terminates without being sampled, and thread 312 is spawned. When thread dump N+2 is printed, thread 312 emits stack trace (A,B,E). As can be seen from Figure 3, the (A,B,D) thread type is the first type of thread observed, and the intensity of the (A,B,D) thread type is greater than the (A,C) or (A,B,E) thread types.

[0071] After thread dump N, the control system may register one SegmentInfo(A,B,D) node as a classification signature for stack trace (A,B,D). The control system may then associate a SeasonalTrendInfo(A,B,D) object with the SegmentInfo(A,B,D) node and update the state encapsulated by this node.

[0072] SegmentInfo(A,B,D).numOfOccur = 1. SegmentInfo(A,B,D).totalNumOfOccur = 1. 4 illustrates a set of classification signatures 400, including one classification signature 450 registered in response to a stack trace (A, B, D). As can be seen in FIG. 4, the classification signature 450 includes one node 402 that corresponds to SegmentInfo(A, B, D). SegmentInfo(A, B, D) is shown as the combined node of all stack frames a1-d3 in the stack trace.

[0073] If stack trace (A,B,D) is observed again in thread dump N+1, the control system may update the SegmentInfo(A,B,D) node as follows:

[0074] SegmentInfo(A,B,D).numOfOccur = 1. SegmentInfo(A,B,D).totalNumOfOccur = 2. When stack trace (A,C) is observed for the first time in thread dump N+1, the control system determines that the entire set of stack frames in stack segment (A,B,D) is no longer linearly connected. Now, a bifurcation exists between the last stack frame in the set of stack frames represented by "A" (e.g., going from the top to the bottom of the stack trace) and the first stack frame in the set of stack frames represented by "B,D" because the next stack frame following the last stack frame in any stack trace is either (1) the first stack frame in (B,D) or (2) the first stack frame in the set of stack frames represented by "C". Thus, the control system divides stack segment (A,B,D) into stack segment (A) and stack segment (B,D) by creating nodes SegmentInfo(A) and SegmentInfo(B,D) and assigning these two nodes to children of SegmentInfo(A,B,D). For stack trace (A,C), the control system may initialize a stack segment (C) by creating a node SegmentInfo(C) and register an ordered tuple that includes SegmentInfo(A) and SegmentInfo(C) as a classification signature for stack trace (A,C).

[0075] In some embodiments, the control system may clone the SeasonalTrendInfo(A,B,D) object to create SeasonalTrendInfo(A) and SeasonalTrendInfo(B,D) objects for nodes SegmentInfo(A) and SegmentInfo(B,D), respectively, and create a new SeasonalTrendInfo(C) for SegmentInfo(C), as follows:

[0076] SeasonalTrendInfo(A) <-SeasonalTrendInfo(A,B,D) SeasonalTrendInfo(B,D) <-SeasonalTrendInfo(A,B,D) SeasonalTrendInfo(C) <- new SeasonalTrendInfo The control system may also update the SegmentInfo node as follows:

[0077] SegmentInfo(A).numOfOccur = 2 SegmentInfo(A).totalNumOfOccur = 3 SegmentInfo(C).numOfOccur = 1 SegmentInfo(C).totalNumOfOccur = 1 5 illustrates a set of classification signatures 500, including classification signature 450 and a new classification signature 550 generated in response to observing stack trace (A,C) for the first time. As can be seen from FIG. 5, classification signature 450 now includes three nodes: node 402, node 502, and node 504. Node 402 corresponds to SegmentInfo(A,B,D), which is the combined node of node 502 and node 504. Node 502 corresponds to SegmentInfo(A) and combines stack frames a1-a3. Node 504 corresponds to SegmentInfo(B,D) and combines stack frames b1-d3. The classification signature 550 includes two nodes: a node 506 corresponding to SegmentInfo(A) indicated as combining stack frames a1 to a3, and a node 508 corresponding to SegmentInfo(C) indicated as combining stack frames c1 to c3.

[0078] When stack trace (A,B,E) is observed for the first time in thread dump N+2, the control system determines that the entire set of stack frames in stack segment (B,D) is no longer linearly connected. Now, a bifurcation exists between the last stack frame in the set of stack frames denoted by "B" and the first stack frame in the set of stack frames denoted by "D", because the next stack frame following the last stack frame in any stack trace is either (1) the first stack frame in (D) or (2) the first stack frame in the set of stack frames denoted by "E". Thus, the control system may split stack segment (B,D) into stack segment (B) and stack segment (D) by creating nodes SegmentInfo(B) and SegmentInfo(D) and assigning these two nodes to child nodes of SegmentInfo(B,D). For stack trace (A,B,E), the control system may initialize stack segment "E" by creating a node SegmentInfo(E) and assigning SegmentInfo(A), SegmentInfo(D), and assigning these two nodes to child nodes of SegmentInfo(B,D). A user may register an ordered tuple containing (A,B,E) as a classification signature for stack trace (A,B,E).

[0079] In some embodiments, the control system may clone the SeasonalTrendInfo(B,D) object to create SeasonalTrendInfo(B) and SeasonalTrendInfo(D) objects for nodes SegmentInfo(B) and SegmentInfo(D), respectively, and create a new SeasonalTrendInfo(E) for SegmentInfo(E), as follows:

[0080] SeasonalTrendInfo(B) <-SeasonalTrendInfo(B,D) SeasonalTrendInfo(D) <-SeasonalTrendInfo(B,D) SeasonalTrendInfo(E) <- new SeasonalTrendInfo The control system may also update the SegmentInfo node as follows:

[0081] SegmentInfo(A).numOfOccur = 1 SegmentInfo(A).totalNumOfOccur = 4 SegmentInfo(B).numOfOccur = 1 SegmentInfo(B).totalNumOfOccur = 3 SegmentInfo(E).numOfOccur = 1 SegmentInfo(E).totalNumOfOccur = 1 FIG. 6 illustrates a classification signature 600 including classification signatures 450 and 550 and a new classification signature 650 generated in response to stack trace (A,B,E). As can be seen from FIG. 6, here classification signature 450 includes five nodes: node 402, node 502, node 504, node 602, and node 604. Node 504 corresponds to SegmentInfo(B,D), which is the combined node of node 602 and node 604. Node 602 corresponds to SegmentInfo(B), which combines stack frames b1-b3. Node 604 corresponds to SegmentInfo(D), which is the combined node of stack frames d1-d3. Classification signature 550 is unchanged. The classification signature 650 includes three nodes: a node 606 corresponding to SegmentInfo(A) indicated as combining stack frames a1-a3, a node 608 corresponding to SegmentInfo(B) indicated as combining stack frames b1-b3, and a node 610 corresponding to SegmentInfo(E) indicated as combining stack frames e1-e3.

[0082] As shown in FIG. 6, the classification signature of stack trace (A,B,D) may consist of one SegmentInfo node at the root of classification signature 450. That is, stack trace (A,B,D), being the stack trace with the highest strength, has the most compact representation. Meanwhile, stack trace (A,C) is assigned the second shortest classification signature with two order nodes (A) and (C). The last encountered stack trace (A,B,E) is assigned the third shortest classification signature with three order nodes (A), (B), and (E). As shown in FIGS. 4-6, a ThreadClassificationInfo object may correspond to a set of SegmentInfo nodes, where one SegmentInfo node represents a binary tree (or subtree that is binary) of a set of other SegmentInfo nodes and / or other StackFrameInfo objects. The ThreadClassificationInfo objects, SegmentInfo nodes, and StackFrameInfo objects together may comprise a generation grammar.

[0083] Thread1 -> (A,B,D) Thread2 ->(A)(C) Thread3 ->(A)(B)(E) (A,B,D) -> (A)(B,D) (B,D) -> (B)(D) A -> a1,a2,a3 B -> b1,b2,b3 C -> c1,c2,c3 D -> d1,d2,d3 E -> e1,e2,e3 As can be seen above, the individual stack frames ai, bi, ci, di, ei are terminals of the grammar, whereas the SegmentInfo node is a non-terminal. Some embodiments can parse the stack frames of a stack trace from the bottom up the stack trace (left to right in the notation below).

[0084] a1, a2, a3, b1, b2, b3, d1, d2, d3 (A),b1,b2,b3,d1,d2,d3 Using production rule (A)->a1,a2,a3 (A),(B),d1,d2,d3 Using production rule (B)->b1,b2,b3 (A),(B),(D) Using production rule (D) -> d1,d2,d3 (A),(B,D) using the production rule (B,D) -> (B)(D) (A,B,D) Use the production rule (A,B,D) -> (A),(B,D) Thread1 uses the production rule Thread1 -> (A,B,D) As can be seen from the above, some embodiments can parse the stack frames by upward parsing. The upward parsing can be similar to shift-reduce parsing or left-to-right "LR" parsing. This parsing can require shift-reduce the stack frames and SegmentInfo nodes to construct a parse tree for the stack trace by parsing from the leaves of the tree toward the root. Some embodiments parse another occurrence of the thread's stack trace by combining parse trees for previous occurrences of the thread's stack trace and reducing them to the same parse tree (i.e., shift-reduce parsing, "LR" parsing reading left-to-right). Each node of the classification tree can be a compact label for the class of the stack trace, and the root of the classification tree can be a compact label for the class of the thread.

[0085] 7 is a flowchart 700 of a process for generating and / or modifying one or more thread classification signatures in response to a thread dump, according to some embodiments. In some embodiments, the process illustrated in flowchart 700 may be performed by a computer system having one or more processors (e.g., computer system 1700 of FIG. 17). The one or more processors may perform the steps based on computer code stored on a computer-readable medium. The steps described in FIG. 7 may be performed in any order, with or without other steps.

[0086] Flowchart 700 begins at step 702, where embodiments perform a thread dump during execution of a multi-threaded program. In particular, some embodiments may support one or more monitoring processes that monitor the software execution environment in which the multi-threaded program runs. A software execution environment may support multiple multi-threaded processes, including a multi-threaded program. In some cases, the software execution environment may be a virtual machine that supports the output of thread dumps. In some embodiments, one or more monitoring processes may run in the virtual machine alongside the multi-threaded program. In some embodiments, the one or more monitoring processes may run separately from the virtual machine, on the same set of machines or on a different set of machines. The one or more monitoring processes may periodically initiate thread dumps of the virtual machine. For a particular thread dump, a stack trace may be taken for each thread running on behalf of (e.g., generated by) the multi-threaded program at the time the particular thread dump is output.

[0087] In step 704, embodiments receive a stack trace for each thread that was running during the thread dump. The stack trace for a particular thread may correspond to one or more lines of text describing the call stack of the thread. Each line in the stack trace corresponds to a particular stack frame on the call stack of the thread and may describe the code block for that stack frame. In some embodiments, the stack frame may include a source code file, a line number pointing to the code block, and a class name and / or method name for the code block.

[0088] At decision 706, embodiments determine whether another stack trace needs to be analyzed. If not, the flowchart ends at step 716. Specifically, once all of the stack traces in a thread dump have been analyzed by one or more monitoring processes, some embodiments may update the intensity statistical data encapsulated in memory by one or more objects. For example, member variables (e.g., rawMeasure, rawDeseasonalizedMeasure, smoothedWeekdaySeasonalFactor, and / or smoothedWeekendSeasonalFactor) of one or more SeasonalTrendInfo objects may be updated based on what types of stack traces were obtained from the thread dump.

[0089] If it is necessary to determine whether another stack trace needs to be analyzed, then in step 708, embodiments determine whether an existing trace signature represents a sequence of stack frames that the stack trace contains. Specifically, some embodiments may use the existing set of classification signatures created based on stack frames received from a previous thread dump as a generative grammar to determine whether the sequence of stack frames can be represented by one of the existing signatures. This may require one or more shift-and-reduce operations in which parts of the stack trace are collapsed into leaf SegmentInfo nodes, which themselves are collapsed into combined nodes. If the shift-and-reduce operations result in an ordered tuple that is registered as a classification signature, then the classification signature represents a sequence of stack frames that the stack trace contains.

[0090] If, at decision 710, such a trace (i.e., classification) signature exists, the flowchart proceeds to step 714. If not, at step 712, embodiments generate a new trace signature that represents the sequence of stack frames that the stack trace contains. That is, a branch point is discovered within a set of stack frames that were previously thought to be linearly connected. Some embodiments may then generate one or more SegmentInfo nodes, modify one or more binary trees, and / or modify one or more ordered tuples to generate a new classification signature that represents the (formerly) linearly connected set of stack frames that the stack trace contains. The technique for achieving this is described in more detail below with reference to FIG.

[0091] In step 714, embodiments increment counters for the trace signature before returning to decision 706. Specifically, certain counters that are members (e.g., numOfOccur and / or totalNumOfOccur) of ThreadClassificationInfo objects, SegmentInfo objects, and / or StackFrameInfo objects may be incremented to track the number of stack traces, stack segments, and stack frames by their type as they are received and discovered.

[0092] 8 is a flowchart 800 of a process for generating or modifying a thread classification signature in response to detecting a branch point, according to some embodiments. In some embodiments, the process illustrated in flowchart 800 may be implemented by a computer system having one or more processors (e.g., computer system 1700 of FIG. 17). The one or more processors may perform the steps based on computer code stored on a computer-readable medium. The steps described in FIG. 8 may be performed in any order, with or without other steps.

[0093] Flowchart 800 begins at step 802. In step 802, an embodiment determines whether one or more SegmentInfo nodes have been previously generated. If so, the flowchart proceeds to step 804. If not, the flowchart proceeds to step 814. Unless the stack trace currently being analyzed is the first stack trace received for the data set, the set of classification signatures may include one or more classification signatures that contain SegmentInfo nodes that were previously generated for previous stack traces. Because stack trace types received from the same process may share stack segments with each other, a stack trace of any type received for the first time may lead to the discovery of a branch point.

[0094] In step 804, embodiments may determine a subsequence of one or more stack frames included in the sequence of stack frames included in the stack trace that are not represented by a previously generated node. Specifically, some embodiments may examine the classification signatures and SegmentInfo nodes present while attempting to compress the sequence of stack frames included in the stack trace by a series of shift-and-reduce operations. Any subsequence of stack frames in the sequence that cannot be reduced may be determined to be a stack segment of the new type. In this case, some embodiments may determine that a SegmentInfo node needs to be generated to represent a stack segment of the new type.

[0095] In step 806, embodiments may create one or more additional nodes to represent a subsequence of the one or more stack frames. Specifically, a new StackFrameInfo object may be created for each stack frame included in the stack segment of the new type. A new SegmentInfo node may be created corresponding to the stack segment of the new type, where the new SegmentInfo node references each of the new StackFrameInfo objects.

[0096] In step 808, embodiments incorporate at least one of the one or more additional nodes into one or more previously generated binary trees contained in one or more previously generated sets. One or more binary trees of one or more existing classification signatures may be modified and / or expanded to take into account the newly discovered branch point. Stack segments represented by leaf SegmentInfo nodes of the existing binary trees may be updated by the new branch point. If a leaf node is split into two new leaf SegmentInfo nodes, then the leaf node may become the join node of the two new leaf SegmentInfo nodes.

[0097] In step 810, an embodiment generates one or more additional binary trees. At least one or more of the one or more binary trees includes at least one of the one or more additional nodes. In most cases, the one or more additional binary trees will be single-level trees with one node. One of the newly generated binary trees may include the new SegmentInfo node generated in step 806.

[0098] In step 812, embodiments generate additional tuples including the one or more additional binary trees to represent the stack traces. The additional tuples may correspond to classification signatures representing stack traces of newly discovered types. Some tuples may include one node and be ordered sets of single-level binary trees, similar to a list of nodes. Other tuples may correspond to a multilevel binary tree. Yet other tuples may include a combination of single-level and multilevel binary trees. In general, as more types of stack traces are discovered, subsequent classification signatures generated may correspond to longer ordered tuples. However, longer classification signatures tend to represent stack traces that occur less frequently, since stack traces of common types are more likely to be encountered first. This will ensure that a higher percentage of stack traces are compressed into shorter classification signatures. After step 812, the flowchart ends in step 820.

[0099] In step 814, embodiments generate a tuple containing one binary tree containing one node to represent the stack trace. Since no SegmentInfo node is found, the stack trace currently being parsed is assumed to be the first SegmentInfo node. As a result, some embodiments may generate a classification signature corresponding to one binary tree with only one SegmentInfo node. After step 814, the flowchart ends in step 820. In the future, as stack traces of different types are encountered, the binary tree may be expanded with new SegmentInfo nodes to represent newly encountered branch points.

[0100] IV. Measuring Heap Usage at Irregular Intervals Some embodiments may have a control system monitor heap allocation (i.e., heap usage) over time to estimate trends and predict future memory usage within virtual machines. By detecting seasonal trends and predicting storage capacity requirements, some embodiments may dynamically reallocate shared system memory between virtual machines, allowing for flexible resource allocation. Predicting capacity requirements may require estimating the heap growth rate. To ensure accuracy of the sample, heap allocation may be measured during a full garbage collection (GC) cycle. GC cycles occur at irregular intervals. Estimating the heap growth rate may require division by random intervals, complicated by irregular intervals that intermittently approach zero in a haphazard manner. The noise in the growth rate measurements is a ratio of two Gaussian distributions resulting in a Cauchy distribution, and Gaussian distributions are difficult to filter. The mean and standard deviation of the Cauchy distribution are undefined, in the sense that the mean and standard deviation cannot be estimated more accurately with a large number of data points than with a single data point. A larger pool of samples may increase the likelihood of encountering sample points with large absolute values ​​that correspond to divisions by time-close intervals.

[0101] Note that unlike heap size measurements, which are sampled at irregular intervals due to the irregularity of full GC cycles, thread intensity measurements are sampled at regular intervals, avoiding time-close intervals. Even so, the heap allocation trends described herein can be used to The same techniques for seasonal trending can be applied to seasonally trending and forecasting thread intensity and stack segment intensity measurements. In some embodiments, the techniques can accommodate for indefinite latencies due to CPU scheduling of threads and interruptions of full GC cycles. The techniques can also accommodate for indefinite sampling intervals due to indefinite computation time required to classify stack segments. In cases where a particular thread or stack segment has not been observed in a thread dump, some embodiments may leave the numOfOccur member variable of the associated ThreadClassificationInfo object or the associated SegmentInfo object at zero. Zero may indicate that measurements are not available for the particular thread or stack segment. Such embodiments may not update the rawMeasure variable of the SeasonalTrendInfo object. Such embodiments may update the rawMeasure member variable of the SeasonalTrendInfo object only if the numOfOccur member variable of the associated ThreadClassificationInfo object or the associated SegmentInfo object is not zero. Such embodiments may track the number of thread dumps "N" since rawMeasure was last updated. The thread intensity measurements may correspond to a time series with irregular intervals.

[0102] The Holt-Winters triple exponential filter, published in 1957 and 1960, can be used for seasonal trending and forecasting. CC Holt, "Forecasting Trends and Seasonal by Exponentially Weighted Averages," Office of Naval Research Memorandum No. 52, 1957, is incorporated herein by reference. PR Winters, "Forecasting Sales by Exponentially Weighted Moving Averages," Management Science Vol. 6, No. 3, pp. 324-342, 1960, which is incorporated herein by reference. Wright extended the Holt-Winters method in 1986 to accommodate irregular intervals. DJ Wright, "Forecasting data published at irregular time intervals using an extension of Holt's method," Management Science, Vol. 32, No. 4, pp. 499-510, 1986, which is incorporated herein by reference. In 2008, Hanzak proposed an adjustment factor for time-proximate intervals. "Improved Holt Method for Irregular Time Series" by T. Hanzak, WDS'08 Proceedings, Part 1, pp. 62-67, 2008, which is incorporated herein by reference.

[0103] The time proximity interval adjustment factor, which compensates for the relative strength of noise added by the random time proximity interval in the rate estimate, may erroneously blunt the rate of change estimate if the time interval decreases monotonically during congestion caused by memory leaks or deadlocks. The nonlinear or polynomial time complexity of the full GC algorithm may cause thread runtime intervals to become smaller as congestion worsens. In the case of memory leaks, a shorter time interval may result in shorter runtimes but longer measurement times because the virtual machine may freeze for longer periods due to more frequent full GCs. If the virtual machine freezes during a full GC, new requests may be queued outside the virtual machine. During subsequent runtimes, the backlog may accelerate the rate of change of heap usage. In some embodiments, the Hansack time proximity interval adjustment factor is used for heap allocation trending and forecasting to track accelerating heap growth rates.

[0104] In an embodiment of the present invention, a Holt-Winter triple exponential filter can be applied for seasonal trending and forecasting of heap usage to effectively realize flexible memory allocation. A standard Holt-Winter triple exponential filter that can be applied to request forecasts from regular time series can be specially tailored to be utilized for random intervals with irregular time-proximate intervals. An embodiment of the present invention can apply Wright's formula for irregular intervals and Hanzack's adjustment factor for time-proximate intervals for trending and forecasting of heap allocation. A non-trivial choice of filter structure suitable for irregular intervals caused by full GC can be made. The structure of the Holt-Winter-Wright-Hanzack filter can be derived from first principles and systematically devised to adapt to match the time series produced by full GC cycles.

[0105] In some embodiments, an exponential moving average equation is fitted to smooth the time series data, local linear trends, seasonal trends, forecast error residuals, and forecast absolute deviations to monitor and forecast resource utilization measures such as heap memory usage and thread intensity. In some embodiments, the equation may be based on Brown's exponential filter (1956), Holt's double exponential filter (1957), Winters' triple exponential filter (1960), Wright's extensions for irregular intervals (1986), Hanzack's time-proximal interval adjustment factor (2008), and outlier detection and clipping. The following publications are incorporated herein by reference: "Exponential Smoothing for Predicting Demand" by R. G. Brown, Cambridge, Massachusetts, 1999; "Exponential Smoothing for Predicting Demand" by R. G. Brown, Cambridge, Massachusetts, 1999; "Exponential Smoothing for Predicting Demand" by R. G. Brown, Cambridge, Massachusetts, 1999; "Exponential Smoothing for Predicting Demand" by R. G. Brown, Cambridge, Massachusetts, 1999. Arthur D. Little Inc., 1956, p. 15, C "Forecasting Trends and Seasonal by Exponentially Weighted Averages" by C. Holt (Office of Naval Research Memorandum No. 52, 1957); "Forecasting Sales by Exponentially Weighted Moving Averages" by P. R. Winters (Management Management Science, Vol. 6, No. 3, pp. 324-342, 1960; DJ Wright, "Forecasting data published at irregular time intervals using an extension of Holt's method," Management Science, Vol. 32, No. 4, pp. 499-510, 1986; T. Hanzack, "Improved Holt Method for Irregular Time Series," World Science Data System (WDS) 2008 Proceedings, Part 1, pp. 62-67, 2008; S. Maung, SW Butler, and SA Henck, "Method and Apparatus for process Endpoint Prediction based on Actual Thickness Measurements," Management Science, Vol. 6, No. 3, pp. 324-342, 1960; "Method and Apparatus for Processing Endpoint Predictions Based on Actual Thickness Measurements" (U.S. Pat. No. 5,503,707, 1996).

[0106] V. Correlating Thread Intensity with Heap Usage Various embodiments provide techniques for identifying stack traces (i.e., classes of threads) that are draining the heap within an application by correlating trends between intensity statistics and heap usage statistics of various thread classes generated by a multithreaded application. In doing so, some embodiments may identify seasons during which high heap usage tends to be greater (i.e., high heap usage seasons) within a period during which one or more multithreaded applications are executing within a software execution environment based on the heap usage statistics. As explained above, some embodiments then correlate the intensity statistics of multiple thread classes by analysis of thread dumps taken from the software execution environment during the same period during which the high heap usage seasons occur. Some embodiments may then identify classes of "heap-draining" threads (i.e., heap-draining stack traces) from among the threads of the identified classes by ranking the classes of threads according to the degree to which their intensity statistics correlate with high heap usage trends.

[0107] Some embodiments may refer to this class of threads as hoarding the heap because the code they are executing is likely to be inefficient in terms of heap memory usage. In other words, erroneous and / or unoptimized code executed by these threads can cause the threads to hoard large amounts of heap memory, contributing significantly to the trend of high heap usage.

[0108] It should be noted that such memory hotspots are significant in terms of running a cloud-based service in a production environment over time. Thus, by enabling continuous detection and mitigation of such hotspots, some embodiments may directly impact the operational efficiency of a cloud service. Also, such embodiments may be advantageous over using memory profiler tools to profile such applications, because such tools may add excessive overhead to the application. Thus, memory profiler tools may not be practical for profiling applications running in a production environment all the time.

[0109] A. Inefficient heap usage in your code One common cause of inefficient memory usage is due to local variables defined in a thread's stack frame. Generally, when a running thread instantiates an object, that object occupies heap memory until the number of stack frames that reference the object (directly or indirectly) becomes zero, at which point the heap memory is freed at the next garbage collection. Therefore, local variables that reference large objects from stack frames that remain alive for a long time can unintentionally contribute significantly to heap memory usage, because such local variables do not allow the object to be garbage collected.

[0110] Some embodiments assume that a fraction "P" of the total heap usage "G" bytes can be attributed to a class of threads "C". Further, some embodiments may assume that the average heap usage among this class of threads "C" (i.e., the heap usage per thread) is "M" bytes. In this case, "T C Let " denote the expected number of threads of thread class "C". By the following relationship, "T C " is obtained. "T C " is defined as thread strength in the statistical model.

[0111]

number

[0112] In response to identifying the class of thread that is draining the heap, particular embodiments may communicate the class of the thread to developers, performance engineers, and other interested parties (e.g., via a notification or alert). As a result, code associated with such types of threads may be subject to detailed code review and code profiling. In some cases, specific associated stack frames may be inspected. For example, an investigation may examine stack frames included in the stack trace of the thread that is draining the heap. It may be necessary to output a heap dump during a time close to a seasonal peak in heap usage to inspect the stack frames. The stack frames may contain local variables that reference objects that contribute to high heap usage (e.g., objects occupying large amounts of heap memory). This type of code inspection and optimization is done by visual code review, automated code review, profiling of identified threads, just-in-time (JIT) compiler optimizations, dynamic bytecode injection, or a combination of these techniques. In some embodiments, the classes of threads hoarding the heap may be communicated to other automated code optimization tools to leverage the code optimization capabilities of these tools.

[0113] Some embodiments may automatically redesign or rewrite application code to make it more efficient in its memory usage. For example, some embodiments may automatically rewrite the code to free large objects in local variables as soon as possible without changing the behavior or correctness of the application. In some cases, this may require careful analysis of the code paths involved in threads that are hoarding the heap.

[0114] For example, consider the following code: fileOS.write(buffer.toString().getBytes()); Some embodiments may determine that the above code is inefficient in terms of memory usage because three objects, buffer, buffer.toString(), and buffer.toString().getBytes(), are held by local variables in the stack frame of the thread that is draining the heap, which prevents these three objects from being garbage collected while the thread is blocked in a file system call.

[0115] Some embodiments can modify the code as follows so that at least two objects, buffer and buffer.toString(), are garbage collected while the thread is blocked in the file system call.

[0116]

number

[0117] Some embodiments can use a non-disruptive method to inspect stack frames of stack traces that are accumulating heap.

[0118] B. Initialization of seasonal factors for weekday and weekend periods To identify stack traces that are draining the heap, some embodiments may (1) identify seasons of high heap usage by estimating seasonal trends in heap usage statistics for the execution environment, and (2) for each of one or more thread classes, estimate seasonal trends in the thread intensity statistics for the class of threads. Several techniques for determining seasonal trends in heap usage statistics and seasonal trends in thread intensity statistics for regular or irregular intervals are described in patent applications 14 / 109,578, 14 / 109,546, and 14 / 705,304. No. 6,399,433, filed on Oct. 23, 2006. The same application is incorporated herein by reference for all purposes.

[0119] To determine the seasonal trend of statistical data, a period and an interval to which the seasonal trend is associated may be defined. Specifically, a period may be divided into multiple non-overlapping intervals. Each interval of the period may be associated with one seasonal index. For example, if the period is one day and the interval is one hour, there should be 24 seasonal indexes to cover this period. As another example, if the period is one year and the interval is one month, there should be 12 seasonal indexes.

[0120] Some embodiments may model weekdays, weekends, and holidays as separate periods. If weekday and weekend periods are separated, one cycle of weekend periods may be interleaved with five cycles of weekday periods such that one weekend period is processed after five consecutive weekday periods. Thus, the frequency of the consecutive weekday periods is one weekday period every 24 hours, and the frequency of the weekend periods is one weekend period every seven days. In embodiments where individual holidays (e.g., Christmas Day and New Year's Day) are modeled as separate periods, the frequency of a particular holiday period is once a year.

[0121] The seasonal index may be a multiplicative seasonal factor or an additive seasonal period applied to the interval associated with the seasonal index. For example, in an embodiment where a multiplicative seasonal factor is used to represent the seasonal index, if the interval "9:00AM-10:00AM" is associated with a seasonal factor of 1.3, any measurements sampled during the interval 9:00AM-10:00AM are adjusted 30% higher (i.e., multiplied by 1.3). In an embodiment where the seasonal index is represented by an additive seasonal period, the additive seasonal period is added to the measurements.

[0122] A season is a classification of a set of intervals based on some criteria. For example, considering a period of one year, the twelve intervals of January, February, March, April, May, June, July, August, September, October, November, and December can be classified into four northern meteorological seasons as follows:

[0123] December, January, and February are classified as the winter season. The months of March, April, and May are classified as the spring season.

[0124] The months of June, July, and August are classified as the summer season. The months of September, October, and November are classified as the fall season.

[0125] Some embodiments may divide the weekday period into 96 15-minute intervals. In this regard, 96 seasonal indices are derived, where each of the 96 weekday seasonal indices (i.e., weekday factors) corresponds to a different one of the 96 weekday intervals. Similarly, some embodiments may divide the weekend period into 192 15-minute intervals. This results in 192 seasonal indices, where each of the 192 weekend seasonal indices (i.e., weekend factors) corresponds to a different one of the 192 weekend intervals.

[0126] To separate seasonal patterns for weekday periods from seasonal patterns for weekend periods, certain embodiments may apply a multi-period trending filter to weekday periods separately from applying such a filter to weekend periods. Some embodiments may then renormalize the weekday and weekend factors such that a seasonal factor of 1 represents a common baseline level for weekday and weekend periods. As a result, a seasonal factor greater than 1 may represent higher than average heap usage during the period to which the seasonal factor is applied, while another seasonal factor less than 1 may represent lower than average heap usage during another period to which the other seasonal factor is applied.

[0127] In some embodiments, a multi-period trending approach may be deployed to separate holidays (e.g., Labor Day, Christmas, New Year's Day, etc.) into separate periods, where the holiday periods repeat once every 12 months, while the weekday periods repeat once every 24 hours and the weekend periods repeat once every 7 days. In such embodiments, the holiday period seasonal factor, weekday period seasonal factor, and weekend period seasonal factor may all be convolved together such that a seasonal factor of 1 represents a common baseline level for the weekday periods, weekend periods, and holiday periods.

[0128] Given a time period (e.g., weekday period, weekend period, or holiday period / year period), let P denote the number of cycles in the period contained in a given measurement dataset (e.g., a time series of heap usage measurements over a certain period), and let K denote the number of intervals in the number of periods contained in the given dataset. If L denotes the number of seasonal indices in a period, then K=P×L. For example, if there are at least three years of data in the dataset, with one period corresponding to a year and one interval corresponding to a month, then the number of cycles P in the period is three and the number of monthly intervals is 36.

[0129]

number

[0130] Specifically, the seasonal factor for a given interval is calculated by the average heap usage for that interval over the entire dataset (for example, for a dataset that spans a full week). It may be equal to the percentage of the average heap usage over all the same intervals (e.g., all 9AM-10AM intervals), and the average heap usage over the entire data set.

[0131] C. Renormalization As mentioned above, some embodiments may renormalize weekday and weekend seasonal factors such that a seasonal factor of "1" represents a common baseline level for weekday and weekend periods.

[0132] In general, certain embodiments may perform the renormalization by calculating a weighted average of the seasonal factors over all periods and dividing each of the seasonal factors by the weighted average. Consider the following example, which requires seasonal indices for multiple periods of different lengths, where each period is divided into 15-minute intervals.

[0133] Weekday Seasonal Index: D i , i=1 , 2 , …96 Weekend Seasonal Index: E i , i=1 , 2 , … , 192 Seasonal index of 10 holidays: H k,i , i=1 , 2 , … , 96;k=1 , 2 , …10 In a particular year, there are 253 weekdays (excluding holidays), 50.5 weekend days, and 10 holidays. 253+50.5×2+10=364 days. In this example, some embodiments may calculate a weighted average "A" of the seasonal factors using the following formula, where the weights are proportional to the number of cycles of each period (e.g., weekday period, weekend period, and 10 holiday periods) in a year:

[0134]

number

[0135] Some embodiments may include, for each seasonal factor D i , E i , and H k,i By dividing by A, we can derive a new renormalized seasonal factor for each period.

[0136]

number

[0137] After renormalization of the weekday and weekend seasonal factors, a seasonal factor of 1 should represent a common base level for the weekday and weekend factors.

[0138] D. Smoothing spline fit As discussed above, some embodiments may fit a smoothing spline across multiple time periods to smooth the transition between cycles of a period (e.g., between two weekday periods) or between cycles of two adjacent periods (e.g., between a weekday period and a weekend period). Specifically, fitting a spline may require concatenating seasonal indices for one or more periods to smooth the transitions between those periods.

[0139] In general, certain embodiments (e.g., filters) may be used to filter out period A, B, C, D, E, F, E, G, H, I, J, or I as they cycle through weekdays, such as transitioning from Monday to Tuesday, Tuesday to Wednesday, Wednesday to Thursday, and Thursday to Friday. i reaches the end of one cycle of period A i When starting a new cycle of seasonal index A, such an embodiment i The three sequences can be concatenated and a smoothing spline can be fitted end-to-end to the entire sequence. Some embodiments then take the middle segment of the smoothed sequence to obtain a new smoothed seasonal index A i may be represented as:

[0140] When a particular implementation (e.g., a filter) transitions from Friday to Saturday, i reaches the end of one cycle of the adjacent period B i When starting a new cycle of the seasonal index A i and one sequence of seasonal indices B i and period B i Seasonal index C for the period following i Then, some embodiments may take the middle segment of the smoothed sequence to generate a new smoothed seasonal index B i Also, some embodiments may take the first segment of the smoothed sequence to obtain the smoothed seasonal index A i may be represented as:

[0141] When a particular implementation (e.g., a filter) transitions from Sunday to Monday, period B i reaches the end of one cycle of the adjacent period C i When starting a new cycle of period B, some embodiments i Seasonal index A for the period preceding i and one sequence of seasonal indices B i and a sequence of seasonal indices C iThen, some embodiments take the middle segment of the smoothed sequence to find a new smoothed seasonal index B i Also, some embodiments may take the third segment of the smoothed sequence to obtain a new smoothed seasonal index C i may be represented as:

[0142] For cloud services, the load cycle during weekends and holidays is often different from the load cycle during weekdays. Traditional seasonal trending solutions would usually only represent one period of seasonal indices. To separate weekend seasonal indices from normal weekday seasonal indices, such traditional solutions would vary by extending the period range to a full week or month. In addition, such traditional solutions would treat holidays separately.

[0143] Returning to the step of identifying stack traces that are filling up the heap, to smooth the weekday seasonal factors, some embodiments may construct an array of seasonal factors by concatenating three sequences of weekday factors. For example, some embodiments may generate such an array by executing the following code in the R programming language:

[0144] factors <- c(smoothedWeekdaySeasonalFactor, smoothedWeekdaySeasonalFactor, smoothedWeekdaySeasonalFactor) Some embodiments may then apply a spline to smooth the array of weekday factors, for example, some embodiments may call the smooth.spline function in the R language with a smoothing parameter of 0.3 to smooth the factors.

[0145] extendedWeekdayIndices <- 1:(3 * 96) f <- smooth.spline(extendedWeekdayIndices, factors, spar = 0.3) Some embodiments may then designate the intermediate sequence in the array (i.e., the middle 96 weekday factors) as the smoothed weekday factors. For example, some embodiments may obtain the smoothed weekday factors by executing the following code in the R programming language:

[0146] sandwichWeekdayIndices <- (96 + 1):(96 * 2) smoothedWeekdaySeasonalFactor <- predict(f, sandwichWeekdayIndices)$y In a manner similar to the manner in which the weekday factors are smoothed, some embodiments may apply splines to smooth the weekend factors. Specifically, some embodiments may construct an array of seasonal factors by concatenating a sequence of weekend factors between two sequences of weekday factors. For example, some embodiments may generate this array by executing the following code in the R programming language:

[0147] factors <- c(smoothedWeekdaySeasonalFactor, smoothedWeekendSeasonalFactor, smoothedWeekdaySeasonalFactor) Some embodiments may then apply a spline to smooth the alignment of weekday and weekend factors, for example, some embodiments may call the smooth.spline function in the R language with a smoothing parameter of 0.3 to smooth these factors.

[0148] extendedWeekendIndices <- 1:(2 * 96 + 192) f <- smooth.spline(extendedWeekendIndices, factors, spar = 0.3) Some embodiments may then designate the intermediate sequence in the array (i.e., the intermediate 192 seasonal factors in the array that are weekend factors) as the smoothed weekend factors. For example, some embodiments may obtain the smoothed weekend factors by executing the following code in the R programming language:

[0149] sandwichWeekendIndices <- (96 + 1):(96 + 192) smoothedWeekendSeasonalFactor <- predict(f, sandwichWeekendIndices)$y However, some embodiments may represent the 96 weekday seasonal indices and the 192 weekend seasonal indices separately to separate seasonal patterns observed during weekdays from seasonal patterns observed during weekends. In some embodiments, sequential filtering of the time series of heap usage statistics may require five sets of exponential filters, including an exponential filter for heap usage measurements, an exponential filter for seasonal factors, an exponential filter for linear trends, an exponential filter for accelerating trends, and an exponential filter for residuals.

[0150] As mentioned above, heap allocations may be measured during full garbage collection (GC) cycles that occur at irregular intervals to ensure accuracy of the samples. In situations where heap usage is particularly high, the constant garbage collections will cause the sampling interval to approach zero on a whim. As the prediction requires an estimate of the rate of change, as this irregular interval approaches zero on a whim, the rate of change may become a Cauchy distributed random variable with undefined mean and standard deviation. Thus, some embodiments may employ Holt's double exponential filter, Winters' triple exponential filter, Wright's extensions for irregular intervals, Hansac's time proximity interval adjustment factor, clipping with outlier detection and adaptive scaling of the outlier cutoff to overcome the Cauchy distribution problem to determine seasonal trends in the statistical data determined in association with full GC. In some embodiments, five sets of this exponential filter are applied sequentially to the time series. Then, the weekday and weekend factors can be estimated.

[0151]

number

[0152] After the end of each period, some embodiments may apply a spline to smooth the seasonal factors. For example, when reaching the end of a weekday period that precedes another weekday period (i.e., transitioning from Monday to Tuesday, Tuesday to Wednesday, Wednesday to Thursday, or Thursday to Friday), some embodiments may construct an array of seasonal factors by concatenating three sequences of weekday factors. For example, some embodiments may generate such an array by executing the following code in the R programming language:

[0153] factors <- c(smoothedWeekdaySeasonalFactor, smoothedWeekdaySeasonalFactor, smoothedWeekdaySeasonalFactor) Some embodiments may then apply a spline to smooth the array of weekday factors, for example, some embodiments may call the smooth.spline function in the R language with a smoothing parameter of 0.3 to smooth the factors.

[0154] extendedWeekdayIndices <- 1:(3 * 96) f <- smooth.spline(extendedWeekdayIndices, factors, spar = 0.3) Some embodiments may then designate the intermediate sequence in the array (i.e., the middle 96 weekday factors) as the smoothed weekday factors. For example, some embodiments may obtain the smoothed weekday factors by executing the following code in the R programming language:

[0155] sandwichWeekdayIndices <- (96 + 1):(96 * 2) smoothedWeekdaySeasonalFactor <- predict(f, sandwichWeekdayIndices)$y In another case, when the end of a weekday period preceding a weekend period is reached (i.e., a Friday to Saturday transition), some embodiments may construct an array of seasonal factors by concatenating a sequence of weekend seasonal factors between two sequences of weekday seasonal factors. For example, some embodiments may generate such an array by executing the following code in the R programming language:

[0156] factors <- c(smoothedWeekdaySeasonalFactor, smoothedWeekendSeasonalFactor, smoothedWeekdaySeasonalFactor) Some embodiments may then apply a spline to smooth the array of weekday and weekend factors. For example, some embodiments may smooth the factors by calling the smooth.spline function in the R language with a smoothing parameter of 0.3. good.

[0157] extendedWeekendIndices <- 1:(2 * 96 + 192) f <- smooth.spline(extendedWeekendIndices, factors, spar = 0.3) Some embodiments may then specify the left sequence in the array (i.e., the first 96 seasonal factors in the array, which are the weekday factors) as the smoothed weekday factors. For example, some embodiments may obtain the smoothed weekday factors by executing the following code in the R programming language:

[0158] leftsideWeekendIndices <- 1:96 smoothedWeekdaySeasonalFactor <- predict(f, leftsideWeekendIndices)$y In another case, when the end of a weekend period is reached (i.e., the transition from Sunday to Monday), some embodiments may construct an array of seasonal factors by concatenating a sequence of weekend seasonal factors between two sequences of weekday seasonal factors. For example, some embodiments may generate such an array by executing the following code in the R programming language:

[0159] factors <- c(smoothedWeekdaySeasonalFactor, smoothedWeekendSeasonalFactor, smoothedWeekdaySeasonalFactor) Some embodiments may then apply a spline to smooth the array of weekday and weekend factors, for example, some embodiments may call the smooth.spline function in the R language with a smoothing parameter of 0.3 to smooth the factors.

[0160] extendedWeekendIndices <- 1:(2 * 96 + 192) f <- smooth.spline(extendedWeekendIndices, factors, spar = 0.3) Some embodiments may then designate the intermediate sequence in the array (i.e., the intermediate 192 seasonal factors in the array that are weekend factors) as the smoothed weekend factors. For example, some embodiments may obtain the smoothed weekend factors by executing the following code in the R programming language:

[0161] sandwichWeekendIndices <- (96 + 1):(96 + 192) smoothedWeekendSeasonalFactor <- predict(f, sandwichWeekendIndices)$y Some embodiments may also specify the right sequence in the array (i.e., the last 96 seasonal factors in the array, which are the weekday factors) as the smoothed weekday factors. For example, some embodiments may obtain the smoothed weekday factors by executing the following code in the R programming language:

[0162] rightsideWeekendIndices <- (96 + 192+ 1):( 2 * 96 + 192) smoothedWeekdaySeasonalFactor <- predict(f, rightsideWeekendIndices)$y It should be noted that some embodiments may perform the above-mentioned convolution and smoothing spline fit each time the sequential filter (1) reaches the end of one cycle of a period and begins a new cycle of the same period (e.g., the sequential filter reaches the end of a Monday), or (2) reaches the end of one cycle of a period and begins a new cycle of an adjacent period (e.g., the sequential filter reaches the end of a Friday).

[0163] E. Seasonal cycle test Some embodiments may include a method for determining whether a seasonal cycle exists in one or more candidate periods of a dataset. In general, to determine whether a dataset exhibits seasonal cycles for a particular period, some embodiments may perform the following steps:

[0164] Let Q denote the number of seasonal indices in a period, P denote the number of cycles in a period, and K denote the number of intervals in a cycle of a period, where K = P x Q.

[0165] Some embodiments may calculate the average measurement for each interval of a cycle of a period. To do so, some embodiments may enumerate these intervals from 0 to (K-1) and calculate the average measurement for each interval of the period using the following formula:

[0166]

number

[0167] Some embodiments may then calculate the average measurement for each cycle in this period. To do so, some embodiments may enumerate the cycles in this period from 0 to (P-1) and calculate the average measurement for each cycle in the period using the following formula:

[0168]

number

[0169] Some embodiments may then apply a null hypothesis test to find out whether a seasonal cycle exists in the time period. In this regard, the null hypothesis to be tested is whether the correlation coefficient r between the seasonal index of the most recent cycle "u" and the seasonal index of the preceding cycle "v" is u , v may correspond to the assumption that r is zero. Specifically, some embodiments may calculate the correlation coefficient r using the following formula: u , v may be determined.

[0170]

number

[0171] Some embodiments employ various techniques to determine the correlation coefficient r u , v is large enough to indicate above a significance level that there is a common seasonal cycle between cycles "u" and "v." For example, some embodiments may employ a Student's t-test, a permutation test, or a Fisher transformation.

[0172] To test the hypotheses, some embodiments may define one or more test statistics. The test statistics may be functions of the parameters. In this case, the correlation coefficient r u , v The test statistic t below has a Student's t distribution with n-2 degrees of freedom and is u , v In some embodiments, the null hypothesis r u , v = 0, which assumes that the seasonal indices are uncorrelated across cycles of one period. Some embodiments may choose to reject the null hypothesis (i.e., r u ,v = 0).

[0173]

number

[0174] Let F(t) represent a probability distribution. If the significance level is 0.1, then we can determine the value of the random variable t such that F(t) = 0.9. 0.9,(n-2) Assume that the alternative hypothesis is a biased condition.

[0175]

number

[0176] If this condition is true, the alternative hypothesis is accepted, which indicates that there is a common seasonal cycle between yearly cycles "u" and "v". If there is a common seasonal cycle between the most recent cycle and the previous cycle, some embodiments may proceed to calculate a seasonal factor for each seasonal index of the cycle. Some embodiments may apply the above formula to detect the presence of an annual seasonal cycle in heap usage by the software execution environment, as described below.

[0177] F. Detecting annual seasonal cycles in heap usage When analyzing multi-year heap usage statistics of a software execution environment, some embodiments may detect two or more seasonal trends, each with a different time scale. For example, such embodiments may detect a multi-year time series of heap usage statistics, a yearly seasonal trend, and a daily seasonal trend. The yearly seasonal trend and the daily seasonal trend are superimposed into a multi-seasonal trend. Thus, some embodiments may adopt a time scale appropriate for analyzing the yearly seasonal trend. This time scale has a period corresponding to one year and an interval corresponding to one month. In this manner, the one-year period may be divided into twelve one-month-long intervals.

[0178] To determine whether a data set exhibits an annual seasonal cycle, some embodiments may first determine the multiplicative factors of the monthly indices included in the data set.

[0179] In a concrete case, let P denote the number of years in the data set (i.e., the number of cycles in a year). In addition, let Q denote the number of months in the data set (i.e., the number of intervals in the cycle). Thus, Q=12×P. Let K denote the number of weekdays or weekends in the data set. To represent the enumeration of these weekdays or weekends, let the index k range from 0 to (K-1). N k Let denote the number of samples in the kth weekday or weekend. Using the formula below, some embodiments can apply the formula below to calculate the average heap usage for each weekday or weekend in the dataset:

[0180]

number

[0181] Some embodiments may define the function H as follows: H:(year x integer)->index, which maps sequential pairs containing the index of a year and an integer corresponding to the index of a weekday or weekend of that year: Some embodiments may then calculate the average heap usage for each year from the average heap usage for each weekday or weekend of that year, using the formula below:

[0182]

number

[0183] Some embodiments may define the function G as follows: G:(month x integer)->index, which maps sequential pairs containing a month index and an integer corresponding to the index of a weekday or weekend of that month. Using the formula below, some embodiments may calculate the average heap usage for each monthly interval of the period from the average heap usage for weekdays or weekends of that month.

[0184]

number

[0185] After determining the multiplicative factor of the monthly index, some embodiments may apply a null hypothesis test to find out whether there is an annual seasonal cycle. In this regard, the null hypothesis to be tested is whether the correlation coefficient r between the monthly index of the most recent year "u" and the monthly index of the previous year "v" is u , v may correspond to the assumption that r is zero. Specifically, some embodiments may calculate the correlation coefficient r using the following formula: u , v may be determined.

[0186]

number

[0187] Some embodiments employ various techniques to determine the correlation coefficient r u , v is large enough to indicate above a significance level that there is a common seasonal cycle between years "u" and "v." For example, some embodiments may employ a Student's t-test, a permutation test, or a Fisher transformation.

[0188] If the null hypothesis is true (i.e., r u , v =0), then the test statistic t below has a Student's t-distribution with n-2 degrees of freedom.

[0189]

number

[0190] Let F(t) represent a probability distribution. If the significance level is 0.1, then we can determine the value of the random variable t such that F(t) = 0.9. 0.9,(n-2)Assume that the alternative hypothesis is a biased condition.

[0191]

number

[0192] The condition for accepting the alternative hypothesis is that there is a common seasonal cycle between years "u" and "v".

[0193] G. Determining the season of year with high heap usage If it is determined that there is a common seasonal cycle between the most recent year and the prior year, some embodiments may calculate the monthly seasonal factors enumerated by monthly seasonal index 0-11 by employing the following formula:

[0194]

number

[0195] Note that the recursion requires an unbound variable s, which can be used to break ties. In some embodiments, by default, s=1.

[0196] In a particular embodiment, the closure V of the seasonal index classifies the seasons with high annual heap usage. The threshold T can be set to a percentage, such as 85 percent, of the range of the seasonal factor. For example, the seasonal factors of the 12 monthly seasonal indexes for a period of one year are as shown in the table below.

[0197] [Table 3]

[0198] The range of the multiplicative seasonal factor is (1.34-0.76), which is 0.58. Therefore, 85 percent of the range of the seasonal factor is (0.76+0.85×0.58), which is 1.253. Applying this 85 percent to the threshold T, T=1.25. As a result, such an embodiment would classify May, June, and July as being in the high heap usage season of the year.

[0199] Some embodiments may select a segment of the data set that spans the most recent cycle of a one-year period. For example, from a cycle of 2013, 2014, 2015, and 2016, such an embodiment may select a segment of data that includes 2015-2016. The selected data segment may span two or more weeks of heap usage statistics data within a high annual heap usage season. For example, if the seasonal factors are as in the table below, a data segment may be selected from November 2015, December 2015, and January 2016.

[0200] [Table 4]

[0201] H. Regression to determine filter constants and time zone offsets Some embodiments may include an estimate of the time zone offset. If the time zone offset is not available, some embodiments may perform a non-linear regression on a segment of the data set to estimate the time zone offset and use this time zone offset to filter the data. By providing an estimate of the time zone offset, some embodiments may improve the estimation of seasonal indices at transitions between time periods.

[0202] Specifically, some embodiments use the following filter constants: measureFilterConstant α, rateFilterConstant β, accelerationFilterConstant κ, seasonalFactorFilterConstant A nonlinear regression may be performed using filter constants (i.e., regression parameters that are independent variables) γ, errorResidualFilterConstant δ, and timeZoneOffset tz to minimize the mean squared error (MSE) and / or mean absolute deviation (MAD) of the residuals of the one-step predictions. In some embodiments, the timestamps may be shifted by the time zone offset tz in the regression. Some embodiments may apply the nonlinear multivariate regression using an optimization routine (e.g., an optimal routine provided by the R programming language). Some embodiments may use such an implementation. The weekday and weekend seasonal factors may be derived using optimal values ​​of α, β, κ, γ, δ, and tz shown in the following equations employed by the present invention:

[0203]

number

[0204] Some embodiments include a time zone offset as a regression parameter so that the transition between cycles of periods or between two adjacent periods is as accurate as possible.

[0205] I. Ranking thread classes by degree of correlation Once the high annual heap usage season is determined, some embodiments may calculate and / or obtain weekday / weekend factors that represent the daily / weekly seasonal cycles included in the most recent (e.g., most recent) high annual heap usage season. Note that the daily / weekly seasonal cycles in this segment of the data set (i.e., during the high annual heap usage season) may be more pronounced than at other times (i.e., outside the high annual heap usage season). Thus, a determination of the degree of correlation between seasonal trends in heap usage and seasonal trends in intensity statistics for one or more thread classes may be based on this segment of the data set. In other words, for correlation analysis, some embodiments may derive seasonal trends for various thread classes using the same intervals included in the most recent high annual heap usage season.

[0206] Note that to determine seasonal trends in intensity statistics for a particular class of threads, some embodiments may employ the techniques used to determine seasonal trends in heap usage, as described above, that is, seasonal trending of thread intensity statistics and heap usage statistics may require using the same number of seasonal indexes for weekday and weekend periods (e.g., 96 seasonal indexes for one weekday period and 192 seasonal indexes for one weekend period).

[0207] Once the seasonal trends of the heap usage and the intensity statistics of one or more thread classes are determined, some embodiments may then calculate a correlation degree between the seasonal trends of the heap usage and the intensity statistics of the thread class for each of the one or more thread classes. Specifically, the correlation degree for a sequence of 96 seasonal factors or 192 seasonal factors may be calculated. Note that calculating the correlation degree between seasonal trends may be more efficient than calculating the correlation degree between a sequence of heap usage measurements and a sequence of thread intensity measurements, because the sequence of measurements is likely to be significantly longer.

[0208] Let H denote a sequence of N seasonal factors of heap usage. Let T denote a sequence of N seasonal factors of thread intensity of a class of threads. The correlation coefficient between these two sequences of seasonal factors is given by the correlation coefficient (H,T) defined below:

[0209]

number

[0210] Some embodiments may derive the weekday and weekend seasonal factors of heap usage by performing a regression on the heap usage statistics data included in the most recent annual high heap usage season. The time interval of this segment of the dataset is denoted by (t1, t2). To analyze the correlation between the seasonal factors of the thread class's intensity statistics data and the seasonal factors of the heap usage, some embodiments may obtain the seasonal factors from the same time interval (t1, t2) in the seasonal factor time series included in the SeasonalTrendInfo associated with the thread class. Specifically, the seasonal factor time series may be stored in the smoothedWeekdaySeasonalFactor and smoothedWeekendSeasonalFactor member variables included in the associated SeasonalTrendingInfo object.

[0211] Some embodiments may iterate through all ThreadClassificationInfo objects for a class of threads and recursively traverse the SegmentInfo objects contained in each of the ThreadClassificationInfo objects to collect the ThreadClassificationInfo objects and the SeasonalTrendInfo objects contained within the SegmentInfo objects. When calculating the correlation coefficient (H,T) between each of the classes of threads and the heap usage using the above formula, some embodiments may extract a weekday factor or weekend factor contained in each of the SeasonalTrendInfo objects. Once the correlation degree for each thread class is calculated, some embodiments may rank the thread classes by their correlation degree with the heap usage seasonal trend. The highest ranked thread class may then be classified as the heap-draining thread class. Some embodiments may then analyze code and stack traces associated with the heap-draining thread class to identify inefficient memory usage that can be manually or automatically corrected and / or improved.

[0212] However, some embodiments may be extended to determine the correlation coefficient based on time periods other than weekday and weekend periods (eg, end-of-quarter periods).

[0213] FIG. 9 is a flowchart 900 of a process for identifying code that may be contributing to high heap usage in a software execution environment, according to some embodiments. In some embodiments, the process illustrated in flowchart 900 may be implemented by a computer system having one or more processors (e.g., computer system 1700 of FIG. 17). The one or more processors may execute steps based on computer code stored on a computer-readable medium. The steps described in FIG. 9 may be executed in any order, with or without other steps.

[0214] Flowchart 900 begins at step 902. In step 902, an embodiment determines the length of time that heap usage by one or more processes is above a threshold. The length of time may correspond to a season of high heap usage throughout the year, and the threshold may correspond to a percentage of the range of seasonal factors assigned to an interval (e.g., a 15-minute interval) relative to one or more overall periods (e.g., weekday and weekend periods). In some embodiments, the threshold may be set by selecting a percentage. Once a percentage is selected, the threshold may be represented by the product of the seasonal factor range and the percentage plus the minimum seasonal factor. For example, if the selected percentage is 85 percent, the minimum seasonal factor is 0.76, and the maximum seasonal factor is 1.34, then the threshold may be represented by (0.76+0.85×(1.34−0.76)), which is 1.253. As a result, any interval with a multiplicative seasonal factor greater than 1.25 will be determined to be part of the length of time that heap usage is above the threshold.

[0215] In step 904, embodiments determine heap information for one or more processes during the length of time. The heap information may correspond to an amount of heap memory being used by one or more processes within the software execution environment at different times during the length of time. For example, the heap information may be based on heap usage measurements obtained from the software execution environment at irregular intervals (e.g., during full GCs). In addition, the software execution environment may correspond to a production environment that includes one or more virtual machines (e.g., JVMs), and the one or more processes may support one or more cloud services.

[0216] In step 906, embodiments determine thread information for the one or more processes for the length of time. In some embodiments, the thread information may include a thread intensity seasonality factor for each of a plurality of intervals for each of the one or more classes of threads determined from the analyzed thread dump.

[0217] In some embodiments, the heap information may include a heap usage seasonality factor for each of a plurality of intervals. Specifically, this length of time may span one or more cycles of a first period (e.g., weekday periods) having a first length and one or more cycles of a second period (e.g., weekend periods) having a second length. Each period may be divided into a plurality of intervals. For example, a weekday period may be divided into 96 15-minute intervals, and a weekend period may be divided into 192 15-minute intervals.

[0218] It should be noted that each of the multiple intervals may be associated with a particular season (i.e., a seasonal index) for one of the periods. For each seasonal index, some embodiments may determine a heap usage seasonal factor and for each identified class of threads, a thread intensity seasonal factor. Each interval is thereby associated with one heap usage seasonal factor and multiple thread intensity seasonal factors (one for each class of threads). For example, assuming three different classes of threads are discovered, a weekday period may have 96 heap usage seasonal factors and 288 thread intensity seasonal factors (96 thread intensity seasonal factors for each of the three classes of threads), and a weekend period may have 192 heap usage seasonal factors and 576 thread intensity seasonal factors.

[0219] In step 908, embodiments correlate the heap information with the thread information to identify one or more lines of code in one or more processes that correspond to heap usage above a threshold. Correlating group information with thread information is described in more detail below with reference to FIG.

[0220] In response to identifying one or more lines of code at step 910, embodiments may initiate one or more actions related to the one or more lines of code. For example, embodiments may generate an alert related to the one or more lines of code that is sent to an interested party or a code optimization tool. In response, the identified lines of code may be investigated and / or optimized. Alternatively, some embodiments may optimize the one or more lines of code to more efficiently utilize heap memory.

[0221] 10 is a flowchart 1000 of a process for calculating the degree of correlation between various classes of threads and high heap usage, according to some embodiments. In some embodiments, the process illustrated in flowchart 1000 may be implemented by a computer system having one or more processors (e.g., computer system 1700 of FIG. 17). The one or more processors may perform the steps based on computer code stored on a computer-readable medium. The steps described in FIG. 10 may be performed in any order, with or without other steps.

[0222] Flowchart 1000 begins at step 1002. In step 1002, an embodiment obtains one or more thread dumps of one or more processes. As described above, the control system may periodically cause the software execution environment to output thread dumps that include one or more stack traces of threads generated by one or more processes executing within the software execution environment.

[0223] In step 1004, embodiments may receive one or more threads from one or more thread dumps and obtain a class of one or more threads by classifying each of the received threads based on a stack trace corresponding to the received thread. Once all of the thread dumps have been received and processed, embodiments may analyze each of the one or more thread classes in steps 1006-1016 to determine a degree of correlation between each of the thread classes and high heap usage.

[0224] In decision 1006, embodiments may determine whether there is another class of thread among the one or more thread classes and determine how it correlates with high heap usage. If there is, embodiments may proceed to step 1008. If not, embodiments may proceed to step 1018.

[0225] In optional step 1008, embodiments calculate an average of the heap usage seasonal factor for the multiple intervals. In step 1010, embodiments calculate an average of the thread class and thread intensity seasonal factor for the multiple intervals. In optional step 1012, embodiments calculate a variance of the heap usage seasonal factor for the multiple intervals. In step 1014, embodiments calculate a variance of the thread class and thread intensity seasonal factor for the multiple intervals. In step 1016, embodiments calculate a degree of correlation between thread class and heap usage above a threshold.

[0226] In step 1018, embodiments select a given thread class from one or more thread classes that has the highest correlation to the heap usage above the threshold. Specifically, once the correlation for each thread class is calculated, some embodiments may rank the thread classes by their correlation with the seasonal heap usage trend. The highest ranked thread class may then be selected as the given thread class.

[0227] In step 1020, embodiments may identify one or more lines of code that may be a significant contributor to high heap usage based on the given thread's class. Specifically, some embodiments may then analyze the file names and lines specified by the stack trace to identify lines of code associated with the class of thread that is draining the heap, where each thread of one or more processes that belongs to the given thread's class executes the one or more lines of code.

[0228] VI. Overcoming Weak Exogeneity and Heteroscedasticity in Forecasting As mentioned above, to ensure accuracy of the samples, heap allocations may be measured during full garbage collection (GC) cycles, which occur at irregular intervals. In situations where heap usage is particularly high, the constant garbage collections will cause the sampling interval to approach zero at random. As a result, time series data based on measurements of heap allocations will exhibit weak exogeneity and heterogeneity of variance. Weak exogeneity means that the process generating the residuals depends somewhat on the process generating the time intervals between full GC samples. Heterogenous variance means that the variance of the residuals is not constant over time.

[0229] Traditionally, generating an ordinary least squares regression of a linear trend assumes that the predictor and response variables are generated by an exogenous and homoscedastic process. However, for a dataset based on values ​​measured during full GCs, the predictor (i.e., irregular intervals) and response (i.e., heap usage measured during full GCs) variables are not independent because as heap usage increases, the frequency at which full GCs are issued also increases. Some embodiments may overcome the weak exogeneity and heteroscedasticity of the dataset using robust and resistant regression methods.

[0230] Certain embodiments may utilize robust least squares regression to overcome the weak exogeneity and heterogeneity of variance exhibited in such data sets. Specifically, some embodiments may (1) decompose the measurement time series into a deseasonalized measurement component (i.e., the deseasonalized component) and a seasonal factor component (i.e., the seasonal effector), (2) apply robust linear regression to the deseasonalized measurement component, (3) apply a smoothing spline filter to the seasonal factor component, and (4) reconstruct the linear regression line and the smoothed seasonal factor into a seasonal linear trend model.

[0231] The Least-Trimmed Squares (LTS) estimator is sensitive to outliers. It is a robust regression method without a fixed square residual. Given a set of N samples, the LTS estimator minimizes the sum of the least square residuals by trimming the 50% of the samples corresponding to the largest square residuals as outliers. The LTS estimator performs one iteration of ordinary least square regression on all N samples, sorts the residuals, and selects the smallest N / 2 residual (i.e., the trimmed sample). The LTS estimator then iteratively reruns the regression by updating the trimmed sample to reduce the mean of the square residuals. However, compared to the specific embodiment described below, the time complexity of the LTS algorithm may be relatively high.

[0232] The generalized weighted least-squares (WLS) estimator is a robust regression method that multiplies the squared error residual of each sample by a weight that is inversely proportional to the sample variance. The employment of the WLS estimator can depend on weights determined by prior knowledge of the data. For example, the prior knowledge may specify (1) the accuracy of different tools used to measure different sample points, (2) the variance between duplicate measurements corresponding to the same time, or (3) the variance between measurements of nearest neighbor groups. If the weights cannot be determined by prior knowledge, the WLS estimator may estimate the residuals with one iteration of ordinary least-squares regression, and then iteratively rerun the regression using the inverse of the residuals as weights, producing a stable estimate of the linear model. However, compared to the specific embodiments described below, the time complexity of the WLS algorithm is relatively high.

[0233]

number

[0234]

number

[0235] Some embodiments then calculate the raw growth rate with seasonal variation removed using the following formula: The addition rate may also be calculated.

[0236]

number

[0237] Some embodiments may then use the formula to update the moving average.

[0238]

number

[0239] Some embodiments use a rate filter parameter to trim the data points. Trimming the data points helps to make the density of sample points uniform over the entire time domain, thereby improving the robustness of the linear regression algorithm. For data points that represent measurements of heap usage in a software execution environment during a full GC cycle, data points that are close to each other will correspond to periods of higher heap usage (e.g., during load spikes) when full GCs are performed more frequently.

[0240] Some embodiments compare the rate filter parameter to a threshold and, if the rate filter parameter is less than the threshold, exclude (i.e., trim) the corresponding data point from the robust linear regression. Some embodiments may use the median or mean of the rate filter parameter as the threshold. In particular, some embodiments may trim data points that are close to each other, since they may represent load spikes or outliers. As a result, some embodiments may mitigate the weak exogeneity condition by making the density of data points uniform along the time axis, which reduces the correlation between irregular time intervals and the residuals.

[0241]

number

[0242] Some embodiments may generate the error residual of the predicted measurement using the following formula:

[0243]

number

[0244] The following example code (written in the R programming language) shows how the trimmed subset of samples and sample weights may be calculated. As shown in the example code below, some embodiments may use the R function "rlm". The R function "rlm" allows certain embodiments to specify the trimmed subset of samples and sample weights for generating a weighted least squares regression. Note that the rateFilterParameter, seasonalFactor, absoluteErrorResidual, measure, and time vector in the example code are time series with the same time domain.

[0245]

number

[0246]

number

[0247]

number

[0248] To reduce the impact of outliers and short-term spikes in heap usage on the linear regression, some embodiments may combine a uniform density of data points with a smaller weighting of outlying sample values. In doing so, some embodiments may increase the robustness of the linear regression, which may make it easier to capture long-term trends (e.g., in heap usage). Note that by using these two techniques together, the linear regression line may be better fitted to the data and may be more efficient than traditional LTS or WLS estimators, which typically require multiple iterations of the regression.

[0249] To further improve the robustness of the regression, some embodiments further identify transition states and prune sample points that are in the transition states, and prune run-to-run segments that are outliers (e.g., data segments that correspond to memory leaks, out-of-memory events, or software execution environments experiencing very high growth rates). Good too.

[0250] 12 shows three trend graphs derived by different linear regression methods for determining heap usage in a production environment. The blue trend line 1205 can be derived by a standard linear regression algorithm that assigns equal weights to each sample point. The brown trend line 1210 can be derived by a conventional robust regression algorithm. The red line 1215 represents the regression provided by the present embodiment described above and is located close to the brown trend line.

[0251] Figure 13 shows additional graphs illustrating how traditional regression methods can produce erroneous results. As shown in the graph, the brown trend line 1305, which represents traditional regression, closely fits two clusters of densely populated sample points. In contrast, the red line 1215 accurately tracks the trends in the sample points and provides a long-term forecast of heap usage in a software execution environment.

[0252] 14 is a flowchart 1400 of a process for generating a prediction of a signal, according to some embodiments. In some embodiments, the process illustrated in flowchart 1400 may be implemented by a computer system having one or more processors (e.g., computer system 1700 of FIG. 17). The one or more processors may perform steps based on computer code stored on a computer-readable medium. The steps described in FIG. 14 may be performed in any order, with or without other steps.

[0253] Flowchart 1400 begins at step 1402. In step 1402, an embodiment receives a signal from an environment in which one or more processes are executing, the signal including a plurality of measurements sampled over a time span. In some embodiments, the plurality of measurements may be heap usage measured by a control system monitoring heap usage in a software execution environment (e.g., a production environment) that includes one or more running processes.

[0254] In step 1404, embodiments extract from the signal a deseasonalized component and a seasonal effector (step 1404). In some embodiments, the seasonal effector may correspond to a seasonal factor identified for each interval of time assigned to the data set. In some embodiments, the deseasonalized component may be obtained by applying a seasonal factor to the signal.

[0255] In step 1406, embodiments apply one or more spline functions to the seasonal effector to generate the first model. In this regard, some embodiments may give relatively little weight to sample values ​​that deviate significantly from expected values ​​that represent a convolution of nearby groups of samples.

[0256] In step 1408, embodiments may apply a linear regression method to the components with the seasonal variations removed to generate a second model. Specifically, to compensate for the relatively short intervals experienced during high heap usage, some embodiments may adjust the filter parameters to give a relatively small weight to samples taken during short intervals. Some embodiments may use the rate filter parameters to trim the data points included in the data set. Trimming the data points helps to make the density of sample points uniform across the entire time domain, thereby improving the robustness of the linear regression algorithm.

[0257] In step 1410, embodiments generate a prediction of the signal based on the first model and the second model. In some embodiments, the prediction of the signal is performed in steps 1406 and It may correspond to a regression line produced using the techniques described in 1408. In particular, the predictions generated will provide a better fit to the signal.

[0258] In step 1412, embodiments may initiate one or more actions related to the environment based at least in part on the prediction. For example, if the prediction indicates that heap usage will increase in the future, some embodiments may allocate additional resources (e.g., memory, RAM) to the software execution environment.

[0259] 15 is a schematic diagram of a distributed system 1500 for implementing an embodiment. In the illustrated embodiment, the distributed system 1500 comprises one or more client computing devices 1502, 1504, 1506, and 1508. The one or more client computing devices 1502, 1504, 1506, and 1508 are configured to execute and operate client applications, such as web browsers, proprietary clients (e.g., Oracle Forms), etc., over one or more network(s) 1510. A server 1512 may be communicatively coupled to the client computing devices 1502, 1504, 1506, and 1508 via the network 1510.

[0260] In various embodiments, server 1512 may be adapted to run one or more services or software applications. In particular embodiments, server 1512 may also provide other services or software applications, which may include non-virtual and virtual environments. In some embodiments, these services may be provided to users of client computing devices 1502, 1504, 1506, and / or 1508 as web-based or cloud services, or under a Software as a Service (SaaS) model. Users operating client computing devices 1502, 1504, 1506, and / or 1508 can utilize one or more client applications to interact with server 1512 to utilize the services provided by these components.

[0261] In the configuration shown in FIG. 15, the software components 1518, 1520, and 1522 of the system 1500 are shown as being implemented on a server 1512. In other embodiments, one or more of the components of the system 1500 and / or one or more of the services provided by these components may also be implemented by one or more of the client computing devices 1502, 1504, 1506, and / or 1508. A user operating a client computing device may then utilize one or more client applications to use the services provided by these components. These components may be implemented in hardware, firmware, software, or a combination thereof. It should be understood that a variety of different system configurations are possible and may differ from the distributed system 1500. Thus, the embodiment shown in FIG. 15 is an example of a distributed system for implementing the example system and is not intended to be limiting.

[0262] Client computing devices 1502, 1504, 1506, and / or 1508 may include various types of computing systems. For example, client computing devices may be palm-sized portable devices (e.g., iPhones, mobile phones, iPads, computing tablets, Personal Digital Assistants (PDAs)) or wearable devices that run software such as Microsoft Windows Mobile and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 10, Palm OS, etc. The device may include a head mounted display (e.g., a Google Glass® head mounted display). The device may support a variety of applications, such as various Internet-related apps, email, and short message service (SMS) applications. or may use a variety of other communication protocols. The client computing devices may also include general purpose personal computers, including, by way of example, personal and / or laptop computers running various versions of Microsoft Windows, Apple Macintosh, and / or Linux operating systems. The client computing devices may also be workstation computers running various commercially available UNIX or UNIX-like operating systems, including, but not limited to, various GNU / Linux operating systems, such as Google Chrome OS. The client computing devices may also include electronic devices capable of communicating over the network(s) 1510, such as thin client computers, Internet-enabled gaming systems (e.g., Microsoft Xbox gaming consoles with or without Kinect gesture input devices), and / or personal messaging devices.

[0263] 15 is shown with four client computing devices, any number of client computing devices may be supported. Other devices, such as sensor-equipped devices, may also interact with the server 1512.

[0264] The network(s) 1510 in the distributed system 1500 may include, but are not limited to, TCP / IP (Transmission Control Protocol / Internet Protocol), SNA The network(s) 1510 may be any type of network familiar to those skilled in the art that can support data communications using available protocols including the Systems Network Architecture (ISNA), Internet Packet Exchange (IPX), AppleTalk, etc. By way of example only, the network(s) 1510 may be a Local Area Network (LAN), an Ethernet-based network, a Token Ring, a Wide Area Network, the Internet, a Virtual Private Network (VPN), an Intranet, an Extranet, etc. , PSTN (Public Switched Telephone Network), infrared network, wireless Networks (e.g., IEEE (Institute of Electrical and Electronics) 80 The network may be a virtual network including any combination of these, and / or other networks, such as networks operating under any of the 2.11 suite of protocols, Bluetooth, and / or other wireless protocols.

[0265] The server 1512 may be one or more general-purpose computers, dedicated server computers (for example, a PC (Personal Computer) server, a UNIX (registered trademark) server, a mid-range The servers 1512 may be comprised of a single virtual machine (including servers, mainframe computers, rack-mounted servers, etc.), a server farm, a server cluster, or any other suitable arrangement and / or combination. The servers 1512 may include one or more virtual machines running a virtual operating system, or other computing architecture requiring virtualization. One or more flexible pools of logical storage may be virtualized to maintain virtual storage for the servers. Virtual networks may be controlled by the servers 1512 using Software-Defined Networking (SDN). In various embodiments, the servers Server 1512 may be adapted to execute one or more of the services or software applications described in the above disclosure. For example, server 1512 may correspond to a server for performing the above-mentioned processes according to an embodiment of the present disclosure.

[0266] Server 1512 may run any of the operating systems mentioned above, as well as any commercially available server operating system. Server 1512 may also run a variety of additional server and / or mid-tier applications, including a Hypertext Transport Protocol (HTTP) server, a File Transfer Protocol (FTP) server, a Common Gateway Interface (CGI) server, a JAVA server, a database server, and the like. Exemplary database servers include those from Oracle, Microsoft, Sybase, IBM (International Business Machines), Examples of database servers sold by companies such as, but not limited to, these include database servers sold by companies such as, but not limited to,

[0267] In some implementations, the server 1512 may include one or more applications for analyzing and consolidating data feeds and / or event updates received from users of the client computing devices 1502, 1504, 1506, and 1508. By way of example, the data feeds and / or event updates may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates received from one or more third-party information sources and continuous data streams. The data feeds and / or event updates may include real-time events related to sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and the like. The server 1512 may also include one or more applications for displaying the data feeds and / or real-time events via one or more display devices of the client computing devices 1502, 1504, 1506, and 1508.

[0268] The distributed system 1500 may also include one or more databases 1514 and 1516. These databases may provide a mechanism for storing information such as user interaction information, usage pattern information, matching rule information, and other information used by embodiments of the present disclosure. The databases 1514 and 1516 may reside in a variety of locations. As an example, one or more of the databases 1514 and 1516 may reside on a non-transitory storage medium local to (and / or resident in) the server 1512. Alternatively, the databases 1514 and 1516 may reside in a location remote from the server 1512 and communicate with the server 1512 through a network-based or dedicated connection. In one set of embodiments, the databases 1514 and 1516 may reside in a Storage-Area Network (SAN). Similarly, any files necessary to perform functions attributed to the server 1512 may be stored locally on the server 1512 and / or remotely from the server 1512, as appropriate. In one set of embodiments, databases 1514 and 1516 may include relational databases adapted to store, update, and retrieve data in response to SQL-formatted commands, such as those provided by Oracle.

[0269] In some embodiments, the cloud environment may provide one or more services. FIG. 16 is a simplified block diagram of one or more components of a system environment 1600 in which services may be provided as cloud services, according to an embodiment of the present disclosure. In the embodiment shown in FIG. 16, the system environment 1600 includes one or more client computing devices 1604, 1606, and 1608. The one or more client computing devices 1604, 1606, and 1608 may be used by users to interact with a cloud infrastructure system 1602 that provides cloud services. The cloud infrastructure system 1602 may include one or more computers and / or servers, which may include those described above with respect to server 1612. stomach.

[0270] It should be understood that the cloud infrastructure system 1602 shown in Figure 16 may have components other than those shown. Additionally, the embodiment shown in Figure 16 is only one example of a cloud infrastructure system that may incorporate embodiments of the present disclosure. In other embodiments, the cloud infrastructure system 1602 may have more or fewer components than those shown in the figure, may combine two or more components, or may have a different configuration or arrangement of components.

[0271] Client computing devices 1604, 1606, and 1608 may be devices similar to those described above. Client computing devices 1604, 1606, and 1608 may be configured to operate a client application, such as a web browser, a proprietary client application (e.g., Oracle Forms), or other application. The client application may be used by a user of the client computing device to interact with cloud infrastructure system 1602 to utilize services provided by cloud infrastructure system 1602. Although exemplary system environment 1600 is shown with three client computing devices, any number of client computing devices may be supported. Other devices, such as sensored devices, may interact with cloud infrastructure system 1602.

[0272] Network(s) 1610 may facilitate communication and exchange of data between client computing devices 1604, 1606, and 1608 and cloud infrastructure system 1602. Each network may be any type of network familiar to those skilled in the art capable of supporting data communication using any of a variety of commercially available protocols, including those discussed above with respect to network(s) 1610.

[0273] In particular embodiments, the services provided by cloud infrastructure system 1602 may include a host of services that are available upon request to users of the cloud infrastructure system. A variety of other services may be provided including, but not limited to, online data storage and backup solutions, web-based email services, hosted office suite document collaboration services, database processing, managed technical support services, etc. The services provided by the cloud infrastructure system may be dynamically scaled to meet the needs of its users.

[0274] In certain embodiments, a specific instantiation of a service provided by cloud infrastructure system 1602 may be referred to herein as a "service instance." In general, any service from a cloud service provider's system that is made available to a user over a communications network, such as the Internet, is referred to as a "cloud service." Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are distinct from the customer-owned on-premise servers and systems. For example, the cloud service provider's system may host an application, and a user may order and use the application on a request basis over a communications network, such as the Internet.

[0275] In some instances, the cloud infrastructure of a computer network The services in this case may include storage, hosted databases, hosted web servers, protected computer network access to software applications, or other services that a cloud vendor provides to users, or other services not mentioned above that are known in the art. For example, a service may include password-protected access to remote storage on a cloud over the Internet. As another example, a service may include a web-services-based hosted relational database and scripting language middleware engine for private use by networked developers. As another example, a service may include access to an email software application hosted on a cloud vendor's website.

[0276] In particular embodiments, cloud infrastructure system 1602 may include a suite of application, middleware, and database service offerings that are delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such a cloud infrastructure system is the Oracle Public Cloud offered by the assignee of the present application.

[0277] The cloud infrastructure system 1602 may also provide computing and analytical services on "big data." The term "big data" generally refers to extremely large data sets that can be stored and manipulated by analysts and researchers to visualize, detect trends, and / or interact with large amounts of data. This big data and related applications may be hosted and / or manipulated by the infrastructure system at many levels and different scales. Dozens, hundreds, or thousands of processors linked in parallel may act on such data to present it or to simulate external forces on it or what it represents. These data sets may require structured data, such as structured data organized in a database or according to a structured model, and / or unstructured data (e.g., emails, images, data blobs (binary large objects), web pages, complex event processing). By leveraging the capabilities of embodiments to relatively quickly assemble more (or fewer) computing resources to a target, the cloud infrastructure system may be better able to perform tasks on large data sets based on requests from companies, government agencies, research institutes, private individuals, like-minded groups or organizations, or other entities.

[0278] In various embodiments, cloud infrastructure system 1602 may be adapted to automatically provision, manage, and track customer subscriptions to the services it provides. Cloud infrastructure system 1602 may provide cloud services through different deployment models. For example, the services may be provided under a public cloud model, where an organization that sells cloud services (e.g., owned by Oracle Corporation) owns cloud infrastructure system 1602 and the services are made available to the general public or to different industry enterprises. As another example, the services may be provided under a private cloud model, where cloud infrastructure system 1602 is run solely for one organization and cloud infrastructure system 1602 may provide services for one or more entities within the organization. Cloud services may also be provided under a community cloud model, where cloud infrastructure system 1602 and the services it provides are shared by several organizations within an associated community. Cloud services may be offered under a hybrid cloud model, which is a combination of two or more different models.

[0279] In some embodiments, the services provided by the cloud infrastructure system 1602 are provided under the Software as a Service (SaaS) category, the Platform as a Service (PaaS) category, and the Infrastructure as a Service (IaaS) category. A subscription order may include one or more services, or other categories of services, including hybrid services. A customer may order one or more services offered by cloud infrastructure system 1602 through a subscription order. Cloud infrastructure system 1602 then performs processing to provide the services in the customer's subscription order.

[0280] In some embodiments, the services provided by the cloud infrastructure system 1602 may include, but are not limited to, application services, platform services, and infrastructure services. In some examples, application services may be provided by the cloud infrastructure system through SaaS services. A SaaS platform may be configured to provide cloud services that fall under the SaaS category. For example, the SaaS platform may provide functionality for building and delivering a suite of on-demand applications to an integrated development / deployment platform. The SaaS platform may manage and control the underlying software and infrastructure for providing the SaaS services. By utilizing the services provided by the SaaS platform, customers can utilize applications that run on the cloud infrastructure system. Customers can obtain application services without having to purchase licenses and support separately. A variety of different SaaS services may be provided. Examples include, but are not limited to, services that provide solutions for sales performance management, enterprise integration, and business flexibility for large organizations.

[0281] In some embodiments, platform services may be provided by cloud infrastructure system 1602 through a PaaS platform. The PaaS platform may be configured to provide cloud services that fall under the PaaS category. Platform services include, but are not limited to, services that allow an organization (such as Oracle) to consolidate existing applications onto a shared common architecture, and the ability to create new applications that leverage the shared services provided by the platform. The PaaS platform may manage and control the underlying software and infrastructure to provide the PaaS services. Customers can obtain the PaaS services provided by cloud infrastructure system 1602 without having to purchase separate licenses and support. Platform services include, but are not limited to, JCS (Oracle Java Cloud Service), DBCS (Oracle Database Cloud Service), and others.

[0282] The services provided by the PaaS platform allow customers to employ programming languages ​​and tools supported by the cloud infrastructure system and to manage the deployed services. In some embodiments, the platform services provided by the cloud infrastructure system may include database cloud services, middleware cloud services (e.g., Oracle Fusion Middleware services), and Java cloud services. In one embodiment, the database cloud services include: It may support a shared services deployment model that allows organizations to pool database resources and provide database as a service to customers in the form of a database cloud. In a cloud infrastructure system, a middleware cloud service may provide a platform for customers to develop and deploy various business applications, and a Java cloud service may provide a platform for customers to deploy Java applications.

[0283] In a cloud infrastructure system, a variety of different infrastructure services may be provided by the IaaS platform that facilitates management and control of underlying computing resources such as storage, network, and other fundamental computing resources for customers utilizing the services provided by the SaaS and PaaS platforms.

[0284] Additionally, in certain embodiments, cloud infrastructure system 1602 may include infrastructure resources 1630 for providing resources used to provide various services to customers of the cloud infrastructure system. In one embodiment, infrastructure resources 1630 may include a pre-integrated optimized combination of hardware, such as servers, storage, and networking resources for running the services and other resources offered by the PaaS and SaaS platforms.

[0285] In some embodiments, resources in cloud infrastructure system 1602 may be shared by multiple users and dynamically reallocated upon demand. In addition, resources may be allocated to users in different time zones. For example, cloud infrastructure system 1602 may allow a first set of users in a first time zone to utilize resources in the cloud infrastructure system for a specified number of hours and then reallocate the same resources to another set of users located in a different time zone, thereby maximizing utilization of the resources.

[0286] In particular embodiments, a number of internal shared services 1632 may be provided that are shared by different components or modules of cloud infrastructure system 1602 to enable the provision of services by cloud infrastructure system 1602. These internal shared services may include, but are not limited to, security / identity services, integration services, enterprise repository services, enterprise manager services, virus scanning / whitelist services, highly available backup / recovery services, services to enable cloud support, email services, notification services, file transfer services, etc.

[0287] In particular embodiments, cloud infrastructure system 1602 may provide comprehensive management of cloud services (e.g., SaaS services, PaaS services, and IaaS services) in the cloud infrastructure system. In one embodiment, cloud management functions may include, for example, functionality for provisioning, managing, and tracking customer subscriptions, etc., received by cloud infrastructure system 1602.

[0288] In one embodiment, as shown in FIG. 16, the cloud management functionality includes an order management module 1620, an order orchestration module 1622, an order provisioning module 1624, and an order management module 1626. The functionality may be provided by one or more modules, such as a tracking module 1624, an order management / monitoring module 1626, and an identity management module 1628. These modules may include or be provided using one or more computers and / or servers. The one or more computers and / or servers may be general purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0289] In an exemplary operation, at step 1634, a customer using a client device, such as client computing device 1604, 1606, or 1608, may interact with cloud infrastructure system 1602 by requesting one or more services offered by cloud infrastructure system 1602 and ordering a subscription for one or more services offered by cloud infrastructure system 1602. In particular embodiments, the customer may access a cloud UI, such as cloud user interface (UI) 1612, cloud UI 1614, and / or cloud UI 1616, and place the subscription order via these UIs. Order information received by cloud infrastructure system 1602 in response to a customer placing an order may include information identifying the customer and one or more services offered by cloud infrastructure system 1602 for which the customer subscribes.

[0290] In step 1636, the order information received from the customer may be stored in order database 1618. If this is a new order, a new record may be created for the order. In one embodiment, order database 1618 may be one of several databases operated by cloud infrastructure system 1618 and operated along with other system elements.

[0291] In step 1638, the order information may be forwarded to the order management module 1620. The order management module 1620 may be configured to perform billing and accounting functions for the order, such as confirming the order and registering the order once confirmed.

[0292] At step 1640, information regarding the order may be transmitted to an order orchestration module 1622. The order orchestration module 1622 is configured to orchestrate the provisioning of services and resources related to the orders placed by the customer. In some cases, the order orchestration module 1622 may use the services of the order provisioning module 1624 for provisioning. In certain embodiments, the order orchestration module 1622 enables management of business processes associated with each order and applies business logic to determine whether the order should proceed with provisioning.

[0293] As shown in the embodiment depicted in FIG. 16, in step 1642, upon receiving a new subscription order, the order orchestration module 1622 allocates the resources required to fulfill the subscription order and sends a request to the order provisioning module 1624 to configure the resources. The order provisioning module 1624 enables the allocation of resources for the services subscribed to by the customer. The order provisioning module 1624 provides a level of abstraction between the cloud services provided by the cloud infrastructure system 1600 and the physical implementation layer used to provision the resources to provide the requested services. This allows for control over whether the services and resources are actually provisioned on the fly or pre-provisioned and allocated only when requested. This allows the order orchestration module 1622 to be decoupled from implementation details, such as whether the order was created or not.

[0294] Once the services and resources have been provisioned, in step 1644, a notification may be sent to the subscribing customer indicating that the requested service is ready for use. In some cases, information (e.g., a link) may be sent to the customer that allows the customer to begin using the requested service.

[0295] In step 1646, the customer's subscription order may be managed and tracked by the order management / monitoring module 1626. In some cases, the order management / monitoring module 1626 may be configured to collect usage statistics regarding the customer's use of the subscribed services. For example, statistics may be collected about the amount of storage used, the amount of data transferred, the number of users, system uptime and system downtime, etc.

[0296] In certain embodiments, cloud infrastructure system 1600 may include identity management module 1628. Identity management module 1628 is configured to provide identity services, such as access management and access authorization services in cloud infrastructure system 1600. In some embodiments, identity management module 1628 may control / manage information about customers who wish to utilize services provided by cloud infrastructure system 1602. Such information may include information authenticating the identity of such customers and information describing what operations those customers are authorized to perform on various system resources (e.g., files, directories, applications, communication ports, memory segments, etc.). Identity management module 1628 may also include management of descriptive information about each customer, and how that descriptive information may be accessed and modified and by whom.

[0297] FIG. 17 illustrates an exemplary computer system 1700 that may be used to implement embodiments of the present disclosure. In some embodiments, the computer system 1700 may be used to implement any of the various servers and computer systems described above. As shown in FIG. 17, the computer system 1700 includes various subsystems including a processing unit 1704 that communicates with several peripheral subsystems via a bus subsystem 1702. These peripheral subsystems may include a processing acceleration unit 1706, an I / O subsystem 1708, a storage subsystem 1718, and a communication subsystem 1724. The storage subsystem 1718 may include a tangible computer readable storage medium 1722 and a system memory 1710.

[0298] The bus subsystem 1702 provides a mechanism for allowing the various components and subsystems of the computer system 1700 to communicate with one another as intended. Although the bus subsystem 1702 is illustrated as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 1702 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using various bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus, which may be implemented as a Mezzanine bus manufactured in accordance with the IEEE P1386.1 standard, or the like.

[0299] The processing subsystem 1704 controls the operation of the computer system 1700 and may include one or more processing units 1732, 1734, etc. The processing units may include one or more processors, including single-core processors or multi-core processors, one or more cores of a processor, or a combination thereof. In some embodiments, the processing subsystem 1704 may include one or more special-purpose co-processors, such as a graphics processor, digital signal processor (DSP), etc. In some embodiments, some or all of the processing units of the processing subsystem 1704 may be implemented using custom circuitry, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0300] In some embodiments, a processing unit included in the processing subsystem 1704 can execute instructions stored in the system memory 1710 or on the computer readable storage medium 1722. In various embodiments, the processing unit can execute various program or code instructions and maintain multiple simultaneously running programs or processes. At any time, some or all of the program code being executed may reside in the system memory 1710 and / or on the computer readable storage medium 1722, potentially including on one or more storage devices. Through suitable programming, the processing subsystem 1704 can provide a variety of functions.

[0301] In particular embodiments, a processing acceleration unit 1706 may be provided to perform customized processing or to offload some of the processing performed by the processing subsystem 1704 to speed up the overall processing performed by the computer system 1700.

[0302] I / O subsystem 1708 may include devices and mechanisms for inputting information into computer system 1700 and / or devices and mechanisms for outputting information via computer system 1700. In general, use of the term "input device" is intended to include any type of device and mechanism for inputting information into computer system 1700. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touch pads or touch screens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as a Microsoft Kinect® motion sensor that allows a user to control and interact with the input device, a Microsoft Xbox® 360 game controller, and devices that provide an interface for accepting input using gesture and voice commands. The user interface input device may also detect the user's eye actions (e.g., "blinking" while taking a photo and / or making a menu selection) and use the eye gestures as input to the input device (e.g., Google Glass®). In addition, the user interface input devices may include voice recognition sensing devices that allow a user to interact with a voice recognition system (e.g., the Siri® navigator) via voice commands.

[0303] Other examples of user interface input devices include three-dimensional (3D) mice, joysticks or pointing sticks, game pads and graphic tablets, as well as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, etc. Examples of user interface input devices include, but are not limited to, audio / visual devices such as, for example, a digital camera, a 3D printer, a laser range finder, and an eye-tracking device. In addition, user interface input devices may include medical imaging input devices, for example, computed tomography, magnetic resonance imaging, positron emission tomography, and ultrasound devices. User interface input devices may also include audio input devices, for example, MIDI keyboards, digital musical instruments, and the like.

[0304] User interface output devices may include a display subsystem, indicator lights, or non-visual display devices such as audio output devices, etc. The display subsystem may be a flat panel display such as one using a cathode ray tube (CRT), liquid crystal display (LCD) or plasma display, a projection device, a touch screen, etc. In general, use of the term "output device" is intended to include any type of device or mechanism for outputting information from computer system 1700 to a user or to another computer. For example, user interface output devices may include, but are not limited to, a variety of display devices that visually convey textual, graphical, and audio / visual information, such as monitors, printers, speakers, headphones, automobile navigation systems, drawing devices, audio output devices, and modems.

[0305] The storage subsystem 1718 provides a repository or data store for storing information used by the computer system 1700. The storage subsystem 1718 provides a tangible, non-transitory, computer-readable storage medium for storing basic programming and data constructs that provide the functionality of some embodiments. Software (programs, code modules, instructions) that, when executed by the processing subsystem 1704, provide the functionality described above may be stored in the storage subsystem 1718. The software may be executed by one or more processing units of the processing subsystem 1704. The storage subsystem 1718 may also provide a repository for storing data used in accordance with the present disclosure.

[0306] The storage subsystem 1718 may include one or more non-transient memory elements, including volatile and non-volatile memory elements. As shown in FIG. 17, the storage subsystem 1718 includes a system memory 1710 and a computer-readable storage medium 1722. The system memory 1710 may include several memories, including a volatile main RAM (Random Access Memory) for storing instructions and data during program execution, and a non-volatile ROM (Read Only Memory) or flash memory in which fixed instructions are stored. In some implementations, a Basic Input / Output System (BIOS), which contains the basic routines that help transfer information between elements in the computer system 1700, such as during start-up, may typically be stored in ROM. The RA M typically contains the data and / or program modules currently being operated on and executed by the processing subsystem 1704. In some implementations, the system memory 1710 may include a number of different types of memory, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0307] 17, system memory 1710 may store application programs 1712, which may include client applications, web browsers, mid-tier applications, relational database management systems (RDBMS), and the like, program data 1714, and operating system 1716. By way of example, operating system 1716 may include various versions of Microsoft Windows, Apple Macintosh, and / or Linux operating systems, various commercially available UNIX or UNIX operating systems, and the like. The operating systems may include operating systems similar to those described above (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® 10 OS, and Palm® OS operating systems.

[0308] The computer readable storage medium 1722 may store programming and data structures that provide the functionality of some embodiments. Software (programs, code modules, instructions) that, when executed by the processing subsystem 1704, provide the processor with the functionality described above may be stored in the storage subsystem 1718. By way of example, the computer readable storage medium 1722 may include non-volatile memory such as a hard disk drive, a magnetic disk drive, an optical disk drive such as a CD ROM, a DVD, a Blu-Ray® disk, or other optical media. The computer readable storage medium 1722 may include, but is not limited to, a Zip® drive, a flash memory card, a Universal Serial Bus (USB) flash drive, a Secure Digital (SD) card, a DVD disk, a digital video tape, and the like. The computer readable storage media 1722 may also include non-volatile memory based Solid-State Drives (SSDs), such as flash memory based Solid-State Drives, enterprise flash drives, solid state ROM, volatile memory based SSDs, such as solid state RAM, dynamic RAM, static RAM, DRAM based SSDs, Magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM and flash memory based SSDs. The computer readable media 1722 may provide non-volatile storage of computer readable instructions, data structures, program modules, and other data for the computer system 1700.

[0309] Additionally, in certain embodiments, storage subsystem 1700 may include a computer readable storage medium reader 1720 that may be further coupled to a computer readable storage medium 1722. Alternatively, in conjunction with system memory 1710, or in combination with system memory 1710 as appropriate, computer readable storage medium 1722 may comprehensively represent remote, local, fixed, and / or removable storage devices, as well as storage media for storing computer readable information.

[0310] In certain embodiments, computer system 1700 may provide support for running one or more virtual machines. Computer system 1700 may execute programs such as a hypervisor to facilitate configuration and management of virtual machines. Each virtual machine may be assigned memory, computational (e.g., processors, cores), I / O, and networking resources. Each virtual machine typically runs its own operating system. This operating system may be the same or different from the operating systems executed by other virtual machines executed by computer system 1700. Thus, multiple operating systems may be executed simultaneously by computer system 1700. Each virtual machine generally runs separately from other virtual machines.

[0311] The communications subsystem 1724 provides an interface to other computer systems and networks. The communications subsystem 1724 serves as an interface for receiving data from the computer system 1700 and for transmitting data from the computer system 1700 to other systems. For example, the communications subsystem 1724 may include one or more The computer system 1700 may be capable of establishing a communication channel with a client computing device over the Internet to send and receive information to the client computing device.

[0312] The communications subsystem 1724 may support wired and / or wireless communications protocols. For example, in certain embodiments, the communications subsystem 1724 may include a Radio Frequency (RF) transceiver component for accessing a wireless voice or / and data network (e.g., using cellular technology, next generation data network technologies such as 3G, 4G, or Enhanced Data Rates For Global Evolution (EDGE), WiFi (IEEE 802.11 family of standards), other mobile communications technologies, or any combination thereof), a GPS, (Global Positioning System) receiver components and / or other components In some embodiments, the communications subsystem 1724 can provide wired network connectivity (e.g., Ethernet) in addition to or instead of a wireless interface.

[0313] The communications subsystem 1724 can send and receive data in a variety of forms. For example, in some embodiments, the communications subsystem 1724 may receive incoming communications in the form of structured and / or unstructured data feeds 1726, event streams 1728, event updates 1730, etc. For example, the communications subsystem 1724 may be configured to receive (or send) data feeds 1726 in real time from users of social media networks and / or other communications services, such as web feeds, such as Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third party information sources.

[0314] In particular embodiments, the communications subsystem 1724 may be configured to receive data in the form of a continuous data stream. The continuous data stream may include an event stream 1728 of real-time events and / or event updates 1730, and may in fact be a continuous or infinite stream with no clear end. Applications that generate continuous data may include, for example, sensor data applications, tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and the like.

[0315] The communications subsystem 1724 may also be configured to output structured and / or unstructured data feeds 1726, event streams 1728, event updates 1730, etc. to one or more databases that may be in communication with one or more streaming data source computers connected to the computer system 1700.

[0316] The computer system 1700 may be one of a variety of types, including a palm-sized portable device (e.g., an iPhone® mobile phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head-mounted display), a personal computer, a workstation, a mainframe, a kiosk, a server rack, or other data processing system.

[0317] Due to the ever-changing nature of computers and networks, the description of computer system 1700 shown in FIG. 17 is illustrative only. Many other configurations are possible having more or fewer components than the system shown in FIG. 17. Based on the teachings, a person of ordinary skill in the art will appreciate other ways and / or methods to implement the various embodiments.

[0318] Although specific embodiments of the present disclosure have been described, various modifications, alternatives, alternative configurations, and equivalents are also encompassed within the scope of the present disclosure. Modifications include any suitable combination of the disclosed features. The embodiments of the present disclosure are not limited to operating in a particular data processing environment, but can freely operate in multiple data processing environments. In addition, while the embodiments of the present disclosure have been described using a particular sequence of transactions and steps, it should be apparent to one skilled in the art that the scope of the present disclosure is not limited to the sequence of transactions and steps described. Various features and aspects of the above-described embodiments may be used individually or jointly.

[0319] Furthermore, while embodiments of the present disclosure have been described using a particular combination of hardware and software, it should be appreciated that other combinations of hardware and software are within the scope of the present disclosure. Embodiments of the present disclosure may be implemented solely in hardware, solely in software, or using a combination thereof. The various processes described herein may be implemented on the same processor or on different processors in any combination. Thus, where a component or module is described as being configured to perform a particular operation, such configuration may be achieved, for example, by designing an electronic circuit to perform the operation, by programming a programmable electronic circuit (such as a microprocessor) to perform the operation, or by any combination thereof. Processes may communicate using a variety of techniques, including, but not limited to, conventional techniques for inter-process communication, where different pair processes may use different techniques, and the same pair of processes may use different techniques at different times.

[0320] The specification and drawings should appropriately be regarded as illustrative rather than strict. However, it will be apparent that additions, subtractions, deletions, and other modifications and variations may be made thereto without departing from the broader spirit and scope of the appended claims. Thus, although specific embodiments have been described, they are not intended to be limiting. Various modifications and equivalents are within the scope of the appended claims.

Claims

1. 1. A computer-implemented method comprising: Receiving a first stack trace including a first sequence of stack frames; generating a first trace signature representing the first stack frame sequence based at least in part on the first stack frame sequence, the first trace signature corresponding to a first tuple including a first node representing the first stack frame sequence, the method further comprising: receiving one or more subsequent stack traces; performing a plurality of actions for at least one of the one or more subsequent stack traces, the plurality of actions including: identifying a second stack trace from at least one of the one or more subsequent stack traces, the second stack trace including a second stack frame sequence; generating a second node based at least in part on comparing the first stack frame sequence to the second stack frame sequence; creating a second tuple that includes the second node; generating a second trace signature representative of the second stack frame sequence, the second trace signature corresponding to the second tuple, the plurality of actions further comprising: generating a subsequent trace signature to represent a subsequent stack frame sequence based at least in part on the second trace signature.

2. 2. The computer-implemented method of claim 1, wherein the first tuple comprises a binary tree that includes the first node as a root node.

3. determining whether the subsequent trace signature was generated to represent the subsequent stack frame sequence included in the at least one of the one or more subsequent stack traces; 3. The computer-implemented method of claim 2, wherein based on a determination that the subsequent trace signature has not been generated, the actions are performed for the at least one subsequent stack trace of the one or more subsequent stack traces.

4. determining whether the subsequent trace signature has been generated to represent the subsequent stack frame sequence includes determining whether one or more previously generated nodes represent the subsequent stack frame sequence, the one or more previously generated nodes including the first node; Generating the subsequent trace signature to represent the subsequent stack frame sequence includes: identifying one or more stack frame subsequences in the subsequent stack frame sequence that are not represented by the one or more previously generated nodes; generating one or more additional nodes to represent the one or more stack frame subsequences included in the subsequent stack frame sequence; and coupling at least one of the one or more additional nodes to one or more previously generated binary trees of one or more previously generated tuples, the one or more previously generated binary trees including the binary tree and the previously generated plurality of tuples including the tuple, and generating the subsequent trace signature to represent the subsequent stack frame sequence further comprises: generating one or more additional binary trees, at least one of the one or more binary trees including at least one of the one or more additional nodes, and generating the subsequent trace signature to represent the subsequent stack frame sequence further comprises:

4. The computer-implemented method of claim 3, further comprising generating an additional tuple that includes the one or more additional binary trees, the subsequent trace signature corresponding to the additional tuple.

5. 5. The computer-implemented method of claim 4, wherein each tuple, each node, and each stack frame is uniquely identified by an integer.

6. the binary tree is a first binary tree, Generating a second node based at least in part on comparing the first stack frame sequence to the second stack frame sequence includes: determining that the first stack frame sequence and the second stack frame sequence share matching stack frame subsequences and that non-matching stack frame subsequences in the first stack frame sequence are different from non-matching stack frame subsequences in the second stack frame sequence; creating the second node representing the matching stack frame subsequence; generating a third node representing the non-matching stack frame subsequence in the second stack frame sequence; generating a second binary tree including the second node and a third binary tree including the third node; and generating the second tuple including the second binary tree and the third binary tree, the second node and the third node being siblings, and the second trace signature corresponding to the second tuple.

7. Generating the second trace signature further includes: generating a fourth node representing the matching stack frame subsequence; generating a fifth node representing the non-matching stack frame subsequence included in the first stack frame sequence; and adding the fourth node and the fifth node to the first binary tree as children of the first node.

8. determining that the first stack frame sequence and the second stack frame sequence share a matching stack frame subsequence; and generating the second trace signature based at least in part on the matching stack frame subsequence.

9. Generating the second trace signature includes: determining that a non-matching stack frame sub-sequence in the first stack frame sequence is different from a non-matching stack frame sub-sequence in the second stack frame sequence; generating a first segment signature representing the matching stack frame subsequence; generating a second segment signature representing the non-matching stack frame subsequence in the second stack frame sequence; and generating the second trace signature, the second trace signature including the first segment signature and the second segment signature.

10. Generating the second trace signature includes: generating a third segment signature classifying the non-matching stack frame subsequences included in the first stack frame sequence; and modifying the first trace signature to include the first segment signature and the third segment signature.

11. 10. The computer-implemented method of claim 9, wherein the first trace signature is associated with a first counter that is incremented when a stack trace containing the first stack frame sequence is received, and the second trace signature is associated with a second counter that is incremented when a stack trace containing the second stack frame sequence is received.

12. when a stack trace containing the matching stack frame subsequence is received, the first segment signature is associated with a third counter that is incremented; the second segment signature is associated with a fourth counter that is incremented when a stack trace is received that includes the non-matching stack frame subsequence in the second stack frame sequence; when the first segment signature is generated, the third counter is duplicated from the first counter; 12. The computer-implemented method of claim 11, wherein the fourth counter is replicated from the second counter when the second segment signature is generated.

13. A program for causing a computer to execute the method according to any one of claims 1 to 12.

14. A memory storing the program according to claim 13; and one or more processors for executing the program.

Images