Method, system and storage medium for performance profiling of distributed services

Abstract

The application relates to a performance profiling method, system and storage medium of a distributed service, wherein the distributed service comprises a plurality of work nodes, and each work node is respectively provided with a local profiler; the performance profiling method comprises the following steps: receiving stack trace data collected by each local profiler for the work node, and performing aggregation processing on the stack trace data to obtain aggregated stack data; generating a target flame graph according to the aggregated stack data, and generating a target performance profiling result for the distributed service based on the target flame graph. Through the application, the problem that performance profiling is difficult to adapt to large-scale distributed services is solved.

Description

Technical Field

[0001] This application relates to the field of computer science, and in particular to methods, systems and storage media for performance profiling of distributed services. Background Technology

[0002] Performance profiling refers to analyzing a program's runtime behavior by recording hardware or software metrics, such as function call frequency, execution time, and resource usage, in order to accurately pinpoint performance bottlenecks, memory leaks, or other efficiency issues. Existing performance analysis tools can only perform performance analysis on a single machine and cannot handle diverse, distributed workloads.

[0003] Currently, no effective solution has been proposed to address the problem that performance profiling in related technologies is difficult to adapt to large-scale distributed services. Summary of the Invention

[0004] This application provides a method, system, and storage medium for performance profiling of distributed services, in order to at least solve the problem that performance profiling in related technologies is difficult to adapt to large-scale distributed services.

[0005] In a first aspect, embodiments of this application provide a performance profiling method for a distributed service, applied to a controller; the distributed service includes multiple worker nodes, and each worker node is equipped with a local profiler; the local profiler is connected to the controller; the method includes:

[0006] Receive stack trace data for the working node collected by each of the local profilers, and aggregate the stack trace data to obtain aggregated stack data;

[0007] A target flame graph is generated based on the aggregated stack data, and a target performance profile for the distributed service is generated based on the target flame graph.

[0008] In some embodiments, the method further includes:

[0009] The local profiler acquires first stack trace data at a first sampling frequency, generates a first target flame graph based on the first stack trace data, and generates a first performance profiling result based on the first target flame graph; wherein the first sampling frequency is higher than a preset sampling frequency;

[0010] Based on the first performance profiling result, the first sampling frequency is adjusted to obtain a second sampling frequency; the second stack trace data of the local profiler at the second sampling frequency is obtained, and a second target flame graph is generated based on the second stack trace data; and a second performance profiling result is generated based on the second target flame graph.

[0011] In some embodiments, after generating the second performance profiling result, the method further includes:

[0012] Similarity is calculated based on the first performance analysis result and the second performance analysis result to obtain the similarity result;

[0013] If the similarity result is less than a preset similarity threshold, the first performance profiling result is used as the target performance profiling result.

[0014] In some embodiments, receiving stack trace data for the working node collected by each of the local profilers includes:

[0015] Receive initial flame graphs sent by each of the local profilers; wherein the initial flame graphs are generated by the local profilers based on the stack trace data;

[0016] The initial flame graph is parsed to obtain the stack trace data.

[0017] In some embodiments, the local profiler further includes an agent and a collection kernel; the initial flame graph is generated by the collection kernel collecting the stack trace data and the agent processing the stack trace data.

[0018] In some embodiments, the number of acquisition kernels is multiple; receiving the initial flame map sent by each of the local profilers includes:

[0019] The operation information of the distributed service is detected, and based on the operation information, the corresponding target kernel is determined from each of the collected kernels; wherein, the agent processes the stack trace data collected from the target kernel and generates the initial flame graph.

[0020] Thirdly, this application provides a performance profiling method for distributed services, applied to a local profiler; wherein the distributed service includes multiple worker nodes, and each worker node is equipped with a local profiler; the method includes:

[0021] Collect stack trace data for the aforementioned working node;

[0022] The stack trace data is sent to the controller; the controller aggregates the stack trace data to obtain aggregated stack data; a target flame graph is generated based on the aggregated stack data, and a target performance profile result for the distributed service is generated based on the target flame graph.

[0023] In some embodiments, the acquisition of stack trace data on the worker node includes:

[0024] The distributed service is sampled using multiple threads, and the threads are pruned based on the number of thread samples to obtain the target thread.

[0025] Obtain the stack trace data obtained by sampling the target thread.

[0026] Thirdly, embodiments of this application provide a performance profiling system for a distributed service, wherein the distributed service includes multiple worker nodes; the performance profiling system includes:

[0027] A local profiler, deployed on the corresponding worker node, is used to collect stack trace data on the worker node;

[0028] The controller is used to receive the stack trace data sent by each of the local profilers, and to aggregate the stack trace data to obtain aggregated stack data.

[0029] The controller is also configured to generate a target flame graph based on the aggregated stack data, and generate a target performance profile for the distributed service based on the target flame graph.

[0030] In some embodiments, the performance profiling system further includes:

[0031] A monitoring server is used to receive hotspot function metrics sent by the local profiler; wherein the hotspot function metrics are calculated and generated by the local profiler based on the stack trace data;

[0032] The monitoring server is also used to generate tracking and storage data for each hot function in the distributed service based on the hot function metric.

[0033] Fourthly, embodiments of this application provide a storage medium storing a computer program, wherein the computer program is configured to execute the performance profiling method for distributed services as described in the first or second aspect above.

[0034] Compared to related technologies, the performance profiling method, system, and storage medium for distributed services provided in this application embodiment address the issue of distributed services comprising multiple worker nodes, each worker node having a local profiler deployed. The performance profiling method includes: receiving stack trace data collected by each local profiler for the worker nodes, aggregating the stack trace data to obtain aggregated stack data; generating a target flame graph based on the aggregated stack data; and generating a target performance profiling result for the distributed service based on the target flame graph. This method enables low-cost and non-intrusive hotspot function profiling using profiling tools. Furthermore, unlike traditional tools that generate flame graphs for only a single instance, this invention effectively and accurately locates the true hotspot functions by aggregating flame graphs across all instances, going beyond just library function profiling. This achieves highly versatile, accurate, and low-cost performance profiling in containerized and distributed environments in a non-intrusive manner, effectively solving the problem of performance profiling being difficult to adapt to large-scale distributed services.

[0035] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0036] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0037] Figure 1 This is a hardware structure block diagram of a terminal for a distributed service performance profiling method according to an embodiment of this application;

[0038] Figure 2 This is a flowchart of a distributed service performance profiling method according to an embodiment of this application;

[0039] Figure 3A This is a schematic diagram of an initial flame diagram according to an embodiment of this application;

[0040] Figure 3B This is a schematic diagram of another initial flame diagram according to an embodiment of this application;

[0041] Figure 3C This is a schematic diagram of a target flame diagram according to an embodiment of this application;

[0042] Figure 4 This is a schematic diagram of a hotspot function change according to an embodiment of this application;

[0043] Figure 5 This is a schematic diagram of the evaluation result of a frequency dynamic adjustment algorithm according to an embodiment of this application;

[0044] Figure 6 This is a flowchart of an automatic selection and acquisition kernel method according to an embodiment of this application;

[0045] Figure 7 This is a flowchart of another method for performance profiling of a distributed service according to an embodiment of this application;

[0046] Figure 8 This is a schematic diagram illustrating the number of thread samples according to an embodiment of this application;

[0047] Figure 9 This is a schematic diagram illustrating the aggregation cost and error of a hotspot function according to an embodiment of this application;

[0048] Figure 10 This is a structural block diagram of a distributed service performance profiling system according to an embodiment of this application;

[0049] Figure 11 This is a schematic diagram of the structure of a performance profiling system for a distributed service according to an embodiment of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0051] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0052] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application means two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The terms “first,” “second,” “third,” etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0053] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. Taking running on a terminal as an example, Figure 1 This is a hardware structure block diagram of a terminal for a distributed service performance profiling method according to an embodiment of this application. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. Optionally, the terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0054] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the distributed service performance profiling method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thus implementing the aforementioned method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0055] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0056] Cloud products, such as large-scale online services like Google Search and Amazon Online Shopping, are typically deployed in data centers to ensure continuous availability. To effectively handle massive request traffic, these services are divided into numerous instances or replicas, ranging from hundreds to tens of thousands. For example, according to data released by Alibaba, an eight-day statistical analysis conducted in 2018 on a cluster of 4,000 machines showed that the number of instances for a single service ranged from hundreds to tens of thousands, with a peak of 99,583.

[0057] Given the sheer scale of these services, even small performance improvements can lead to significant cost savings. Therefore, identifying and optimizing hotspot functions within distributed services in the data center helps improve resource utilization. However, large-scale distributed services differ significantly from traditional single applications in terms of hardware and software architecture and deployment methods. First, at the hardware level, due to their massive scale, service instances are distributed across different worker nodes in the data center. Second, at the software level, services are developed using different types of programming languages, including compiled and interpreted ones. Finally, with the emergence and development of container technology, it has gradually replaced virtual machines as the primary method for deploying services in the data center. These characteristics indicate that to discover and optimize the performance bottlenecks of numerous services within the data center, a non-intrusive, high-performance profiler is needed that supports multiple languages ​​and is adaptable to containerization.

[0058] Performance profiling analyzes a program's runtime behavior by recording hardware or software metrics to identify performance bottlenecks. Profilers typically use code instrumentation for greater accuracy or sampling methods to improve efficiency. To better understand this application, the relevant concepts are explained below:

[0059] Code instrumentation refers to inserting additional code into a target program. When this modified program is executed, it generates performance data for report generation. This method ensures accuracy by collecting metrics only when relevant functions are executed, thus avoiding issues such as the shadow effect and skid. However, this technique typically requires restarting or recompiling the program, resulting in considerable overhead. Furthermore, the additional instructions in the code profiling can slow down service response times and consume more resources. These drawbacks are particularly pronounced in efficiency-driven data center environments.

[0060] Sampling-based profiling, also known as stack tracing, utilizes a hardware performance monitoring unit (PMU) to periodically capture function call stack snapshots of the target program. After accumulating sufficient samples, the frequency of each function's appearance in the stack trace indicates its resource consumption during profiling. Unlike code instrumentation, sampling only interrupts the program upon receiving certain signals, thus minimizing the introduced Central Processing Unit (CPU) overhead. Furthermore, since it does not require any modifications to the profiled program, this method is considered non-intrusive. While this method may be slightly inaccurate (because the functions captured in the sampling may not always be the actually executed functions), its high efficiency makes it the preferred choice for commonly used profiling tools such as Perf, HPCtoolkit, and Intel VTune. Since modifying source code or initiating recompilation and service restarts is impractical in production environments, this application achieves a trade-off between accuracy and efficiency through optimized sampling methods based on non-intrusive sampling profiling.

[0061] It's also worth noting that current performance profiling tools can be broadly categorized into single-machine profilers and multi-machine profilers. However, both types have limitations when applied to large-scale distributed services. First, single-machine profilers struggle to handle the large scale, distributed nature of services, and diverse workloads. Second, to overcome the shortcomings of single-machine profilers, researchers have integrated single-machine collectors, data links, databases, control panels, and other modules into multi-machine performance profiling systems to simultaneously profile numerous services and manage and analyze metrics in a unified manner. However, current multi-machine profilers are mostly closed-source and proprietary, and may not be directly attachable to running processes. Furthermore, while users can create their own multi-machine profilers through customized monitoring systems, directly deploying a single-machine profiler incurs significant performance overhead, which is generally unacceptable for distributed microservices composed of numerous instances.

[0062] Based on this, this embodiment provides a performance profiling method for a distributed service, applied to a controller. The distributed service includes multiple worker nodes, and each worker node is equipped with a local profiler; the local profiler is connected to the controller. Figure 2 This is a flowchart of a performance profiling method for a distributed service according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0063] Step S210: Receive stack trace data collected by each local profiler for the working node, and aggregate the stack trace data to obtain aggregated stack data.

[0064] A local profiler, deployed on a corresponding worker node, is used to collect stack trace data of service instances on that worker node. A local profiler is a profiling tool that runs on a local environment. One local profiler is deployed on each worker node. Specifically, this local profiler responds to profiling commands sent by the controller to the target worker node and is responsible for collecting stack trace data of the microservices on that worker node. This stack trace data typically contains key information such as function call sequences and execution times, forming the basis of performance analysis.

[0065] In this embodiment, the distributed service performance profiling method is applied to a controller, which is connected to an ontology profiler. The controller acts as a central hub, responsible for transmitting user commands to various components of the system. After system startup, the controller enters a busy-wait state, ready to accept commands via the command-line interface (CLI). When a user initiates profiling, the controller parses the configuration, establishes communication with local profilers to receive stack trace data collected by these local profilers, and assigns analysis tasks according to the user configuration. Furthermore, users can flexibly manage existing analysis tasks through the controller.

[0066] Step S220: Generate a target flame graph based on the aggregated stack data, and generate a target performance profile result for the service instance based on the target flame graph.

[0067] It should be noted that focusing solely on analyzing resource-intensive functions may lead to insufficient understanding of performance bottlenecks. To address this limitation, this application employs a combination of flame graphs and numerical metrics to effectively and accurately pinpoint the true hotspots. The flame graph graphically displays the performance data collected by the performance analyzer, enabling statistical analysis and facilitating the identification of performance hotspots.

[0068] Performance profiling tools typically rank functions based on resource consumption, but this can be misleading. High resource utilization may be attributed to low-level library functions, often used for basic tasks like resource locking or random number generation, and called by higher-level business logic functions. High resource utilization in these library functions doesn't necessarily mean they are primary targets for optimization. Instead, business logic functions often require more careful scrutiny and optimization, as they may contain logical errors and over-call the aforementioned low-level functions. An effective performance profiler should analyze function call relationships to reveal which user-defined functions frequently trigger resource-intensive library functions. This invention generates flame graphs for each instance, such as the target service, to visually represent function call relationships.

[0069] For a single instance of a service, a flame graph can be generated by collecting call stack data. However, a large-scale application (such as buy2) often has multiple instances, and these instances may have different performance characteristics due to differences in accessed resources and frequencies. To address this issue, this application aggregates the sampling results of all sampled instances of the target service to generate a comprehensive flame graph covering the entire service. Specifically, the controller merges and aggregates the stack trace data sent by each local profiler; the purpose of aggregation is to summarize and statistically analyze information such as function call relationships and execution times from multiple stack trace data to obtain aggregated stack data. This merged aggregated stack data represents the overall behavior of all instances. Finally, the controller uses the aggregated stack trace data as input to generate an overall flame graph using a flame graph generation tool; the flame graph generation tool processes and analyzes the input data, drawing the flame graph based on information such as function call relationships and execution times. This generated overall flame graph can intuitively display the performance bottlenecks and hot functions of the entire distributed system, helping developers quickly locate and optimize problems.

[0070] On the other hand, the aforementioned local profiler and controller can also transmit stack trace data in the form of flame graphs. Specifically, the local profiler generates an initial flame graph based on the stack trace data; the controller receives the initial flame graphs sent by each local profiler and parses them to obtain the stack trace data. Each local profiler generates an initial flame graph based on its collected stack trace data and transmits it to the controller. The controller parses each initial flame graph and extracts the stack trace data. This data may include key information such as the order of function calls, the number of calls, and the call duration. By parsing the obtained stack trace data, the controller can further analyze the program's performance, providing strong support for performance tuning and troubleshooting. Through this method, the local profiler processes the raw stack trace data and generates the initial flame graph, enabling data transmission in the form of a flame graph. This minimizes data transmission overhead between the local profiler and the controller, while effectively reducing the computational burden on the controller node.

[0071] Please see Figure 3A and Figure 3B The images show the initial flame graphs for Service Instance 1 and Service Instance 2, generated by stack traces and computation on their respective local profilers. The initial flame graph for Service Instance 1 includes functions a(), b(), c(), d(), e(), and f(), with the call time percentage or frequency of each function displayed graphically. The initial flame graph for Service Instance 2 includes functions a(), b(), c(), d(), and g(). Please refer to [link to relevant documentation]. Figure 3CAfter the controller tracks the aggregated stack data, it generates a target flame graph based on the merged aggregated stack data. This flame graph includes the percentage of call time for the functions a(), b(), c(), d(), e(), f(), g(), and h() mentioned above.

[0072] In existing technologies, single-machine performance profiling tools cannot handle diverse and distributed workloads; some profiling tools operate in an intrusive manner, requiring modification of the target service's source code or binary code, resulting in inconvenience and increased costs; furthermore, multi-machine profiling tools are mostly closed-source and proprietary, limiting their use by external users. Additionally, existing tools typically sort hot functions based on resource consumption, which may lead to insufficient understanding of performance bottlenecks, and even when using flame graphs, they often only support a single instance of the target service.

[0073] This application provides a distributed profiling framework through the aforementioned distributed service performance profiling method. This framework can utilize profiling tools to conduct low-cost and non-intrusive hotspot function profiling. Furthermore, unlike traditional tools that only generate flame graphs for a single instance, this invention effectively and accurately locates the real hotspot functions by aggregating flame graphs across all instances, rather than being limited to the profiling of library functions. Thus, it achieves highly versatile, highly accurate, and low-cost performance profiling in a non-intrusive manner in containerized and distributed environments, effectively solving the problem that performance profiling is difficult to adapt to large-scale distributed services.

[0074] In some embodiments, to maintain accuracy while keeping profiling costs low, a dynamic frequency adjustment scheme is provided, which adaptively adjusts the sampling frequency based on previous analysis results. This scheme is based on two important observations: the relationship between sampling frequency and accuracy, and the periodic variation of the hotspot function.

[0075] To better understand this application, the implementation principle of the frequency dynamic adjustment method is explained here. First, it is generally believed that performance profiling with a higher sampling frequency usually yields more accurate results (because more samples are obtained), but this also incurs higher costs. For a single program, a user might set a higher sampling frequency, sacrificing some performance for higher analysis accuracy. However, when multiple instances are involved, this leads to a significant increase in cost. Therefore, there is a trade-off between profiling cost and accuracy.

[0076] Secondly, typical workloads exhibit cyclical patterns in their hot functions, as seen in the widely adopted Java benchmark Specjbb2015. See also... Figure 4The changes in the top 5 hottest functions were monitored during the benchmark test run. Specifically, the time cost proportions of the top 5 hottest functions were treated as a probability distribution, and the difference between the current distribution and the previous distribution was calculated every 10 seconds using Jensen-Shannon Divergence (JS divergence); JS divergence is a similarity measure between probability distributions. The results are as follows: Figure 4 As shown, two key trends are revealed: First, JS divergence remains high at the beginning and end of the application lifecycle. This indicates that hotspot functions change rapidly at the start and end. Second, when the application is running stably, hotspot functions exhibit a cyclical pattern. That is, repeating peaks in JS divergence can be observed after similar intervals, indicating that hotspots have changed, before falling back to lower levels before the next cycle begins.

[0077] Based on the above analysis, the dynamic sampling frequency adjustment scheme can be summarized as follows: using high-frequency sampling results as a baseline, and using this to correct low-frequency sampling results. Furthermore, the performance profiling method for the aforementioned distributed system may also include the following steps:

[0078] The system acquires first stack trace data of the local profiler at a first sampling frequency, generates a target flame graph based on the first stack trace data, and generates a first performance profiling result based on the first target flame graph. The first sampling frequency is higher than a preset sampling frequency. Based on the first performance profiling result, the first sampling frequency is adjusted to obtain a second sampling frequency. The system acquires second stack trace data of the local profiler at the second sampling frequency, generates a target flame graph based on the second stack trace data, and generates a second performance profiling result based on the second target flame graph.

[0079] Furthermore, in some embodiments, after generating the second performance profiling result, the performance profiling method further includes:

[0080] The similarity is calculated based on the first performance profile result and the second performance profile result to obtain the similarity result; the controller is also used to take the first performance profile result as the target performance profile result when the detected similarity result is less than the preset similarity threshold.

[0081] Specifically, while maintaining a consistently high sampling frequency ensures accuracy, it is also more costly. However, if the sampling rate remains consistently low, the accuracy of the results cannot be guaranteed. Therefore, one can first use a high sampling rate to obtain some relatively accurate results, and then try using a low sampling rate. If the difference between the low-frequency and high-frequency sampling results is not significant, it proves that the hotspot function has not changed significantly, so the results saved at the high-frequency time can be used to replace the current sampling results. Otherwise, it means that the application itself has changed, and the results obtained previously at a high frequency are no longer applicable. In this case, the sampling rate should be increased to obtain baseline results again.

[0082] Table 1 shows one of the pseudocodes for the frequency dynamic adjustment algorithm, as shown in Table 1:

[0083] Table 1 Frequency Dynamic Adjustment Algorithm

[0084]

[0085] Using the pseudocode above as an example, to evaluate the similarity of the sampling results, we utilize the JS divergence of the hotspot function distribution mentioned above. Users can set a threshold θ to adjust the strictness of the similarity judgment. When the D of two distributions... JS When the value is greater than θ, they are considered significantly different. Since the number of functions in the sampling results is very large, to control computational costs, θ is usually calculated using the top 5 hot functions instead of all functions, as these represent the main performance bottlenecks of the target program.

[0086] The evaluation results of the above algorithm are as follows: Figure 5 As shown, several findings can be drawn. First, as the sampling frequency increases, the sampling cost generally rises, but the sampling error decreases accordingly, confirming the trend mentioned above. Second, under the dynamic frequency adjustment strategy, the CPU's performance consumption is close to that at 5000Hz, but reaches a mean absolute error (MAE) far lower than at 5000Hz and comparable to that at 10000Hz. Notably, it only consumes 12.4% of the sampling cost at 10000Hz.

[0087] Through the above embodiments, a dynamic frequency adjustment scheme is proposed. This scheme dynamically adjusts the acquisition stack tracking frequency according to the analysis results, allowing users to enjoy the high accuracy of high sampling rate while maintaining a low sampling rate at low cost most of the time, thereby ensuring the accuracy of the analysis while maintaining a low cost.

[0088] In some embodiments, the local profiler described above also includes an agent and a collection kernel. The initial flame graph described above is generated by the collection kernel collecting stack trace data, and the agent processing the stack trace data.

[0089] Specifically, the aforementioned acquisition kernel is used to collect stack trace data. In this embodiment, stack tracing is performed using dedicated acquisition kernels deployed on worker nodes without modifying or interrupting the target microservice. These kernels utilize the hardware's Performance Monitoring Unit (PMU) to collect runtime stack traces triggered by specific events. Each stack trace captures the sequence of function calls up to the sampling point. For example, in a trace like Func1→Func2→Func3, Func3 is the executing function, called by function Func2, which in turn is called by function Func1. Identical stack traces are counted to reflect the frequency of target performance events.

[0090] Additionally, it should be noted that in the embodiments of this application, the selected acquisition kernel should meet some key criteria, such as: considering the difficulty of restarting large online services, the acquisition kernel should not require program modification or restart; in order to reduce profiling costs, the acquisition kernel should be based on sampling rather than code insertion; and it needs to provide support for the characteristics of the target language.

[0091] The aforementioned agent processes stack trace data and generates an initial flame graph. The agent is a software component or service that acts as an intermediary between the three core components of a performance profiling system: the controller, the local profiler, and the monitoring server. First, the agent automatically activates the appropriate kernels based on the controller's instructions and dynamically adjusts the frequency. Then, the agent processes the raw stack traces of these kernels and transforms the aggregated stack traces into metrics recognizable by the Prometheus server. Furthermore, the agent periodically generates flame graphs for global aggregation.

[0092] In some embodiments, there are multiple acquisition kernels; receiving the initial flame map sent by each of the local profilers includes the following steps:

[0093] The system detects the runtime information of distributed services and, based on this information, determines the corresponding target kernel from among the collected kernels. The agent processes the stack trace data collected from the target kernel and generates an initial flame graph.

[0094] Large-scale distributed services typically consist of multiple microservices, each potentially developed in a different programming language. However, current performance profiling tools struggle to handle this characteristic. Different programming languages ​​have different built-in performance profilers, thus existing multi-machine performance profilers require users to have information about the target service's language to select the appropriate profiler. In this embodiment, programming languages ​​are categorized into interpreted and compiled languages. For compiled languages, since the binary code executes directly on the CPU, stack traces can be obtained by directly reading the target service's stack space. For interpreted languages, since interpreters typically have language-specific memory models, the application programming interface (API) provided by the interpreter needs to be used to reconstruct the memory space to obtain stack traces. Essentially, this is a process of mapping from a programming language to a profiling kernel. For example, in practical applications, the async-profiler kernel might be chosen for Java, the py-spy kernel for Python, and the Perf kernel for compiled languages.

[0095] The following describes the process of automatically selecting the acquisition kernel. Please refer to [link / reference]. Figure 6 The process includes the following steps:

[0096] Step S601: Identify the target microservice and determine whether the target microservice is running through an interpreter.

[0097] Step S602: If the judgment result of step S601 is yes, it indicates that the target microservice is developed using an interpreted language; at this time, an appropriate collection kernel designed specifically for the language will be selected to capture the stack trace.

[0098] In step S603, if the judgment result of step S602 is negative, then in the absence of an interpreter detected, it can be assumed that the target microservice is developed using a compiled language; in this case, a system-level profiler such as Perf will be used by default to obtain the stack trace.

[0099] In existing technologies, large-scale distributed services are typically composed of multiple microservices, each of which may be developed in different programming languages, making it difficult for users to specify the target language in advance. However, this application, through the above embodiments, automatically selects a suitable data collection kernel for services developed in different programming languages, thereby enabling seamless analysis of services written in multiple languages. This achieves a language-independent adaptation mechanism to support multilingual microservices, reducing the prerequisite knowledge users need about the target service and providing a language-barrier-free experience.

[0100] In this embodiment, a performance profiling method for distributed services is also provided, applied to a local profiler; wherein the distributed service includes multiple worker nodes, and each worker node deploys a local profiler; please refer to [link to relevant documentation]. Figure 7 The method includes the following steps:

[0101] Step S710: Collect stack trace data on the worker node;

[0102] Step S720: The stack trace data is sent to the controller; wherein the controller aggregates the stack trace data to obtain aggregated stack data; a target flame graph is generated based on the aggregated stack data, and a target performance profile result for the distributed service is generated based on the target flame graph.

[0103] In some embodiments, the above-described collection of stack trace data on the worker node may further include the following steps:

[0104] Multiple threads are used to perform stack trace sampling on the service instance, and the threads are pruned based on the number of thread samples to obtain the target thread; the stack trace data obtained from the target thread is then acquired.

[0105] Since microservices are typically multithreaded, it's necessary to aggregate stack trace data from all child threads of the target service to fully reflect its hot functions. However, the aggregation process doesn't need to utilize all child threads. This is because when a process runs, it generates numerous daemon threads to serve user threads. For example, in Java, the Java Virtual Machine (JVM) starts daemon threads and user threads in the background. These daemon threads handle tasks such as garbage collection and resource monitoring. Compared to user threads, daemon threads consume fewer resources but are far more numerous. Although the sample size of daemon threads is small and doesn't significantly impact hot functions, the large number of daemon threads significantly increases the overall aggregation cost.

[0106] Taking the typical Java workload Specjbb2015 as an example, Specjbb2015 is a benchmark used to evaluate the performance of multi-threaded Java applications based on Java business application servers on the Java platform. Figure 8This graph shows the number of samples for each child thread during execution. Threads are arranged in descending order of sample count, with the horizontal axis representing the thread's ranking by sample count. The left vertical axis represents the sample count for each thread, while the right vertical axis shows the percentage of the total sample count from the cumulative sample count of the top n threads. The graph shows that only about 20% of the threads, specifically the first 368 threads, generated a large number of samples. Most of the other threads are inactive, running as daemon threads. This pattern is also reflected in the proportional distribution: out of a total of 1840 threads, the first 444 threads generated 90% of the samples, corresponding to the P90 pruning standard. Meanwhile, the p99 threads are the first 838 threads, meaning the remaining 1002 threads contribute only 1% of the total sample count, corresponding to the P99 pruning standard.

[0107] In this embodiment, after the agent obtains the stack traces of the sampled threads, it sorts them according to the number of samples for each thread. Given that the number of threads is far less than the number of stack traces, this sorting process incurs almost no cost. The agent then retains the threads with the most samples, such as the top 99%, and discards the remaining samples, primarily from inactive daemon threads. This approach allows for focused analysis of the most active threads, improving the efficiency of the analysis process while avoiding being overwhelmed by relatively unimportant data. Experiments show that the error introduced by this method is negligible, yet it saves considerable costs.

[0108] Please see Figure 9 For the top 5, 10, 20, and 50 hot functions, the percentage time saved by the system under the P70, P80, P90, P95, and P99 pruning criteria was more significant than the error it introduced. In particular, when using the p99 pruning criterion, only a 0.58% mean absolute percentage error (MAPE) was introduced for the top 50 hot functions, while aggregation time was reduced by 6.8%. Therefore, the p99 criterion was adopted as the system's default pruning threshold. Specifically, the agent in the local profiler compares the number of detected thread samples with the default pruning threshold and discards thread samples with a number less than the default pruning threshold, retaining only target threads with a number exceeding the default pruning threshold.

[0109] In existing technologies, directly deploying a single-machine profiler in a multi-machine system leads to significant performance overhead, which is generally unacceptable for distributed microservices composed of a large number of instances. To address this issue, this application, through the above embodiments, considers that stack traces from a large number of daemon threads may not provide useful information for identifying hot functions. Therefore, during the aggregation of multi-threaded stack trace data, stack traces originating from these daemon threads are pruned to reduce the cost of computing hot functions, effectively improving the efficiency of the aggregation process while having almost no impact on the aggregation results.

[0110] This embodiment also provides a performance profiling system for distributed services; please refer to [link / reference]. Figure 10 The distributed service comprises multiple worker nodes; this performance profiling system includes:

[0111] Local profiler 20, deployed on the corresponding worker node, is used to collect stack trace data on the worker node;

[0112] The controller 10 is used to receive stack trace data sent by each local profiler and to aggregate the stack trace data to obtain aggregated stack data.

[0113] The controller 10 is also used to generate a target flame graph based on the aggregated stack data, and to generate target performance profiling results for the distributed service based on the target flame graph.

[0114] The above embodiments provide a performance profiling system architecture adapted to large-scale distributed microservices, which includes deploying a local profiler 20 on each worker node and establishing a control node based on the controller 10 to perform control and data processing in a unified manner, thereby realizing a performance profiling method that aggregates flame graphs across all instances, which can effectively and accurately locate the real hot functions.

[0115] In some of these embodiments, please refer to Figure 10 The aforementioned performance profiling system further includes a monitoring server 30, used to receive hotspot function metrics sent by the local profiler 20; these hotspot function metrics are calculated and generated by the local profiler based on stack trace data. In this embodiment, the local profiler 20 is responsible for extracting and processing stack traces from the corresponding target microservice, and then providing metrics for exposing hotspot functions to the monitoring server 30. The monitoring server 30 is also used to generate trace storage data for each hotspot function in the service instance based on the hotspot function metrics.

[0116] Specifically, the monitoring server 30 can be a server device like Prometheus, an open-source service monitoring system. Specifically, the monitoring server 30 pulls hotspot function metrics from the monitoring ports of each worker node, obtained by the local profiler, and provides data storage, analysis, and visualization services. These hotspot function metrics represent the performance consumption of hotspot functions. They are calculated by the local profiler 20 by aggregating raw stack trace data and formatted according to the monitoring server 30's standards. The final generated function hotspot metrics include a metric name and a set of descriptive labels. The monitoring server 30's strategy for handling hotspot functions is as follows: if the monitoring server 30 detects a new hotspot function, it will include it in its tracking scope; this means that the monitoring server 30 will begin collecting and storing the function's performance metrics for subsequent analysis and querying. If a function detected by the monitoring server 30 already has a track record, the existing track record for that function will be updated when new performance analysis data is received; in this way, the server can maintain real-time tracking of hotspot function performance changes. Therefore, the monitoring server 30 provides a complete data management suite for the tracked hot functions, including persistent storage, querying, analysis and visualization capabilities, enabling users to track resource consumption trends or export data for more in-depth evaluation.

[0117] The above embodiments provide a performance profiling system that also includes a monitoring server 30. By establishing a central node for unified control and data processing, the system's workload is primarily concentrated on the server node, ensuring that the local profiler 21 consumes less CPU time and minimizing interference with the target microservice. Furthermore, on the monitoring server 30 node, users can query and visualize these results for further analysis, providing a reference for code optimization and service status monitoring.

[0118] The embodiments of this application will be described below in conjunction with specific application scenarios. Figure 11 This is a schematic diagram of the structure of a distributed service performance profiling system according to an embodiment of this application, such as... Figure 11As shown, the system consists of three key components: a LocalProfiler, a Controller, and a Prometheus Server. The architectural design principle of this invention is to adapt to the distributed architecture and diverse workloads of large-scale microservices. This includes deploying a LocalProfiler with multiple acquisition kernels on each worker node, and establishing a Controller node and a Prometheus Server for unified control and data processing. This design has two main advantages. First, the system's workload is primarily concentrated on the Prometheus Server node; by ensuring that the LocalProfiler consumes minimal CPU time, this design minimizes interference with the target microservice. Second, the loose coupling of functions across different nodes enhances the system's maintainability and scalability.

[0119] The following workflow describes the process of initiating the profiling task in this system:

[0120] Users create configuration files specifying target nodes, target services, metrics to sample, sampling frequency, etc. The controller reads the configuration file, establishes a connection with the target node, and starts the profiling task while providing the necessary parameters to the local profiler. The local profiler continuously samples stack traces; it aggregates the raw stack traces obtained during profiling and calculates certain metrics that cannot be directly sampled. It then transforms the profiling results into formatted Prometheus metrics and uploads them to an available port. The Prometheus server monitors the port exposed by the local profiler, pulls the metrics, and stores them in its built-in time-series database. Users can view the visualized data in the Prometheus dashboard or export data from the database for further analysis.

[0121] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0122] Furthermore, in conjunction with the performance profiling methods for distributed services in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the performance profiling methods for distributed services in the above embodiments.

[0123] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0124] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0125] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for performance profiling of distributed services, characterized in that, Applied to the controller; the distributed service includes multiple worker nodes, and each worker node is equipped with a local profiler. The local profiler is connected to the controller; the method includes: Receive stack trace data collected by each of the local profilers for the working node, and aggregate the stack trace data to obtain aggregated stack data; A target flame graph is generated based on the aggregated stack data, and a target performance profile for the distributed service is generated based on the target flame graph. The method further includes: The local profiler acquires first stack trace data at a first sampling frequency, generates a first target flame graph based on the first stack trace data, and generates a first performance profiling result based on the first target flame graph; wherein the first sampling frequency is higher than a preset sampling frequency; Based on the first performance profiling result, the first sampling frequency is reduced and adjusted to obtain a second sampling frequency; the second stack trace data of the local profiler at the second sampling frequency is obtained, and a second target flame diagram is generated based on the second stack trace data; and a second performance profiling result is generated based on the second target flame diagram. Similarity is calculated based on the first performance analysis result and the second performance analysis result to obtain the similarity result; If the similarity result is less than a preset similarity threshold, the first performance profiling result is used as the target performance profiling result.

2. The performance profiling method according to claim 1, characterized in that, The receiving of stack trace data for the working node collected by each of the local profilers includes: Receive initial flame graphs sent by each of the local profilers; wherein the initial flame graphs are generated by the local profilers based on the stack trace data; The initial flame graph is parsed to obtain the stack trace data.

3. The performance profiling method according to claim 2, characterized in that, The local profiler also includes an agent and a collection kernel; the initial flame graph is generated by the collection kernel collecting the stack trace data and the agent processing the stack trace data.

4. The performance profiling method according to claim 3, characterized in that, The number of acquisition kernels is multiple; receiving the initial flame map sent by each of the local profilers includes: The operation information of the distributed service is detected, and based on the operation information, the corresponding target kernel is determined from each of the collected kernels; wherein, the agent processes the stack trace data collected from the target kernel and generates the initial flame graph.

5. A method for performance profiling of distributed services, characterized in that, The method is applied to a local profiler; wherein the distributed service includes multiple worker nodes, and each worker node is equipped with a local profiler; the method includes: Collect stack trace data for the aforementioned working node; The stack trace data is sent to the controller; the controller aggregates the stack trace data to obtain aggregated stack data; a target flame graph is generated based on the aggregated stack data, and a target performance profile result for the distributed service is generated based on the target flame graph. The controller acquires the first stack trace data of the local profiler at a first sampling frequency, generates a first target flame graph based on the first stack trace data, and generates a first performance profiling result based on the first target flame graph; wherein, the first sampling frequency is higher than a preset sampling frequency; Based on the first performance profiling result, the controller adjusts the first sampling frequency by reducing it to obtain a second sampling frequency; it acquires the second stack trace data of the local profiler at the second sampling frequency, and generates a second target flame graph based on the second stack trace data, and generates a second performance profiling result based on the second target flame graph. Similarity is calculated based on the first performance analysis result and the second performance analysis result to obtain the similarity result; If the similarity result is less than a preset similarity threshold, the first performance profiling result is used as the target performance profiling result.

6. The performance profiling method according to claim 5, characterized in that, The collection of stack trace data on the working node includes: The distributed service is sampled using multiple threads, and the threads are pruned based on the number of thread samples to obtain the target thread. Obtain the stack trace data obtained by sampling the target thread.

7. A performance profiling system for distributed services, characterized in that, The distributed service includes multiple worker nodes; the performance profiling system includes: A local profiler, deployed on the corresponding worker node, is used to collect stack trace data on the worker node; The controller is used to receive the stack trace data sent by each of the local profilers, and to aggregate the stack trace data to obtain aggregated stack data. The controller is also configured to generate a target flame graph based on the aggregated stack data, and generate a target performance profile for the distributed service based on the target flame graph; The controller is further configured to acquire first stack trace data of the local profiler at a first sampling frequency, and generate a first target flame graph based on the first stack trace data, and generate a first performance profiling result based on the first target flame graph; wherein, the first sampling frequency is higher than a preset sampling frequency; The controller is further configured to: reduce the first sampling frequency based on the first performance profiling result to obtain a second sampling frequency; acquire second stack trace data of the local profiler at the second sampling frequency, and generate a second target flame graph based on the second stack trace data; generate a second performance profiling result based on the second target flame graph; perform similarity calculation based on the first performance profiling result and the second performance profiling result to obtain a similarity result; and if the similarity result is detected to be less than a preset similarity threshold, use the first performance profiling result as the target performance profiling result.

8. The performance profiling system according to claim 7, characterized in that, The performance profiling system also includes: A monitoring server is used to receive hotspot function metrics sent by the local profiler; wherein the hotspot function metrics are calculated and generated by the local profiler based on the stack trace data; The monitoring server is also used to generate tracking and storage data for each hot function in the distributed service based on the hot function metric.

9. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the performance profiling method for distributed services according to any one of claims 1-6 at runtime.

Images