Distributed service performance analysis method and system and storage medium

Abstract

The invention relates to a distributed service performance analysis method and system and a storage medium, and the distributed service comprises a plurality of working nodes, and each working node is provided with a local analyzer. The performance analysis method comprises the following steps: receiving stack tracking data aiming at a working node and collected by each local analyzer, and carrying out aggregation processing on the stack tracking data to obtain aggregated stack data; and generating a target flame map according to the aggregated stack data, and generating a target performance analysis result for the distributed service based on the target flame map. According to the invention, the problem that performance analysis is difficult to adapt to large-scale distributed services is solved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a performance analysis method, system and storage medium for distributed services. Background Art

[0002] Performance profiling refers to analyzing the runtime behavior of a program by recording hardware or software indicators, such as function call frequency, execution time, and resource usage, so as to accurately locate performance bottlenecks, memory leaks, or other efficiency issues. Existing performance analysis tools can only perform performance analysis on a single machine and cannot handle diverse, distributed workloads.

[0003] Currently, no effective solution has been proposed to address the problem that performance analysis in related technologies is difficult to adapt to large-scale distributed services. Summary of the invention

[0004] The embodiments of the present application provide a performance analysis method, system and storage medium for distributed services, so as to at least solve the problem in the related art that performance analysis is difficult to adapt to large-scale distributed services.

[0005] In a first aspect, an embodiment of the present application provides a performance analysis method for a distributed service, which is applied to a controller; the distributed service includes a plurality of working nodes, and each of the working nodes is respectively deployed with a local analyzer; the local analyzer is connected to the controller; the method includes:

[0006] receiving stack trace data for the working node collected by each of the local profilers, and performing aggregation processing on the stack trace data to obtain aggregated stack data;

[0007] A target flame graph is generated according to the aggregated stack data, and a target performance analysis result for the distributed service is generated based on the target flame graph.

[0008] In some embodiments, the method further comprises:

[0009] Obtaining first stack trace data of the local profiler at a first sampling frequency, generating a first target flame graph according to the first stack trace data, and generating a first performance profiling result based on the first target flame graph; wherein the first sampling frequency is higher than a preset sampling frequency;

[0010] Based on the first performance profiling result, the first sampling frequency is adjusted to obtain a second sampling frequency; second stack trace data of the local profiler at the second sampling frequency is obtained, and a second target flame graph is generated according to the second stack trace data, and a second performance profiling result is generated based on the second target flame graph.

[0011] In some embodiments, after generating the second performance analysis result, the method further includes:

[0012] Performing similarity calculation based on the first performance analysis result and the second performance analysis result to obtain a similarity result;

[0013] When it is detected that the similarity result is less than a preset similarity threshold, the first performance analysis result is used as the target performance analysis result.

[0014] In some embodiments, the receiving stack trace data collected by each of the local profilers for the working node includes:

[0015] Receiving an initial flame graph sent by each of the local profilers; wherein the initial flame graph is generated by the local profiler according to the stack trace data;

[0016] The initial flame graph is parsed to obtain the stack trace data.

[0017] In some embodiments, the local profiler further includes an agent and a collection kernel; the initial flame graph is generated by the collection kernel collecting the stack trace data and the agent processing the stack trace data.

[0018] In some embodiments, there are multiple collection kernels; receiving the initial flame graph sent by each of the local profilers includes:

[0019] Detecting the running information of the distributed service, and determining a corresponding target kernel from each of the collecting kernels based on the running information; wherein the agent processes the stack trace data collected by the target kernel and generates the initial flame graph.

[0020] In a third aspect, the present application provides a performance analysis method for a distributed service, which is applied to a local analyzer; wherein the distributed service includes a plurality of working nodes, and each of the working nodes is respectively deployed with a local analyzer; the method includes:

[0021] Collecting stack trace data on the working node;

[0022] The stack trace data is sent to a controller; wherein the controller aggregates the stack trace data to obtain aggregated stack data; a target flame graph is generated according to the aggregated stack data, and a target performance analysis result for the distributed service is generated based on the target flame graph.

[0023] In some embodiments, the collecting is for stack trace data on the working node, including:

[0024] Performing stack trace sampling processing on the distributed service using multiple threads, and performing pruning processing on the threads based on the number of thread samples to obtain a target thread among the threads;

[0025] The stack trace data sampled from the target thread is obtained.

[0026] In a third aspect, an embodiment of the present application provides a performance analysis system for a distributed service, wherein the distributed service includes a plurality of working nodes; the performance analysis system includes:

[0027] A local profiler, deployed on the corresponding working node, for collecting stack trace data on the working node;

[0028] A controller, configured to receive the stack trace data sent by each of the local profilers, and aggregate the stack trace data to obtain aggregated stack data;

[0029] The controller is further configured to generate a target flame graph according to the aggregated stack data, and generate a target performance analysis result for the distributed service based on the target flame graph.

[0030] In some embodiments, the performance analysis system further comprises:

[0031] A monitoring server, configured to receive a hotspot function indicator sent by the local profiler; wherein the hotspot function indicator is calculated and generated by the local profiler based on the stack trace data;

[0032] The monitoring server is also used to generate tracking and storage data for each hot function in the distributed service based on the hot function indicators.

[0033] In a fourth aspect, an embodiment of the present application provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the performance analysis method of the distributed service as described in the first aspect or the second aspect above when running.

[0034] Compared with the related art, the performance analysis method, system and storage medium of the distributed service provided by the embodiment of the present application, the distributed service includes multiple working nodes, and each working node is respectively deployed with a local analyzer; the performance analysis method includes: receiving the stack trace data collected by each local analyzer on the working node, and aggregating the stack trace data to obtain aggregated stack data; generating a target flame graph according to the aggregated stack data, and generating a target performance analysis result for the distributed service based on the target flame graph. Through the above method, it is possible to use the analysis tool to carry out low-cost and non-invasive hotspot function analysis; and, unlike traditional tools that only generate flame graphs for a single instance, the present invention effectively and accurately locates the real hotspot function by aggregating flame graphs across all instances, and is not limited to the analysis of library functions, thereby achieving high versatility, high accuracy and low-cost performance analysis in a non-invasive manner in a containerized and distributed environment, effectively solving the problem that performance analysis is difficult to adapt to large-scale distributed services.

[0035] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0037] Figure 1 It is a hardware structure block diagram of a terminal of a performance analysis method of a distributed service according to an embodiment of the present application;

[0038] Figure 2 is a flow chart of a performance analysis method for a distributed service according to an embodiment of the present application;

[0039] Figure 3A is a schematic diagram of an initial flame graph according to an embodiment of the present application;

[0040] Figure 3B is a schematic diagram of another initial flame graph according to an embodiment of the present application;

[0041] Figure 3C is a schematic diagram of a target flame graph according to an embodiment of the present application;

[0042] Figure 4 is a schematic diagram of a hotspot function change according to an embodiment of the present application;

[0043] Figure 5 is a schematic diagram of an evaluation result of a frequency dynamic adjustment algorithm according to an embodiment of the present application;

[0044] Figure 6 is a flow chart of a method for automatically selecting an acquisition kernel according to an embodiment of the present application;

[0045] Figure 7 is a flow chart of another distributed service performance analysis method according to an embodiment of the present application;

[0046] Figure 8 is a schematic diagram of the number of thread samples according to an embodiment of the present application;

[0047] Fig. 9 is a schematic diagram of a hotspot function aggregation cost and error according to an embodiment of the present application;

[0048] Fig.10 is a structural block diagram of a performance analysis system for distributed services according to an embodiment of the present application;

[0049] Fig.11 It is a structural diagram of a performance analysis system for distributed services according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present application, some changes such as design, manufacturing or production based on the technical contents disclosed in the present application are only conventional technical means, and should not be understood as insufficient contents disclosed in the present application.

[0051] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0052] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to greater than or equal to two. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The terms "first", "second", "third" and the like involved in the present application are merely used to distinguish similar objects and do not represent a specific ordering of the objects.

[0053] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. Taking running on a terminal as an example, Figure 1 1 is a hardware structure block diagram of a terminal of a performance analysis method for a distributed service according to an embodiment of the present application. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0054] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the performance analysis method of the distributed service in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0055] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0056] Cloud products represented by large online services, such as Google Search and Amazon online shopping, are usually deployed in data centers to ensure continuous availability. In order to effectively handle the huge request traffic, these services are divided into a large number of instances or replicas, ranging from hundreds to tens of thousands. For example, according to data released by Alibaba, an 8-day statistics conducted on a cluster of 4,000 machines in 2018 showed that the number of instances of a single service ranged from hundreds to tens of thousands, with a maximum of 99,583.

[0057] Given the sheer scale of these services, even a small performance improvement can lead to significant cost savings. Therefore, identifying and optimizing hot functions in distributed services in data centers can help improve resource utilization. However, compared with traditional single applications, large-scale distributed services present significant differences in both hardware and software architectures as well as deployment methods. First, at the hardware level, due to their large scale, service instances are distributed across different worker nodes in a data center. Second, at the software level, services are developed using different types of programming languages, including compiled and interpreted. Finally, with the emergence and development of container technology, it has gradually replaced virtual machines and become the primary way to deploy services in data centers. These characteristics indicate that in order to discover the performance bottlenecks of a large number of services in a data center and then optimize them, a non-intrusive, high-performance profiler that can support multiple languages ​​and adapt to containerization is needed.

[0058] Performance profiling analyzes the runtime behavior of a program by recording hardware or software metrics to identify performance bottlenecks. Profilers typically use code instrumentation to achieve higher accuracy or sampling methods to improve efficiency. To better understand this application, the following are some explanations of the relevant concepts:

[0059] Code instrumentation refers to inserting additional code into the target program. When this modified program is executed, it produces performance data for report generation. This method ensures accuracy by collecting metrics only when the relevant functions are executed, thus avoiding problems such as shadow effects and skids. However, this technique usually requires restarting or recompiling the program, resulting in considerable overhead. In addition, the additional instructions of the profiled code slow down service response time and consume more resources. These disadvantages are particularly prominent in data center environments that pursue efficiency.

[0060] Sampling-based profiling, also known as stack tracing: This method uses a hardware performance monitoring unit (PMU) to periodically capture a snapshot of the function call stack of the target program. After accumulating enough samples, the frequency of each function appearing in the stack trace indicates its resource consumption during profiling. Unlike code instrumentation, sampling only interrupts the program when certain signals are received, so the introduction of the central processing unit (CPU) overhead is minimal. In addition, since it does not require any modification to the program being profiled, this method is also considered non-intrusive. Although this method may be slightly inaccurate (because the functions captured in the sampling may not always be the functions actually executed), its high efficiency makes it the first choice for commonly used profiling tools such as Perf, HPCtoolkit, and Intel VTune. Since modifying the source code or initiating recompilation and restarting the service is unrealistic in a production environment, this application is based on non-intrusive sampling profiling, and achieves a trade-off between accuracy and efficiency by optimizing the sampling method.

[0061] It should also be noted that current performance profiling tools can be divided into two categories: single-machine profilers and multi-machine profilers. However, both categories have limitations when they are applied to large-scale distributed services. First, single-machine profilers have difficulty handling the large-scale, distributed characteristics of services and diverse workloads. Second, in order to overcome the shortcomings of single-machine profilers, researchers have integrated single-machine collectors, data links, databases, control panels, and other modules into multi-machine performance profiling systems to profile a large number of services at the same time and manage and analyze metrics in a unified manner. However, current multi-machine profilers are mostly closed-source proprietary and may not be directly attached to running processes. In addition, although users can create their own multi-machine profilers through customized monitoring systems, directly deploying single-machine profilers will result in considerable performance overhead, which is usually unacceptable for distributed microservices consisting of a large number of instances.

[0062] Based on this, this embodiment provides a performance analysis method for a distributed service, which is applied to a controller. The distributed service includes multiple working nodes, and each working node is respectively deployed with a local analyzer; the local analyzer is connected to the controller. Figure 2 is a flow chart of a performance analysis method for a distributed service according to an embodiment of the present application. Figure 2 As shown, the process includes the following steps:

[0063] Step S210: receiving stack trace data collected by each local profiler on the working node, and performing aggregation processing on the stack trace data to obtain aggregated stack data.

[0064] The local profiler is deployed on the corresponding working node and is used to collect stack trace data of the service instance on the working node. Among them, the local profiler refers to a profiling tool that runs on the local environment. A local profiler is deployed on each working node. Specifically, the local profiler responds to the profiling instruction sent by the controller to the target working node, and is responsible for collecting the stack trace data of the microservice on the working node where it is located. The stack trace data usually contains key information such as function call sequence and execution time, and is the basis of performance analysis.

[0065] In this embodiment, the performance analysis method of distributed services is applied to the controller, and the controller is connected to the ontology analyzer. The controller acts as a central hub and is responsible for delivering user commands to various components of the system. After the system is started, the controller enters a busy waiting state and is ready to accept commands through the command line interface (CLI). When the user initializes the analysis, the controller parses the configuration and establishes communication with the local analyzer to receive the stack trace data collected by the above local analyzers and dispatch analysis tasks according to the user configuration. In addition, the user can also flexibly manage existing analysis tasks through the controller.

[0066] Step S220: Generate a target flame graph according to the aggregated stack data, and generate a target performance analysis result for the service instance based on the target flame graph.

[0067] It should be noted that focusing on analyzing resource-intensive functions may lead to insufficient understanding of performance bottlenecks; in order to address this limitation, the embodiments of the present application use a flame graph in combination with numerical indicators to effectively and accurately locate the real hotspot functions. Among them, the flame graph uses a graphical method to display the performance data collected by the performance analyzer, and performs statistics and analysis on the data to facilitate the identification of performance hotspots.

[0068] Performance analysis tools typically sort functions based on resource consumption, but this can be misleading because high resource usage can be attributed to underlying library functions, which are often used to perform basic tasks such as resource locking or generating random numbers and are called by high-level business logic functions. The high resource usage exhibited by these library functions does not mean that they are the primary targets for optimization. On the contrary, business logic functions often require more careful inspection and optimization because these functions may have logical errors and excessively call the aforementioned underlying functions. An effective performance profiler should combine function call relationships to reveal which user-defined functions frequently trigger resource-intensive library functions. The present invention generates a flame graph for each instance such as a target service to intuitively reflect the function call relationship.

[0069] For a single instance of a service, a flame graph can be generated by calling the stack collection results. However, a large-scale application (such as buy2) often has multiple instances, and different instances may have different performance characteristics due to different access resource content and access frequency. In order to solve this problem, the present application generates a comprehensive flame graph covering the entire service by aggregating the sampling results of all sampled instances of the target service. Specifically, the controller merges and aggregates the stack trace data sent by each local profiler; the purpose of aggregation is to summarize and count the function call relationship, execution time and other information in multiple stack trace data to obtain aggregated stack data. These merged aggregated stack data represent the comprehensive behavior of all instances. Finally, the controller uses the aggregated stack trace data as input to generate an overall flame graph through a flame graph generation tool; wherein, the flame graph generation tool processes and analyzes the input data, and draws a flame graph based on the function call relationship, execution time and other information. The generated overall flame graph can intuitively display the performance bottlenecks and hot functions of the entire distributed system, helping developers to quickly locate and optimize problems.

[0070] On the other hand, stack trace data can also be transmitted between the local profiler and the controller in the form of a flame graph. The local profiler is also used to generate an initial flame graph based on the stack trace data; the controller is also used to receive the initial flame graph sent by each local profiler, and parse the initial flame graph to obtain stack trace data. Specifically, each local profiler generates an initial flame graph based on the stack trace data collected by each local profiler, and transmits it to the controller. The controller parses each initial flame graph and extracts stack trace data from it. These data may include key information such as the order of function calls, the number of calls, and the time taken to call. By parsing the obtained stack trace data, the controller can further analyze the performance of the program and provide strong support for performance tuning and troubleshooting. In the above manner, the local profiler processes the original stack trace data and generates an initial flame graph, so that the data can be transmitted in the form of a flame graph, which minimizes the data transmission overhead between the local profiler and the controller, and effectively reduces the computing burden of the controller node.

[0071] See also Figure 3A and Figure 3B , respectively showing the initial flame graphs for service instance 1 and service instance 2, which are generated by stack tracing and calculation on the corresponding local profiler. The initial flame graph of service instance 1 includes functions a(), b(), c(), d(), e(), and f(), and the call time proportion or frequency of each function is intuitively displayed through a graph. The initial flame graph of service instance 2 includes functions a(), b(), c(), d(), and g(). And, please refer to Figure 3CAfter aggregating the stack trace data, the controller generates a target flame graph based on the merged aggregated stack data. The flame graph contains the call time proportions of the above functions a(), b(), c(), d(), e(), f(), g(), and h().

[0072] In the existing technology, single-machine performance analysis tools cannot cope with diverse and distributed workloads; some analysis tools work in an intrusive way and need to modify the source code or binary code of the target service, which is inconvenient to use and increases costs; and multi-machine analysis tools are mostly closed-source proprietary, and external users are restricted from using them. In addition, existing tools usually sort hot functions according to resource consumption, which may lead to insufficient understanding of performance bottlenecks. Even if flame graphs are used, they often only support a single instance of the target service.

[0073] The embodiment of the present application provides a distributed profiling framework through the above-mentioned performance profiling method for distributed services, and the framework can use profiling tools to carry out low-cost and non-invasive hotspot function analysis; and, unlike traditional tools that only generate flame graphs for a single instance, the present invention aggregates flame graphs across all instances to effectively and accurately locate the real hotspot functions, and is not limited to the analysis of library functions, thereby achieving highly versatile, highly accurate and low-cost performance analysis in a containerized and distributed environment in a non-invasive manner, effectively solving the problem that performance analysis is difficult to adapt to large-scale distributed services.

[0074] In some of the embodiments, in order to ensure accuracy while keeping the analysis cost low, a frequency dynamic adjustment scheme is also provided to adaptively adjust the sampling frequency according to the previous analysis results. The scheme is proposed based on two important observations: the relationship between sampling frequency and accuracy, and the periodic change of hotspot functions.

[0075] In order to better understand the present application, the implementation principle of the frequency dynamic adjustment method is described here. First, it is generally believed that the performance analysis of high sampling frequency usually gives more accurate results (because more samples are obtained), but it also brings higher costs. For a single program, the user may set a higher sampling frequency to sacrifice some performance in exchange for higher analysis accuracy. But when it comes to multiple instances, this will cause a large increase in costs. Therefore, there is a trade-off between analysis cost and accuracy.

[0076] Second, typical workloads exhibit periodic changes in their hot functions, such as the widely used Java benchmark Specjbb2015. Figure 4, we monitored the changes of the top 5 hot functions during the benchmark run. Specifically, we treated the time cost ratio of the top 5 hot functions as a probability distribution, and used the Jensen-Shannon Divergence (JS Divergence) to calculate the difference between the current distribution and the previous distribution every 10 seconds; JS Divergence is a similarity measure between probability distributions. The results are shown in Figure 2. Figure 4 As shown, two key trends are revealed: First, JS divergence remains at a high level at the beginning and end of the application life cycle. This indicates that the hotspot function changes rapidly at the beginning and end stages. Second, when the application runs stably, the hotspot function shows a periodic change pattern. That is, it can be observed that JS divergence has repeated peaks after similar intervals, indicating that the hotspot has changed, and then falls back to a lower level before the next cycle begins.

[0077] Based on the above analysis, the dynamic adjustment scheme of sampling frequency can be summarized as: taking the high-frequency sampling results as the baseline, and using this to correct the low-frequency sampling results. Among them, the above-mentioned performance analysis method of the distributed system can also include the following steps:

[0078] Obtain first stack trace data of the local profiler at a first sampling frequency, generate a target flame graph based on the first stack trace data, and generate a first performance profiling result based on the first target flame graph; the first sampling frequency is higher than a preset sampling frequency; based on the first performance profiling result, adjust the first sampling frequency to obtain a second sampling frequency; obtain second stack trace data of the local profiler at a second sampling frequency, generate a target flame graph based on the second stack trace data, and generate a second performance profiling result based on the second target flame graph.

[0079] Furthermore, in some of the embodiments, after generating the second performance analysis result, the performance analysis method further includes:

[0080] A similarity calculation is performed based on the first performance analysis result and the second performance analysis result to obtain a similarity result; the controller is further configured to use the first performance analysis result as a target performance analysis result when it is detected that the similarity result is less than a preset similarity threshold.

[0081] Specifically, always maintaining a high sampling frequency is accurate but more expensive. However, if the sampling rate is always kept low, there is no guarantee that the results obtained are accurate. Therefore, you can first use a high sampling rate to obtain some relatively accurate results, and then try to use a low sampling rate to obtain the results. If the difference between the low-frequency and high-frequency sampling results is not significant, it proves that the hotspot function has not changed significantly, so the results saved at a high frequency can be used to replace the current sampling results. Otherwise, it means that the application itself has changed, and the results previously obtained at a high frequency are no longer applicable. The sampling rate should be increased to regain the baseline results.

[0082] Among them, Table 1 shows one pseudo code of the frequency dynamic adjustment algorithm, as shown in Table 1:

[0083] Table 1 Frequency dynamic adjustment algorithm

[0084]

[0085] Taking the above pseudo code as an example, in order to evaluate the similarity of the sampling results, we use the JS divergence of the hotspot function distribution mentioned above. The user can set the threshold θ to adjust the strictness of the similarity judgment. JS >θ, they are considered significantly different. Since the number of functions in the sampling results is very large, in order to control the computational cost, θ is usually calculated using the top 5 hot functions instead of all functions, which represent the main performance bottlenecks of the target program.

[0086] The evaluation results of the above algorithm are as follows Figure 5 As shown, several findings can be drawn. First, as the sampling frequency increases, the sampling cost generally increases, but the sampling error decreases accordingly, which confirms the trend mentioned above. Second, under the dynamic frequency adjustment strategy, the CPU performance consumption is close to 5000Hz, but it reaches a mean absolute error (MAE) far lower than that at 5000Hz and comparable to that at 10000Hz. It is worth noting that it only consumes 12.4% of the sampling cost at 10000Hz.

[0087] Through the above embodiments, a frequency dynamic adjustment scheme is proposed, which dynamically adjusts the acquisition stack trace frequency according to the profiling results, allowing the user to maintain a low sampling rate at a low cost most of the time while enjoying the high accuracy of a high sampling rate, thereby ensuring profiling accuracy while maintaining a low cost.

[0088] In some embodiments, the local profiler further includes an agent and a collection kernel. The initial flame graph is generated by the collection kernel collecting stack trace data and the agent processing the stack trace data.

[0089] Specifically, the above-mentioned acquisition kernel is used to collect stack trace data. In this embodiment, a dedicated acquisition kernel deployed on a working node is used to perform stack tracing without modifying or interrupting the target microservice. These kernels use the hardware's performance monitoring unit (PMU) to collect runtime stack traces triggered by specific events. Each stack trace captures the sequence of function calls up to the sampling point. For example, in a trace like Func1→Func2→Func3, Func3 is the function being executed, called by function Func2, and function Func2 is called by function Func1. The same stack trace is counted to reflect the frequency of occurrence of the target performance event.

[0090] In addition, it should be noted that in the embodiments of the present application, the selected acquisition kernel should meet some key criteria, for example: considering the difficulty of restarting large online services, the acquisition kernel cannot require program modification or restart; in order to reduce the cost of analysis, the acquisition kernel should be based on sampling rather than code insertion; support needs to be provided for the target language characteristics.

[0091] The above-mentioned agent is used to process the stack trace data and generate the initial flame graph. Among them, the agent is a software component or service that can act as an intermediary between the three core components of the performance analysis system: the controller, the local profiler, and the monitoring server. First, the agent automatically activates the appropriate kernel according to the instructions of the controller and dynamically adjusts the frequency. Then, the agent processes the original stack traces of these kernels and converts the aggregated stack traces into indicators that can be recognized by the Prometheus server. In addition, the agent also generates flame graphs regularly for global aggregation.

[0092] In some embodiments, there are multiple acquisition kernels; the receiving of the initial flame graph sent by each of the local profilers comprises the following steps:

[0093] The running information of the distributed service is detected, and based on the running information, a corresponding target kernel is determined from each collected kernel; wherein the agent processes the stack trace data collected by the target kernel and generates an initial flame graph.

[0094] Among them, large-scale distributed services are usually composed of multiple microservices, each of which may be developed in a different programming language. However, current performance analysis tools are difficult to cope with this feature. The built-in performance analyzers of different programming languages ​​are different, so the existing multi-machine performance analyzers require users to have information about the language used by the target service in order to select a suitable analyzer. In this embodiment, the programming language is divided into two categories: interpreted and compiled. For compiled languages, since the binary code is executed directly on the CPU, the stack trace can be obtained by directly reading the stack space of the target service. For interpreted languages, since the interpreter usually has a language-specific memory model, it is necessary to use the application programming interface (Application Programming Interface, API) provided by the interpreter to rebuild the memory space to obtain the stack trace. In essence, this is a process of mapping from programming language to analysis kernel. For example, in the actual application process, the async-profiler acquisition kernel can be selected for the Java programming language, the py-spy acquisition kernel can be selected for Python, and the Perf acquisition kernel can be selected for compiled languages.

[0095] The following is an explanation of the process of automatically selecting the acquisition kernel. Figure 6 , the process includes the following steps:

[0096] Step S601, determine the target microservice, and determine whether the target microservice is run through an interpreter.

[0097] Step S602: If the judgment result of the above step S601 is yes, it indicates that the target microservice is developed using an interpreted language; at this time, an appropriate acquisition kernel designed specifically for a specific language will be selected to capture the stack trace.

[0098] Step S603, if the judgment result of the above step S602 is no, then when no interpreter is detected, it can be considered that the target microservice is developed using a compiled language; at this time, a system-level profiler such as Perf will be used by default to obtain the stack trace.

[0099] In the prior art, large-scale distributed services are usually composed of multiple microservices, each of which may be developed in a different programming language, making it difficult for users to specify the target language in advance. However, through the above embodiments, the present application automatically selects appropriate acquisition kernels for services developed in different programming languages, thereby being able to seamlessly analyze services written in multiple languages, and implements a language-independent adaptation mechanism to support multi-language microservices, reducing the user's required prior knowledge of the target service, and providing the user with an experience without language barriers.

[0100] In this embodiment, a performance analysis method for a distributed service is also provided, which is applied to a local analyzer; wherein the distributed service includes a plurality of working nodes, and each working node is respectively deployed with a local analyzer; see Figure 7 , the method comprises the following steps:

[0101] Step S710, collecting stack trace data on the working node;

[0102] Step S720, sending the stack trace data to the controller; wherein the controller aggregates the stack trace data to obtain aggregated stack data; generates a target flame graph according to the aggregated stack data, and generates a target performance analysis result for the distributed service based on the target flame graph.

[0103] In some embodiments, the above-mentioned collection of stack trace data on the working node may further include the following steps:

[0104] A stack trace sampling process is performed on a service instance using multiple threads, and threads are pruned based on the number of thread samples to obtain a target thread among the threads; and stack trace data obtained by sampling the target thread is obtained.

[0105] Since microservices are usually multi-threaded, it is necessary to aggregate the stack trace data of all child threads of the target service to fully reflect the hot functions of the target service. However, the aggregation process does not require the use of all child threads. This is because when a process runs, many daemon threads are generated to serve user threads. Taking Java as an example, the Java Virtual Machine (JVM) starts daemon threads and user threads in the background. These daemon threads are responsible for tasks such as garbage collection and resource monitoring. Compared with user threads, daemon threads consume fewer resources, but their number is large. Although the number of samples in the daemon thread is small and does not have a significant impact on the hot functions, a large number of daemon threads significantly increases the overall aggregation cost.

[0106] Taking the typical Java workload Specjbb2015 as an example, Specjbb2015 is a benchmark used to evaluate the performance of multi-threaded Java applications based on Java business application servers on the Java platform. Figure 8It shows the number of samples for each subthread during its execution. The threads are arranged in descending order of the number of samples, and the horizontal axis represents the ranking of the number of samples of the threads. The left vertical axis represents the number of samples for each thread, while the right vertical axis shows the proportion of the cumulative number of samples of the first n threads to the total number of samples. As can be seen from the figure, only about 20% of the threads, specifically the first 368 threads, generate a large number of samples. Most of the threads outside of this are not active, that is, they run as daemon threads. This pattern is also reflected in the proportional distribution: out of a total of 1840 threads, the first 444 threads generate 90% of the samples, corresponding to the P90 pruning standard. At the same time, the p99 threads are the first 838 threads, that is, the remaining 1002 threads contribute only 1% of the total number of samples, corresponding to the P99 pruning standard.

[0107] In this embodiment, after the agent obtains the stack traces of the sampled threads, the agent sorts them according to the number of samples of each thread. Since the number of threads is much lower than the number of stack traces, this sorting process is almost cost-free. The agent then retains the threads with the largest number of samples, such as the top 99% of threads, and then discards the remaining samples, which are mainly from inactive daemon threads. This method allows the most active threads to be analyzed intensively, improving the efficiency of the analysis process while avoiding being overwhelmed by relatively unimportant data. Experiments show that the error introduced by this method is almost negligible, but it can save considerable costs.

[0108] See also Fig. 9 , for the top 5, 10, 20 and 50 hot functions, under the P70, P80, P90, P95, and P99 pruning criteria, the percentage of time saved by the system is more significant than the error it produces. In particular, when the p99 pruning criterion is used, only 0.58% mean absolute percentage error (MAPE) is introduced for the top 50 hot functions, while the aggregation time is reduced by 6.8%. Therefore, the p99 criterion is adopted as the default pruning threshold for the system. Specifically, the agent in the local profiler compares the number of detected thread samples with the default pruning threshold, and discards thread samples whose number of thread samples is less than the default pruning threshold, and only retains target threads whose number exceeds the default paper-cutting threshold.

[0109] In the prior art, directly deploying a single-machine profiler in a multi-machine system will result in a large performance overhead, which is usually unacceptable for distributed microservices consisting of a large number of instances. In order to improve this problem, the present application uses the above embodiments to consider that stack traces from a large number of daemon threads may not provide useful information to identify hot functions. Therefore, in the process of aggregating multi-threaded stack trace data, stack traces from these daemon threads are pruned to reduce the cost of calculating hot functions, effectively improving the efficiency of the aggregation process while hardly affecting the aggregation results.

[0110] In this embodiment, a performance analysis system for distributed services is also provided. Fig.10 , the distributed service includes multiple working nodes; the performance analysis system includes:

[0111] A local profiler 20, deployed on a corresponding working node, for collecting stack trace data on the working node;

[0112] The controller 10 is used to receive the stack trace data sent by each local analyzer, and aggregate the stack trace data to obtain aggregated stack data;

[0113] The controller 10 is further configured to generate a target flame graph according to the aggregated stack data, and generate a target performance analysis result for the distributed service based on the target flame graph.

[0114] Through the above embodiments, a performance analysis system architecture that is adaptable to large-scale distributed microservices is provided, including deploying a local analyzer 20 on each working node, and establishing a control node based on the controller 10 to perform control and data processing in a unified manner, thereby realizing a performance analysis method that aggregates flame graphs across all instances, which can effectively and accurately locate the real hotspot functions.

[0115] In some of these examples, see Fig.10 The above-mentioned performance analysis system also includes: a monitoring server 30, which is used to receive the hot function index sent by the local profiler 20; the hot function index is calculated and generated by the local profiler based on the stack trace data. In this embodiment, the local profiler 20 is responsible for extracting and processing the stack trace from the corresponding target microservice, and then exposing the index of the hot function to the monitoring server 30. The monitoring server 30 is also used to generate tracking storage data for each hot function in the service instance based on the hot function index.

[0116] Specifically, the above-mentioned monitoring server 30 can adopt a server device such as a Prometheus server for service monitoring; wherein the Prometheus server is an open source service monitoring system. Specifically, the monitoring server 30 pulls the hot function indicators obtained by the local analyzer from the monitoring port of each working node, and provides data storage, analysis and visualization services; these hot function indicators represent the performance occupancy of the hot function, which is calculated by the local analyzer 20 by aggregating the original stack trace data, and formatted according to the monitoring server 30 standard, and the function hot indicators finally generated include the indicator name and a set of descriptive tags. Among them, when processing hot functions, the strategy of the monitoring server 30 is as follows: if the monitoring server 30 detects a new hot function, it will be included in the tracking range; this means that the monitoring server 30 will start to collect and store the performance indicators of the function for subsequent analysis and query. If a function detected by the monitoring server 30 already has a trace, then when new performance analysis data is received, the existing trace record of the function will be updated; in this way, the server can keep real-time tracking of the performance changes of the hot function. Therefore, the monitoring server 30 provides a complete data management suite for tracked hotspot functions, including persistent storage, query, analysis and visualization capabilities, enabling users to track resource consumption trends or export data for deeper evaluation.

[0117] Through the above embodiment, a performance analysis system is provided which also includes a monitoring server 30; by establishing a central node for controlling and processing data in a unified manner, the system workload is mainly concentrated on the server node, thereby ensuring that the local profiler 21 consumes less CPU time and minimizes interference with the target microservice. In addition, at the monitoring server 30 node, users can query and visualize these results for further analysis, providing a reference for code optimization and service status monitoring.

[0118] The following describes the embodiments of the present application in conjunction with specific usage scenarios. Fig.11 is a schematic diagram of the structure of a performance analysis system for distributed services according to an embodiment of the present application, such as Fig.11As shown, the system consists of three key components: a local profiler (LocalProfiler), a controller (Controller), and a Prometheus server (Prometheus Server). The architectural design principle of the present invention is to adapt to the distributed architecture and diversified workloads of large-scale microservices. Among them, it includes deploying a local profiler with multiple acquisition kernels on each working node, and establishing a control node and a Prometheus server to control and process data in a unified manner. This design has two main advantages. First, the workload of the system is mainly concentrated on the Prometheus server node; by ensuring that the local profiler consumes less CPU time, this design minimizes interference with the target microservice. Secondly, the functions of different nodes are loosely coupled, which enhances the maintainability and scalability of the system.

[0119] The following workflow describes the process of starting a profiling task in this system:

[0120] The user creates a configuration file, specifies the target node, target service, metrics to be sampled, sampling frequency, etc. The controller reads the configuration file, establishes a connection with the target node, and starts the profiling task while providing the necessary parameters to the local profiler. Among them, the local profiler continuously samples stack traces; the local profiler aggregates the original stack traces obtained during the profiling and calculates certain metrics that cannot be sampled directly. It then converts the profiling results into formatted Prometheus metrics and then uploads them to an available port. The Prometheus server monitors the ports exposed by the local profiler, pulls metrics, and stores them in its built-in time series database. Users can view visualized data in the Prometheus dashboard or export data from the database for further analysis.

[0121] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0122] In addition, in combination with the performance analysis method of the distributed service in the above embodiment, the embodiment of the present application can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, any one of the performance analysis methods of the distributed service in the above embodiment is implemented.

[0123] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0124] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0125] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A performance analysis method for distributed services, characterized in that: Applied to a controller; the distributed service includes a plurality of working nodes, and each of the working nodes is respectively deployed with a local analyzer; The local analyzer is connected to the controller; the method comprises: Receiving stack trace data collected by each of the local profilers on the working node, and performing aggregation processing on the stack trace data to obtain aggregated stack data; A target flame graph is generated according to the aggregated stack data, and a target performance analysis result for the distributed service is generated based on the target flame graph.

2. The performance analysis method according to claim 1, characterized in that: The method further comprises: Obtaining first stack trace data of the local profiler at a first sampling frequency, generating a first target flame graph according to the first stack trace data, and generating a first performance profiling result based on the first target flame graph; wherein the first sampling frequency is higher than a preset sampling frequency; Based on the first performance profiling result, the first sampling frequency is adjusted to obtain a second sampling frequency; second stack trace data of the local profiler at the second sampling frequency is obtained, and a second target flame graph is generated according to the second stack trace data, and a second performance profiling result is generated based on the second target flame graph.

3. The performance analysis method according to claim 2, characterized in that: After generating the second performance analysis result, the method further includes: Performing similarity calculation based on the first performance analysis result and the second performance analysis result to obtain a similarity result; When it is detected that the similarity result is less than a preset similarity threshold, the first performance analysis result is used as the target performance analysis result.

4. The performance analysis method according to any one of claims 1 to 3, characterized in that: The receiving stack trace data collected by each of the local profilers on the working node includes: Receiving an initial flame graph sent by each of the local profilers; wherein the initial flame graph is generated by the local profiler according to the stack trace data; The initial flame graph is parsed to obtain the stack trace data.

5. The performance analysis method according to claim 4, characterized in that: The local profiler also includes an agent and a collection kernel; the initial flame graph is generated by the collection kernel collecting the stack trace data and the agent processing the stack trace data.

6. The performance analysis method according to claim 5, characterized in that: There are multiple collection cores; receiving the initial flame graph sent by each of the local analyzers includes: Detecting the running information of the distributed service, and determining a corresponding target kernel from each of the collecting kernels based on the running information; wherein the agent processes the stack trace data collected by the target kernel and generates the initial flame graph.

7. A performance analysis method for distributed services, characterized in that: Applied to a local analyzer; wherein the distributed service includes a plurality of working nodes, and each of the working nodes is respectively deployed with a local analyzer; the method includes: Collecting stack trace data on the working node; The stack trace data is sent to a controller; wherein the controller aggregates the stack trace data to obtain aggregated stack data; a target flame graph is generated according to the aggregated stack data, and a target performance analysis result for the distributed service is generated based on the target flame graph.

8. The performance analysis method according to claim 7, characterized in that: The collecting of stack trace data on the working node includes: Performing stack trace sampling processing on the distributed service using multiple threads, and performing pruning processing on the threads based on the number of thread samples to obtain a target thread among the threads; The stack trace data sampled from the target thread is obtained.

9. A performance analysis system for distributed services, characterized in that: The distributed service includes multiple working nodes; The performance analysis system comprises: A local profiler, deployed on the corresponding working node, for collecting stack trace data on the working node; A controller, configured to receive the stack trace data sent by each of the local profilers, and aggregate the stack trace data to obtain aggregated stack data; The controller is further configured to generate a target flame graph according to the aggregated stack data, and generate a target performance analysis result for the distributed service based on the target flame graph.

10. The performance analysis system according to claim 9, characterized in that: The performance analysis system also includes: A monitoring server, configured to receive a hotspot function indicator sent by the local profiler; wherein the hotspot function indicator is calculated and generated by the local profiler based on the stack trace data; The monitoring server is also used to generate tracking and storage data for each hot function in the distributed service based on the hot function indicators.

11. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the performance analysis method for distributed services according to any one of claims 1 to 8 when running.

Images