Performance analysis method, electronic equipment, storage medium and program product

Abstract

The invention provides a performance analysis method, electronic equipment, a storage medium and a program product. The method is applied to an agent side, a user mode program and a kernel mode program run in the agent side, and the method comprises the steps that the user mode program reads an executable file executed by the agent side in a user mode, and backtracking information used for backtracking a target service is extracted from the executable file; the user mode program generates a backtracking table based on the backtracking information, and transmits the backtracking table to a kernel mode where the kernel mode program is located; when a performance analysis event is triggered, the kernel mode program backtracks a function call process in a target service execution process according to backtracking information recorded in a backtracking table in a kernel mode to generate statistical data of a function call stack; and the user mode program obtains statistical data of the function call stack generated by the kernel mode program, and converts the statistical data of the function call stack into performance analysis data.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of Internet technology, and in particular, to a performance analysis method, electronic device, storage medium, and program product. Background Art

[0002] Performance analysis is a common requirement in the process of program development and program iteration. In related technologies, Linux perf is usually used to perform performance analysis on large services.

[0003] However, during the performance analysis process, Linux perf needs to copy a large amount of stack content from kernel state to user state for stack backtracing. The copying of a large amount of stack content consumes a lot of CPU resources and takes a long time, making it impossible to complete continuous performance analysis. Summary of the invention

[0004] In view of this, one or more embodiments of this specification provide a performance analysis method, an electronic device, a computer-readable storage medium, and a computer program product to provide an efficient performance analysis method.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of an embodiment of this specification, a performance analysis method is provided, which is applied to an agent, wherein a user-mode program and a kernel-mode program are running on the agent, and the method includes:

[0007] The user state program reads the executable file executed by the agent in the user state, and extracts the backtracking information used for backtracking the target business from the executable file;

[0008] The user state program generates a backtrace table based on the backtrace information, and transmits the backtrace table to the kernel state where the kernel state program is located;

[0009] When a performance analysis event is triggered, the kernel state program, in the kernel state, traces back the function call process in the target service execution process according to the traceback information recorded in the traceback table, and generates statistical data of the function call stack;

[0010] The user state program obtains statistical data of the function call stack generated by the kernel state program, and converts the statistical data of the function call stack into performance analysis data.

[0011] Optionally, data is transferred between the user state program and the kernel state program via eBPF Map;

[0012] The step of transferring the backtrace table to the kernel state where the kernel state program is located comprises:

[0013] The backtrace table is written into the eBPF Map, and the kernel state program reads the eBPF Map to store the backtrace table in the kernel state;

[0014] The user state program obtains statistical data of the function call stack generated by the kernel state program, including:

[0015] The user state program reads the statistical data of the function call stack written by the kernel state program into the eBPF Map.

[0016] Optionally, when the number of executable files is greater than 1, a corresponding backtrace table is generated for each executable file; data is transferred between the user state program and the kernel state program through two eBPF Maps; the first eBPF Map is used as a data table for writing the backtrace table, and the second eBPF Map is used as an index table for recording the first and last addresses of each backtrace table in the first eBPF Map;

[0017] The step of transferring the backtrace table to the kernel state where the kernel state program is located comprises:

[0018] The backtrace table generated by each executable file is written into the first eBPF Map, and the head and tail addresses of the written backtrace table located in the first eBPF Map are written into the second eBPF Map, and the head and tail addresses of each backtrace table in the second eBPF Map are further read by the kernel state program, and the backtrace table in the first eBPF Map is read based on the head and tail addresses of each backtrace table, so as to store each backtrace table in the kernel state;

[0019] The user state program obtains statistical data of the function call stack generated by the kernel state program, including:

[0020] The user state program reads the head and tail addresses of each backtrace table in the second eBPF Map, and reads the statistical data of the function call stack corresponding to each backtrace table written by the kernel state program in the first eBPF Map based on the head and tail addresses of each backtrace table.

[0021] Optionally, the backtracking of the function call process in the target service execution process according to the backtracking information recorded in the backtracking table includes:

[0022] According to the traceback information recorded in the traceback table, the function call process in the target service execution process is traced back using the eh_frame method.

[0023] Optionally, before backtracing the function call process in the target service execution process in the eh_frame manner according to the backtracing information recorded in the backtracing table, the method further includes:

[0024] Determine the tracing method for the target service;

[0025] If the backtracking mode is the frame_pointer mode, the frame_pointer mode is used to backtrack the function call process in the target service execution process;

[0026] If the backtracking mode is not the frame_pointer mode, the backtracking information recorded in the backtracking table is executed, and the function calling process in the target service execution process is backtracked using the eh_frame mode.

[0027] Optionally, the user state program reads an executable file executed by the agent in the user state, and extracts the backtracking information used for backtracking the target service from the executable file, including:

[0028] The user-mode program starts at least one performance analysis process, and reads an executable file that uses memory resources in the memory address distribution according to the memory address distribution of the performance analysis process;

[0029] Backtrace information for backtracing a target function is extracted from the read executable file.

[0030] Optionally, converting the statistical data of the function call stack into performance analysis data includes:

[0031] The function symbol information in the executable file is parsed, and corresponding function symbols are associated with each function in the function call stack.

[0032] Optionally, parsing function symbol information in the executable file and associating corresponding function symbols for each function in the function call stack includes:

[0033] Parsing function symbol information in the executable file, and associating corresponding function symbols with kernel functions in the function call stack;

[0034] Extracting segments of user function symbols from the function symbol information to generate a simplified debugging file;

[0035] The debugging file, the function call stack of the function symbol associated with the kernel function, and the statistical data of the function call stack are sent as intermediate files to the collection end corresponding to the agent end; the collection end further parses the debugging file using a symbol parsing tool to associate the corresponding function symbol with the user function in the function call stack.

[0036] Optionally, the proxy terminal corresponds to a host machine where any service node in the distributed cluster is located.

[0037] Optionally, the distributed cluster includes a Kubernetes cluster.

[0038] According to a second aspect of an embodiment of this specification, a performance analysis device is provided, which is applied to an agent end, wherein a user state program and a kernel state program are running in the agent end, and the device includes:

[0039] A reading unit, wherein the user state program reads the executable file executed by the agent in the user state, and extracts the backtracking information used for backtracking the target business from the executable file;

[0040] A transmission unit, wherein the user state program generates a backtrace table based on the backtrace information, and transmits the backtrace table to the kernel state where the kernel state program is located;

[0041] A backtracking unit, wherein when a performance analysis event is triggered, the kernel state program, in the kernel state, backtracks the function call process in the target service execution process according to the backtracking information recorded in the backtracking table to generate statistical data of the function call stack;

[0042] The analysis unit is configured such that the user-mode program obtains statistical data of the function call stack generated by the kernel-mode program, and converts the statistical data of the function call stack into performance analysis data.

[0043] According to a third aspect of the embodiments of this specification, an electronic device is provided, including:

[0044] processor;

[0045] a memory for storing processor-executable instructions;

[0046] The processor implements the performance analysis method described in any embodiment of this specification by running the executable instructions.

[0047] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the performance analysis method described in any embodiment of this specification are implemented.

[0048] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program, which, when executed by a processor, implements the performance analysis method described in any embodiment of this specification.

[0049] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0050] In the embodiments of the present specification, a backtrace table containing backtrace information for backtracing a target service is generated by reading an executable file, and the backtrace table is passed to the kernel state, so that the function call process during the execution of the target service is backtraced in the kernel state according to the backtrace table to obtain statistical data of the function call stack; since the stack backtrace is implemented in the kernel state, there is no need to copy a large amount of stack content to the user state, thereby avoiding the copying and consuming a large amount of CPU resources, and the time consumption is short, so real-time or near real-time performance analysis can be achieved.

[0051] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a schematic diagram of the architecture of a performance analysis system provided by an exemplary embodiment;

[0053] Figure 2 is a flow chart of a performance analysis method provided by an exemplary embodiment;

[0054] Figure 3 is a schematic diagram of backtracking provided by an exemplary embodiment;

[0055] Figure 4 is a schematic diagram of two eBPF Maps provided by an exemplary embodiment;

[0056] Figure 5 is a schematic diagram of a symbol parsing process provided by an exemplary embodiment;

[0057] Figure 6 is a logical schematic diagram of a collection end provided by an exemplary embodiment;

[0058] Figure 7 is a schematic diagram of multi-party interaction provided by an exemplary embodiment;

[0059] Figure 8 is a structural schematic diagram of a performance analysis device provided by an exemplary embodiment;

[0060] Fig. 9 It is a schematic structural diagram of an electronic device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0061] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0062] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0063] The embodiments of this specification provide a performance analysis method, which can be targeted at any program, application, service, applet or other software or a combination of software and hardware.

[0064] A performance analysis method provided in an embodiment of the present specification can be applied to an electronic device, which can be a server, a terminal or other device.

[0065] As mentioned above, when using traditional Linux perf for performance analysis, due to the large CPU resource overhead, only a sampling frequency of 10 seconds per hour can be achieved. Even some large services cannot complete performance sampling, and performance analysis is unavailable.

[0066] The reason for the high CPU resource overhead is that a large amount of stack content is copied from kernel state to user state for stack backtracing. This process consumes a lot of CPU resources and is time-consuming, making it impossible to complete continuous performance analysis.

[0067] In response to the above problems, the performance analysis solution provided in this specification generates a backtrace table containing backtrace information for the target service by reading the executable file, and passes the backtrace table to the kernel state, so as to backtrace the function call process during the execution of the target service in the kernel state according to the backtrace table to obtain statistical data of the function call stack; since the stack backtrace is implemented in the kernel state, there is no need to copy a large amount of stack content to the user state, so the copying can avoid consuming a large amount of CPU resources, and the time consumption is short, so real-time or near real-time performance analysis can be achieved.

[0068] See below Figure 1The performance analysis system may include at least one agent terminal 11 , a collection terminal 12 and a processing terminal 13 .

[0069] Among them, the proxy end 11 can be called Agent, which can be deployed on the host machine of the business, and non-invasively start the performance analysis process related to the business through ebpf (extended Berkeley Packet Filter) technology, and complete stack backtracking and statistics in the kernel state when the performance collection time is triggered, and convert the statistical results into intermediate data and pass them to the collection end 12.

[0070] The acquisition end 12 may be called a Collector, which is used to symbolically associate the intermediate data transmitted by the agent end 11 to supplement the missing function information of the intermediate data, and further persistently store the supplemented intermediate data as performance analysis data.

[0071] The processing end 13 may be an online diagnosis platform or other platform for visual display based on the performance analysis data persisted by the acquisition end 12. The visual display may, for example, generate real-time or near-real-time statistical graphs based on the performance analysis data, such as flame graphs related to performance, comparison graphs of month-on-month performance statistics, etc.

[0072] Exemplarily, the performance analysis system may include multiple agents 11. Taking a distributed cluster as an example, the agent 11 may correspond to a host machine where any service node in the distributed cluster is located.

[0073] Among them, the distributed cluster may include but is not limited to Kubernetes (K8S) cluster, Openshift cluster, Hasicorp Nomad cluster, Docker Swarm cluster, etc.

[0074] Figure 2 : is a flowchart of a performance analysis method provided by an exemplary embodiment. The method of this embodiment is applied to the aforementioned agent end, in which a user-mode program and a kernel-mode program are running. The method may include the following processing:

[0075] Step 210: The user state program reads the executable file executed by the agent in the user state, and extracts the backtracking information used for backtracking the target service from the executable file.

[0076] Exemplarily, a user-mode program may start at least one performance analysis process, and based on the memory address distribution of the performance analysis process, read an executable file that uses memory resources in the memory address distribution, and then extract backtrace information for backtracing a target function from the read executable file.

[0077] Taking the k8s cluster as an example, a single business node in the k8s cluster supports starting multiple performance analysis processes at the same time, thereby realizing performance analysis of multiple business services at the same time.

[0078] The performance analysis process may include a profiling process. Profiling is a technology used to analyze the performance of a program or service. Profiling can be used to count and calculate the resource usage of a program or service, such as CPU time, memory usage, function calls, etc.

[0079] Since performance analysis requires the execution of programs or services, the performance analysis process will use the executable files required for the execution of the programs or services; based on this, these executable files can be read by determining the memory address distribution of the performance analysis process, and then the backtrace information for backtracing the target business can be extracted from these executable files.

[0080] Step 220: The user-mode program generates a backtrace table based on the backtrace information, and transmits the backtrace table to the kernel mode where the kernel-mode program is located.

[0081] In this specification, the user-mode program and the kernel-mode program may communicate and transmit data via the eBPF Map.

[0082] eBPF (Extended Berkeley Packet Filter) is a kernel technology that allows users to run custom programs in the kernel to achieve efficient networking, monitoring, security and other functions. In eBPF, "Map" can support communication and data exchange between kernel-mode programs and user-mode programs.

[0083] eBPF Map is a data structure that can store key-value pairs for use by kernel-mode programs and user-mode programs. Common eBPF Map types include but are not limited to hash, array, lru_hash, percpu_array, etc.

[0084] User-mode programs can write data to the eBPF Map, and this data can be shared with kernel-mode programs, thereby transferring data from user-mode to kernel-mode. Correspondingly, kernel-mode programs can also write data to the eBPF Map, and this data can also be shared with user-mode programs, thereby transferring data from kernel-mode to user-mode.

[0085] Based on this, the above-mentioned transferring the backtrace table to the kernel state where the kernel state program is located may further include:

[0086] The backtrace table is written into the eBPF Map, and the kernel-state program reads the eBPF Map to store the backtrace table in the kernel state.

[0087] In the above embodiment, the communication between the user-mode program and the kernel-mode program can be realized through the eBPF Map, and the backtrace table can be transferred non-invasively.

[0088] Step 230, when the performance analysis event is triggered, the kernel state program, in the kernel state, traces back the function call process in the target service execution process according to the traceback information recorded in the traceback table, and generates statistical data of the function call stack.

[0089] The performance analysis event may be triggered periodically, for example, the performance analysis event may be triggered periodically according to CPU cycles, so as to continuously trace back the function call process during the execution of the target service in the kernel state and continuously generate statistical data of the function call stack.

[0090] The statistical data of the function call stack in this specification may include the function call process during the execution of the target service, the number of times each function is called, and other information.

[0091] In an exemplary embodiment, the backtracking of the function call process in the target service execution process according to the backtracking information recorded in the backtracking table includes:

[0092] According to the traceback information recorded in the traceback table, the function call process in the target service execution process is traced back using the eh_frame method.

[0093] In the related art, each function call during the execution of the target service will generate a CFI (call frame information) instruction in the user state, and the principle of eh_frame backtracing is to parse these CFI instructions to find the function call process. However, in the kernel state, since the kernel state has restrictions on the CIF instruction data, such as limiting the number of CFI instructions that may be stored, the CFI in the kernel state is not complete, so parsing CFI in the kernel state through the eh_frame method cannot truly backtrack the function call process.

[0094] Therefore, in the embodiments of this specification, when backtracing is performed using the eh_frame method, the cfi instruction is no longer parsed, but the backtracing table transferred to the kernel state is used to backtrace the function call process in the target service execution process based on the backtracing information recorded in the backtracing table. This can avoid the inability to truly backtrace the function call process due to incomplete cfi (call frame information) instructions.

[0095] In addition to the eh_frame method, this specification also supports frame_pointer backtracing. Among them, frame pointer backtracing is relatively simple and can be implemented without the help of a backtracing table; however, it requires specific parameters to be enabled during compilation and requires special registers to support frame pointers, so this method has a greater impact on program performance.

[0096] Below Figure 3 As shown, before backtracking the function call process in the target service execution process using the eh_frame method according to the backtracking information recorded in the backtracking table, the method further includes:

[0097] Determine the tracing method for the target service;

[0098] If the backtracking mode is the frame_pointer mode, the frame_pointer mode is used to backtrack the function call process in the target service execution process;

[0099] If the backtracking mode is not the frame_pointer mode, the backtracking information recorded in the backtracking table is executed, and the function calling process in the target service execution process is backtracked using the eh_frame mode.

[0100] In this embodiment, by being compatible with both frame_pointer and eh_frame, even when the eh_frame mode is unavailable, backtracking can be performed through frame_pointer, thus achieving better compatibility.

[0101] Step 240: the user-mode program obtains statistical data of the function call stack generated by the kernel-mode program, and converts the statistical data of the function call stack into performance analysis data.

[0102] like Figure 3As shown, after the kernel-mode program traces back the function call process during the execution of the target service and generates statistical data of the function call stack, the statistical data of the function call stack can be written into the eBPF Map. Furthermore, the user-mode program can read the statistical data of the function call stack written into the eBPF Map by the kernel-mode program.

[0103] In an exemplary embodiment, since the eBPF Map only supports key-value pairs of fixed size, and as shown above, the agent can perform performance analysis on multiple target services at the same time, this will involve executable files corresponding to different target services, and the sizes of backtrace tables of different executable files vary greatly. Therefore, it is difficult for a single eBPF Map to manage different backtrace tables at the same time.

[0104] Based on this, this specification designs a method to efficiently store and search multiple backtrace tables based on the index of eBPF Map. Specifically, when the number of executable files is greater than 1, a corresponding backtrace table is generated for each executable file.

[0105] like Figure 4 As shown, data is transferred between the user-mode program and the kernel-mode program via two eBPF Maps; the first eBPF Map is used as a data table for writing the backtrace table, and the second eBPF Map is used as an index table for recording the first and last addresses of each backtrace table in the first eBPF Map.

[0106] Exemplarily, in the aforementioned step 220, transferring the backtrace table to the kernel state where the kernel state program is located may include:

[0107] The backtrace table generated by each executable file is written into the first eBPF Map, and the head and tail addresses of the written backtrace table located in the first eBPF Map are written into the second eBPF Map, and the head and tail addresses of each backtrace table in the second eBPF Map are further read by the kernel state program, and the backtrace table in the first eBPF Map is read based on the head and tail addresses of each backtrace table, so as to store each backtrace table in the kernel state;

[0108] Accordingly, the aforementioned step 240 in which the user-mode program obtains the statistical data of the function call stack generated by the kernel-mode program may include:

[0109] The user state program reads the head and tail addresses of each backtrace table in the second eBPF Map, and reads the statistical data of the function call stack corresponding to each backtrace table written by the kernel state program in the first eBPF Map based on the head and tail addresses of each backtrace table.

[0110] Still Figure 4 For example, assuming that the address passed to the user-state program is 0x49400, the user-state program first matches the start and end address range of the address 0x49400 in the index table as 0x493e0-0x61a80; after determining the start and end addresses, it then reads the statistical data of the 100,000 function call stacks with exeid=2 written in the backtrace table corresponding to 0x493e0-0x61a80 in the data table.

[0111] In summary, through the cooperation of the index table and the data table, the traceback tables corresponding to different target services can be efficiently stored and read.

[0112] In this specification, after obtaining the statistical data of the function call stack, the user-mode program may also convert the statistical data of the function call stack into performance analysis data.

[0113] The step of converting the statistical data of the function call stack into performance analysis data may include:

[0114] The user state program parses the function symbol information in the executable file and associates corresponding function symbols with each function in the function call stack.

[0115] Although the function call stack obtained by backtracing the kernel-mode program records the function call relationship, each function lacks a function symbol, resulting in poor readability of the function call stack and the inability to effectively use the function call stack for further refined performance analysis.

[0116] To this end, this embodiment completes the missing function symbols of each function in the function call stack by parsing the function symbol information in the executable file.

[0117] In actual applications, the functions in the function call stack are different depending on the execution object. For example, the function executed in the user state can be called a user function, and the function executed in the kernel state can be called a kernel function.

[0118] Since the symbols of kernel functions on different agents are different, the symbol association of kernel functions needs to be performed locally on each agent. In addition, the parsing and searching of kernel functions are relatively simple and lightweight, so performing symbol association of kernel functions locally on the agent will not take up too much CPU resources.

[0119] User functions are different. The symbol resolution of user functions needs to rely on specific symbol resolution tools such as dwarf, which is a heavyweight resolution and search process. It takes up more CPU resources and is too heavy for the proxy.

[0120] In addition, for different proxy terminals, the function symbols of unified user functions are the same, so user symbol resolution on the proxy terminal is a completely reinventing the wheel process.

[0121] After comprehensive consideration, this specification proposes a method to put user symbol analysis uniformly at the collection end. Figure 1 The Collector in the implementation example.

[0122] Specifically, the user-mode program parses the function symbol information in the executable file and associates the corresponding function symbol with each function in the function call stack, which may include:

[0123] The user state program parses the function symbol information in the executable file, and associates the corresponding function symbol with the kernel function in the function call stack;

[0124] The user state program extracts the segments of the user function symbol from the function symbol information to generate a simplified debugging file;

[0125] The user state program sends the debugging file, the function call stack associated with the function symbol of the kernel function, and the statistical data of the function call stack as intermediate files to the collection end corresponding to the agent end;

[0126] The acquisition end further uses a symbol parsing tool to parse the debugging file to associate corresponding function symbols with user functions in the function call stack.

[0127] Please combine the following Figure 5 The schematic diagram of the symbol resolution process shown in the figure shows that after the user-mode program reads the statistical data of the function call stack (the statistical data is recorded in the pprof file), it combines the function symbol information in the parsed executable file to associate the corresponding function symbol with the kernel function. Then, the user function symbol is extracted from the function symbol information to generate a streamlined debuginfo file (i.e., debugging file). Then, the pprof file and debuginfo file associated with the kernel function symbol are sent to the collection end as intermediate files through the HTTP API provided by the collection end.

[0128] The collection end can receive the intermediate files sent by each agent end, and associate the corresponding function symbols with the user functions in these intermediate files. As mentioned above, since the same user functions in different intermediate files have the same function symbols, each user function only needs to be resolved once, and the resolved function symbols can be cached locally, so that the same user symbols in other intermediate files can be associated with corresponding function symbols. In this way, the cache can be used more effectively to avoid repeated resolution of function symbols.

[0129] In this manual, the acquisition end supports a variety of symbol analysis tools such as symtab, dwarf, etc.

[0130] Figure 6 The logic diagram of the collection end shown in the figure takes dwarf as an example. Dwarf contains a tree structure defined by the code, and the nodes in the tree structure can be found according to the function address. After receiving the intermediate file sent by the agent, the collection end can generate the following according to the debugging file in it: Figure 6 The symbol index (dwarf index) shown in FIG. 1 includes nodes with address range attributes (ranges) and nodes with function symbols (funcs).

[0131] It should be noted that by building a symbol index, compared with the original search logic, additional support for binary search is provided, thereby speeding up the symbol search speed, and by retaining the streamlined function information, the original dwarf data structure is released, greatly reducing memory usage.

[0132] After constructing the above symbol index, according to the function address of the user function in the intermediate file, Figure 6 The 0x44010 shown matches the corresponding address range attributes: attribute 1 (start: 0x43000, end: 0x45000, funcIndex: 0), attribute 2 (start: 0x43000, end: 0x44500, funcIndex: 1), and attribute 3 (start: 0x44000, end: 0x44200, funcIndex: 2), and further matches the function symbol to which each attribute belongs: symbol 1 corresponding to attribute 1 (Server::Run, @server.c:120, cuOffs et:1), symbol 2 corresponding to attribute 2 (Task::task_run, @worker.c:22, cuOffset:2), symbol 3 corresponding to attribute 3 (Sample::read, @lib.c:15, cuOffset:3); in this way, the corresponding function symbols can be associated with the user function, so as to obtain the complete function calling relationship Run@server.c:100 calls task_run@worker.c:22, and task_run@worker.c:22 further calls read@lib.c:15.

[0133] After completing the function symbol, the collection end can store the complete performance analysis data persistently locally or in a third-party data device, thus facilitating further analysis and processing. Figure 1 The processing end 13 shown in FIG. 1 performs a visual display of the performance analysis results.

[0134] For example, the processing end reads the performance analysis data that is continuously and persistently stored by the collection end, and can generate a real-time or near real-time performance flame graph. The horizontal axis of the flame graph can be time, and the vertical axis can be statistical data in each performance analysis data, such as the number of times each function is called. The flame graph can intuitively show the changes in statistical data in different time periods.

[0135] For another example, the processing end reads the performance analysis data that is continuously and persistently stored by the collection end, and can generate a diff analysis report of the current and historical performance of the target service. The diff analysis report can intuitively show the changes in service performance, thus providing a reference for analyzing the causes of performance degradation.

[0136] In an exemplary embodiment, since there are different proxy terminals, and each proxy terminal can also start multiple performance analysis processes at the same time, in order to facilitate the subsequent refined management of performance analysis data, such as Figure 5 As shown in:

[0137] User-mode programs can attach metadata obtained during the process of starting the performance analysis process as tags to the performance analysis data. Taking the k8s cluster as an example, each performance analysis process started by a business node can be located in the corresponding container pod, so the user-mode program can attach metadata information such as the pod where the started performance analysis process is located, the image version, and the availability zone as tags to the performance analysis data.

[0138] For the processing end, when performing visual display, you can use tags to filter the performance analysis data to be displayed, such as filtering the performance analysis data of the target pod, image version, and availability zone that you want to display.

[0139] The above embodiments can more flexibly present diverse visual display effects.

[0140] After introducing the method embodiment with the agent end as the execution subject, the following introduces the embodiment of the performance analysis method under multi-party interaction in combination with the agent end, the collection end and the processing end. Figure 7 As shown, it may include:

[0141] Step 310: The user-mode program of the agent reads the executable file and generates a backtrace table.

[0142] The user state program starts at least one performance analysis process, and reads an executable file that uses memory resources in the memory address distribution according to the memory address distribution of the performance analysis process, then extracts backtrace information for backtracing a target function from the read executable file, and finally generates a corresponding backtrace table according to the backtrace information. The backtrace table records the extracted backtrace information for backtracing a target function.

[0143] Step 311: The user state program transfers the backtrace table to the kernel state.

[0144] User-mode programs can write backtrace tables into eBPF Map. Since eBPF Map provides a data transmission channel between kernel-mode programs and user-mode programs, kernel-mode programs can read the backtrace tables written by user-mode programs from eBPF Map, thereby storing the read backtrace tables in kernel mode.

[0145] Step 312: The kernel-mode program of the proxy performs backtracing according to the backtracing table to generate statistical data of the function call stack.

[0146] In response to the triggered performance analysis event, the kernel state program traces back the function call process during the execution of the target service according to the traceback information recorded in the traceback table stored in the kernel state, and generates statistical data of the function call stack. The statistical data of the function call stack may include the function call process during the execution of the target service, the number of times each function is called, and other information.

[0147] Among them, the backtracking of the function call process in the target service execution process can be performed using the frame_pointer method or the eh_frame method. The specific backtracking process can be referred to the aforementioned Figure 2 The embodiments in .

[0148] Step 313: The kernel-mode program transfers the statistical data of the function call stack to the user-mode program.

[0149] The kernel-mode program writes the statistical data of the function call stack to the eBPF Map, so that the user-mode program can read the statistical data of the function call stack written by the kernel-mode program from the eBPF Map and store the statistical data of the function call stack in the user mode.

[0150] Considering that a user-mode program can start multiple performance analysis processes at the same time, and each performance analysis process responds to its own backtrace table, in order to facilitate the storage and reading of eBPF Map data, the user-mode program and the kernel-mode program can transfer data through two eBPF Maps. The first eBPF Map is used as a data table to write the backtrace table, and the second eBPF Map is used as an index table to record the first and last addresses of each backtrace table in the first eBPF Map. The coordinated use of the index table and the data table can be found in the above Figure 2 The embodiments in .

[0151] Step 314: The user-mode program performs symbol association on the kernel functions in the function call stack and generates a simplified debugging file.

[0152] Although the function call stack obtained by backtracing the kernel-mode program records the function call relationship, each function lacks a function symbol, resulting in poor readability of the function call stack and the inability to effectively use the function call stack for further refined performance analysis. Therefore, it is necessary to complete the missing function symbols of each function in the function call stack.

[0153] Furthermore, since the symbol resolution of kernel functions needs to be performed locally on the agent side, and the symbol resolution of user functions is relatively complex and heavily quantized, the local execution efficiency on the agent side is inefficient and repetitive, the symbol resolution of kernel functions is performed on the agent side, while the symbol resolution of user functions is performed on the acquisition side.

[0154] Specifically, the user state program can associate the corresponding function symbol with the kernel function in the function call stack by parsing the function symbol information in the executable file, and the user state program can extract the segmentation of the user function symbol from the function symbol information to generate a simplified debugging file.

[0155] Step 315: the user-mode program sends the debugging file and the statistical data of the function symbols associated with the kernel symbols to the collection end as an intermediate file.

[0156] Step 320: The acquisition end uses a symbol parsing tool to parse the debugging file and associates corresponding function symbols with the user functions in the function call stack.

[0157] Since the same user functions in different intermediate files have the same function symbols, each user function only needs to be resolved once, and the resolved function symbols can be cached locally, so that the same user symbols in other intermediate files can be associated with corresponding function symbols. This can make more effective use of the cache and avoid repeated resolution of function symbols.

[0158] Step 321: The collection end persistently stores the performance analysis data supplemented with the function symbols of the user functions.

[0159] After completing the function symbol, the collection end can store the complete performance analysis data persistently locally or in a third-party data device, thus facilitating further analysis and processing.

[0160] Step 330: The collection end transmits the persistent performance analysis data to the processing end.

[0161] When the processing end needs to display the performance analysis results, it can request the collection end to obtain the performance analysis data; or, the collection end can actively send the persistent performance analysis data to the processing end continuously.

[0162] Step 331: The processing end generates a visualized performance analysis result based on the performance analysis data.

[0163] The processing end reads the performance analysis data that is continuously and persistently stored by the collection end, and can generate a real-time or near-real-time performance flame graph. The flame graph can intuitively show the changes in statistical data in different time periods.

[0164] Alternatively, the processing end can read the performance analysis data stored continuously and persistently by the collection end to generate a diff analysis report of the current and historical performance of the target service. The diff analysis report can intuitively show the changes in service performance, thus providing a reference for analyzing the causes of performance degradation.

[0165] Figure 8 An example of a structural block diagram of a performance analysis device is provided. The performance analysis device corresponds to the aforementioned Figure 2 Specifically, the performance analysis device is applied to an agent end, in which a user state program and a kernel state program are running, and the device includes: a reading unit 710 , a transmission unit 720 , a backtracking unit 730 and an analysis unit 740 .

[0166] A reading unit 710, in which a user state program reads an executable file executed by an agent in a user state, and extracts backtracking information for backtracking a target service from the executable file;

[0167] A transmission unit 720, wherein the user state program generates a backtrace table based on the backtrace information, and transmits the backtrace table to the kernel state where the kernel state program is located;

[0168] The backtracking unit 730, when the performance analysis event is triggered, in the kernel state, the kernel state program backtracks the function call process in the target service execution process according to the backtracking information recorded in the backtracking table, and generates statistical data of the function call stack;

[0169] The analysis unit 740, the user state program obtains the statistical data of the function call stack generated by the kernel state program, and converts the statistical data of the function call stack into performance analysis data.

[0170] In an exemplary embodiment, data is transferred between the user-mode program and the kernel-mode program via an eBPF Map;

[0171] When the transfer unit 720 transfers the backtrace table to the kernel state where the kernel state program is located, the transfer unit 720 is further used to write the backtrace table into the eBPF Map, and the kernel state program reads the eBPF Map to store the backtrace table in the kernel state;

[0172] When acquiring the statistical data of the function call stack generated by the kernel-state program, the analyzing unit 740 is further configured to read the statistical data of the function call stack written by the kernel-state program into the eBPF Map.

[0173] In an exemplary embodiment, when the number of executable files is greater than 1, a corresponding backtrace table is generated for each executable file; data is transferred between the user state program and the kernel state program through two eBPF Maps; the first eBPF Map is used as a data table for writing the backtrace table, and the second eBPF Map is used as an index table for recording the first and last addresses of each backtrace table in the first eBPF Map;

[0174] When the transfer unit 720 transfers the backtrace table to the kernel state where the kernel state program is located, it is further used to write the backtrace table generated by each executable file into the first eBPF Map, and write the written backtrace table located at the head and tail addresses of the first eBPF Map into the second eBPF Map, and further read the head and tail addresses of each backtrace table in the second eBPF Map by the kernel state program, and read the backtrace table in the first eBPF Map based on the head and tail addresses of each backtrace table, so as to store each backtrace table in the kernel state;

[0175] When obtaining the statistical data of the function call stack generated by the kernel-state program, the analysis unit 740 is further used to read the head and tail addresses of each backtrace table in the second eBPF Map, and read the statistical data of the function call stack corresponding to each backtrace table written by the kernel-state program in the first eBPF Map based on the head and tail addresses of each backtrace table.

[0176] In an exemplary embodiment, when the backtrace unit 730 backtraces the function call process in the execution process of the target service according to the backtrace information recorded in the backtrace table, it is further used to backtrace the function call process in the execution process of the target service according to the backtrace information recorded in the backtrace table using the eh_frame method.

[0177] In an exemplary embodiment, the backtrace unit 730 is further used to determine a backtrace mode for backtracing the target service before backtracing the function call process in the target service execution process in the eh_frame mode according to the backtrace information recorded in the backtrace table; if the backtrace mode is the frame_pointer mode, the frame_pointer mode is used to backtrace the function call process in the target service execution process; if the backtrace mode is not the frame_pointer mode, the eh_frame mode is used to backtrace the function call process in the target service execution process according to the backtrace information recorded in the backtrace table.

[0178] In an exemplary embodiment, the reading unit 710 further includes:

[0179] A reading subunit, in which a user-mode program starts at least one performance analysis process, and reads an executable file that uses memory resources in the memory address distribution according to the memory address distribution of the performance analysis process;

[0180] The extraction subunit extracts the backtrace information used for backtracing the target function from the read executable file.

[0181] In an exemplary embodiment, when converting the statistical data of the function call stack into performance analysis data, the analysis unit 740 is further used to parse the function symbol information in the executable file and associate corresponding function symbols with each function in the function call stack.

[0182] In an exemplary embodiment, when the analysis unit 740 parses the function symbol information in the executable file and associates corresponding function symbols with each function in the function call stack, it is further used to parse the function symbol information in the executable file and associate corresponding function symbols with kernel functions in the function call stack; extract segments of user function symbols from the function symbol information to generate a streamlined debugging file; send the debugging file and the function call stack associated with the function symbol of the kernel function and statistical data of the function call stack as intermediate files to a collection end corresponding to the agent end; and the collection end further parses the debugging file using a symbol resolution tool to associate corresponding function symbols with user functions in the function call stack.

[0183] In an exemplary embodiment, the proxy terminal corresponds to a host machine where any service node in the distributed cluster is located.

[0184] In an exemplary embodiment, the distributed cluster includes a Kubernetes cluster.

[0185] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0186] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0187] like Fig. 9 As shown, Fig. 9 The hardware structure diagram of an electronic device according to an embodiment of the present specification is shown, and the device may include: a processor 1710, a memory 1720, an input / output interface 1730, a communication interface 1740, and a bus 1750. The processor 1710, the memory 1720, the input / output interface 1730, and the communication interface 1740 are connected to each other in communication within the device through the bus 1750.

[0188] The processor 1710 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification. The processor implements the above method by running executable instructions.

[0189] The memory 1720 for storing processor executable instructions can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1720 can store operating systems and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1720.

[0190] The input / output interface 1730 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0191] The communication interface 1740 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0192] The bus 1750 includes a path that transmits information between the various components of the device (eg, the processor 1710, the memory 1720, the input / output interface 1730, and the communication interface 1740).

[0193] It should be noted that, although the above device only shows the processor 1710, the memory 1720, the input / output interface 1730, the communication interface 1740 and the bus 1750, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0194] The embodiments of this specification also provide a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the performance analysis method of any of the above embodiments is implemented.

[0195] Computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0196] An embodiment of the present specification also provides a computer program product, which, when executed by a processor, is used to implement the performance analysis method described in any embodiment of the present specification.

[0197] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0198] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0199] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0200] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A performance analysis method, characterized in that: Applied to an agent, wherein a user-mode program and a kernel-mode program are running on the agent, the method comprises: The user state program reads the executable file executed by the agent in the user state, and extracts the backtracking information used for backtracking the target business from the executable file; The user state program generates a backtrace table based on the backtrace information, and transmits the backtrace table to the kernel state where the kernel state program is located; When a performance analysis event is triggered, the kernel state program, in the kernel state, traces back the function call process in the target service execution process according to the traceback information recorded in the traceback table, and generates statistical data of the function call stack; The user state program obtains statistical data of the function call stack generated by the kernel state program, and converts the statistical data of the function call stack into performance analysis data.

2. The method according to claim 1, characterized in that Data is transferred between the user state program and the kernel state program via eBPF Map; The step of transferring the backtrace table to the kernel state where the kernel state program is located comprises: The backtrace table is written into the eBPF Map, and the kernel state program reads the eBPF Map to store the backtrace table in the kernel state; The user state program obtains statistical data of the function call stack generated by the kernel state program, including: The user state program reads the statistical data of the function call stack written by the kernel state program into the eBPF Map.

3. The method according to claim 1, characterized in that When the number of executable files is greater than 1, a corresponding backtrace table is generated for each executable file; data is transferred between the user state program and the kernel state program through two eBPF Maps; the first eBPF Map is used as a data table for writing the backtrace table, and the second eBPF Map is used as an index table for recording the first and last addresses of each backtrace table in the first eBPF Map; The step of transferring the backtrace table to the kernel state where the kernel state program is located comprises: The backtrace table generated by each executable file is written into the first eBPF Map, and the head and tail addresses of the written backtrace table located in the first eBPF Map are written into the second eBPF Map, and the head and tail addresses of each backtrace table in the second eBPF Map are further read by the kernel state program, and the backtrace table in the first eBPF Map is read based on the head and tail addresses of each backtrace table, so as to store each backtrace table in the kernel state; The user state program obtains statistical data of the function call stack generated by the kernel state program, including: The user state program reads the head and tail addresses of each backtrace table in the second eBPF Map, and reads the statistical data of the function call stack corresponding to each backtrace table written by the kernel state program in the first eBPF Map based on the head and tail addresses of each backtrace table.

4. The method according to claim 1, characterized in that: The step of backtracking the function call process in the target service execution process according to the backtracking information recorded in the backtracking table includes: According to the traceback information recorded in the traceback table, the function call process in the target service execution process is traced back using the eh_frame method.

5. The method according to claim 4, characterized in that Before backtracing the function call process in the target service execution process in the eh_frame mode according to the backtracing information recorded in the backtracing table, the method further includes: Determine the tracing method for the target service; If the backtracking mode is the frame_pointer mode, the frame_pointer mode is used to backtrack the function call process in the target service execution process; If the backtracking mode is not the frame_pointer mode, the backtracking information recorded in the backtracking table is executed, and the function calling process in the target service execution process is backtracked using the eh_frame mode.

6. The method according to claim 5, characterized in that The user state program reads an executable file executed by the agent in the user state, and extracts backtracking information for backtracking the target business from the executable file, including: The user-mode program starts at least one performance analysis process, and reads an executable file that uses memory resources in the memory address distribution according to the memory address distribution of the performance analysis process; Backtrace information for backtracing a target function is extracted from the read executable file.

7. The method according to claim 1, characterized in that The converting the statistical data of the function call stack into performance analysis data comprises: The function symbol information in the executable file is parsed, and corresponding function symbols are associated with each function in the function call stack.

8. The method according to claim 7, characterized in that Parsing the function symbol information in the executable file and associating corresponding function symbols for each function in the function call stack, including: Parsing function symbol information in the executable file, and associating corresponding function symbols with kernel functions in the function call stack; Extracting segments of user function symbols from the function symbol information to generate a simplified debugging file; The debugging file, the function call stack of the function symbol associated with the kernel function, and the statistical data of the function call stack are generated as an intermediate file and sent to the collection end corresponding to the agent end; the collection end further parses the debugging file using a symbol parsing tool to associate the corresponding function symbol with the user function in the function call stack.

9. The method according to claim 1, characterized in that: The proxy terminal corresponds to the host machine where any service node in the distributed cluster is located.

10. The method according to claim 9, characterized in that The distributed cluster includes a Kubernetes cluster.

11. A performance analysis device, characterized in that: Applied to an agent, the agent runs a user-mode program and a kernel-mode program, and the device includes: A reading unit, wherein the user state program reads the executable file executed by the agent in the user state, and extracts the backtracking information used for backtracking the target business from the executable file; A transmission unit, wherein the user state program generates a backtrace table based on the backtrace information, and transmits the backtrace table to the kernel state where the kernel state program is located; A backtracking unit, wherein when a performance analysis event is triggered, the kernel state program, in the kernel state, backtracks the function call process in the target service execution process according to the backtracking information recorded in the backtracking table to generate statistical data of the function call stack; The analysis unit is configured such that the user-mode program obtains statistical data of the function call stack generated by the kernel-mode program, and converts the statistical data of the function call stack into performance analysis data.

12. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 10 by running the executable instructions.

13. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images